![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1424766%2F3690bc48fab4ed74631ebd292ad0e421%2FScreenshot%202024-04-09%20at%2018.39.18.png?generation=1712680785774576&alt=media\")
Thanks to kaggle and everyone involved for hosting such an interesting competition. We learned a lot about EEG data and how to create strong models for it. Special thanks to @darraghdog who carried most of the workload in the last weeks of the competition, when I was busy with training my first real life neural network.
\nOur solution is an ensemble of multiple models from two diverse modeling approaches. The first approach is to apply pretrained 2d-CNN architectures to the MelSpectrogram transformation of the data. The second approach uses 1D-Convolutions to encode the raw eeg data before modeling with Squeezeformer blocks.
\nKey ingredients for our solution were a robust cross validation based on data selection and creative augmentations.
Having a solid cross validation was certainly key to doing well in this competition! So we spend a lot of time figuring out a good validation scheme. In general we used a 4fold validation setup, split by patient id. One key ingredient was filtering the val dataset for those rows only having more than 9 votes. The SPaRCNet paper most likely explained how the data was created.
\n
Not only was the test data filtered by having more than 9 votes, but also the data has been augmented by using the same labels but shifting the eeg data. This means that the given 106k training rows are highly redundant and can/ should be filtered.
\n\n\nWe expanded the high- and low-quality sets of EEG segments by adding additional segments belonging to the
\n
\n same stationary period of the EEG
This basically means the authors applied a shift-like augmentation to the raw eeg data to create more data and used same labels for the newly created data. This explains why there were so many rows for the same EEG_id having the same label. We believe that separating the original data from the augmented data makes the data cleaner and improved our cross validation. So we put some effort into reverse-engineering this process and finding the original “true” data point belonging to a given label. So we ended up with a filtered dataset of only 6350 rows which we used not only for validation using our 4fold scheme but also mainly used that for training, and discarded the other 100k rows. At the end we had a quite good cv/ lb correlation
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1424766%2Fccecc2d899a329f4d796640a966d5f53%2FScreenshot%202024-04-09%20at%2018.47.16.png?generation=1712681252795117&alt=media\")
For 2D models, we had pretrained weights and we observed high data quality was important and we got no benefit from using pseudo labels and data with <8 votes in the label. All models were trained with high quality data.
\nOur 1D models were trained from scratch so it helped to also use the low count data. See below in the training procedure how low vote data was used in training. Pseudo labelling was not helpful for us - we tried a lot of things here.
\nIn the end we did not use the 10 minute spectrograms. They may have helped slightly in the 2d model in some experiments but not in any major way.
\nNo preprocessing of the data to disk was made, which really sped up experiments and allowed flexibility in approaches. We used torchaudio‘s GPU implementation to create the MelSpectrogram on the fly and normalized the signal.
\nThe double banana montage, explained by @cdeotte was used throughout our solution. As mentioned by others, stacking 16 signals next to one another had limitations in that the each node would interact more strongly in the model with it’s neighbours. We experimented with ordering and found this order worked well, which essentially looks at the left and right side of the brain for each node together,
\nFp1>F7 Fp2>F8 F7>T3 F8>T4 T3>T5 T4>T6 T5>O1 T6>O2 Fp1>F3 Fp2>F4 F3>C3 F4>C4 C3>P3 C4>P4 P3>O1 P4>O2
Scipy implementation of butter bandpass filter was used with an order of 2, and lowpass of between 0 and 1.5 Hz on different models, and a highpass of between 20 and 30 Hz. No notchfilter was used.
\nWe observed it was important not to normalize the data batch-wise or sample-wise. We saw this in @medali1992 Resnet 1D GRU implementation - Not sure if this is where the idea originally came from. Instead, after the butter filter, we used the logic implemented x = x.clip(-1024, 1024) / 32..
For 2D models, we had pretrained weights and we observed high data quality was more important - not all augmentations helped. Our 1D models were trained from scratch so heavy augmentations were used. In both models, augmentation was highly effective.
\nThe following augmentations were used by boths models,
\nIn the 1D models only,
\nIn the 2D models only,
\nLike most other competitors we used pretrained 2D CNN backbones which we fed with melspec transformation of the data. However, we made several improvements to publicly known approaches. The first major boost comes from not combining the data into regions of the EEG electrodes (LL, LP, RP, RL) but creating 16 individual mel spectrograms for the double banana montage. We then concatenated the 16 images into one large image. In the last week we found another big improvement, by “zooming” onto the center 10sec. Zooming was applied with different window and hop lengths to increase diversity. Mel-Spectrogram transformation are done as part of the model directly on GPU using torchaudios Mel-Spectrogram.
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1424766%2F2088a92e1f8bf11dbff9735ce5c9be94%2FScreenshot%202024-04-09%20at%2018.53.19.png?generation=1712681633116409&alt=media\")
Our 1d CNN was heavily inspired by the work of Yuri Sun @sunyuri , and particularly his implementation of a lightweight CNN for seizure detection (code and paper here). As shown on the architecture the model focuses on grouped convolutions timewise and then channel wise. The first two blocks focus time-wise, keeping the channel information independent with grouped convolutions. After pooling the timewise information a channel wise convolution is made to interact across the nodes on the brain, and then another timewise convoltion. Technically we use conv2d, but the effect is like conv1d, only we can convolve over all channels in parallel. The original implementation used a CBAM attention block, we swapped this out and instead used 3 layers of squeezeformer. In addition we added pointwise convolutions before and after the depthwise convolutions which helped stabilite and learn better within the net.
\nAs we train from scratch on relatively small data, keeping parameter count low was important. During ASL competition we had spent a lot of time on that, so leveraged our squeezeformer implementation from there.
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1424766%2Fdc57c62c4a5e83a98435a20b780b6860%2FScreenshot%202024-04-09%20at%2018.55.46.png?generation=1712681758605758&alt=media\")
Training parameters used were generally 12 to 16 epochs, cosine learning rate decay with initial lr 0.0012 and batch size 32. Drop path was quite effective and set at 0.2. The main difference in the training was the granularity of the window lengths in the magnified 10 second window, different models were trained with different granularity and a mixnet_l and mixnet_xl backbones.
\nTraining parameters used were ~32 epochs, cosine learning rate decay with initial lr 0.001 and batch size 64. When we used low vote samples, we increased batch size to 256. We used a hidden dimension of 128 and dropout (attention, ff, head, conv) were all 0.1, going deeper on hidden dimension or more layers did not help.
\nWe trained 3 different version of the 1d model with respect to data.
For diversity, we trained some 1d models with butter bandpass order 0 and 1. They were not as strong but added some diversity in the blend.
\nUp until the last day we had been using mean blend of weights, with a higher weighting on 2d models. 15 hours before the end, with the help of GPT4, we created a simple neural net to learn the best blend weight on the 23 base models. 5 of the models were 1D CNN with different training procedures. 18 of the models were Mixnet 2d CNNs, with different backbones, and 10 second magnification windows.
\nOn CV the learned blend weights lifted the score but affect on public LB was not so large.
\nEarlier, we had been thinking that the distribution of the predictions were different to the distribution seen in the >8 votes portion of the training dataset. So about 10 hours from the end we added a bias term for each of the 6 classes to the nnet. This bias would be added to the logits. CV difference was small, but when the Public LB score came through it jumped us from 9th to 4th position which was exciting. The nnet is quite simple, see below. On private score, over the final days, we had been in the 0.27 LB prize zone with most of our blends even with mean weights.
(nn.Module):\n ():\n (Net, ).__init__()\n .fc = nn.Linear(n_models, , bias = False) \n .fc_c = torch.nn.Parameter(torch.zeros()[None,,None])\n ():\n . fc(x) + .fc_c\nNo supplemental data was used
\nBest single 1D model - 4 folds, 2 seed per fold - CV/Public/Private 0.257 / 0.25 / 0.30
\nBest single 2D model - 4 folds, 2 seed per fold - CV/Public/Private 0.232 / 0.24 / 0.29
\nFinal blend 4 folds, 2 seed, 23 model per fold - CV/Public/Private 0.207 / 0.22 / 0.27
Pytorch
\nTimm, huggingface, albumentations.
\n@darraghdog used a 4090 GPU instance rented from runpod.io
\nNeptune.ai was our MLOps stack to track, compare and share models. It was heavily used and allowed easy 4 fold grouped view of models and mean of the val scores for all folds - which made tracking a lot easier.
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1424766%2F7ef0101b2bb47fdcc1b99688221b47b9%2FScreenshot%202024-04-09%20at%2019.02.09.png?generation=1712682149182806&alt=media\")
Thanks for reading. Questions are welcome.
\nEdit:
\nInference kernel: https://www.kaggle.com/code/darraghdog/3rd-place-solution
\ngithub: https://github.com/darraghdog/kaggle-hms-3rd-place-solution
Thanks to kaggle and everyone involved for hosting such an interesting competition. We learned a lot about EEG data and how to create strong models for it. Special thanks to @darraghdog who carried most of the workload in the last weeks of the competition, when I was busy with training my first real life neural network.
\nOur solution is an ensemble of multiple models from two diverse modeling approaches. The first approach is to apply pretrained 2d-CNN architectures to the MelSpectrogram transformation of the data. The second approach uses 1D-Convolutions to encode the raw eeg data before modeling with Squeezeformer blocks.
\nKey ingredients for our solution were a robust cross validation based on data selection and creative augmentations.
Having a solid cross validation was certainly key to doing well in this competition! So we spend a lot of time figuring out a good validation scheme. In general we used a 4fold validation setup, split by patient id. One key ingredient was filtering the val dataset for those rows only having more than 9 votes. The SPaRCNet paper most likely explained how the data was created.
\n
Not only was the test data filtered by having more than 9 votes, but also the data has been augmented by using the same labels but shifting the eeg data. This means that the given 106k training rows are highly redundant and can/ should be filtered.
\n\n\nWe expanded the high- and low-quality sets of EEG segments by adding additional segments belonging to the
\n
\n same stationary period of the EEG
This basically means the authors applied a shift-like augmentation to the raw eeg data to create more data and used same labels for the newly created data. This explains why there were so many rows for the same EEG_id having the same label. We believe that separating the original data from the augmented data makes the data cleaner and improved our cross validation. So we put some effort into reverse-engineering this process and finding the original “true” data point belonging to a given label. So we ended up with a filtered dataset of only 6350 rows which we used not only for validation using our 4fold scheme but also mainly used that for training, and discarded the other 100k rows. At the end we had a quite good cv/ lb correlation
\n
For 2D models, we had pretrained weights and we observed high data quality was important and we got no benefit from using pseudo labels and data with <8 votes in the label. All models were trained with high quality data.
\nOur 1D models were trained from scratch so it helped to also use the low count data. See below in the training procedure how low vote data was used in training. Pseudo labelling was not helpful for us - we tried a lot of things here.
\nIn the end we did not use the 10 minute spectrograms. They may have helped slightly in the 2d model in some experiments but not in any major way.
\nNo preprocessing of the data to disk was made, which really sped up experiments and allowed flexibility in approaches. We used torchaudio‘s GPU implementation to create the MelSpectrogram on the fly and normalized the signal.
\nThe double banana montage, explained by @cdeotte was used throughout our solution. As mentioned by others, stacking 16 signals next to one another had limitations in that the each node would interact more strongly in the model with it’s neighbours. We experimented with ordering and found this order worked well, which essentially looks at the left and right side of the brain for each node together,
\nFp1>F7 Fp2>F8 F7>T3 F8>T4 T3>T5 T4>T6 T5>O1 T6>O2 Fp1>F3 Fp2>F4 F3>C3 F4>C4 C3>P3 C4>P4 P3>O1 P4>O2
Scipy implementation of butter bandpass filter was used with an order of 2, and lowpass of between 0 and 1.5 Hz on different models, and a highpass of between 20 and 30 Hz. No notchfilter was used.
\nWe observed it was important not to normalize the data batch-wise or sample-wise. We saw this in @medali1992 Resnet 1D GRU implementation - Not sure if this is where the idea originally came from. Instead, after the butter filter, we used the logic implemented x = x.clip(-1024, 1024) / 32..
For 2D models, we had pretrained weights and we observed high data quality was more important - not all augmentations helped. Our 1D models were trained from scratch so heavy augmentations were used. In both models, augmentation was highly effective.
\nThe following augmentations were used by boths models,
\nIn the 1D models only,
\nIn the 2D models only,
\nLike most other competitors we used pretrained 2D CNN backbones which we fed with melspec transformation of the data. However, we made several improvements to publicly known approaches. The first major boost comes from not combining the data into regions of the EEG electrodes (LL, LP, RP, RL) but creating 16 individual mel spectrograms for the double banana montage. We then concatenated the 16 images into one large image. In the last week we found another big improvement, by “zooming” onto the center 10sec. Zooming was applied with different window and hop lengths to increase diversity. Mel-Spectrogram transformation are done as part of the model directly on GPU using torchaudios Mel-Spectrogram.
\n
Our 1d CNN was heavily inspired by the work of Yuri Sun @sunyuri , and particularly his implementation of a lightweight CNN for seizure detection (code and paper here). As shown on the architecture the model focuses on grouped convolutions timewise and then channel wise. The first two blocks focus time-wise, keeping the channel information independent with grouped convolutions. After pooling the timewise information a channel wise convolution is made to interact across the nodes on the brain, and then another timewise convoltion. Technically we use conv2d, but the effect is like conv1d, only we can convolve over all channels in parallel. The original implementation used a CBAM attention block, we swapped this out and instead used 3 layers of squeezeformer. In addition we added pointwise convolutions before and after the depthwise convolutions which helped stabilite and learn better within the net.
\nAs we train from scratch on relatively small data, keeping parameter count low was important. During ASL competition we had spent a lot of time on that, so leveraged our squeezeformer implementation from there.
Training parameters used were generally 12 to 16 epochs, cosine learning rate decay with initial lr 0.0012 and batch size 32. Drop path was quite effective and set at 0.2. The main difference in the training was the granularity of the window lengths in the magnified 10 second window, different models were trained with different granularity and a mixnet_l and mixnet_xl backbones.
\nTraining parameters used were ~32 epochs, cosine learning rate decay with initial lr 0.001 and batch size 64. When we used low vote samples, we increased batch size to 256. We used a hidden dimension of 128 and dropout (attention, ff, head, conv) were all 0.1, going deeper on hidden dimension or more layers did not help.
\nWe trained 3 different version of the 1d model with respect to data.
For diversity, we trained some 1d models with butter bandpass order 0 and 1. They were not as strong but added some diversity in the blend.
\nUp until the last day we had been using mean blend of weights, with a higher weighting on 2d models. 15 hours before the end, with the help of GPT4, we created a simple neural net to learn the best blend weight on the 23 base models. 5 of the models were 1D CNN with different training procedures. 18 of the models were Mixnet 2d CNNs, with different backbones, and 10 second magnification windows.
\nOn CV the learned blend weights lifted the score but affect on public LB was not so large.
\nEarlier, we had been thinking that the distribution of the predictions were different to the distribution seen in the >8 votes portion of the training dataset. So about 10 hours from the end we added a bias term for each of the 6 classes to the nnet. This bias would be added to the logits. CV difference was small, but when the Public LB score came through it jumped us from 9th to 4th position which was exciting. The nnet is quite simple, see below. On private score, over the final days, we had been in the 0.27 LB prize zone with most of our blends even with mean weights.
(nn.Module):\n ():\n (Net, ).__init__()\n .fc = nn.Linear(n_models, , bias = False) \n .fc_c = torch.nn.Parameter(torch.zeros()[None,,None])\n ():\n . fc(x) + .fc_c\nNo supplemental data was used
\nBest single 1D model - 4 folds, 2 seed per fold - CV/Public/Private 0.257 / 0.25 / 0.30
\nBest single 2D model - 4 folds, 2 seed per fold - CV/Public/Private 0.232 / 0.24 / 0.29
\nFinal blend 4 folds, 2 seed, 23 model per fold - CV/Public/Private 0.207 / 0.22 / 0.27
Pytorch
\nTimm, huggingface, albumentations.
\n@darraghdog used a 4090 GPU instance rented from runpod.io
\nNeptune.ai was our MLOps stack to track, compare and share models. It was heavily used and allowed easy 4 fold grouped view of models and mean of the val scores for all folds - which made tracking a lot easier.
Thanks for reading. Questions are welcome.
\nEdit:
\nInference kernel: https://www.kaggle.com/code/darraghdog/3rd-place-solution
\ngithub: https://github.com/darraghdog/kaggle-hms-3rd-place-solution
Congrats! Thanks for sharing.
", "rawMarkdown": "Congrats! Thanks for sharing.", "votes": null }, { "id": "2743979", "postDate": "04/09/2024 17:27:04", "content": "Great solution (as always). I'll reread in more details, but first reaction is that you explain why one hing I tried did not work: I tried to sample eeg raw data from other locations than the middle and did not see any significant change.
", "rawMarkdown": "Great solution (as always). I'll reread in more details, but first reaction is that you explain why one hing I tried did not work: I tried to sample eeg raw data from other locations than the middle and did not see any significant change.", "votes": null }, { "id": "2744026", "postDate": "04/09/2024 18:00:28", "content": "I was busy with training my first real life neural network.
\nCongratulations!
congratulations!
\nNice work and nice solution writing
\nThanks for your sharing
thank you for the explanation post. Amazing work!
", "rawMarkdown": "thank you for the explanation post. Amazing work!", "votes": null }, { "id": "2744065", "postDate": "04/09/2024 18:27:44", "content": "Congrats! Thank you very much for sharing. Amazing work!
", "rawMarkdown": "Congrats! Thank you very much for sharing. Amazing work!", "votes": null }, { "id": "2744247", "postDate": "04/09/2024 20:06:25", "content": "Congrats @christofhenkel on achieveing 3rd place.
", "rawMarkdown": "Congrats @christofhenkel on achieveing 3rd place.", "votes": null }, { "id": "2744248", "postDate": "04/09/2024 20:06:30", "content": "Thank you very much for sharing this magnificent work,
\nis there any chance for that final code released ?
\nregards
Thanks for sharing!
\nRegarding cv, we thought based on the data section that votes in test rounding from 3 to 20 so our validation was stratifiedgroupkfold stratifying on votes 3-20 and grouping by patients. We optimized this a lot and reached cv 0.45 using 1 stage only and trusted this too much and stopped working on 2 stages entirely. Really sad that all our work went into dust at the end due to wrong cv.
\nOne interesting thing we found is how high is the impact of correct normalization. We dropped the normalization-per-image and adopted global normalization with some modifications. That boosted our cv by around 0.1 and public lb by 0.05.
Thanks @mohammad2012191 and yes, we had similar experiences on normalisation. Also, the effort you spent will payoff in different ways with experience gained 💪🏻, I always find that the best learning comes when things do not go as well as expected
", "rawMarkdown": "Thanks @mohammad2012191 and yes, we had similar experiences on normalisation. Also, the effort you spent will payoff in different ways with experience gained 💪🏻, I always find that the best learning comes when things do not go as well as expected", "votes": null }, { "id": "2744329", "postDate": "04/09/2024 20:44:01", "content": "A repo with source code will be added in the thread 🧵 in the couple of weeks
", "rawMarkdown": "A repo with source code will be added in the thread 🧵 in the couple of weeks", "votes": null }, { "id": "2744395", "postDate": "04/09/2024 21:52:52", "content": "Lol it took me a sec to get what this meant 😂
", "rawMarkdown": "Lol it took me a sec to get what this meant 😂", "votes": null }, { "id": "2744424", "postDate": "04/09/2024 22:48:27", "content": "Wow congratulations this is very helpful
", "rawMarkdown": "Wow congratulations this is very helpful", "votes": null }, { "id": "2744455", "postDate": "04/09/2024 23:26:49", "content": "Congratulations Dieter and Darragh
\n\n\nBest single 1D model - 4 folds, 2 seed per fold - CV/Public/Private 0.257 / 0.25 / 0.30
\n
Wow. That's a very powerful 1D model. Great job!
", "rawMarkdown": "Congratulations Dieter and Darragh\n>Best single 1D model - 4 folds, 2 seed per fold - CV/Public/Private 0.257 / 0.25 / 0.30\n\nWow. That's a very powerful 1D model. Great job!", "votes": null }, { "id": "2744526", "postDate": "04/10/2024 01:01:08", "content": "Congrats! These single models are powerful. Excellent!
", "rawMarkdown": "Congrats! These single models are powerful. Excellent!", "votes": null }, { "id": "2744675", "postDate": "04/10/2024 04:20:40", "content": "Congratulations on 3rd place in this competition. Thanks for sharing the details of your solutions with colorful diagrams.
", "rawMarkdown": "Congratulations on 3rd place in this competition. Thanks for sharing the details of your solutions with colorful diagrams.", "votes": null }, { "id": "2744918", "postDate": "04/10/2024 07:58:38", "content": "Have fun with the little one
", "rawMarkdown": "Have fun with the little one", "votes": null }, { "id": "2745291", "postDate": "04/10/2024 14:27:48", "content": "Congratulations on 3rd place in this competition. It's very nice to sharing with us !
", "rawMarkdown": "Congratulations on 3rd place in this competition. It's very nice to sharing with us !", "votes": null }, { "id": "2745364", "postDate": "04/10/2024 15:16:42", "content": "Congratulations!! 🎉🎉, Love to see Squeezeformer again…
", "rawMarkdown": "Congratulations!! 🎉🎉, Love to see Squeezeformer again...", "votes": null }, { "id": "2745791", "postDate": "04/10/2024 20:08:38", "content": "Congratulations! I am so surprised that you understood that the authors applied a shift-like augmentation to the rEEGeeg data and identified 6350 original data. While the data section of the competition site says
Many of these samples overlapped and have been consolidated.
And I thought that they gathered the 106,800 data (I was honestly surprised that they would collect them experimentally) as the row numbers and merged them.
\nAlong with the high-quality data being used for the test, I wonder why the completion host would not mention it.
", "rawMarkdown": "Congratulations! I am so surprised that you understood that `the authors applied a shift-like augmentation to the rEEGeeg data` and identified 6350 original data. While the data section of the competition site says\n\n`Many of these samples overlapped and have been consolidated. `\n\nAnd I thought that they gathered the 106,800 data (I was honestly surprised that they would collect them experimentally) as the row numbers and merged them.\n\nAlong with the high-quality data being used for the test, I wonder why the completion host would not mention it.", "votes": null }, { "id": "2746009", "postDate": "04/11/2024 01:40:44", "content": "Congratulations. I especially liked idea of zooming center region of EEG.
\nAnd our team also noticed the trick of processing test data, but only my team mate @yujiariyasu successfully squeezed gain from that. I should have dig deeper on that.
\nThanks for sharing exciting report.
", "rawMarkdown": "Congratulations. I especially liked idea of zooming center region of EEG.\n\nAnd our team also noticed the trick of processing test data, but only my team mate @yujiariyasu successfully squeezed gain from that. I should have dig deeper on that.\n\nThanks for sharing exciting report.", "votes": null }, { "id": "2746279", "postDate": "04/11/2024 06:20:51", "content": "Congratulations! I have learned a lot.
", "rawMarkdown": "Congratulations! I have learned a lot.", "votes": null }, { "id": "2746401", "postDate": "04/11/2024 07:43:26", "content": "When extra precision was released we saw the difference of the postprocessing bias term was CV/Public/Private 0.001 / 0.004 / 0.003
", "rawMarkdown": "When extra precision was released we saw the difference of the postprocessing bias term was CV/Public/Private 0.001 / 0.004 / 0.003", "votes": null }, { "id": "2746403", "postDate": "04/11/2024 07:44:41", "content": "\n\nSo we put some effort into reverse-engineering this process and finding the original “true” data point belonging to a given label. So we ended up with a filtered dataset of only 6350 rows which we used not only for validation using our 4fold scheme but also mainly used that for training, and discarded the other 100k rows.
\n
Anyway, could you share us more details of reverse engineering true labels?
\nOur approach is simple:
\nDo you use more sophisticated ways?
", "rawMarkdown": "> So we put some effort into reverse-engineering this process and finding the original “true” data point belonging to a given label. So we ended up with a filtered dataset of only 6350 rows which we used not only for validation using our 4fold scheme but also mainly used that for training, and discarded the other 100k rows.\n\nAnyway, could you share us more details of reverse engineering *true* labels?\n\nOur approach is simple:\n1. grouping nearby labels into sub-groups which has same 6-dim vote vectors\n2. take middle point of each sub-groups\n3. filter only middle point\n\nDo you use more sophisticated ways?", "votes": null }, { "id": "2746407", "postDate": "04/11/2024 07:45:47", "content": "The CBAM work from you & your team really helped a lot. It is a clever way to approach the problem as 1D model.
", "rawMarkdown": "The CBAM work from you & your team really helped a lot. It is a clever way to approach the problem as 1D model.", "votes": null }, { "id": "2746447", "postDate": "04/11/2024 08:10:01", "content": "Here is the main block.
\n eeg_id with same votes\ntrain = train + label_columns](str)(, axis=)\n\n each unique eeg combination calculate overlaping time frames of same label and take the one row that has the highest overlap to all \nrows = \n eeg_id_l (train()):\n df0 = train==eeg_id_l](drop=True)()\n offsets = df0(int)\n x = np(offsets()+)\n o offsets:\n x += \n best_idx = np(() o offsets])\n rows += ]\n\ndf = pd(rows)\n (although few) eeg have multiple labels, just take the first one - this last part is not essential\ndf2 = df(subset=)()\nMost of the time take middle point of each sub-groups is good enough. But only if there is only one \"true\" point for the eeg_id.
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1424766%2Fe5db07f3b4bf72f51f41c9b80dd27ac4%2FScreenshot%202024-04-11%20at%2010.11.20.png?generation=1712823564444253&alt=media\")
\nThis would be the resulting x from the block above for a more complex example. As you can see there are 2 local maxima. Those are most likely the offsets of the true datapoints. The plot also shows why in this example middle point is not the optimal way for filtering
Thanks. It all make sense.
", "rawMarkdown": "Thanks. It all make sense.", "votes": null }, { "id": "2746710", "postDate": "04/11/2024 12:16:26", "content": "btw, our pp got public: 0.0045 / private: 0.005 gain
", "rawMarkdown": "btw, our pp got public: 0.0045 / private: 0.005 gain", "votes": null }, { "id": "2746714", "postDate": "04/11/2024 12:18:47", "content": "Thanks for sharing exciting report.
", "rawMarkdown": "Thanks for sharing exciting report.", "votes": null }, { "id": "2746930", "postDate": "04/11/2024 14:44:32", "content": "that is a good jump!
", "rawMarkdown": "that is a good jump!", "votes": null }, { "id": "2748296", "postDate": "04/12/2024 11:07:32", "content": "Congrats! This is actually very similar to our solution. I tried melspectrograms but I couldn't make them work. They were way worse compared to cusignal STFT spectrograms. Our best models were MaxViT tiny and ConvNeXt base. I think you can get first place using those models.
", "rawMarkdown": "Congrats! This is actually very similar to our solution. I tried melspectrograms but I couldn't make them work. They were way worse compared to cusignal STFT spectrograms. Our best models were MaxViT tiny and ConvNeXt base. I think you can get first place using those models.", "votes": null }, { "id": "2748429", "postDate": "04/12/2024 12:44:33", "content": "Congratulations on the 3rd place and thanks for sharing!
\nVery interesting approach with the split. I spent most of the competition using all data, because I could see no reason for using only samples with votes >=10, even if they worked well in the LB. In the last weeks I finally took time to read some papers, starting with those on Sparcnet. It was an eye opener. It finally made sense to use only the high quality samples. I followed an approach slightly different from yours:
\nI ended up with 6366 samples. I used all for training, applying weights to account for eeg_ids with a very large number of samples, which didn't exist in the test data, and the fact that the share of \"Other\" in the test sample was lower, according to the paper. For validation I picked only one sample per eeg_id: the one with the most votes or the longest in case of a tie.
\nWhat puzzled me is that adding low quality samples with pseudo labels to the training data, I was able to significantly improve the CV (0.23 for single model), but only get a small LB improvement. Moreover, after a certain point, the CV kept going down but the LB would start degrading just slightly. The public notebooks I looked at had a leak (they sampled high quality and low quality separately, thus mixing eegs from the same patient in train and test), but I couldn't find any leak in my case. I still don't know why CV improved so much, but LB didn't. Maybe this approach introduced a leak because of labeler bias. Any thoughts from your experience using pseudo labels for 1D models?
", "rawMarkdown": "Congratulations on the 3rd place and thanks for sharing!\n\nVery interesting approach with the split. I spent most of the competition using all data, because I could see no reason for using only samples with votes >=10, even if they worked well in the LB. In the last weeks I finally took time to read some papers, starting with those on Sparcnet. It was an eye opener. It finally made sense to use only the high quality samples. I followed an approach slightly different from yours: \n1. pick only the eeg_ids whose maximum number of votes >=10\n2. take all consecutive samples with the same share of votes (and more than 10 votes) and pick the middle of those samples (10m, 50s, 20s, and 10s). I wanted to use the whole range and randomize, but given the kaggle limits on memory and GPU hours, I ended up going with just the middle. Samples with less than 10 votes were ignored.\n\nI ended up with 6366 samples. I used all for training, applying weights to account for eeg_ids with a very large number of samples, which didn't exist in the test data, and the fact that the share of \"Other\" in the test sample was lower, according to the paper. For validation I picked only one sample per eeg_id: the one with the most votes or the longest in case of a tie.\n\nWhat puzzled me is that adding low quality samples with pseudo labels to the training data, I was able to significantly improve the CV (0.23 for single model), but only get a small LB improvement. Moreover, after a certain point, the CV kept going down but the LB would start degrading just slightly. The public notebooks I looked at had a leak (they sampled high quality and low quality separately, thus mixing eegs from the same patient in train and test), but I couldn't find any leak in my case. I still don't know why CV improved so much, but LB didn't. Maybe this approach introduced a leak because of labeler bias. Any thoughts from your experience using pseudo labels for 1D models?", "votes": null }, { "id": "2749092", "postDate": "04/12/2024 20:30:31", "content": "@vialactea thanks for your comments; re: the pseudo, we had a very similar experience - very promising CV score, but then on LB the pseudo trained model did worse than the base model which generated the pseudo. I just checked private LB scores and they were also the same.
\nI do not see any convincing reason; one possible idea I thought is that diversity among folds is higher with non-pseudo training, so when averaged the score is better - as opposed to with pseudo, where we may get similar predictions per fold which do not blend as well even though each individual checkpoint is better. I'm not sure if this is the reason though. Perhaps as you say it could have been a leak because of labeler bias. When a lot of effort was spent on it and score didnt improve, we switched to something else - there were a lot of other things to try.
\ncongratulations on your result in the competition.
Hi @vialactea When you use pseudo are you careful to make 5 sets of pseudo labels? one for each fold?
\nFor example we use fold 1 to pseudo label the vote<10 on fold 1 train data only (i.e don't use patients outside for fold 1) and save that as pseudo set 1. Then we use fold 2 to pseudo label the vote<10 on fold 2 and save that as pseudo set 2 etc.
Then in stage 2 training, we add the pseudo set 1 to fold 1 during training. Doing this prevents leaks and prevents optimistic CV score.
I experienced the same experience (improve CV drastically and deteriorated LB) with pseudo label. I pseudo label to OOF: first, I trained fold0 train with all data and make pseudo label to fold0 valid set (vote<10). I did this because I believed pseudo label to OOF set is safe since it only leaks features, not labels.
\nHowever, @cdeotte's approach is safest because there are no leaks of information about validation set including features. Then I wonder what is the mechanism of leaks in pseudo labeling OOF set.
\n@cdeotte Do you have in mind some particular case of leak when pseudo labeling OOF set?
", "rawMarkdown": "I experienced the same experience (improve CV drastically and deteriorated LB) with pseudo label. I pseudo label to OOF: first, I trained fold0 train with all data and make pseudo label to fold0 valid set (vote<10). I did this because I believed pseudo label to OOF set is safe since it only leaks features, not labels. \n\nHowever, @cdeotte's approach is safest because there are no leaks of information about validation set including features. Then I wonder what is the mechanism of leaks in pseudo labeling OOF set.\n\n@cdeotte Do you have in mind some particular case of leak when pseudo labeling OOF set?", "votes": null }, { "id": "2749223", "postDate": "04/12/2024 23:13:03", "content": "Yes. Each fold has subset of patient_id. Therefore we cannot pseudo label all data. We must use fold 1 to only pseudo label the patient_id in fold 1. Otherwise there is a leak.
Congratulations Dieter and Darragh!!!!!!!!
", "rawMarkdown": "Congratulations Dieter and Darragh!!!!!!!!", "votes": null }, { "id": "2751855", "postDate": "04/14/2024 15:01:38", "content": "Seems so 😏 will know next time. We had an idea in the final weeks that our 2d model was weaker but there were so many things to try. Congrats on your great work!
", "rawMarkdown": "Seems so 😏 will know next time. We had an idea in the final weeks that our 2d model was weaker but there were so many things to try. Congrats on your great work!", "votes": null }, { "id": "2761071", "postDate": "04/19/2024 16:48:49", "content": "Hi @cdeote. Yes, I made sure that the pseudo labels for a stage 2 fold came from models trained without any of the patients in that fold. Moreover, all stage 1 and stage 2 models used the exact same 5 folds split of patients, with the caveat that some patients didn't have high quality hard labels and thus were not used in stage 1 training.
\nI couldn´t understand how come the stage 2 score for a fold would improve significantly when using pseudo labels produced from models that had been trained without seeing any of the patients in that fold, but that improvement would vanish in the LB. I expected ensembling to be affected due to a potential loss of diversity, as @darraghdog mentioned, but LB of single models didn't improve either.
", "rawMarkdown": "Hi @cdeote. Yes, I made sure that the pseudo labels for a stage 2 fold came from models trained without any of the patients in that fold. Moreover, all stage 1 and stage 2 models used the exact same 5 folds split of patients, with the caveat that some patients didn't have high quality hard labels and thus were not used in stage 1 training.\n\nI couldn´t understand how come the stage 2 score for a fold would improve significantly when using pseudo labels produced from models that had been trained without seeing any of the patients in that fold, but that improvement would vanish in the LB. I expected ensembling to be affected due to a potential loss of diversity, as @darraghdog mentioned, but LB of single models didn't improve either.", "votes": null }, { "id": "2764109", "postDate": "04/20/2024 23:00:07", "content": "I know, right. I even read the original paper and saw that chart, but discounted it in thinking \"they surely have given us original data and would tell us if anything was augmentation.\"
", "rawMarkdown": "I know, right. I even read the original paper and saw that chart, but discounted it in thinking \"they surely have given us original data and would tell us if anything was augmentation.\"", "votes": null }, { "id": "2799058", "postDate": "05/07/2024 15:17:56", "content": "Congratulations on 3rd place! I'm sorry for the late question.
\nCurrently, I'm tracing your released code on github. And there's one thing I can't understand. I notice that you use masking in each SqueezeformerBlock. I categorize these masking operations into 3 types,
mask_pad for LlamaAttention.mask for convolution module.mask_flat for those commented with Skip/Unskip pad.Could you provide some hints about the intuition behind these design choices? Thanks a lot!
", "rawMarkdown": "Congratulations on 3rd place! I'm sorry for the late question.\n\nCurrently, I'm tracing your released code on github. And there's one thing I can't understand. I notice that you use **masking** in each `SqueezeformerBlock`. I categorize these masking operations into 3 types,\n1. `mask_pad` for `LlamaAttention`.\n2. `mask` for convolution module.\n3. `mask_flat` for those commented with `Skip/Unskip pad`.\n\nCould you provide some hints about the intuition behind these design choices? Thanks a lot!", "votes": null }, { "id": "2807608", "postDate": "05/11/2024 18:28:11", "content": "The masking is not used as our inputs have constant length. The mask was all 1, so just a dummy. We leveraged from a previous competition and discussed how it is used there.
", "rawMarkdown": "The masking is not used as our inputs have constant length. The mask was all 1, so just a dummy. We leveraged from a previous competition and [discussed](https://www.kaggle.com/competitions/asl-fingerspelling/discussion/434485) how it is used there.", "votes": null }, { "id": "2807992", "postDate": "05/12/2024 03:25:53", "content": "Hi @darraghdog,
\nThanks for your explanation. I'll check how it's used in the fingerspelling competition!
", "rawMarkdown": "Hi @darraghdog,\n\nThanks for your explanation. I'll check how it's used in the fingerspelling competition!", "votes": null } ], "comments": [ { "id": 2743973, "author_name": "haohantsao", "author_url": "", "post_date": "04/09/2024 17:24:49", "content": "Congrats! Thanks for sharing.
", "votes": null, "replies": [] }, { "id": 2743979, "author_name": "cpmpml", "author_url": "", "post_date": "04/09/2024 17:27:04", "content": "Great solution (as always). I'll reread in more details, but first reaction is that you explain why one hing I tried did not work: I tried to sample eeg raw data from other locations than the middle and did not see any significant change.
", "votes": null, "replies": [] }, { "id": 2744026, "author_name": "sohier", "author_url": "", "post_date": "04/09/2024 18:00:28", "content": "I was busy with training my first real life neural network.
\nCongratulations!
Lol it took me a sec to get what this meant 😂
", "votes": null, "replies": [] }, { "id": 2744918, "author_name": "nymfree", "author_url": "", "post_date": "04/10/2024 07:58:38", "content": "Have fun with the little one
", "votes": null, "replies": [] } ] }, { "id": 2744036, "author_name": "horikitasaku", "author_url": "", "post_date": "04/09/2024 18:06:34", "content": "congratulations!
\nNice work and nice solution writing
\nThanks for your sharing
thank you for the explanation post. Amazing work!
", "votes": null, "replies": [] }, { "id": 2744065, "author_name": "zlemglsmklkaya", "author_url": "", "post_date": "04/09/2024 18:27:44", "content": "Congrats! Thank you very much for sharing. Amazing work!
", "votes": null, "replies": [] }, { "id": 2744247, "author_name": "adnanalaref", "author_url": "", "post_date": "04/09/2024 20:06:25", "content": "Congrats @christofhenkel on achieveing 3rd place.
", "votes": null, "replies": [] }, { "id": 2744248, "author_name": "letemoin", "author_url": "", "post_date": "04/09/2024 20:06:30", "content": "Thank you very much for sharing this magnificent work,
\nis there any chance for that final code released ?
\nregards
A repo with source code will be added in the thread 🧵 in the couple of weeks
", "votes": null, "replies": [] } ] }, { "id": 2744312, "author_name": "mohammad2012191", "author_url": "", "post_date": "04/09/2024 20:33:31", "content": "Thanks for sharing!
\nRegarding cv, we thought based on the data section that votes in test rounding from 3 to 20 so our validation was stratifiedgroupkfold stratifying on votes 3-20 and grouping by patients. We optimized this a lot and reached cv 0.45 using 1 stage only and trusted this too much and stopped working on 2 stages entirely. Really sad that all our work went into dust at the end due to wrong cv.
\nOne interesting thing we found is how high is the impact of correct normalization. We dropped the normalization-per-image and adopted global normalization with some modifications. That boosted our cv by around 0.1 and public lb by 0.05.
Thanks @mohammad2012191 and yes, we had similar experiences on normalisation. Also, the effort you spent will payoff in different ways with experience gained 💪🏻, I always find that the best learning comes when things do not go as well as expected
", "votes": null, "replies": [] } ] }, { "id": 2744424, "author_name": "rashidrk", "author_url": "", "post_date": "04/09/2024 22:48:27", "content": "Wow congratulations this is very helpful
", "votes": null, "replies": [] }, { "id": 2744455, "author_name": "cdeotte", "author_url": "", "post_date": "04/09/2024 23:26:49", "content": "Congratulations Dieter and Darragh
\n\n\nBest single 1D model - 4 folds, 2 seed per fold - CV/Public/Private 0.257 / 0.25 / 0.30
\n
Wow. That's a very powerful 1D model. Great job!
", "votes": null, "replies": [] }, { "id": 2744526, "author_name": "zijiangyang1116", "author_url": "", "post_date": "04/10/2024 01:01:08", "content": "Congrats! These single models are powerful. Excellent!
", "votes": null, "replies": [] }, { "id": 2744675, "author_name": "crsuthikshnkumar", "author_url": "", "post_date": "04/10/2024 04:20:40", "content": "Congratulations on 3rd place in this competition. Thanks for sharing the details of your solutions with colorful diagrams.
", "votes": null, "replies": [] }, { "id": 2745291, "author_name": "zephyrus1", "author_url": "", "post_date": "04/10/2024 14:27:48", "content": "Congratulations on 3rd place in this competition. It's very nice to sharing with us !
", "votes": null, "replies": [] }, { "id": 2745364, "author_name": "gowrishankarp", "author_url": "", "post_date": "04/10/2024 15:16:42", "content": "Congratulations!! 🎉🎉, Love to see Squeezeformer again…
", "votes": null, "replies": [] }, { "id": 2745791, "author_name": "makio323", "author_url": "", "post_date": "04/10/2024 20:08:38", "content": "Congratulations! I am so surprised that you understood that the authors applied a shift-like augmentation to the rEEGeeg data and identified 6350 original data. While the data section of the competition site says
Many of these samples overlapped and have been consolidated.
And I thought that they gathered the 106,800 data (I was honestly surprised that they would collect them experimentally) as the row numbers and merged them.
\nAlong with the high-quality data being used for the test, I wonder why the completion host would not mention it.
", "votes": null, "replies": [ { "id": 2764109, "author_name": "idithhaber", "author_url": "", "post_date": "04/20/2024 23:00:07", "content": "I know, right. I even read the original paper and saw that chart, but discounted it in thinking \"they surely have given us original data and would tell us if anything was augmentation.\"
", "votes": null, "replies": [] } ] }, { "id": 2746009, "author_name": "tatamikenn", "author_url": "", "post_date": "04/11/2024 01:40:44", "content": "Congratulations. I especially liked idea of zooming center region of EEG.
\nAnd our team also noticed the trick of processing test data, but only my team mate @yujiariyasu successfully squeezed gain from that. I should have dig deeper on that.
\nThanks for sharing exciting report.
", "votes": null, "replies": [ { "id": 2746401, "author_name": "darraghdog", "author_url": "", "post_date": "04/11/2024 07:43:26", "content": "When extra precision was released we saw the difference of the postprocessing bias term was CV/Public/Private 0.001 / 0.004 / 0.003
", "votes": null, "replies": [] }, { "id": 2746403, "author_name": "tatamikenn", "author_url": "", "post_date": "04/11/2024 07:44:41", "content": "\n\nSo we put some effort into reverse-engineering this process and finding the original “true” data point belonging to a given label. So we ended up with a filtered dataset of only 6350 rows which we used not only for validation using our 4fold scheme but also mainly used that for training, and discarded the other 100k rows.
\n
Anyway, could you share us more details of reverse engineering true labels?
\nOur approach is simple:
\nDo you use more sophisticated ways?
", "votes": null, "replies": [ { "id": 2746447, "author_name": "darraghdog", "author_url": "", "post_date": "04/11/2024 08:10:01", "content": "Here is the main block.
\n eeg_id with same votes\ntrain = train + label_columns](str)(, axis=)\n\n each unique eeg combination calculate overlaping time frames of same label and take the one row that has the highest overlap to all \nrows = \n eeg_id_l (train()):\n df0 = train==eeg_id_l](drop=True)()\n offsets = df0(int)\n x = np(offsets()+)\n o offsets:\n x += \n best_idx = np(() o offsets])\n rows += ]\n\ndf = pd(rows)\n (although few) eeg have multiple labels, just take the first one - this last part is not essential\ndf2 = df(subset=)()\nCongratulations on the 3rd place and thanks for sharing!
\nVery interesting approach with the split. I spent most of the competition using all data, because I could see no reason for using only samples with votes >=10, even if they worked well in the LB. In the last weeks I finally took time to read some papers, starting with those on Sparcnet. It was an eye opener. It finally made sense to use only the high quality samples. I followed an approach slightly different from yours:
\nI ended up with 6366 samples. I used all for training, applying weights to account for eeg_ids with a very large number of samples, which didn't exist in the test data, and the fact that the share of \"Other\" in the test sample was lower, according to the paper. For validation I picked only one sample per eeg_id: the one with the most votes or the longest in case of a tie.
\nWhat puzzled me is that adding low quality samples with pseudo labels to the training data, I was able to significantly improve the CV (0.23 for single model), but only get a small LB improvement. Moreover, after a certain point, the CV kept going down but the LB would start degrading just slightly. The public notebooks I looked at had a leak (they sampled high quality and low quality separately, thus mixing eegs from the same patient in train and test), but I couldn't find any leak in my case. I still don't know why CV improved so much, but LB didn't. Maybe this approach introduced a leak because of labeler bias. Any thoughts from your experience using pseudo labels for 1D models?
", "votes": null, "replies": [ { "id": 2749092, "author_name": "darraghdog", "author_url": "", "post_date": "04/12/2024 20:30:31", "content": "@vialactea thanks for your comments; re: the pseudo, we had a very similar experience - very promising CV score, but then on LB the pseudo trained model did worse than the base model which generated the pseudo. I just checked private LB scores and they were also the same.
\nI do not see any convincing reason; one possible idea I thought is that diversity among folds is higher with non-pseudo training, so when averaged the score is better - as opposed to with pseudo, where we may get similar predictions per fold which do not blend as well even though each individual checkpoint is better. I'm not sure if this is the reason though. Perhaps as you say it could have been a leak because of labeler bias. When a lot of effort was spent on it and score didnt improve, we switched to something else - there were a lot of other things to try.
\ncongratulations on your result in the competition.
Hi @vialactea When you use pseudo are you careful to make 5 sets of pseudo labels? one for each fold?
\nFor example we use fold 1 to pseudo label the vote<10 on fold 1 train data only (i.e don't use patients outside for fold 1) and save that as pseudo set 1. Then we use fold 2 to pseudo label the vote<10 on fold 2 and save that as pseudo set 2 etc.
Then in stage 2 training, we add the pseudo set 1 to fold 1 during training. Doing this prevents leaks and prevents optimistic CV score.
I experienced the same experience (improve CV drastically and deteriorated LB) with pseudo label. I pseudo label to OOF: first, I trained fold0 train with all data and make pseudo label to fold0 valid set (vote<10). I did this because I believed pseudo label to OOF set is safe since it only leaks features, not labels.
\nHowever, @cdeotte's approach is safest because there are no leaks of information about validation set including features. Then I wonder what is the mechanism of leaks in pseudo labeling OOF set.
\n@cdeotte Do you have in mind some particular case of leak when pseudo labeling OOF set?
", "votes": null, "replies": [ { "id": 2749223, "author_name": "cdeotte", "author_url": "", "post_date": "04/12/2024 23:13:03", "content": "Yes. Each fold has subset of patient_id. Therefore we cannot pseudo label all data. We must use fold 1 to only pseudo label the patient_id in fold 1. Otherwise there is a leak.
Hi @cdeote. Yes, I made sure that the pseudo labels for a stage 2 fold came from models trained without any of the patients in that fold. Moreover, all stage 1 and stage 2 models used the exact same 5 folds split of patients, with the caveat that some patients didn't have high quality hard labels and thus were not used in stage 1 training.
\nI couldn´t understand how come the stage 2 score for a fold would improve significantly when using pseudo labels produced from models that had been trained without seeing any of the patients in that fold, but that improvement would vanish in the LB. I expected ensembling to be affected due to a potential loss of diversity, as @darraghdog mentioned, but LB of single models didn't improve either.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2746463, "author_name": "christofhenkel", "author_url": "", "post_date": "04/11/2024 08:24:00", "content": "Most of the time take middle point of each sub-groups is good enough. But only if there is only one \"true\" point for the eeg_id.
\n
\nThis would be the resulting x from the block above for a more complex example. As you can see there are 2 local maxima. Those are most likely the offsets of the true datapoints. The plot also shows why in this example middle point is not the optimal way for filtering
Thanks. It all make sense.
", "votes": null, "replies": [ { "id": 2746710, "author_name": "yujiariyasu", "author_url": "", "post_date": "04/11/2024 12:16:26", "content": "btw, our pp got public: 0.0045 / private: 0.005 gain
", "votes": null, "replies": [ { "id": 2746930, "author_name": "darraghdog", "author_url": "", "post_date": "04/11/2024 14:44:32", "content": "that is a good jump!
", "votes": null, "replies": [] } ] } ] } ] } ] } ] }, { "id": 2746279, "author_name": "sunyuri", "author_url": "", "post_date": "04/11/2024 06:20:51", "content": "Congratulations! I have learned a lot.
", "votes": null, "replies": [ { "id": 2746407, "author_name": "darraghdog", "author_url": "", "post_date": "04/11/2024 07:45:47", "content": "The CBAM work from you & your team really helped a lot. It is a clever way to approach the problem as 1D model.
", "votes": null, "replies": [] } ] }, { "id": 2746714, "author_name": "ganeshtalwar", "author_url": "", "post_date": "04/11/2024 12:18:47", "content": "Thanks for sharing exciting report.
", "votes": null, "replies": [] }, { "id": 2748296, "author_name": "gunesevitan", "author_url": "", "post_date": "04/12/2024 11:07:32", "content": "Congrats! This is actually very similar to our solution. I tried melspectrograms but I couldn't make them work. They were way worse compared to cusignal STFT spectrograms. Our best models were MaxViT tiny and ConvNeXt base. I think you can get first place using those models.
", "votes": null, "replies": [ { "id": 2751855, "author_name": "darraghdog", "author_url": "", "post_date": "04/14/2024 15:01:38", "content": "Seems so 😏 will know next time. We had an idea in the final weeks that our 2d model was weaker but there were so many things to try. Congrats on your great work!
", "votes": null, "replies": [] } ] }, { "id": 2749553, "author_name": "aadityaporwal", "author_url": "", "post_date": "04/13/2024 05:45:43", "content": "Congratulations Dieter and Darragh!!!!!!!!
", "votes": null, "replies": [] }, { "id": 2799058, "author_name": "abaojiang", "author_url": "", "post_date": "05/07/2024 15:17:56", "content": "Congratulations on 3rd place! I'm sorry for the late question.
\nCurrently, I'm tracing your released code on github. And there's one thing I can't understand. I notice that you use masking in each SqueezeformerBlock. I categorize these masking operations into 3 types,
mask_pad for LlamaAttention.mask for convolution module.mask_flat for those commented with Skip/Unskip pad.Could you provide some hints about the intuition behind these design choices? Thanks a lot!
", "votes": null, "replies": [ { "id": 2807608, "author_name": "darraghdog", "author_url": "", "post_date": "05/11/2024 18:28:11", "content": "The masking is not used as our inputs have constant length. The mask was all 1, so just a dummy. We leveraged from a previous competition and discussed how it is used there.
", "votes": null, "replies": [ { "id": 2807992, "author_name": "abaojiang", "author_url": "", "post_date": "05/12/2024 03:25:53", "content": "Hi @darraghdog,
\nThanks for your explanation. I'll check how it's used in the fingerspelling competition!
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2743949": "Thanks to kaggle and everyone involved for hosting such an interesting competition. We learned a lot about EEG data and how to create strong models for it. Special thanks to @darraghdog who carried most of the workload in the last weeks of the competition, when I was busy with training my first real life neural network. \n\n## TLDR\n\nOur solution is an ensemble of multiple models from two diverse modeling approaches. The first approach is to apply pretrained 2d-CNN architectures to the MelSpectrogram transformation of the data. The second approach uses 1D-Convolutions to encode the raw eeg data before modeling with Squeezeformer blocks. \nKey ingredients for our solution were a robust cross validation based on data selection and creative augmentations. \n\n## Cross validation and Data filtering\n\nHaving a solid cross validation was certainly key to doing well in this competition! So we spend a lot of time figuring out a good validation scheme. In general we used a 4fold validation setup, split by patient id. One key ingredient was filtering the val dataset for those rows only having more than 9 votes. The [SPaRCNet paper](https://github.com/bdsp-core/IIIC-SPaRCNet/blob/main/IIIC_Classification-Supplemental.pdf) most likely explained how the data was created.\n\n\n\nNot only was the test data filtered by having more than 9 votes, but also the data has been augmented by using the same labels but shifting the eeg data. This means that the given 106k training rows are highly redundant and can/ should be filtered. \n\n> We expanded the high- and low-quality sets of EEG segments by adding additional segments belonging to the\nsame stationary period of the EEG\n\nThis basically means the authors applied a shift-like augmentation to the raw eeg data to create more data and used same labels for the newly created data. This explains why there were so many rows for the same EEG_id having the same label. We believe that separating the original data from the augmented data makes the data cleaner and improved our cross validation. So we put some effort into reverse-engineering this process and finding the original “true” data point belonging to a given label. So we ended up with a filtered dataset of **only 6350 rows** which we used not only for validation using our 4fold scheme but also mainly used that for training, and discarded the other 100k rows. At the end we had a quite good cv/ lb correlation\n\n\n\n## Data sources\nFor 2D models, we had pretrained weights and we observed high data quality was important and we got no benefit from using pseudo labels and data with <8 votes in the label. All models were trained with high quality data. \n\nOur 1D models were trained from scratch so it helped to also use the low count data. See below in the training procedure how low vote data was used in training. Pseudo labelling was not helpful for us - we tried a lot of things here. \n\nIn the end we did not use the 10 minute spectrograms. They may have helped slightly in the 2d model in some experiments but not in any major way. \n\n## Data preprocessing\nNo preprocessing of the data to disk was made, which really sped up experiments and allowed flexibility in approaches. We used torchaudio‘s GPU implementation to create the MelSpectrogram on the fly and normalized the signal. \n\nThe double banana montage, explained by @cdeotte was used throughout our solution. As mentioned by others, stacking 16 signals next to one another had limitations in that the each node would interact more strongly in the model with it’s neighbours. We experimented with ordering and found this order worked well, which essentially looks at the left and right side of the brain for each node together, \nFp1>F7 Fp2>F8 F7>T3 F8>T4 T3>T5 T4>T6 T5>O1 T6>O2 Fp1>F3 Fp2>F4 F3>C3 F4>C4 C3>P3 C4>P4 P3>O1 P4>O2\n\nScipy implementation of butter bandpass filter was used with an order of 2, and lowpass of between 0 and 1.5 Hz on different models, and a highpass of between 20 and 30 Hz. No notchfilter was used. \n \nWe observed it was important not to normalize the data batch-wise or sample-wise. We saw this in @medali1992 Resnet 1D GRU implementation - Not sure if this is where the idea originally came from. Instead, after the butter filter, we used the logic implemented `x = x.clip(-1024, 1024) / 32.`. \n\n## Augmentations\nFor 2D models, we had pretrained weights and we observed high data quality was more important - not all augmentations helped. Our 1D models were trained from scratch so heavy augmentations were used. In both models, augmentation was highly effective. \n\nThe following augmentations were used by boths models,\n- Overwrite with zeros between 1 to 8 melspec nodes, from the total 16, in 50% of samples.\n- Randomly choose a different narrower butter bandpass range in between 1 and 8 melspec nodes, from the total 16, in 20% of samples.\n- In 50% of samples, randomly shift the 50s window around the center point by up to 20 seconds. \n\nIn the 1D models only, \n\n- In 50% of samples, left-right flip the signal time wise. \n- In 50% of samples, switch the sides of the brain left-right. \n\nIn the 2D models only, \n\n- In 50% of samples for the zoomed center, using a static 50s window, randomly shift the 10 second center point within the window by up to 5 seconds. \n\n## Models\n\n### Melspec + 2D-CNN backbones\n\nLike most other competitors we used pretrained 2D CNN backbones which we fed with melspec transformation of the data. However, we made several improvements to publicly known approaches. The first major boost comes from not combining the data into regions of the EEG electrodes (LL, LP, RP, RL) but creating 16 individual mel spectrograms for the double banana montage. We then concatenated the 16 images into one large image. In the last week we found another big improvement, by “zooming” onto the center 10sec. Zooming was applied with different window and hop lengths to increase diversity. Mel-Spectrogram transformation are done as part of the model directly on GPU using torchaudios Mel-Spectrogram. \n\n\n\n### 1D-CNN + Squeezeformer blocks\n\nOur 1d CNN was heavily inspired by the work of Yuri Sun @sunyuri , and particularly his implementation of a lightweight CNN for seizure detection ([code and paper here](https://github.com/ThreePoundUniverse/2023JNE/blob/main/The_proposed_model.py)). As shown on the architecture the model focuses on grouped convolutions timewise and then channel wise. The first two blocks focus time-wise, keeping the channel information independent with grouped convolutions. After pooling the timewise information a channel wise convolution is made to interact across the nodes on the brain, and then another timewise convoltion. Technically we use conv2d, but the effect is like conv1d, only we can convolve over all channels in parallel. The original implementation used a CBAM attention block, we swapped this out and instead used 3 layers of squeezeformer. In addition we added pointwise convolutions before and after the depthwise convolutions which helped stabilite and learn better within the net. \nAs we train from scratch on relatively small data, keeping parameter count low was important. During ASL competition we had spent a lot of time on that, so leveraged our squeezeformer implementation from [there](https://github.com/ChristofHenkel/kaggle-asl-fingerspelling-1st-place-solution/tree/main). \n\n\n\n## Training procedure\n\n### Melspec + 2d CNN backbones\n\nTraining parameters used were generally 12 to 16 epochs, cosine learning rate decay with initial lr 0.0012 and batch size 32. Drop path was quite effective and set at 0.2. The main difference in the training was the granularity of the window lengths in the magnified 10 second window, different models were trained with different granularity and a mixnet_l and mixnet_xl backbones. \n\n### 1D CNN + SqueezeFormer blocks\n\nTraining parameters used were ~32 epochs, cosine learning rate decay with initial lr 0.001 and batch size 64. When we used low vote samples, we increased batch size to 256. We used a hidden dimension of 128 and dropout (attention, ff, head, conv) were all 0.1, going deeper on hidden dimension or more layers did not help. \nWe trained 3 different version of the 1d model with respect to data. \n- Only 8+ vote samples.\n- 3-8 vote count samples, and 8+ vote count samples. In each epoch the 3-8 votes dataset was subsampled; and a different loss was used for each set. We decayed the weight of the 3-8 batch over time, so it started with 0.8 weight and decayed with linear or cosine schedule to ~0.2 weight. This decay weighted loss was added to the loss of the 8+ vote dataset. \n- Finally, we added 1-2 votes as a separate decayed weight loss, and trained it in the same manner together with the 3-8 vote set and the 8+ set. The 1-2 vote set was very noisy and thus started with a weight on 0.5 and finished with 0. \n\nFor diversity, we trained some 1d models with butter bandpass order 0 and 1. They were not as strong but added some diversity in the blend. \n\n## Ensembling + Postprocessing\nUp until the last day we had been using mean blend of weights, with a higher weighting on 2d models. 15 hours before the end, with the help of GPT4, we created a simple neural net to learn the best blend weight on the 23 base models. 5 of the models were 1D CNN with different training procedures. 18 of the models were Mixnet 2d CNNs, with different backbones, and 10 second magnification windows. \nOn CV the learned blend weights lifted the score but affect on public LB was not so large. \nEarlier, we had been thinking that the distribution of the predictions were different to the distribution seen in the >8 votes portion of the training dataset. So about 10 hours from the end we added a bias term for each of the 6 classes to the nnet. This bias would be added to the logits. CV difference was small, but when the Public LB score came through it jumped us from 9th to 4th position which was exciting. The nnet is quite simple, see below. On private score, over the final days, we had been in the 0.27 LB prize zone with most of our blends even with mean weights. \n\n```\nclass Net(nn.Module):\n def __init__(self, n_models = 23):\n super(Net, self).__init__()\n self.fc = nn.Linear(n_models, 1, bias = False) \n self.fc_c = torch.nn.Parameter(torch.zeros(6)[None,:,None])\n def forward(self, x):\n return self. fc(x) + self.fc_c\n``` \n\n## Supplemental Data\nNo supplemental data was used\n\n## Ablation study (roughly)\n\n- Ordering of the 16 nodes signals -0.02 \n- Including low votes samples in 1D model -0.01\n- Augmentations -0.03 or more\n- Magnify annotated window -0.006; and more by blending different views\n- Remove sample wise and batch wise normalisation -0.02\n- FC layer to learn blend weights -0.003\n- Postprocessing with bias term -0.001\n\n\nBest single 1D model - 4 folds, 2 seed per fold - CV/Public/Private 0.257 / 0.25 / 0.30\nBest single 2D model - 4 folds, 2 seed per fold - CV/Public/Private 0.232 / 0.24 / 0.29\nFinal blend 4 folds, 2 seed, 23 model per fold - CV/Public/Private 0.207 / 0.22 / 0.27\n\n## Used tools/ repos\nPytorch\nTimm, huggingface, albumentations.\n@darraghdog used a 4090 GPU instance rented from runpod.io\nNeptune.ai was our MLOps stack to track, compare and share models. It was heavily used and allowed easy 4 fold **grouped** view of models and mean of the val scores for all folds - which made tracking a lot easier. \n\n\n\nThanks for reading. Questions are welcome. \n\nEdit:\nInference kernel: https://www.kaggle.com/code/darraghdog/3rd-place-solution \ngithub: https://github.com/darraghdog/kaggle-hms-3rd-place-solution", "2743973": "Congrats! Thanks for sharing.", "2743979": "Great solution (as always). I'll reread in more details, but first reaction is that you explain why one hing I tried did not work: I tried to sample eeg raw data from other locations than the middle and did not see any significant change.", "2744026": "`I was busy with training my first real life neural network.`\nCongratulations!", "2744036": "congratulations!\nNice work and nice solution writing\nThanks for your sharing", "2744060": "thank you for the explanation post. Amazing work!", "2744065": "Congrats! Thank you very much for sharing. Amazing work!", "2744247": "Congrats @christofhenkel on achieveing 3rd place.", "2744248": "Thank you very much for sharing this magnificent work,\nis there any chance for that final code released ?\nregards", "2744312": "Thanks for sharing! \nRegarding cv, we thought based on the data section that votes in test rounding from 3 to 20 so our validation was stratifiedgroupkfold stratifying on votes 3-20 and grouping by patients. We optimized this a lot and reached cv 0.45 using 1 stage only and trusted this too much and stopped working on 2 stages entirely. Really sad that all our work went into dust at the end due to wrong cv. \nOne interesting thing we found is how high is the impact of correct normalization. We dropped the normalization-per-image and adopted global normalization with some modifications. That boosted our cv by around 0.1 and public lb by 0.05.", "2744327": "Thanks @mohammad2012191 and yes, we had similar experiences on normalisation. Also, the effort you spent will payoff in different ways with experience gained 💪🏻, I always find that the best learning comes when things do not go as well as expected", "2744329": "A repo with source code will be added in the thread 🧵 in the couple of weeks", "2744395": "Lol it took me a sec to get what this meant 😂", "2744424": "Wow congratulations this is very helpful", "2744455": "Congratulations Dieter and Darragh\n>Best single 1D model - 4 folds, 2 seed per fold - CV/Public/Private 0.257 / 0.25 / 0.30\n\nWow. That's a very powerful 1D model. Great job!", "2744526": "Congrats! These single models are powerful. Excellent!", "2744675": "Congratulations on 3rd place in this competition. Thanks for sharing the details of your solutions with colorful diagrams.", "2744918": "Have fun with the little one", "2745291": "Congratulations on 3rd place in this competition. It's very nice to sharing with us !", "2745364": "Congratulations!! 🎉🎉, Love to see Squeezeformer again...", "2745791": "Congratulations! I am so surprised that you understood that `the authors applied a shift-like augmentation to the rEEGeeg data` and identified 6350 original data. While the data section of the competition site says\n\n`Many of these samples overlapped and have been consolidated. `\n\nAnd I thought that they gathered the 106,800 data (I was honestly surprised that they would collect them experimentally) as the row numbers and merged them.\n\nAlong with the high-quality data being used for the test, I wonder why the completion host would not mention it.", "2746009": "Congratulations. I especially liked idea of zooming center region of EEG.\n\nAnd our team also noticed the trick of processing test data, but only my team mate @yujiariyasu successfully squeezed gain from that. I should have dig deeper on that.\n\nThanks for sharing exciting report.", "2746279": "Congratulations! I have learned a lot.", "2746401": "When extra precision was released we saw the difference of the postprocessing bias term was CV/Public/Private 0.001 / 0.004 / 0.003", "2746403": "> So we put some effort into reverse-engineering this process and finding the original “true” data point belonging to a given label. So we ended up with a filtered dataset of only 6350 rows which we used not only for validation using our 4fold scheme but also mainly used that for training, and discarded the other 100k rows.\n\nAnyway, could you share us more details of reverse engineering *true* labels?\n\nOur approach is simple:\n1. grouping nearby labels into sub-groups which has same 6-dim vote vectors\n2. take middle point of each sub-groups\n3. filter only middle point\n\nDo you use more sophisticated ways?", "2746407": "The CBAM work from you & your team really helped a lot. It is a clever way to approach the problem as 1D model.", "2746447": "Here is the main block.\n```\n#group all eeg_id with same votes\ntrain['eeg_id_l'] = train[['eeg_id'] + label_columns].astype(str).agg('_'.join, axis=1)\n\n#for each unique eeg label combination calculate all overlaping time frames of same label and take the one row that has the highest overlap to all \nrows = []\nfor eeg_id_l in tqdm(train['eeg_id_l'].unique()):\n df0 = train[train['eeg_id_l']==eeg_id_l].reset_index(drop=True).copy()\n offsets = df0['spectrogram_label_offset_seconds'].astype(int).values\n x = np.zeros(offsets.max()+600)\n for o in offsets:\n x[o:o+600] += 1\n best_idx = np.argmax([x[o:o+600].sum() for o in offsets])\n rows += [df0.iloc[best_idx]]\n\ndf = pd.DataFrame(rows)\n#some (although few) eeg have multiple labels, just take the first one - this last part is not essential\ndf2 = df.drop_duplicates(subset='eeg_id').copy()\n```", "2746463": "Most of the time take middle point of each sub-groups is good enough. But only if there is only one \"true\" point for the eeg_id. \n\n\n\nThis would be the resulting `x` from the block above for a more complex example. As you can see there are 2 local maxima. Those are most likely the offsets of the true datapoints. The plot also shows why in this example middle point is not the optimal way for filtering", "2746561": "Thanks. It all make sense.", "2746710": "btw, our pp got public: 0.0045 / private: 0.005 gain", "2746714": "Thanks for sharing exciting report.", "2746930": "that is a good jump!", "2748296": "Congrats! This is actually very similar to our solution. I tried melspectrograms but I couldn't make them work. They were way worse compared to cusignal STFT spectrograms. Our best models were MaxViT tiny and ConvNeXt base. I think you can get first place using those models.", "2748429": "Congratulations on the 3rd place and thanks for sharing!\n\nVery interesting approach with the split. I spent most of the competition using all data, because I could see no reason for using only samples with votes >=10, even if they worked well in the LB. In the last weeks I finally took time to read some papers, starting with those on Sparcnet. It was an eye opener. It finally made sense to use only the high quality samples. I followed an approach slightly different from yours: \n1. pick only the eeg_ids whose maximum number of votes >=10\n2. take all consecutive samples with the same share of votes (and more than 10 votes) and pick the middle of those samples (10m, 50s, 20s, and 10s). I wanted to use the whole range and randomize, but given the kaggle limits on memory and GPU hours, I ended up going with just the middle. Samples with less than 10 votes were ignored.\n\nI ended up with 6366 samples. I used all for training, applying weights to account for eeg_ids with a very large number of samples, which didn't exist in the test data, and the fact that the share of \"Other\" in the test sample was lower, according to the paper. For validation I picked only one sample per eeg_id: the one with the most votes or the longest in case of a tie.\n\nWhat puzzled me is that adding low quality samples with pseudo labels to the training data, I was able to significantly improve the CV (0.23 for single model), but only get a small LB improvement. Moreover, after a certain point, the CV kept going down but the LB would start degrading just slightly. The public notebooks I looked at had a leak (they sampled high quality and low quality separately, thus mixing eegs from the same patient in train and test), but I couldn't find any leak in my case. I still don't know why CV improved so much, but LB didn't. Maybe this approach introduced a leak because of labeler bias. Any thoughts from your experience using pseudo labels for 1D models?", "2749092": "vialactea thanks for your comments; re: the pseudo, we had a very similar experience - very promising CV score, but then on LB the pseudo trained model did worse than the base model which generated the pseudo. I just checked private LB scores and they were also the same. \nI do not see any convincing reason; one possible idea I thought is that diversity among folds is higher with non-pseudo training, so when averaged the score is better - as opposed to with pseudo, where we may get similar predictions per fold which do not blend as well even though each individual checkpoint is better. I'm not sure if this is the reason though. Perhaps as you say it could have been a leak because of labeler bias. When a lot of effort was spent on it and score didnt improve, we switched to something else - there were a lot of other things to try.\ncongratulations on your result in the competition.", "2749106": "Hi @vialactea When you use pseudo are you careful to make 5 sets of pseudo labels? one for each fold?\n\nFor example we use `fold 1` to pseudo label the `vote<10` on `fold 1` train data only (i.e don't use patients outside for fold 1) and save that as `pseudo set 1`. Then we use `fold 2` to pseudo label the `vote<10` on `fold 2` and save that as `pseudo set 2` etc.\n\nThen in stage 2 training, we add the `pseudo set 1` to `fold 1` during training. Doing this prevents leaks and prevents optimistic CV score.", "2749205": "I experienced the same experience (improve CV drastically and deteriorated LB) with pseudo label. I pseudo label to OOF: first, I trained fold0 train with all data and make pseudo label to fold0 valid set (vote<10). I did this because I believed pseudo label to OOF set is safe since it only leaks features, not labels. \n\nHowever, @cdeotte's approach is safest because there are no leaks of information about validation set including features. Then I wonder what is the mechanism of leaks in pseudo labeling OOF set.\n\n@cdeotte Do you have in mind some particular case of leak when pseudo labeling OOF set?", "2749223": "Yes. Each fold has subset of `patient_id`. Therefore we cannot pseudo label all data. We must use `fold 1` to only pseudo label the `patient_id` in fold 1. Otherwise there is a leak.", "2749553": "Congratulations Dieter and Darragh!!!!!!!!", "2751855": "Seems so 😏 will know next time. We had an idea in the final weeks that our 2d model was weaker but there were so many things to try. Congrats on your great work!", "2761071": "Hi @cdeote. Yes, I made sure that the pseudo labels for a stage 2 fold came from models trained without any of the patients in that fold. Moreover, all stage 1 and stage 2 models used the exact same 5 folds split of patients, with the caveat that some patients didn't have high quality hard labels and thus were not used in stage 1 training.\n\nI couldn´t understand how come the stage 2 score for a fold would improve significantly when using pseudo labels produced from models that had been trained without seeing any of the patients in that fold, but that improvement would vanish in the LB. I expected ensembling to be affected due to a potential loss of diversity, as @darraghdog mentioned, but LB of single models didn't improve either.", "2764109": "I know, right. I even read the original paper and saw that chart, but discounted it in thinking \"they surely have given us original data and would tell us if anything was augmentation.\"", "2799058": "Congratulations on 3rd place! I'm sorry for the late question.\n\nCurrently, I'm tracing your released code on github. And there's one thing I can't understand. I notice that you use **masking** in each `SqueezeformerBlock`. I categorize these masking operations into 3 types,\n1. `mask_pad` for `LlamaAttention`.\n2. `mask` for convolution module.\n3. `mask_flat` for those commented with `Skip/Unskip pad`.\n\nCould you provide some hints about the intuition behind these design choices? Thanks a lot!", "2807608": "The masking is not used as our inputs have constant length. The mask was all 1, so just a dummy. We leveraged from a previous competition and [discussed](https://www.kaggle.com/competitions/asl-fingerspelling/discussion/434485) how it is used there.", "2807992": "Hi @darraghdog,\n\nThanks for your explanation. I'll check how it's used in the fingerspelling competition!" }, "source": "meta" }