![\"Cropper\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5570735%2F0bdfe4bb5759dfd538b2e473c9cac962%2Fhms-GeneralArch.jpg?generation=1712621808361384&alt=media\")
Thanks to Harvard Medical School and Kaggle for hosting this fun competition. The competition was very rewarding for us, as there was so much to learn!
\nTLDR; @ihebch and I built a multimodal model based on three feature extractors. These are an improved 1D-wavenet, a trainable STFT, and an image model with attention pooling for the competition spectrograms. We used hgnetv2_b4.ssld_stage2_ft_in1k and tf_efficientnetv2_s.in21k_ft_in1k backbones with different architecture configurations to diversify our ensemble. For more details, keep reading!
One of the most important parts of our pipeline was fast experimentation. This allowed us to try creative and crazy ideas quickly. The key points for us were to use smaller inputs, use smaller backbones, and create spectrograms on GPU using nnAudio. In the last few weeks of the competition, we scaled up from 8x to 24x node differences, increased the size of the model backbones, and concatenated the three feature vectors into one multimodal model.
\nModel 1: Improved 1D Wavenet
\nOur best single architecture is a wavenet model with some modifications. We add Mish as a third activation function and we use residual connections from the input convolution rather than within each layer. These changes allowed us to use deeper wavenets but we found no significant improvements going beyond 7 blocks. Finally, we added a downsample convolution after the last wave block to significantly reduce vRam usage.
\nOurs
\n![\"Cropper\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5570735%2F60bcbfa7f8bcde74c05b35c4b81eb392%2Fhms-WaveBlock_Ours.jpg?generation=1712621883006223&alt=media\")
Original
\n![\"Cropper\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5570735%2F151172231e6f88767c2bcd19a5b86e3b%2Fhms-WaveBlock_Original.jpg?generation=1712621909762554&alt=media\")
Each input sequence is passed through a wavenet with the improved blocks, and the outputs are stacked into a single-channel image. All but two of the output channels are stacked together, with the last two output channels being appended to the end of the image. We did this to create an area where features from all 24 node differences are nearby. The stacked outputs are then passed through a timm backbone to get a feature vector.
\n![\"Cropper\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5570735%2F049d909627d392a2b69bc10430ee048e%2Fhms-WaveNet_stack.jpg?generation=1712621986486094&alt=media\")
Model 2: STFT
\nWe generated STFTs on GPU using nnAudio. This is very fast, and each training run for this part of the model took 3-10 mins (depending on the STFT parameters). After generating the STFT, we restack the frequency bins into 2 columns. Finally, the STFTs for each input are stacked together and passed through an image model to get a feature vector.
\n![\"Cropper\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5570735%2F5c61eda5775994d937aca74efb840e19%2Fhms-STFTStack.jpg?generation=1712622013956043&alt=media\")
We are still unsure why the restacking method worked, but we wanted to share some more thoughts on this. We thought that maybe by reducing the height of the STFT, we were bringing information from different nodes closer together. We tested this by changing the order in which we stacked the STFTs, and saw significant changes to our CV. This indicated to us that the model was learning inter-STFT features and that the order we stacked mattered.
\nBased on these findings, we believed that we were missing some sort of attention mechanism between inputs. We started experimenting with this during the last few days of the competition but unfortunately ran out of time.
\nModel 3: Attention Pooling
\nThe final feature extractor we used was a timm backbone with attention pooling for the 10-min spectrograms. We found that adding the differences between the spectrograms as new images helped, and applying heavy dropout to the edges of images (0-95%) during training also helped.
\nAugmentations
\nThe most important augmentations for us were vertical brain flipping, horizontal brain flipping, and a variant of the CropCat[1] augmentation. Other augmentations we used were brain-size scaling and random noise. We used Brain Flipping during inference as TTA.
\nCropCat
\n![\"Cropper\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5570735%2Fb6df8385512f3fa34cc45942a7cd3977%2Fhms-CropCat.jpg?generation=1712622065747848&alt=media\")
Training Data
\nLike other competitors, we found that training on data with >= 10 votes resulted in better LB scores. We used GKF on this subset to create a correlated CV/LB. To create our training dataset, we grouped by eeg_id, selected the rows with the max number of votes, and then removed overlapping segments. We then sampled one data point per eeg_id per epoch. During training, we used weighted kldivergence to emphasize data points with more votes. We also sampled noisy student labels with a probability of 15% to increase model diversity.
Our solution would not have been possible without the informative notebooks and sharing of @cdeotte. Also, we mimicked the code structure of @christofhenkel's solution from the ASL Fingerspelling Competition here. Thank you both for sharing.
\nOur training code can be found here. Happy Kaggling!
\n[1] CropCat: Data Augmentation for Smoothing the Feature Distribution of EEG Signals. https://arxiv.org/abs/2212.06413
", "messages": [ { "id": "2742505", "postDate": "04/09/2024 00:22:23", "content": "Thanks to Harvard Medical School and Kaggle for hosting this fun competition. The competition was very rewarding for us, as there was so much to learn!
\nTLDR; @ihebch and I built a multimodal model based on three feature extractors. These are an improved 1D-wavenet, a trainable STFT, and an image model with attention pooling for the competition spectrograms. We used hgnetv2_b4.ssld_stage2_ft_in1k and tf_efficientnetv2_s.in21k_ft_in1k backbones with different architecture configurations to diversify our ensemble. For more details, keep reading!
One of the most important parts of our pipeline was fast experimentation. This allowed us to try creative and crazy ideas quickly. The key points for us were to use smaller inputs, use smaller backbones, and create spectrograms on GPU using nnAudio. In the last few weeks of the competition, we scaled up from 8x to 24x node differences, increased the size of the model backbones, and concatenated the three feature vectors into one multimodal model.
\nModel 1: Improved 1D Wavenet
\nOur best single architecture is a wavenet model with some modifications. We add Mish as a third activation function and we use residual connections from the input convolution rather than within each layer. These changes allowed us to use deeper wavenets but we found no significant improvements going beyond 7 blocks. Finally, we added a downsample convolution after the last wave block to significantly reduce vRam usage.
\nOurs
\n
Original
\n
Each input sequence is passed through a wavenet with the improved blocks, and the outputs are stacked into a single-channel image. All but two of the output channels are stacked together, with the last two output channels being appended to the end of the image. We did this to create an area where features from all 24 node differences are nearby. The stacked outputs are then passed through a timm backbone to get a feature vector.
\n
Model 2: STFT
\nWe generated STFTs on GPU using nnAudio. This is very fast, and each training run for this part of the model took 3-10 mins (depending on the STFT parameters). After generating the STFT, we restack the frequency bins into 2 columns. Finally, the STFTs for each input are stacked together and passed through an image model to get a feature vector.
\n
We are still unsure why the restacking method worked, but we wanted to share some more thoughts on this. We thought that maybe by reducing the height of the STFT, we were bringing information from different nodes closer together. We tested this by changing the order in which we stacked the STFTs, and saw significant changes to our CV. This indicated to us that the model was learning inter-STFT features and that the order we stacked mattered.
\nBased on these findings, we believed that we were missing some sort of attention mechanism between inputs. We started experimenting with this during the last few days of the competition but unfortunately ran out of time.
\nModel 3: Attention Pooling
\nThe final feature extractor we used was a timm backbone with attention pooling for the 10-min spectrograms. We found that adding the differences between the spectrograms as new images helped, and applying heavy dropout to the edges of images (0-95%) during training also helped.
\nAugmentations
\nThe most important augmentations for us were vertical brain flipping, horizontal brain flipping, and a variant of the CropCat[1] augmentation. Other augmentations we used were brain-size scaling and random noise. We used Brain Flipping during inference as TTA.
\nCropCat
\n
Training Data
\nLike other competitors, we found that training on data with >= 10 votes resulted in better LB scores. We used GKF on this subset to create a correlated CV/LB. To create our training dataset, we grouped by eeg_id, selected the rows with the max number of votes, and then removed overlapping segments. We then sampled one data point per eeg_id per epoch. During training, we used weighted kldivergence to emphasize data points with more votes. We also sampled noisy student labels with a probability of 15% to increase model diversity.
Our solution would not have been possible without the informative notebooks and sharing of @cdeotte. Also, we mimicked the code structure of @christofhenkel's solution from the ASL Fingerspelling Competition here. Thank you both for sharing.
\nOur training code can be found here. Happy Kaggling!
\n[1] CropCat: Data Augmentation for Smoothing the Feature Distribution of EEG Signals. https://arxiv.org/abs/2212.06413
", "rawMarkdown": "Thanks to Harvard Medical School and Kaggle for hosting this fun competition. The competition was very rewarding for us, as there was so much to learn!\n\nTLDR; @ihebch and I built a multimodal model based on three feature extractors. These are an improved 1D-wavenet, a trainable STFT, and an image model with attention pooling for the competition spectrograms. We used `hgnetv2_b4.ssld_stage2_ft_in1k` and `tf_efficientnetv2_s.in21k_ft_in1k` backbones with different architecture configurations to diversify our ensemble. For more details, keep reading!\n\n\n\n## Fast Experimentation\n\nOne of the most important parts of our pipeline was fast experimentation. This allowed us to try creative and crazy ideas quickly. The key points for us were to use smaller inputs, use smaller backbones, and create spectrograms on GPU using nnAudio. In the last few weeks of the competition, we scaled up from 8x to 24x node differences, increased the size of the model backbones, and concatenated the three feature vectors into one multimodal model.\n\n**Model 1: Improved 1D Wavenet**\n\nOur best single architecture is a wavenet model with some modifications. We add Mish as a third activation function and we use residual connections from the input convolution rather than within each layer. These changes allowed us to use deeper wavenets but we found no significant improvements going beyond 7 blocks. Finally, we added a downsample convolution after the last wave block to significantly reduce vRam usage.\n\nOurs\n\n\n\nOriginal\n\n\n\nEach input sequence is passed through a wavenet with the improved blocks, and the outputs are stacked into a single-channel image. All but two of the output channels are stacked together, with the last two output channels being appended to the end of the image. We did this to create an area where features from all 24 node differences are nearby. The stacked outputs are then passed through a timm backbone to get a feature vector.\n\n\n\n**Model 2: STFT**\n\nWe generated STFTs on GPU using nnAudio. This is very fast, and each training run for this part of the model took 3-10 mins (depending on the STFT parameters). After generating the STFT, we restack the frequency bins into 2 columns. Finally, the STFTs for each input are stacked together and passed through an image model to get a feature vector.\n\n\n\nWe are still unsure why the restacking method worked, but we wanted to share some more thoughts on this. We thought that maybe by reducing the height of the STFT, we were bringing information from different nodes closer together. We tested this by changing the order in which we stacked the STFTs, and saw significant changes to our CV. This indicated to us that the model was learning inter-STFT features and that the order we stacked mattered. \n\nBased on these findings, we believed that we were missing some sort of attention mechanism between inputs. We started experimenting with this during the last few days of the competition but unfortunately ran out of time.\n\n**Model 3: Attention Pooling**\n\nThe final feature extractor we used was a timm backbone with attention pooling for the 10-min spectrograms. We found that adding the differences between the spectrograms as new images helped, and applying heavy dropout to the edges of images (0-95%) during training also helped.\n\n## Other Strategies\n\n**Augmentations**\n\nThe most important augmentations for us were vertical brain flipping, horizontal brain flipping, and a variant of the CropCat[1] augmentation. Other augmentations we used were brain-size scaling and random noise. We used Brain Flipping during inference as TTA.\n\nCropCat\n\n\n\n**Training Data**\n\nLike other competitors, we found that training on data with >= 10 votes resulted in better LB scores. We used GKF on this subset to create a correlated CV/LB. To create our training dataset, we grouped by `eeg_id`, selected the rows with the max number of votes, and then removed overlapping segments. We then sampled one data point per eeg_id per epoch. During training, we used weighted kldivergence to emphasize data points with more votes. We also sampled noisy student labels with a probability of 15% to increase model diversity. \n\n## Final Note\n\nOur solution would not have been possible without the informative notebooks and sharing of @cdeotte. Also, we mimicked the code structure of @christofhenkel's solution from the ASL Fingerspelling Competition [here](https://github.com/ChristofHenkel/kaggle-asl-fingerspelling-1st-place-solution). Thank you both for sharing. \n\nOur training code can be found [here](https://github.com/brendanartley/HMS-Competition). Happy Kaggling!\n\n## Frameworks\n\n- [Pytorch Lightning](https://lightning.ai/docs/pytorch/stable/) (training)\n- [Weights + Biases](https://wandb.ai/site) (logging)\n- [Timm](https://huggingface.co/timm) (backbones)\n- [nnAudio](https://github.com/KinWaiCheuk/nnAudio) (STFTs)\n\n# Sources\n\n[1] CropCat: Data Augmentation for Smoothing the Feature Distribution of EEG Signals. https://arxiv.org/abs/2212.06413",
"votes": null
},
{
"id": "2742534",
"postDate": "04/09/2024 00:41:25",
"content": "Thanks for Sharing !
", "rawMarkdown": "Thanks for Sharing !", "votes": null }, { "id": "2742550", "postDate": "04/09/2024 00:50:26", "content": "Great work! Congrats to you both on your new tiers!!!!
", "rawMarkdown": "Great work! Congrats to you both on your new tiers!!!!", "votes": null }, { "id": "2742610", "postDate": "04/09/2024 01:43:28", "content": "What a solid solution! I adopted attention pooling from https://github.com/openai/CLIP/blob/main/clip/model.py#L58, however it didn't work well. May I ask which attention pooling you used @brendanartley
", "rawMarkdown": "What a solid solution! I adopted attention pooling from https://github.com/openai/CLIP/blob/main/clip/model.py#L58, however it didn't work well. May I ask which attention pooling you used @brendanartley", "votes": null }, { "id": "2742615", "postDate": "04/09/2024 01:49:22", "content": "@brendanartley Nice detail explanation and great achievement and is possible to share the training code.
", "rawMarkdown": "brendanartley Nice detail explanation and great achievement and is possible to share the training code.", "votes": null }, { "id": "2742619", "postDate": "04/09/2024 01:55:26", "content": "Of course!
\nCode uploaded here.
", "rawMarkdown": "Of course! ~~We plan to have our code cleaned and uploaded tomorrow.~~\n\nCode uploaded [here](https://github.com/brendanartley/HMS-Competition).", "votes": null }, { "id": "2742621", "postDate": "04/09/2024 01:58:17", "content": "Thanks @huyduong7101. We used attention pooling in the same way as this model.
", "rawMarkdown": "Thanks @huyduong7101. We used attention pooling in the same way as [this model](https://github.com/brendanartley/UBCO-Competition/blob/a6bc71bfc057be979866cfffc58b2855bcb7870f/ubco_stage2/mil_model/model.py#L5).", "votes": null }, { "id": "2742790", "postDate": "04/09/2024 04:57:09", "content": "Thanks for this detailed post about your solution.
\nMay I ask for more information about this kind of method?
\n", "rawMarkdown": "Thanks for this detailed post about your solution.\nMay I ask for more information about this kind of method? \n>We used GKF on this subset to create a correlated CV/LB.", "votes": null }, { "id": "2743496", "postDate": "04/09/2024 13:37:25", "content": "We used GKF on this subset to create a correlated CV/LB.
\n
Sure. When we calculate our CV locally, we used StratifiedGroupKFold. This meant that each out of fold training set had a similar distribution, and that the eeg_ids were \"unseen\". The reason we say this is correlated, is because when we improved our CV score, we also improved when submitting to the LB.
On the other hand, when we trained + evaluated with data that had total_votes < 10, improvements to our CV did not always lead to improvements on the LB. In this case the CV/LB was not correlated.
", "rawMarkdown": "Sure. When we calculate our CV locally, we used [StratifiedGroupKFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedGroupKFold.html). This meant that each out of fold training set had a similar distribution, and that the `eeg_id`s were \"unseen\". The reason we say this is correlated, is because when we improved our CV score, we also improved when submitting to the LB. \n\nOn the other hand, when we trained + evaluated with data that had total_votes < 10, improvements to our CV did not always lead to improvements on the LB. In this case the CV/LB was not correlated.", "votes": null }, { "id": "2746177", "postDate": "04/11/2024 05:08:49", "content": "Thanks for detailed explanation, it will be very helpful. Congrats to both on your new tiers!!!
", "rawMarkdown": "Thanks for detailed explanation, it will be very helpful. Congrats to both on your new tiers!!!", "votes": null }, { "id": "2752363", "postDate": "04/14/2024 23:46:46", "content": "Hi! Are you sure you used 10min spectrograms instead of 10 sec? Looks like a typo
", "rawMarkdown": "Hi! Are you sure you used 10min spectrograms instead of 10 sec? Looks like a typo", "votes": null }, { "id": "2752365", "postDate": "04/14/2024 23:52:33", "content": "Hi @asimandia, the 10-min spectrogram refers to the spectrograms provided by the competition hosts. See spectrogram_id and spectrogram_sub_id on the data page here. Hope this helps 🙂
Thanks for Sharing !
", "votes": null, "replies": [] }, { "id": 2742550, "author_name": "cody11null", "author_url": "", "post_date": "04/09/2024 00:50:26", "content": "Great work! Congrats to you both on your new tiers!!!!
", "votes": null, "replies": [] }, { "id": 2742610, "author_name": "huyduong7101", "author_url": "", "post_date": "04/09/2024 01:43:28", "content": "What a solid solution! I adopted attention pooling from https://github.com/openai/CLIP/blob/main/clip/model.py#L58, however it didn't work well. May I ask which attention pooling you used @brendanartley
", "votes": null, "replies": [ { "id": 2742621, "author_name": "brendanartley", "author_url": "", "post_date": "04/09/2024 01:58:17", "content": "Thanks @huyduong7101. We used attention pooling in the same way as this model.
", "votes": null, "replies": [] } ] }, { "id": 2742615, "author_name": "seshurajup", "author_url": "", "post_date": "04/09/2024 01:49:22", "content": "@brendanartley Nice detail explanation and great achievement and is possible to share the training code.
", "votes": null, "replies": [ { "id": 2742619, "author_name": "brendanartley", "author_url": "", "post_date": "04/09/2024 01:55:26", "content": "Of course!
\nCode uploaded here.
", "votes": null, "replies": [] } ] }, { "id": 2742790, "author_name": "lmhongkhnh", "author_url": "", "post_date": "04/09/2024 04:57:09", "content": "Thanks for this detailed post about your solution.
\nMay I ask for more information about this kind of method?
\n", "votes": null, "replies": [ { "id": 2743496, "author_name": "brendanartley", "author_url": "", "post_date": "04/09/2024 13:37:25", "content": "We used GKF on this subset to create a correlated CV/LB.
\n
Sure. When we calculate our CV locally, we used StratifiedGroupKFold. This meant that each out of fold training set had a similar distribution, and that the eeg_ids were \"unseen\". The reason we say this is correlated, is because when we improved our CV score, we also improved when submitting to the LB.
On the other hand, when we trained + evaluated with data that had total_votes < 10, improvements to our CV did not always lead to improvements on the LB. In this case the CV/LB was not correlated.
", "votes": null, "replies": [] } ] }, { "id": 2746177, "author_name": "aadityaporwal", "author_url": "", "post_date": "04/11/2024 05:08:49", "content": "Thanks for detailed explanation, it will be very helpful. Congrats to both on your new tiers!!!
", "votes": null, "replies": [] }, { "id": 2752363, "author_name": "asimandia", "author_url": "", "post_date": "04/14/2024 23:46:46", "content": "Hi! Are you sure you used 10min spectrograms instead of 10 sec? Looks like a typo
", "votes": null, "replies": [ { "id": 2752365, "author_name": "brendanartley", "author_url": "", "post_date": "04/14/2024 23:52:33", "content": "Hi @asimandia, the 10-min spectrogram refers to the spectrograms provided by the competition hosts. See spectrogram_id and spectrogram_sub_id on the data page here. Hope this helps 🙂
\n\n## Fast Experimentation\n\nOne of the most important parts of our pipeline was fast experimentation. This allowed us to try creative and crazy ideas quickly. The key points for us were to use smaller inputs, use smaller backbones, and create spectrograms on GPU using nnAudio. In the last few weeks of the competition, we scaled up from 8x to 24x node differences, increased the size of the model backbones, and concatenated the three feature vectors into one multimodal model.\n\n**Model 1: Improved 1D Wavenet**\n\nOur best single architecture is a wavenet model with some modifications. We add Mish as a third activation function and we use residual connections from the input convolution rather than within each layer. These changes allowed us to use deeper wavenets but we found no significant improvements going beyond 7 blocks. Finally, we added a downsample convolution after the last wave block to significantly reduce vRam usage.\n\nOurs\n\n\n\nOriginal\n\n\n\nEach input sequence is passed through a wavenet with the improved blocks, and the outputs are stacked into a single-channel image. All but two of the output channels are stacked together, with the last two output channels being appended to the end of the image. We did this to create an area where features from all 24 node differences are nearby. The stacked outputs are then passed through a timm backbone to get a feature vector.\n\n\n\n**Model 2: STFT**\n\nWe generated STFTs on GPU using nnAudio. This is very fast, and each training run for this part of the model took 3-10 mins (depending on the STFT parameters). After generating the STFT, we restack the frequency bins into 2 columns. Finally, the STFTs for each input are stacked together and passed through an image model to get a feature vector.\n\n\n\nWe are still unsure why the restacking method worked, but we wanted to share some more thoughts on this. We thought that maybe by reducing the height of the STFT, we were bringing information from different nodes closer together. We tested this by changing the order in which we stacked the STFTs, and saw significant changes to our CV. This indicated to us that the model was learning inter-STFT features and that the order we stacked mattered. \n\nBased on these findings, we believed that we were missing some sort of attention mechanism between inputs. We started experimenting with this during the last few days of the competition but unfortunately ran out of time.\n\n**Model 3: Attention Pooling**\n\nThe final feature extractor we used was a timm backbone with attention pooling for the 10-min spectrograms. We found that adding the differences between the spectrograms as new images helped, and applying heavy dropout to the edges of images (0-95%) during training also helped.\n\n## Other Strategies\n\n**Augmentations**\n\nThe most important augmentations for us were vertical brain flipping, horizontal brain flipping, and a variant of the CropCat[1] augmentation. Other augmentations we used were brain-size scaling and random noise. We used Brain Flipping during inference as TTA.\n\nCropCat\n\n\n\n**Training Data**\n\nLike other competitors, we found that training on data with >= 10 votes resulted in better LB scores. We used GKF on this subset to create a correlated CV/LB. To create our training dataset, we grouped by `eeg_id`, selected the rows with the max number of votes, and then removed overlapping segments. We then sampled one data point per eeg_id per epoch. During training, we used weighted kldivergence to emphasize data points with more votes. We also sampled noisy student labels with a probability of 15% to increase model diversity. \n\n## Final Note\n\nOur solution would not have been possible without the informative notebooks and sharing of @cdeotte. Also, we mimicked the code structure of @christofhenkel's solution from the ASL Fingerspelling Competition [here](https://github.com/ChristofHenkel/kaggle-asl-fingerspelling-1st-place-solution). Thank you both for sharing. \n\nOur training code can be found [here](https://github.com/brendanartley/HMS-Competition). Happy Kaggling!\n\n## Frameworks\n\n- [Pytorch Lightning](https://lightning.ai/docs/pytorch/stable/) (training)\n- [Weights + Biases](https://wandb.ai/site) (logging)\n- [Timm](https://huggingface.co/timm) (backbones)\n- [nnAudio](https://github.com/KinWaiCheuk/nnAudio) (STFTs)\n\n# Sources\n\n[1] CropCat: Data Augmentation for Smoothing the Feature Distribution of EEG Signals. https://arxiv.org/abs/2212.06413",
"2742534": "Thanks for Sharing !",
"2742550": "Great work! Congrats to you both on your new tiers!!!!",
"2742610": "What a solid solution! I adopted attention pooling from https://github.com/openai/CLIP/blob/main/clip/model.py#L58, however it didn't work well. May I ask which attention pooling you used @brendanartley",
"2742615": "brendanartley Nice detail explanation and great achievement and is possible to share the training code.",
"2742619": "Of course! ~~We plan to have our code cleaned and uploaded tomorrow.~~\n\nCode uploaded [here](https://github.com/brendanartley/HMS-Competition).",
"2742621": "Thanks @huyduong7101. We used attention pooling in the same way as [this model](https://github.com/brendanartley/UBCO-Competition/blob/a6bc71bfc057be979866cfffc58b2855bcb7870f/ubco_stage2/mil_model/model.py#L5).",
"2742790": "Thanks for this detailed post about your solution.\nMay I ask for more information about this kind of method? \n>We used GKF on this subset to create a correlated CV/LB.",
"2743496": "Sure. When we calculate our CV locally, we used [StratifiedGroupKFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedGroupKFold.html). This meant that each out of fold training set had a similar distribution, and that the `eeg_id`s were \"unseen\". The reason we say this is correlated, is because when we improved our CV score, we also improved when submitting to the LB. \n\nOn the other hand, when we trained + evaluated with data that had total_votes < 10, improvements to our CV did not always lead to improvements on the LB. In this case the CV/LB was not correlated.",
"2746177": "Thanks for detailed explanation, it will be very helpful. Congrats to both on your new tiers!!!",
"2752363": "Hi! Are you sure you used 10min spectrograms instead of 10 sec? Looks like a typo",
"2752365": "Hi @asimandia, the 10-min spectrogram refers to the spectrograms provided by the competition hosts. See `spectrogram_id` and `spectrogram_sub_id` on the data page [here](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/data). Hope this helps 🙂"
},
"source": "meta"
}