![](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Apr-2024/single1.png\")
Very fun competition! I love competitions with opportunities to be creative, explore, and make discoveries! In the past 3 months, I enthusiastically conducted 1000+ experiments!
\nI also love competitions where a single model can achieve Gold medal (instead of a large ensemble).
\nThank you Kagglers for all the great discussion shares and notebook shares!
\nThe task in this competition is to predict annotator opinions. By reading the host's paper here, we learn that there are 119 overall annotators in train data and 20 expert annotators in test data. The paper shows the average Seizure prediction from 119 annotators is 18.8% whereas the 20 experts are less likely to classify as Seizure and their average is 1.5%.
\nWow, this is a big difference therefore we need to train our models with expert opinions instead of overall opinions. We can locate the expert annotators in the train data with the filter expert annotators = train.loc[train.vote_count>=10].
From the paper we see that the annotators see 2D spectrograms and 2D plots of waveforms. So let's input 2D spectrograms and 2D plots of waveforms into our models. (Also inputting 1D raw waveform helped too).
\nFirst we train 3 separate models using train data with vote_count >= 10 (i.e. the expert annotator opinions).
tiny_vit_21m_512 to classify into 6 targets. tiny_vit_21m_224 to classify into 6 targets. EegNet-1D here to classify into 6 targets.
![](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Apr-2024/single2.png\")
Using our 3 single models above, we pseudo label the train data with vote<10. We set the new labels as new_label = 0.1 * old_label + 0.3 * model1 + 0.3 * model2 + 0.3 * model3.
We load our 3 pretrained models above and remove the 3 heads. We combine them into one model, concatenate the final embeddings and add 1 new head. Then we train with 2x copies of vote>=10 data concatenated with 1x copy of vote<10 with pseudo data. (Note the 2D spectrogram model above actually trains with both 10 sec spectrograms and 50 spectrograms in one image as depicted below):
![](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Apr-2024/mega_model.png\")
To successfully accomplish the above and achieve single model Gold Medal Hooray!, there are many small details. Here are some:
\nPreprocess all raw waveform with MNE library
\n mne.filter import filter_data, notch_filter\n sample = data.T[[0,4,5,6, 11,15,16,17, 0,1,2,3, 11,12,13,14]]\\\n - data.T[[4,5,6,7, 15,16,17,18, 1,2,3,7, 12,13,14,18]]\n sample = notch_filter(sample.astype(), 200, 60, =32, =)\n sample = filter_data(sample.astype(), 200, 0.5, 40, =32, =) \n sample = np.clip(sample,-500,500)\n sample = np.nan_to_num(sample, =0)\nUse all 107k rows of train data (instead of only 1 row per eeg_id).
Read all eeg parquets into CPU RAM and create spectrograms inside dataloader with torchaudio.transforms.MelSpectrogram and crop with eeg_label_offset_seconds.
import torchaudio.transforms as T\n\nmake_spec1 = T.MelSpectrogram\n\nmake_spec2 = T.MelSpectrogram\nEach epoch train with 1 sample from each unique eeg_id. But randomly select 1 sample from all eeg_id samples with train = train.loc[train.eeg_id == EEG_ID].sample(1).
Apply data augmentations to raw waveform (not spectrogram) inside dataloader before creating spectrogram.
Use data augmentation \"brain-right-left flip\", \"brain-temporal-parasagittal-flip\", \"waveform-crop-rescale\", \"waveform-invert\", \"frequency-dropout\"
Use tiny_vit_21m_512 for 2D spectrograms and tiny_vit_21m_224 for 2D wave plots.
Use Nischay EegNet-1D here for 1D raw wave data model.
Tune LR and learning schedule of each model individually using my discussion post here
Cross validate with train vote>=10 and observe perfect correlation between CV and LB.
My single model inference notebook is published here. This single model achieves Gold 14th place. My final submission is an ensemble of this model with some other models and achieves Gold 8th place.
\nThe code describes the PyTorch model details, the preprocess details, PyTorch data loader details, and data augmentation details. Enjoy!
", "messages": [ { "id": 2744038, "postDate": "2024-04-09T18:08:31.937Z", "content": "Very fun competition! I love competitions with opportunities to be creative, explore, and make discoveries! In the past 3 months, I enthusiastically conducted 1000+ experiments!
\nI also love competitions where a single model can achieve Gold medal (instead of a large ensemble).
\nThank you Kagglers for all the great discussion shares and notebook shares!
\nThe task in this competition is to predict annotator opinions. By reading the host's paper here, we learn that there are 119 overall annotators in train data and 20 expert annotators in test data. The paper shows the average Seizure prediction from 119 annotators is 18.8% whereas the 20 experts are less likely to classify as Seizure and their average is 1.5%.
\nWow, this is a big difference therefore we need to train our models with expert opinions instead of overall opinions. We can locate the expert annotators in the train data with the filter expert annotators = train.loc[train.vote_count>=10].
From the paper we see that the annotators see 2D spectrograms and 2D plots of waveforms. So let's input 2D spectrograms and 2D plots of waveforms into our models. (Also inputting 1D raw waveform helped too).
\nFirst we train 3 separate models using train data with vote_count >= 10 (i.e. the expert annotator opinions).
tiny_vit_21m_512 to classify into 6 targets. tiny_vit_21m_224 to classify into 6 targets. EegNet-1D here to classify into 6 targets.
Using our 3 single models above, we pseudo label the train data with vote<10. We set the new labels as new_label = 0.1 * old_label + 0.3 * model1 + 0.3 * model2 + 0.3 * model3.
We load our 3 pretrained models above and remove the 3 heads. We combine them into one model, concatenate the final embeddings and add 1 new head. Then we train with 2x copies of vote>=10 data concatenated with 1x copy of vote<10 with pseudo data. (Note the 2D spectrogram model above actually trains with both 10 sec spectrograms and 50 spectrograms in one image as depicted below):
To successfully accomplish the above and achieve single model Gold Medal Hooray!, there are many small details. Here are some:
\nPreprocess all raw waveform with MNE library
\n mne.filter import filter_data, notch_filter\n sample = data.T[[0,4,5,6, 11,15,16,17, 0,1,2,3, 11,12,13,14]]\\\n - data.T[[4,5,6,7, 15,16,17,18, 1,2,3,7, 12,13,14,18]]\n sample = notch_filter(sample.astype(), 200, 60, =32, =)\n sample = filter_data(sample.astype(), 200, 0.5, 40, =32, =) \n sample = np.clip(sample,-500,500)\n sample = np.nan_to_num(sample, =0)\nUse all 107k rows of train data (instead of only 1 row per eeg_id).
Read all eeg parquets into CPU RAM and create spectrograms inside dataloader with torchaudio.transforms.MelSpectrogram and crop with eeg_label_offset_seconds.
import torchaudio.transforms as T\n\nmake_spec1 = T.MelSpectrogram\n\nmake_spec2 = T.MelSpectrogram\nEach epoch train with 1 sample from each unique eeg_id. But randomly select 1 sample from all eeg_id samples with train = train.loc[train.eeg_id == EEG_ID].sample(1).
Apply data augmentations to raw waveform (not spectrogram) inside dataloader before creating spectrogram.
Use data augmentation \"brain-right-left flip\", \"brain-temporal-parasagittal-flip\", \"waveform-crop-rescale\", \"waveform-invert\", \"frequency-dropout\"
Use tiny_vit_21m_512 for 2D spectrograms and tiny_vit_21m_224 for 2D wave plots.
Use Nischay EegNet-1D here for 1D raw wave data model.
Tune LR and learning schedule of each model individually using my discussion post here
Cross validate with train vote>=10 and observe perfect correlation between CV and LB.
My single model inference notebook is published here. This single model achieves Gold 14th place. My final submission is an ensemble of this model with some other models and achieves Gold 8th place.
\nThe code describes the PyTorch model details, the preprocess details, PyTorch data loader details, and data augmentation details. Enjoy!
", "rawMarkdown": "# Thank You Kaggle and Hosts!\nVery fun competition! I love competitions with opportunities to be creative, explore, and make discoveries! In the past 3 months, I enthusiastically conducted 1000+ experiments!\n\nI also love competitions where a single model can achieve Gold medal (instead of a large ensemble).\n\nThank you Kagglers for all the great discussion shares and notebook shares!\n\n# Understand Test Data - What is Competition Task?\nThe task in this competition is to **predict annotator opinions**. By reading the host's paper [here][1], we learn that there are **119 overall annotators** in train data and **20 expert annotators** in test data. The paper shows the average Seizure prediction from 119 annotators is 18.8% whereas the 20 experts are less likely to classify as Seizure and their average is 1.5%. \n\nWow, this is a big difference therefore we need to train our models with **expert opinions** instead of **overall opinions**. We can locate the expert annotators in the train data with the filter `expert annotators = train.loc[train.vote_count>=10]`.\n\n# Which Features Do Annotators See?\nFrom the paper we see that the annotators see 2D spectrograms and 2D plots of waveforms. So let's input 2D spectrograms and 2D plots of waveforms into our models. (Also inputting 1D raw waveform helped too).\n\n# Train 3 Separate Models with Vote>=10\nFirst we train 3 separate models using train data with `vote_count >= 10` (i.e. the **expert annotator opinions**). \n* One model receives 2D spectrograms (created from EEG) as input and uses `tiny_vit_21m_512` to classify into 6 targets. \n* Second model receives 2D matplotlib plots of the montage waveforms converted into a 2D image and uses `tiny_vit_21m_224` to classify into 6 targets. \n* And third model receives 1D raw time series waveform and uses `EegNet-1D` [here][2] to classify into 6 targets.\n\n\n\n\n\n# Pseudo Label Vote<10\nUsing our 3 single models above, we pseudo label the train data with `vote<10`. We set the new labels as `new_label = 0.1 * old_label + 0.3 * model1 + 0.3 * model2 + 0.3 * model3`.\n\n# Build One Mega Model and Train with Vote>=10 plus Pseudo Vote<10\nWe load our 3 pretrained models above and remove the 3 heads. We combine them into one model, concatenate the final embeddings and add 1 new head. Then we train with 2x copies of `vote>=10` data concatenated with 1x copy of `vote<10 with pseudo` data. (Note the 2D spectrogram model above actually trains with both 10 sec spectrograms and 50 spectrograms in one image as depicted below):\n\n\n\n# Single Model Performance\n## ==> 🎉 CV = 0.246, Public LB = 0.243, Private LB = 0.290 🎉\n\n# More Details\nTo successfully accomplish the above and achieve **single model Gold Medal** Hooray!, there are many small details. Here are some:\n* Preprocess all raw waveform with MNE library \n\n from mne.filter import filter_data, notch_filter\n sample = data.T[[0,4,5,6, 11,15,16,17, 0,1,2,3, 11,12,13,14]]\\\n - data.T[[4,5,6,7, 15,16,17,18, 1,2,3,7, 12,13,14,18]]\n sample = notch_filter(sample.astype('float64'), 200, 60, n_jobs=32, verbose='ERROR')\n sample = filter_data(sample.astype('float64'), 200, 0.5, 40, n_jobs=32, verbose='ERROR') \n sample = np.clip(sample,-500,500)\n sample = np.nan_to_num(sample, nan=0)\n\n* Use all 107k rows of train data (instead of only 1 row per eeg_id).\n* Read all eeg parquets into CPU RAM and create spectrograms inside dataloader with `torchaudio.transforms.MelSpectrogram` and crop with `eeg_label_offset_seconds`.\n\n import torchaudio.transforms as T\n # 50 SECOND SPEC\n make_spec1 = T.MelSpectrogram(n_fft=2048, win_length=1280, hop_length=19,\n f_min=0,f_max=20,sample_rate=200,n_mels=64).to(device)\n # MIDDLE 10 SECOND SPEC\n make_spec2 = T.MelSpectrogram(n_fft=1280, win_length=32, hop_length=4,\n f_min=0,f_max=20,sample_rate=200,n_mels=64).to(device)\n\n* Each epoch train with 1 sample from each unique `eeg_id`. But randomly select 1 sample from all `eeg_id` samples with `train = train.loc[train.eeg_id == EEG_ID].sample(1)`.\n\n* Apply data augmentations to raw waveform (not spectrogram) inside dataloader before creating spectrogram.\n* Use data augmentation \"brain-right-left flip\", \"brain-temporal-parasagittal-flip\", \"waveform-crop-rescale\", \"waveform-invert\", \"frequency-dropout\"\n\n* Use `tiny_vit_21m_512` for 2D spectrograms and `tiny_vit_21m_224` for 2D wave plots.\n* Use Nischay EegNet-1D [here][2] for 1D raw wave data model.\n* Tune LR and learning schedule of each model individually using my discussion post [here][3]\n* Cross validate with train `vote>=10` and observe perfect correlation between CV and LB.\n\n# Inference Code Published\nMy single model inference notebook is published [here][4]. This single model achieves **Gold 14th place**. My final submission is an ensemble of this model with some other models and achieves **Gold 8th place**.\n\nThe code describes the PyTorch model details, the preprocess details, PyTorch data loader details, and data augmentation details. Enjoy!\n\n[1]: https://github.com/bdsp-core/IIIC-SPaRCNet/blob/main/IIIC_Classification-Supplemental.pdf \n[2]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/471666\n[3]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/488083\n[4]: https://www.kaggle.com/code/cdeotte/single-model-gold-solution-cv-0-24-lb-0-24", "votes": 159 }, { "id": 2744045, "postDate": "2024-04-09T18:15:33.080Z", "content": "Using plots is an intriguing idea. I never saw it before.
\nGlad you ended in top 10 despite sharing so much. Congrats!
", "rawMarkdown": "Using plots is an intriguing idea. I never saw it before.\n\nGlad you ended in top 10 despite sharing so much. Congrats!\n", "votes": 11, "replies": [ { "id": 2744054, "postDate": "2024-04-09T18:21:59.520Z", "content": "Thanks. I thought about using plots a month ago but didn't try it because it seemed silly. I tried it in the last week and it boost CV LB 0.01. I'm surprised that it works.
Solution 20th also uses plots! I tagged you there.
", "rawMarkdown": "Solution 20th also uses plots! I tagged you there.\n", "votes": 1 } ] } ] }, { "id": 2747902, "postDate": "2024-04-12T06:28:13.390Z", "content": "Hi Chris! 🎉Congratulations on your gold medal, and thank you for your selfless sharing.
\nAs a novice, I am now confused about choosing PyTorch or TensorFlow. Many baselines were produced during the beginning of competition based on TensorFlow (including your WaveNet Starter, EfficientNetB0 Starter), but in the end most gold medals' solutions used PyTorch. I wanna ask if there is any reason?
\nIt seems that PyTorch has a more complete API for implementing many custom aspects:
\ntorch.optim.lr_scheduler.CosineAnnealingLR in PyTorch)entmax instead of softmax (referring to the 1st place solution, I did not find a direct way to implement it in TF)In this competition, I also spent much time solving the problem of different TF versions. Version 2.13 in Kaggle Notebook and my local version 2.15 are incompatible with .h5 weight files🥲.
\nTo sum up, TF seems far less useful than PyTorch. Is this the current situation or is it just my bias haha. Based on this, can you give any suggestions on the use of these two tools?
", "rawMarkdown": "Hi Chris! 🎉**Congratulations** on your gold medal, and thank you for your selfless sharing.\n\nAs a novice, I am now confused about choosing PyTorch or TensorFlow. Many baselines were produced during the beginning of competition based on TensorFlow (including your [WaveNet Starter](https://www.kaggle.com/code/cdeotte/wavenet-starter-lb-0-52), [EfficientNetB0 Starter](https://www.kaggle.com/code/cdeotte/efficientnetb0-starter-lb-0-43)), but in the end most gold medals' solutions used PyTorch. I wanna ask if there is any reason?\n\nIt seems that PyTorch has a more complete API for implementing many custom aspects: \n\n- lr schedule (Cosine annealing is manually implemented in the *Train Scheduler* section of your [EfficientNetB0 Starter](https://www.kaggle.com/code/cdeotte/efficientnetb0-starter-lb-0-43#Train-Scheduler), but it can be simply implemented through `torch.optim.lr_scheduler.CosineAnnealingLR` in PyTorch)\n- Adding sample weights (PyTorch is easier than the TF)\n- Use `entmax` instead of `softmax` (referring to the [1st place solution](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/492560), I did not find a direct way to implement it in TF)\n- And many more...\n\nIn this competition, I also spent much time solving the problem of different TF versions. Version 2.13 in Kaggle Notebook and my local version 2.15 are incompatible with *.h5* weight files🥲. \n\nTo sum up, TF seems far less useful than PyTorch. Is this the current situation or is it just my bias haha. Based on this, can you give any suggestions on the use of these two tools?", "votes": 3, "replies": [ { "id": 2748718, "postDate": "2024-04-12T14:50:18.047Z", "content": "Hi. I still like to use both TensorFlow and PyTorch. My final ensemble (which improves my single model 14th place Gold to 8th place Gold) has models from both libraries. The main reason that I often use PyTorch is because we have access to over 1000+ different backbones. To display all possible backbones use
\nimport timm\nm = timm()\n ,k (m):\n (,k)\nThis prints out over 1000+ models, and we can try each one in PyTorch! It's like being a kid in a candy store! :-) With TensorFlow we cannot access all these models.
\n(For example using tiny_vit_21m_512 gave +0.02 or more boost compared with EffientNetB0 but I could not find this model for TensorFlow. In TensorFlow I was able to find and use ConvNext here and that did better than EfficientNet but not as good as TinyVIT.)
\"Second model receives 2D matplotlib plots of the montage waveforms converted into a 2D image\" 🤯 mind blown
", "rawMarkdown": "\"Second model receives 2D matplotlib plots of the montage waveforms converted into a 2D image\" 🤯 mind blown", "votes": 3, "replies": [ { "id": 2744913, "postDate": "2024-04-10T07:54:38.703Z", "content": "Same here. Congratulations Chris. you should get another gold medal just for sharing alone. I have to admit that I only joined this competition because you were in it and I knew that I would learn a lot!
", "rawMarkdown": "Same here. Congratulations Chris. you should get another gold medal just for sharing alone. I have to admit that I only joined this competition because you were in it and I knew that I would learn a lot! ", "votes": 4, "replies": [ { "id": 2751435, "postDate": "2024-04-14T08:42:51.957Z", "content": "Thank you Dennis and Nymfree!
", "rawMarkdown": "Thank you Dennis and Nymfree!" } ] } ] }, { "id": 2745920, "postDate": "2024-04-10T23:27:25.713Z", "content": "UPDATE I published my single model inference notebook here. It achieves 🎉 CV = 0.246, Public LB = 0.243, Private LB = 0.290 🎉 It describes the PyTorch model details, the preprocess details, PyTorch data loader details, and data augmentation details. Enjoy!
", "rawMarkdown": "**UPDATE** I published my single model inference notebook [here][1]. It achieves 🎉 CV = 0.246, Public LB = 0.243, Private LB = 0.290 🎉 It describes the PyTorch model details, the preprocess details, PyTorch data loader details, and data augmentation details. Enjoy!\n\n[1]: https://www.kaggle.com/code/cdeotte/single-model-gold-solution-cv-0-24-lb-0-24", "votes": 4 }, { "id": 2744085, "postDate": "2024-04-09T18:39:12.913Z", "content": "Hi Chris, Congratulations! I do have to say that my brian has exploded : from this \"The task in this competition is to predict annotator opinions. By reading the host's paper here, we learn that there are 119 overall annotators in train data and 20 expert annotators in test data. The paper shows the average Seizure prediction from 119 annotators is 18.8% whereas the 20 experts are less likely to classify as Seizure and their average is 1.5%.
\nWow, this is a big difference therefore we need to train our models with expert opinions instead of overall opinions. We can locate the expert annotators in the train data with the filter expert annotators = train.loc[train.vote_count>=10].\"
\nI tried reading the original paper and I totally missed that there was a difference amongst annotators - I thought it was all radiology fellows. Furthermore, I had no idea that the experts were those with a vote count of more than 10. Why isn't this on the main page ? Is this what all Kaggle competitions are like?
", "rawMarkdown": "Hi Chris, Congratulations! I do have to say that my brian has exploded : from this \"The task in this competition is to predict annotator opinions. By reading the host's paper here, we learn that there are 119 overall annotators in train data and 20 expert annotators in test data. The paper shows the average Seizure prediction from 119 annotators is 18.8% whereas the 20 experts are less likely to classify as Seizure and their average is 1.5%.\n\nWow, this is a big difference therefore we need to train our models with expert opinions instead of overall opinions. We can locate the expert annotators in the train data with the filter expert annotators = train.loc[train.vote_count>=10].\"\n\nI tried reading the original paper and I totally missed that there was a difference amongst annotators - I thought it was all radiology fellows. Furthermore, I had no idea that the experts were those with a vote count of more than 10. Why isn't this on the main page ? Is this what all Kaggle competitions are like? ", "votes": 4, "replies": [ { "id": 2744092, "postDate": "2024-04-09T18:42:05.470Z", "content": "Personally, this is my biggest complaint about Kaggle. They never tell us what the real task is. In other words, the real task is about predicting the test data but they never tell us what the test data is.
\nIn every competition it is our first task to discover what the test data is and hence what is our real task.
", "rawMarkdown": "Personally, this is my biggest complaint about Kaggle. They never tell us what the real task is. In other words, the real task is about predicting the test data but they never tell us what the test data is.\n\nIn every competition it is our first task to discover what the test data is and hence what is our real task.", "votes": 6, "replies": [ { "id": 2744275, "postDate": "2024-04-09T20:16:43.360Z", "content": "Let's look from the host point of view. What they want is to evaluate ML pipeline quality. Kaggle makes that evaluation through the use of a test dataset. Host designed the test dataset that will provide them what they believe is a good estimate of ML pipelines quality.
\nDisclosing how the test set is constructed would defeat the purpose. Reverse engineering test data creation defeats the purpose.
\nTL;DR we have two conflicting goals at play: host goal is to get an unbiased evaluation of models and pipelines, kagglers goal is to get the best possible score. Hiding test data is a way to align these two goals.
", "rawMarkdown": "Let's look from the host point of view. What they want is to evaluate ML pipeline quality. Kaggle makes that evaluation through the use of a test dataset. Host designed the test dataset that will provide them what they believe is a good estimate of ML pipelines quality.\n\n\nDisclosing how the test set is constructed would defeat the purpose. Reverse engineering test data creation defeats the purpose.\n\nTL;DR we have two conflicting goals at play: host goal is to get an unbiased evaluation of models and pipelines, kagglers goal is to get the best possible score. Hiding test data is a way to align these two goals.\n\n\n\n", "votes": 6 }, { "id": 2744359, "postDate": "2024-04-09T21:00:00.613Z", "content": "In the Coursera couses I took, Andrew Ng emphasized the the test and validation/dev set have to come from the same distribution. This makes sense to me from a theoretical perspective as well. So we were all spending hours fitting to the wrong target (as Andrew Ng describes the validation set) and the results could have been even better. I guess I should have asked in the discussions why the CV scores were different from LB scores (!!) but this was my first competition. It's also here (https://cs230.stanford.edu/blog/split/#:~:text=It%20is%20important%20to%20choose,to%20get%20in%20the%20future.)
", "rawMarkdown": " In the Coursera couses I took, Andrew Ng emphasized the the test and validation/dev set have to come from the same distribution. This makes sense to me from a theoretical perspective as well. So we were all spending hours fitting to the wrong target (as Andrew Ng describes the validation set) and the results could have been even better. I guess I should have asked in the discussions why the CV scores were different from LB scores (!!) but this was my first competition. It's also here (https://cs230.stanford.edu/blog/split/#:~:text=It%20is%20important%20to%20choose,to%20get%20in%20the%20future.)", "votes": 2 } ] }, { "id": 2744419, "postDate": "2024-04-09T22:42:39.847Z", "content": "Agree, actually though I splitted data by nvotes durting the game, but the same as CPMP I tought the novtes < 10 dataset just means data distribtuion diff not by mean of lower quality…
", "rawMarkdown": "Agree, actually though I splitted data by nvotes durting the game, but the same as CPMP I tought the novtes < 10 dataset just means data distribtuion diff not by mean of lower quality...", "votes": 2 } ] }, { "id": 2796609, "postDate": "2024-05-06T10:29:54.577Z", "content": "Thank you for your sharings throughout the competition.
", "rawMarkdown": "Thank you for your sharings throughout the competition.", "votes": 1 }, { "id": 2782804, "postDate": "2024-04-29T13:06:57.963Z", "content": "@cdeotte Hi, thanks a lot for this post, I highly appreciate the work you have done here.
\nI have the following question.
\nAs I understand from this post tiny_vit_21m_224
\nuses that 10seconds data as an input, but it is already pretrained model. How can I find that model that only uses 10seconds data and train separately ?
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F18779914%2F962bd0f42b665d27da4f3ee3c10dc906%2FScreenshot%202024-04-29%20170855.png?generation=1714396404499433&alt=media\")
We download the pretrained tiny_vit_21m_224 model from Timm. Then we finetune it by itself in stage 1. Then we use it to make pseudo label. Then we finetune it again as part of the Mega model in stage 2.
ok, got it, I want to create an additional model, I want to fine tune this model based on tiny_vit_21m_224, and the difference is that this model is gonna be trained on different slice of eeg, not the central 10seconds, then I want to combine those 4 models into one and see the results, So am i in right direction ? could you please a supervise here a little :) I want to use the exact same model as the central 10 seconds data is inputted to, but trained with different dataset.
", "rawMarkdown": "ok, got it, I want to create an additional model, I want to fine tune this model based on tiny_vit_21m_224, and the difference is that this model is gonna be trained on different slice of eeg, not the central 10seconds, then I want to combine those 4 models into one and see the results, So am i in right direction ? could you please a supervise here a little :) I want to use the exact same model as the central 10 seconds data is inputted to, but trained with different dataset.", "votes": 1 } ] } ] }, { "id": 2758405, "postDate": "2024-04-18T06:36:22.543Z", "content": "I ran into some issues where data enhancement became a bottleneck for the training network. I used some data enhancement methods to increase network generalization, which resulted in my cpu running at 100% while my gpu was only using 10%. How can data enhancement also be accelerated using Gpus?
", "rawMarkdown": "I ran into some issues where data enhancement became a bottleneck for the training network. I used some data enhancement methods to increase network generalization, which resulted in my cpu running at 100% while my gpu was only using 10%. How can data enhancement also be accelerated using Gpus?", "votes": 1 }, { "id": 2755002, "postDate": "2024-04-16T10:23:17.137Z", "content": "As described comment here, the eeg net 1D model seems to have a GRU layer which treats the batch dimension as sequence dimension because the layer is instantiated with the default setting batch_first=False.
\nSo there are interactions between samples in batches and I'm confused to interpret this model architecture.
\nThis seemed bug for me at first glance but not modified here.
Here is the result of the one mega model from the solution notebook
\n![\"one](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8363906%2F13f2be8355bd04d98a13c68e9ea04ecf%2F2024-04-16%2018.40.01.png?generation=1713262210701006&alt=media\")
To test there are interactions with samples in a batch, I duplicated the test record and got slightly different result from above.
\n![\"two](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8363906%2F7eb4de8dfdff629a3c7f70fc1a3d1b0e%2F2024-04-16%2018.47.36.png?generation=1713262237564915&alt=media\")
Next I set the parameter batch_first=True, and get the same output for the duplicated row.
\n![\"two](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8363906%2Ff169e7e6215eecfb278b927d51b72eb7%2F2024-04-16%2019.06.12.png?generation=1713262278157549&alt=media\")
I wonder there is an intent behind not setting batch_first=True or this is just a bug.
", "rawMarkdown": "As described comment [here](https://www.kaggle.com/code/nischaydnk/lightning-1d-eegnet-training-pipeline-hbs/comments#2643122), the eeg net 1D model seems to have a GRU layer which treats the batch dimension as sequence dimension because the layer is instantiated with the default setting batch_first=False.\nSo there are interactions between samples in batches and I'm confused to interpret this model architecture.\nThis seemed bug for me at first glance but not modified here.\n\nHere is the result of the one mega model from the [solution notebook](https://www.kaggle.com/code/cdeotte/single-model-gold-solution-cv-0-24-lb-0-24)\n\n\n\nTo test there are interactions with samples in a batch, I duplicated the test record and got slightly different result from above.\n\n\n\nNext I set the parameter batch_first=True, and get the same output for the duplicated row.\n\n\n\nI wonder there is an intent behind not setting batch_first=True or this is just a bug.", "votes": 1, "replies": [ { "id": 2756130, "postDate": "2024-04-16T22:16:33.503Z", "content": "Interesting. I was not aware of this. I will need to think about this and perform some tests to fully understand this. Thanks for sharing.
", "rawMarkdown": "Interesting. I was not aware of this. I will need to think about this and perform some tests to fully understand this. Thanks for sharing.", "votes": 1 } ] }, { "id": 2754948, "postDate": "2024-04-16T09:31:33.417Z", "content": "thank you for sharing this, very helpful
", "rawMarkdown": "thank you for sharing this, very helpful", "votes": 1 }, { "id": 2752717, "postDate": "2024-04-15T05:44:30.063Z", "content": "Thank you for your sharings.
\nI have a question about \"Then we train with 2x copies of vote>=10 data concatenated with 1x copy of vote<10 with pseudo data.\".
\nwhat is the meaning of copies ??
\nI don't understand what exactly it is.
We give vote>=10 weight=2 and vote<10 weight=1 like this. (And for vote>=10 we use original labels and vote<10 we use pseudo labels):
train = pd()\ntrain = train(axis=)\ndf1 = train\ndf2 = train\ntrain_data = pd(,axis=)\nthank you for sharing this, very helpful
", "rawMarkdown": "thank you for sharing this, very helpful ", "votes": 1 }, { "id": 2748258, "postDate": "2024-04-12T10:44:36.703Z", "content": "Very helpful
", "rawMarkdown": "Very helpful", "votes": 1 }, { "id": 2746120, "postDate": "2024-04-11T04:03:59.177Z", "content": "the graphs is looking little dangerous and attracting me to know more!!
\ncongratulations for the gold @cdeotte !!
haha thank you.
", "rawMarkdown": "haha thank you. ", "votes": 1 } ] }, { "id": 2745937, "postDate": "2024-04-10T23:55:37.827Z", "content": "Very helpful!!
", "rawMarkdown": "Very helpful!!", "votes": 1 }, { "id": 2745732, "postDate": "2024-04-10T19:09:06.373Z", "content": "Thanks for sharing this solution it will be very helpful to us.
", "rawMarkdown": "Thanks for sharing this solution it will be very helpful to us.", "votes": 1 }, { "id": 2745654, "postDate": "2024-04-10T18:25:07.657Z", "content": "Great and helpful work👌🏻 congrats☘️
", "rawMarkdown": "Great and helpful work👌🏻 congrats☘️", "votes": 1 }, { "id": 2744907, "postDate": "2024-04-10T07:46:52.740Z", "content": "Great work on HMS - Harmful Brain Activity Classification! Your approach was clear and effective. Have you considered [specific suggestion] for even better results?
", "rawMarkdown": "Great work on HMS - Harmful Brain Activity Classification! Your approach was clear and effective. Have you considered [specific suggestion] for even better results?", "votes": 1 }, { "id": 2744788, "postDate": "2024-04-10T06:04:12.897Z", "content": "Hi Chris! Congrats with the gold, and thank you for your early stages posts and especially 50s spectrogramms, those were very handy!
\nI also used 1D -> 2D plot with the same motivation to show the model ~ same stuff as were shown to annotators, and I also was surprised it works, a nice trick indeed.
\nHave you measured the pseudolabels impact? I found PLs to improve the public score just a bit (on ~0.001 order) on multiple occasions and decided to trust it, but did not do any hold out validation so could not tell more precisely, that would be interesting to see if it helped you more.
", "rawMarkdown": "Hi Chris! Congrats with the gold, and thank you for your early stages posts and especially 50s spectrogramms, those were very handy! \n\nI also used 1D -> 2D plot with the same motivation to show the model ~ same stuff as were shown to annotators, and I also was surprised it works, a nice trick indeed.\n\nHave you measured the pseudolabels impact? I found PLs to improve the public score just a bit (on ~0.001 order) on multiple occasions and decided to trust it, but did not do any hold out validation so could not tell more precisely, that would be interesting to see if it helped you more.", "votes": 1, "replies": [ { "id": 2744817, "postDate": "2024-04-10T06:27:37.483Z", "content": "Pseudo labeling gave large boosts like +0.03. Here is how i performed clean (leakfree) CV. First train 5 KFold models on vote>=10. Then we pseudo label each fold separately. So we take fold 1 model and use it to pseudo label the fold 1 train data vote<10. Next we take the fold 2 model and use it to pseudo label the fold 2 train data, etc etc. (So we have 5 sets of pseudo labels).
Afterward we train another 5 KFold using the vote>=10 data concatenated with vote<10 pseudo label data. When training the new fold 1 model we use the fold 1 pseudo labels. When training the new fold 2 model, we use the fold 2 pseudo labels, etc. Lastly we perform a stage 3 where we finetune each fold model using only vote>=10.
What I describe here is a 3 stage approach. This is a different model than what I describe in my solution post above. We see below that stage 1 cannot score better than 0.31 CV no matter what LR and train schedule we use. However stage 2 (which uses pseudo labels created from stage 1 models) improves the CV score more. And lastly stage 3 improves the CV score even more. This model uses spectrograms and raw waves and has clean CV score 0.27 and LB 0.28. Each stages' LR and train schedule was carefully discovered. Below show the validation KL Divergence loss for fold 1:
\nNote that if we don't create a clean (leakfree) KFold we will never know the correct LR and train schedule because a leaky KFold will just prefer the largest LR possible to overfit the leak.
\n![](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Apr-2024/3-stage.png\")
Also note that pseudo labels allows us to perform knowledge distillation which gives even more boost than simple pseudo labels.
\nWith knowledge distillation, we use pseudo labels from one model (like a 1D EegNet) and transfer that intelligence into another type of model (like 2D VIT) using pseudo labels. Or we can create an ensemble of lots of models, average all the pseudo labels, and transfer that combined knowledge into a new different model.
\nThe reason this helps is because when a teacher trains a student, then the student adds is own \"personality\" to the knowledge. Afterward if we ensemble the new student with the old teacher, the pair will be smarter than the both the teacher and the student alone.
\nI used this technique here in Brain comp and I used it previously in AMEX comp here. In AMEX comp, I transferred the knowledge of LGBM model into NN model. The NN model received the learnings from LGBM and added its own personality thus creating a new representation of the knowledge.
", "rawMarkdown": "Also note that pseudo labels allows us to perform knowledge distillation which gives even more boost than simple pseudo labels.\n\nWith knowledge distillation, we use pseudo labels from one model (like a 1D EegNet) and transfer that intelligence into another type of model (like 2D VIT) using pseudo labels. Or we can create an ensemble of lots of models, average all the pseudo labels, and transfer that combined knowledge into a new different model.\n\nThe reason this helps is because when a teacher trains a student, then the student adds is own \"personality\" to the knowledge. Afterward if we ensemble the new student with the old teacher, the pair will be smarter than the both the teacher and the student alone.\n\nI used this technique here in Brain comp and I used it previously in AMEX comp [here][1]. In AMEX comp, I transferred the knowledge of LGBM model into NN model. The NN model received the learnings from LGBM and added its own personality thus creating a new representation of the knowledge.\n\n[1]: https://www.kaggle.com/competitions/amex-default-prediction/discussion/347641", "votes": 8, "replies": [ { "id": 2752352, "postDate": "2024-04-14T23:26:22.273Z", "content": "Hi @cdeotte , thank you for your detailed explanation about pseudo labeling!
\nI have a question about this part. Why is it necessary to pseudo label each fold separately?
\nWhen you train 5 KFold models on vote >= 10, it seems that vote < 10 records don't belong to any of the folds and you can create pseudo labels for these records using all 5 KFold models (average of all models or something). There seems no leak and pseudo labels could be slightly accurate.
\nAm I missing something?
\nI would appreciate if you could answer to my question. Thank you!
\n", "rawMarkdown": "\nHi @cdeotte , thank you for your detailed explanation about pseudo labeling!\n\n\nI have a question about this part. Why is it necessary to pseudo label each fold separately?\nWhen you train 5 KFold models on vote >= 10, it seems that vote < 10 records don't belong to any of the folds and you can create pseudo labels for these records using all 5 KFold models (average of all models or something). There seems no leak and pseudo labels could be slightly accurate.\nAm I missing something? \nI would appreciate if you could answer to my question. Thank you!\n\n> Here is how i performed clean (leakfree) CV. First train 5 KFold models on vote>=10. Then we pseudo label each fold separately. So we take fold 1 model and use it to pseudo label the fold 1 train data vote<10. Next we take the fold 2 model and use it to pseudo label the fold 2 train data, etc etc. (So we have 5 sets of pseudo labels).\n\n\n", "votes": 2 }, { "id": 2752500, "postDate": "2024-04-15T02:22:56.820Z", "content": "Here is how i performed clean (leakfree) CV. First train 5 KFold models on vote>=10. Then we pseudo label each fold separately. So we take fold 1 model and use it to pseudo label the fold 1 train data vote<10. Next we take the fold 2 model and use it to pseudo label the fold 2 train data, etc etc. (So we have 5 sets of pseudo labels).
\n
@nynyny67 For stage 1, let's imagine that we have Folds A, B, C, D, E. When we train fold 1 model, we train with Folds B, C, D, E and validation with Fold A. Now let's pseudo label ANY DATA with this fold 1 model and call the result pseudo 1. The process of making pseudo label transfers the ground truth of Folds B, C, D, E into the ANY DATA (via knowledge distillation principle even though ANY DATA is different data than Folds B, C, D, E data). So pseudo 1 contains the ground truth from Folds B, C, D, E (from knowledge transfer).
Imagine now that we are training stage 2 fold 2 model, we train with Folds A, C, D, E and validation with Fold B. If we add pseudo 1 to the training of stage 2 fold 2 model, then we leak. Because pseudo 1 contains the ground truth of Folds B, C, D, E and afterward we want to validate fold 2 model with Fold B.
In conclusion, whenever we do pseudo label or knowledge transfer with KFold models, we need to keep K separate pseudo label data for usage in stage 2 to prevent leak. If we have a leak then the stage 2 models will perform poorly on LB because as their fake leaked CV score improve, their real leak-free CV score will be getting worse without us knowing.
", "rawMarkdown": "@nynyny67 For stage 1, let's imagine that we have Folds A, B, C, D, E. When we train `fold 1` model, we train with Folds B, C, D, E and validation with Fold A. Now let's pseudo label ANY DATA with this `fold 1` model and call the result `pseudo 1`. The process of making pseudo label transfers the ground truth of Folds B, C, D, E into the ANY DATA (via knowledge distillation principle even though ANY DATA is different data than Folds B, C, D, E data). So `pseudo 1` contains the ground truth from Folds B, C, D, E (from knowledge transfer).\n\nImagine now that we are training stage 2 `fold 2` model, we train with Folds A, C, D, E and validation with Fold B. If we add `pseudo 1` to the training of stage 2 `fold 2` model, then we leak. Because `pseudo 1` contains the ground truth of Folds B, C, D, E and afterward we want to validate `fold 2` model with Fold B.\n\nIn conclusion, whenever we do pseudo label or knowledge transfer with KFold models, we need to keep K separate pseudo label data for usage in stage 2 to prevent leak. If we have a leak then the stage 2 models will perform poorly on LB because as their **fake** leaked CV score improve, their **real** leak-free CV score will be getting worse without us knowing.", "votes": 5 }, { "id": 2753326, "postDate": "2024-04-15T12:27:56.873Z", "content": "@cdeotte
\nThank you so much for answering my question!
\nI understood why it's necessary to keep K separate pseudo label!
\nOnce we use a pseudo label from a model trained with some ground truth information, we can never use the ground truth data to evaluate the models that used the pseudo label to train because it has a leak!
\nSo if we create pseudo label from all folds, the model trained with it cannot be evaluated locally.
I believe ViT here refers to visual transformers? I had started training one from scratch, should have known better to rely on pre-trained solutions, very nice!
", "rawMarkdown": "I believe ViT here refers to visual transformers? I had started training one from scratch, should have known better to rely on pre-trained solutions, very nice!", "votes": 1, "replies": [ { "id": 2744420, "postDate": "2024-04-09T22:44:59.817Z", "content": "Yes VIT is Vision Transformer. We can print out a full list of the 1000 pretrained models available in timm library with:
\nimport timm\nm = timm()\n ,k (m):\n (,k)\nThen we can try lots of these models to see what works best. For each model we should use our previous LR train schedule but also try LR = LR * 10 and LR = LR/10 because some of these different models require stronger or weaker learning rates.
Congrats @cdeotte on achieve 8th Place.
", "rawMarkdown": "Congrats @cdeotte on achieve 8th Place.", "votes": 1, "replies": [ { "id": 2745393, "postDate": "2024-04-10T15:40:35.260Z", "content": "Thank you Adnan
", "rawMarkdown": "Thank you Adnan", "votes": 1 } ] }, { "id": 2752510, "postDate": "2024-04-15T02:38:09.147Z", "content": "Thank you for your sharings throughout the competition.
\nDid you use larger models instead of tiny-vit and get worse results, or is there a specific reason for using smaller models?
", "rawMarkdown": "Thank you for your sharings throughout the competition.\n\nDid you use larger models instead of tiny-vit and get worse results, or is there a specific reason for using smaller models?", "votes": 2, "replies": [ { "id": 2752513, "postDate": "2024-04-15T02:43:18.823Z", "content": "Yes. I tried many sized models and always got the best performance with tiny vision transformers. Furthermore, tiny-vit is special (among tiny models). It is pretrained with knowledge transfer from larger model into the tiny model.
Thank you for sharing your process selflessly from beginning to end.
\nYou are a true benevolent person who is admired by kagglers.
\nTHX ! !
Thank you Warrenkuo!
", "rawMarkdown": "Thank you Warrenkuo!", "votes": 2 } ] }, { "id": 2744670, "postDate": "2024-04-10T04:19:05.450Z", "content": "Congratulations on securing 8th place in this competition. Thanks for sharing your solution details.
", "rawMarkdown": "Congratulations on securing 8th place in this competition. Thanks for sharing your solution details. ", "votes": 2, "replies": [ { "id": 2744754, "postDate": "2024-04-10T05:19:58.700Z", "content": "Thank you.
", "rawMarkdown": "Thank you. ", "votes": 1 } ] }, { "id": 2744421, "postDate": "2024-04-09T22:45:03.073Z", "content": "Great work, your sharing early in the game make this game much more interesting and helped so much people to join.
\n Could you help share your cv results of your first step 3 models?
Thanks. I enjoyed sharing and engaging in discussions with the community. It did make this competition more challenging.
\n| Model | \nCV | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|
| 2D Eeg Plot Image | \n0.280 | \n0.256 | \n0.317 | \n
| 2D Eeg Spectrogram Image | \n0.281 | \n0.275 | \n0.326 | \n
| 1D Raw Eeg Waveform | \n0.303 | \n0.305 | \n0.363 | \n
| Ordinary Ensemble | \n0.253 | \n??? | \n??? | \n
| One Mega Model | \n0.246 | \n0.242 | \n0.290 | \n
.
\nSurprisingly the 2D Eeg Plot model achieves LB 0.25 by itself wow! We also see using a combined mega model achieves better CV than simple ensemble of 3 models. Note a mega model is like stacking we are training a new model to make predictions from the output of 3 previous models.
Note that these are not my best single modality input models. I have another 1D Raw Eeg Waveform model that achieves CV 0.28 LB 0.28 by modifying the public notebook architecture. And I probably have better versions of the other two single models too. But for my mega model the table above shows which models are utilized for it.
", "rawMarkdown": "Thanks. I enjoyed sharing and engaging in discussions with the community. It did make this competition more challenging.\n\n| Model | CV | Public LB | Private LB |\n| --- | --- | --- | --- |\n| 2D Eeg Plot Image | 0.280 | 0.256 | 0.317 |\n| 2D Eeg Spectrogram Image | 0.281 | 0.275 | 0.326 |\n| 1D Raw Eeg Waveform | 0.303 | 0.305 | 0.363 |\n| Ordinary Ensemble | 0.253 | ??? | ??? |\n| One Mega Model | 0.246 | 0.242 | 0.290 |\n\n.\nSurprisingly the 2D Eeg Plot model achieves LB 0.25 by itself wow! We also see using a combined mega model achieves better CV than simple ensemble of 3 models. Note a mega model is like **stacking** we are training a new model to make predictions from the output of 3 previous models.\n\nNote that these are not my best single modality input models. I have another 1D Raw Eeg Waveform model that achieves CV 0.28 LB 0.28 by modifying the public notebook architecture. And I probably have better versions of the other two single models too. But for my mega model the table above shows which models are utilized for it.\n", "votes": 3 } ] }, { "id": 2744344, "postDate": "2024-04-09T20:51:57.857Z", "content": "Great solution Chris! I also tried to use similar graphs implemented through matplotlib at the very beginning, but gave up on this idea because I couldn't get good results, and all the articles I saw said that this is not a good solution.
", "rawMarkdown": "Great solution Chris! I also tried to use similar graphs implemented through matplotlib at the very beginning, but gave up on this idea because I couldn't get good results, and all the articles I saw said that this is not a good solution.", "votes": 2 }, { "id": 2744261, "postDate": "2024-04-09T20:11:10.023Z", "content": "What post processing did you use? You aid you used some in a past yesterday.
", "rawMarkdown": "What post processing did you use? You aid you used some in a past yesterday.", "votes": 2, "replies": [ { "id": 2744414, "postDate": "2024-04-09T22:40:24.320Z", "content": "My final solution is an ensemble of multiple diverse models (and increases my private rank from 15th to 8th). The model shown in this discussion post by itself achieves gold medal and does not require any post process. Similar to you when I used pseudo label for some models I adjusted the target distribution of pseudo labeled train vote<10 with:
\n\n\nTo avoid this distribution shift I sample pseudo labelled data by target, to keep the same distribution as training data. (Quote from CPMP here)
\n
Some of my other models were trained with different subsets (i.e. different target distributions) of train data and consequently don't have the right prediction distribution of targets.
\nFor example, we can train equally with all vote<10 original targets and vote>=10 original targets then the model will predict too many Seizures and too many Other. Afterward we correct the predictions with the following post process:
\npred = submission[TARGETS].copy()\nfor k,v in [(,),(,)]: \n pred[:,k] = pred[:,k]/(-pred[:,k]) \n pred[:,k] *= v \n pred[:,k] = pred[:,k]/(pred[:,k]+) \npred = pred / pred.sum(axis=,keepdims=) \nsubmission[TARGETS] = pred\nok, I did not have the distribution shift issue indeed.
", "rawMarkdown": "ok, I did not have the distribution shift issue indeed.", "votes": 1 }, { "id": 2753945, "postDate": "2024-04-15T18:14:55.937Z", "content": "Very Nice!!
", "rawMarkdown": "Very Nice!!", "votes": 1 } ] } ] }, { "id": 2824203, "postDate": "2024-05-19T16:24:54.180Z", "content": "Hi Chris … Congratulations on your goldmedal!
", "rawMarkdown": "Hi Chris ... Congratulations on your goldmedal! " }, { "id": 2782824, "postDate": "2024-04-29T13:12:27.030Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2756059, "postDate": "2024-04-16T19:59:23.567Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!", "votes": 1 }, { "id": 2753171, "postDate": "2024-04-15T10:36:46.610Z", "content": "Thank you for sharing
", "rawMarkdown": "Thank you for sharing", "votes": 1 }, { "id": 2746716, "postDate": "2024-04-11T12:19:20.697Z", "content": "Thank you for sharing your process
", "rawMarkdown": "Thank you for sharing your process", "votes": 1 }, { "id": 2768884, "postDate": "2024-04-23T04:40:32.933Z", "content": "Thank you for your sharings.
", "rawMarkdown": "Thank you for your sharings." } ], "comments": [ { "id": 2744045, "author_name": "CPMP", "author_url": "", "post_date": "2024-04-09T18:15:33.080000", "content": "Using plots is an intriguing idea. I never saw it before.
\nGlad you ended in top 10 despite sharing so much. Congrats!
", "votes": 11, "replies": [ { "id": 2744054, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-04-09T18:21:59.520000", "content": "Thanks. I thought about using plots a month ago but didn't try it because it seemed silly. I tried it in the last week and it boost CV LB 0.01. I'm surprised that it works.
Solution 20th also uses plots! I tagged you there.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2747902, "author_name": "Fang Zitao", "author_url": "", "post_date": "2024-04-12T06:28:13.390000", "content": "Hi Chris! 🎉Congratulations on your gold medal, and thank you for your selfless sharing.
\nAs a novice, I am now confused about choosing PyTorch or TensorFlow. Many baselines were produced during the beginning of competition based on TensorFlow (including your WaveNet Starter, EfficientNetB0 Starter), but in the end most gold medals' solutions used PyTorch. I wanna ask if there is any reason?
\nIt seems that PyTorch has a more complete API for implementing many custom aspects:
\ntorch.optim.lr_scheduler.CosineAnnealingLR in PyTorch)entmax instead of softmax (referring to the 1st place solution, I did not find a direct way to implement it in TF)In this competition, I also spent much time solving the problem of different TF versions. Version 2.13 in Kaggle Notebook and my local version 2.15 are incompatible with .h5 weight files🥲.
\nTo sum up, TF seems far less useful than PyTorch. Is this the current situation or is it just my bias haha. Based on this, can you give any suggestions on the use of these two tools?
", "votes": 3, "replies": [ { "id": 2748718, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-04-12T14:50:18.047000", "content": "Hi. I still like to use both TensorFlow and PyTorch. My final ensemble (which improves my single model 14th place Gold to 8th place Gold) has models from both libraries. The main reason that I often use PyTorch is because we have access to over 1000+ different backbones. To display all possible backbones use
\nimport timm\nm = timm()\n ,k (m):\n (,k)\nThis prints out over 1000+ models, and we can try each one in PyTorch! It's like being a kid in a candy store! :-) With TensorFlow we cannot access all these models.
\n(For example using tiny_vit_21m_512 gave +0.02 or more boost compared with EffientNetB0 but I could not find this model for TensorFlow. In TensorFlow I was able to find and use ConvNext here and that did better than EfficientNet but not as good as TinyVIT.)
\"Second model receives 2D matplotlib plots of the montage waveforms converted into a 2D image\" 🤯 mind blown
", "votes": 3, "replies": [ { "id": 2744913, "author_name": "nymfree", "author_url": "", "post_date": "2024-04-10T07:54:38.703000", "content": "Same here. Congratulations Chris. you should get another gold medal just for sharing alone. I have to admit that I only joined this competition because you were in it and I knew that I would learn a lot!
", "votes": 4, "replies": [ { "id": 2751435, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-04-14T08:42:51.957000", "content": "Thank you Dennis and Nymfree!
", "votes": 0, "replies": [] } ] } ] }, { "id": 2745920, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-04-10T23:27:25.713000", "content": "UPDATE I published my single model inference notebook here. It achieves 🎉 CV = 0.246, Public LB = 0.243, Private LB = 0.290 🎉 It describes the PyTorch model details, the preprocess details, PyTorch data loader details, and data augmentation details. Enjoy!
", "votes": 4, "replies": [] }, { "id": 2744085, "author_name": "Idith Haber", "author_url": "", "post_date": "2024-04-09T18:39:12.913000", "content": "Hi Chris, Congratulations! I do have to say that my brian has exploded : from this \"The task in this competition is to predict annotator opinions. By reading the host's paper here, we learn that there are 119 overall annotators in train data and 20 expert annotators in test data. The paper shows the average Seizure prediction from 119 annotators is 18.8% whereas the 20 experts are less likely to classify as Seizure and their average is 1.5%.
\nWow, this is a big difference therefore we need to train our models with expert opinions instead of overall opinions. We can locate the expert annotators in the train data with the filter expert annotators = train.loc[train.vote_count>=10].\"
\nI tried reading the original paper and I totally missed that there was a difference amongst annotators - I thought it was all radiology fellows. Furthermore, I had no idea that the experts were those with a vote count of more than 10. Why isn't this on the main page ? Is this what all Kaggle competitions are like?
", "votes": 4, "replies": [ { "id": 2744092, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-04-09T18:42:05.470000", "content": "Personally, this is my biggest complaint about Kaggle. They never tell us what the real task is. In other words, the real task is about predicting the test data but they never tell us what the test data is.
\nIn every competition it is our first task to discover what the test data is and hence what is our real task.
", "votes": 6, "replies": [ { "id": 2744275, "author_name": "CPMP", "author_url": "", "post_date": "2024-04-09T20:16:43.360000", "content": "Let's look from the host point of view. What they want is to evaluate ML pipeline quality. Kaggle makes that evaluation through the use of a test dataset. Host designed the test dataset that will provide them what they believe is a good estimate of ML pipelines quality.
\nDisclosing how the test set is constructed would defeat the purpose. Reverse engineering test data creation defeats the purpose.
\nTL;DR we have two conflicting goals at play: host goal is to get an unbiased evaluation of models and pipelines, kagglers goal is to get the best possible score. Hiding test data is a way to align these two goals.
", "votes": 6, "replies": [] }, { "id": 2744359, "author_name": "Idith Haber", "author_url": "", "post_date": "2024-04-09T21:00:00.613000", "content": "In the Coursera couses I took, Andrew Ng emphasized the the test and validation/dev set have to come from the same distribution. This makes sense to me from a theoretical perspective as well. So we were all spending hours fitting to the wrong target (as Andrew Ng describes the validation set) and the results could have been even better. I guess I should have asked in the discussions why the CV scores were different from LB scores (!!) but this was my first competition. It's also here (https://cs230.stanford.edu/blog/split/#:~:text=It%20is%20important%20to%20choose,to%20get%20in%20the%20future.)
", "votes": 2, "replies": [] } ] }, { "id": 2744419, "author_name": "gezi", "author_url": "", "post_date": "2024-04-09T22:42:39.847000", "content": "Agree, actually though I splitted data by nvotes durting the game, but the same as CPMP I tought the novtes < 10 dataset just means data distribtuion diff not by mean of lower quality…
", "votes": 2, "replies": [] } ] }, { "id": 2796609, "author_name": "Superman", "author_url": "", "post_date": "2024-05-06T10:29:54.577000", "content": "Thank you for your sharings throughout the competition.
", "votes": 1, "replies": [] }, { "id": 2782804, "author_name": "Datevik", "author_url": "", "post_date": "2024-04-29T13:06:57.963000", "content": "@cdeotte Hi, thanks a lot for this post, I highly appreciate the work you have done here.
\nI have the following question.
\nAs I understand from this post tiny_vit_21m_224
\nuses that 10seconds data as an input, but it is already pretrained model. How can I find that model that only uses 10seconds data and train separately ?
We download the pretrained tiny_vit_21m_224 model from Timm. Then we finetune it by itself in stage 1. Then we use it to make pseudo label. Then we finetune it again as part of the Mega model in stage 2.
ok, got it, I want to create an additional model, I want to fine tune this model based on tiny_vit_21m_224, and the difference is that this model is gonna be trained on different slice of eeg, not the central 10seconds, then I want to combine those 4 models into one and see the results, So am i in right direction ? could you please a supervise here a little :) I want to use the exact same model as the central 10 seconds data is inputted to, but trained with different dataset.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2758405, "author_name": "franticXu", "author_url": "", "post_date": "2024-04-18T06:36:22.543000", "content": "I ran into some issues where data enhancement became a bottleneck for the training network. I used some data enhancement methods to increase network generalization, which resulted in my cpu running at 100% while my gpu was only using 10%. How can data enhancement also be accelerated using Gpus?
", "votes": 1, "replies": [] }, { "id": 2755002, "author_name": "nynyny67", "author_url": "", "post_date": "2024-04-16T10:23:17.137000", "content": "As described comment here, the eeg net 1D model seems to have a GRU layer which treats the batch dimension as sequence dimension because the layer is instantiated with the default setting batch_first=False.
\nSo there are interactions between samples in batches and I'm confused to interpret this model architecture.
\nThis seemed bug for me at first glance but not modified here.
Here is the result of the one mega model from the solution notebook
\n
To test there are interactions with samples in a batch, I duplicated the test record and got slightly different result from above.
\n
Next I set the parameter batch_first=True, and get the same output for the duplicated row.
\n
I wonder there is an intent behind not setting batch_first=True or this is just a bug.
", "votes": 1, "replies": [ { "id": 2756130, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-04-16T22:16:33.503000", "content": "Interesting. I was not aware of this. I will need to think about this and perform some tests to fully understand this. Thanks for sharing.
", "votes": 1, "replies": [] } ] }, { "id": 2754948, "author_name": "Chenghao Peng1013", "author_url": "", "post_date": "2024-04-16T09:31:33.417000", "content": "thank you for sharing this, very helpful
", "votes": 1, "replies": [] }, { "id": 2752717, "author_name": "SeongwookLee", "author_url": "", "post_date": "2024-04-15T05:44:30.063000", "content": "Thank you for your sharings.
\nI have a question about \"Then we train with 2x copies of vote>=10 data concatenated with 1x copy of vote<10 with pseudo data.\".
\nwhat is the meaning of copies ??
\nI don't understand what exactly it is.
We give vote>=10 weight=2 and vote<10 weight=1 like this. (And for vote>=10 we use original labels and vote<10 we use pseudo labels):
train = pd()\ntrain = train(axis=)\ndf1 = train\ndf2 = train\ntrain_data = pd(,axis=)\nthank you for sharing this, very helpful
", "votes": 1, "replies": [] }, { "id": 2748258, "author_name": "Aditya Srivastava", "author_url": "", "post_date": "2024-04-12T10:44:36.703000", "content": "Very helpful
", "votes": 1, "replies": [] }, { "id": 2746120, "author_name": "Anmol gupta", "author_url": "", "post_date": "2024-04-11T04:03:59.177000", "content": "the graphs is looking little dangerous and attracting me to know more!!
\ncongratulations for the gold @cdeotte !!
haha thank you.
", "votes": 1, "replies": [] } ] }, { "id": 2745937, "author_name": "Hoonsuk Bae", "author_url": "", "post_date": "2024-04-10T23:55:37.827000", "content": "Very helpful!!
", "votes": 1, "replies": [] }, { "id": 2745732, "author_name": "Aaditya Porwal", "author_url": "", "post_date": "2024-04-10T19:09:06.373000", "content": "Thanks for sharing this solution it will be very helpful to us.
", "votes": 1, "replies": [] }, { "id": 2745654, "author_name": "Atefeh Mirnaseri", "author_url": "", "post_date": "2024-04-10T18:25:07.657000", "content": "Great and helpful work👌🏻 congrats☘️
", "votes": 1, "replies": [] }, { "id": 2744907, "author_name": "Yugandhar Dasari", "author_url": "", "post_date": "2024-04-10T07:46:52.740000", "content": "Great work on HMS - Harmful Brain Activity Classification! Your approach was clear and effective. Have you considered [specific suggestion] for even better results?
", "votes": 1, "replies": [] }, { "id": 2744788, "author_name": "Mikhail Kotyushev", "author_url": "", "post_date": "2024-04-10T06:04:12.897000", "content": "Hi Chris! Congrats with the gold, and thank you for your early stages posts and especially 50s spectrogramms, those were very handy!
\nI also used 1D -> 2D plot with the same motivation to show the model ~ same stuff as were shown to annotators, and I also was surprised it works, a nice trick indeed.
\nHave you measured the pseudolabels impact? I found PLs to improve the public score just a bit (on ~0.001 order) on multiple occasions and decided to trust it, but did not do any hold out validation so could not tell more precisely, that would be interesting to see if it helped you more.
", "votes": 1, "replies": [ { "id": 2744817, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-04-10T06:27:37.483000", "content": "Pseudo labeling gave large boosts like +0.03. Here is how i performed clean (leakfree) CV. First train 5 KFold models on vote>=10. Then we pseudo label each fold separately. So we take fold 1 model and use it to pseudo label the fold 1 train data vote<10. Next we take the fold 2 model and use it to pseudo label the fold 2 train data, etc etc. (So we have 5 sets of pseudo labels).
Afterward we train another 5 KFold using the vote>=10 data concatenated with vote<10 pseudo label data. When training the new fold 1 model we use the fold 1 pseudo labels. When training the new fold 2 model, we use the fold 2 pseudo labels, etc. Lastly we perform a stage 3 where we finetune each fold model using only vote>=10.
What I describe here is a 3 stage approach. This is a different model than what I describe in my solution post above. We see below that stage 1 cannot score better than 0.31 CV no matter what LR and train schedule we use. However stage 2 (which uses pseudo labels created from stage 1 models) improves the CV score more. And lastly stage 3 improves the CV score even more. This model uses spectrograms and raw waves and has clean CV score 0.27 and LB 0.28. Each stages' LR and train schedule was carefully discovered. Below show the validation KL Divergence loss for fold 1:
\nNote that if we don't create a clean (leakfree) KFold we will never know the correct LR and train schedule because a leaky KFold will just prefer the largest LR possible to overfit the leak.
\n
Also note that pseudo labels allows us to perform knowledge distillation which gives even more boost than simple pseudo labels.
\nWith knowledge distillation, we use pseudo labels from one model (like a 1D EegNet) and transfer that intelligence into another type of model (like 2D VIT) using pseudo labels. Or we can create an ensemble of lots of models, average all the pseudo labels, and transfer that combined knowledge into a new different model.
\nThe reason this helps is because when a teacher trains a student, then the student adds is own \"personality\" to the knowledge. Afterward if we ensemble the new student with the old teacher, the pair will be smarter than the both the teacher and the student alone.
\nI used this technique here in Brain comp and I used it previously in AMEX comp here. In AMEX comp, I transferred the knowledge of LGBM model into NN model. The NN model received the learnings from LGBM and added its own personality thus creating a new representation of the knowledge.
", "votes": 8, "replies": [ { "id": 2752352, "author_name": "nynyny67", "author_url": "", "post_date": "2024-04-14T23:26:22.273000", "content": "Hi @cdeotte , thank you for your detailed explanation about pseudo labeling!
\nI have a question about this part. Why is it necessary to pseudo label each fold separately?
\nWhen you train 5 KFold models on vote >= 10, it seems that vote < 10 records don't belong to any of the folds and you can create pseudo labels for these records using all 5 KFold models (average of all models or something). There seems no leak and pseudo labels could be slightly accurate.
\nAm I missing something?
\nI would appreciate if you could answer to my question. Thank you!
\n", "votes": 2, "replies": [] }, { "id": 2752500, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-04-15T02:22:56.820000", "content": "Here is how i performed clean (leakfree) CV. First train 5 KFold models on vote>=10. Then we pseudo label each fold separately. So we take fold 1 model and use it to pseudo label the fold 1 train data vote<10. Next we take the fold 2 model and use it to pseudo label the fold 2 train data, etc etc. (So we have 5 sets of pseudo labels).
\n
@nynyny67 For stage 1, let's imagine that we have Folds A, B, C, D, E. When we train fold 1 model, we train with Folds B, C, D, E and validation with Fold A. Now let's pseudo label ANY DATA with this fold 1 model and call the result pseudo 1. The process of making pseudo label transfers the ground truth of Folds B, C, D, E into the ANY DATA (via knowledge distillation principle even though ANY DATA is different data than Folds B, C, D, E data). So pseudo 1 contains the ground truth from Folds B, C, D, E (from knowledge transfer).
Imagine now that we are training stage 2 fold 2 model, we train with Folds A, C, D, E and validation with Fold B. If we add pseudo 1 to the training of stage 2 fold 2 model, then we leak. Because pseudo 1 contains the ground truth of Folds B, C, D, E and afterward we want to validate fold 2 model with Fold B.
In conclusion, whenever we do pseudo label or knowledge transfer with KFold models, we need to keep K separate pseudo label data for usage in stage 2 to prevent leak. If we have a leak then the stage 2 models will perform poorly on LB because as their fake leaked CV score improve, their real leak-free CV score will be getting worse without us knowing.
", "votes": 5, "replies": [] }, { "id": 2753326, "author_name": "nynyny67", "author_url": "", "post_date": "2024-04-15T12:27:56.873000", "content": "@cdeotte
\nThank you so much for answering my question!
\nI understood why it's necessary to keep K separate pseudo label!
\nOnce we use a pseudo label from a model trained with some ground truth information, we can never use the ground truth data to evaluate the models that used the pseudo label to train because it has a leak!
\nSo if we create pseudo label from all folds, the model trained with it cannot be evaluated locally.
I believe ViT here refers to visual transformers? I had started training one from scratch, should have known better to rely on pre-trained solutions, very nice!
", "votes": 1, "replies": [ { "id": 2744420, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-04-09T22:44:59.817000", "content": "Yes VIT is Vision Transformer. We can print out a full list of the 1000 pretrained models available in timm library with:
\nimport timm\nm = timm()\n ,k (m):\n (,k)\nThen we can try lots of these models to see what works best. For each model we should use our previous LR train schedule but also try LR = LR * 10 and LR = LR/10 because some of these different models require stronger or weaker learning rates.
Congrats @cdeotte on achieve 8th Place.
", "votes": 1, "replies": [ { "id": 2745393, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-04-10T15:40:35.260000", "content": "Thank you Adnan
", "votes": 1, "replies": [] } ] }, { "id": 2752510, "author_name": "Bartu Hacılar", "author_url": "", "post_date": "2024-04-15T02:38:09.147000", "content": "Thank you for your sharings throughout the competition.
\nDid you use larger models instead of tiny-vit and get worse results, or is there a specific reason for using smaller models?
", "votes": 2, "replies": [ { "id": 2752513, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-04-15T02:43:18.823000", "content": "Yes. I tried many sized models and always got the best performance with tiny vision transformers. Furthermore, tiny-vit is special (among tiny models). It is pretrained with knowledge transfer from larger model into the tiny model.
Thank you for sharing your process selflessly from beginning to end.
\nYou are a true benevolent person who is admired by kagglers.
\nTHX ! !
Thank you Warrenkuo!
", "votes": 2, "replies": [] } ] }, { "id": 2744670, "author_name": "C R Suthikshn Kumar", "author_url": "", "post_date": "2024-04-10T04:19:05.450000", "content": "Congratulations on securing 8th place in this competition. Thanks for sharing your solution details.
", "votes": 2, "replies": [ { "id": 2744754, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-04-10T05:19:58.700000", "content": "Thank you.
", "votes": 1, "replies": [] } ] }, { "id": 2744421, "author_name": "gezi", "author_url": "", "post_date": "2024-04-09T22:45:03.073000", "content": "Great work, your sharing early in the game make this game much more interesting and helped so much people to join.
\n Could you help share your cv results of your first step 3 models?
Thanks. I enjoyed sharing and engaging in discussions with the community. It did make this competition more challenging.
\n| Model | \nCV | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|
| 2D Eeg Plot Image | \n0.280 | \n0.256 | \n0.317 | \n
| 2D Eeg Spectrogram Image | \n0.281 | \n0.275 | \n0.326 | \n
| 1D Raw Eeg Waveform | \n0.303 | \n0.305 | \n0.363 | \n
| Ordinary Ensemble | \n0.253 | \n??? | \n??? | \n
| One Mega Model | \n0.246 | \n0.242 | \n0.290 | \n
.
\nSurprisingly the 2D Eeg Plot model achieves LB 0.25 by itself wow! We also see using a combined mega model achieves better CV than simple ensemble of 3 models. Note a mega model is like stacking we are training a new model to make predictions from the output of 3 previous models.
Note that these are not my best single modality input models. I have another 1D Raw Eeg Waveform model that achieves CV 0.28 LB 0.28 by modifying the public notebook architecture. And I probably have better versions of the other two single models too. But for my mega model the table above shows which models are utilized for it.
", "votes": 3, "replies": [] } ] }, { "id": 2744344, "author_name": "Andrij", "author_url": "", "post_date": "2024-04-09T20:51:57.857000", "content": "Great solution Chris! I also tried to use similar graphs implemented through matplotlib at the very beginning, but gave up on this idea because I couldn't get good results, and all the articles I saw said that this is not a good solution.
", "votes": 2, "replies": [] }, { "id": 2744261, "author_name": "CPMP", "author_url": "", "post_date": "2024-04-09T20:11:10.023000", "content": "What post processing did you use? You aid you used some in a past yesterday.
", "votes": 2, "replies": [ { "id": 2744414, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-04-09T22:40:24.320000", "content": "My final solution is an ensemble of multiple diverse models (and increases my private rank from 15th to 8th). The model shown in this discussion post by itself achieves gold medal and does not require any post process. Similar to you when I used pseudo label for some models I adjusted the target distribution of pseudo labeled train vote<10 with:
\n\n\nTo avoid this distribution shift I sample pseudo labelled data by target, to keep the same distribution as training data. (Quote from CPMP here)
\n
Some of my other models were trained with different subsets (i.e. different target distributions) of train data and consequently don't have the right prediction distribution of targets.
\nFor example, we can train equally with all vote<10 original targets and vote>=10 original targets then the model will predict too many Seizures and too many Other. Afterward we correct the predictions with the following post process:
\npred = submission[TARGETS].copy()\nfor k,v in [(,),(,)]: \n pred[:,k] = pred[:,k]/(-pred[:,k]) \n pred[:,k] *= v \n pred[:,k] = pred[:,k]/(pred[:,k]+) \npred = pred / pred.sum(axis=,keepdims=) \nsubmission[TARGETS] = pred\nok, I did not have the distribution shift issue indeed.
", "votes": 1, "replies": [] }, { "id": 2753945, "author_name": "Kazuma143", "author_url": "", "post_date": "2024-04-15T18:14:55.937000", "content": "Very Nice!!
", "votes": 1, "replies": [] } ] } ] }, { "id": 2824203, "author_name": "Vaishno Kumar Tiwari", "author_url": "", "post_date": "2024-05-19T16:24:54.180000", "content": "Hi Chris … Congratulations on your goldmedal!
", "votes": 0, "replies": [] }, { "id": 2782824, "author_name": "", "author_url": "", "post_date": "2024-04-29T13:12:27.030000", "content": "", "votes": 0, "replies": [] }, { "id": 2756059, "author_name": "", "author_url": "", "post_date": "2024-04-16T19:59:23.567000", "content": "", "votes": 1, "replies": [] }, { "id": 2753171, "author_name": "", "author_url": "", "post_date": "2024-04-15T10:36:46.610000", "content": "", "votes": 1, "replies": [] }, { "id": 2746716, "author_name": "", "author_url": "", "post_date": "2024-04-11T12:19:20.697000", "content": "", "votes": 1, "replies": [] }, { "id": 2768884, "author_name": "", "author_url": "", "post_date": "2024-04-23T04:40:32.933000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2744038": "# Thank You Kaggle and Hosts!\nVery fun competition! I love competitions with opportunities to be creative, explore, and make discoveries! In the past 3 months, I enthusiastically conducted 1000+ experiments!\n\nI also love competitions where a single model can achieve Gold medal (instead of a large ensemble).\n\nThank you Kagglers for all the great discussion shares and notebook shares!\n\n# Understand Test Data - What is Competition Task?\nThe task in this competition is to **predict annotator opinions**. By reading the host's paper [here][1], we learn that there are **119 overall annotators** in train data and **20 expert annotators** in test data. The paper shows the average Seizure prediction from 119 annotators is 18.8% whereas the 20 experts are less likely to classify as Seizure and their average is 1.5%. \n\nWow, this is a big difference therefore we need to train our models with **expert opinions** instead of **overall opinions**. We can locate the expert annotators in the train data with the filter `expert annotators = train.loc[train.vote_count>=10]`.\n\n# Which Features Do Annotators See?\nFrom the paper we see that the annotators see 2D spectrograms and 2D plots of waveforms. So let's input 2D spectrograms and 2D plots of waveforms into our models. (Also inputting 1D raw waveform helped too).\n\n# Train 3 Separate Models with Vote>=10\nFirst we train 3 separate models using train data with `vote_count >= 10` (i.e. the **expert annotator opinions**). \n* One model receives 2D spectrograms (created from EEG) as input and uses `tiny_vit_21m_512` to classify into 6 targets. \n* Second model receives 2D matplotlib plots of the montage waveforms converted into a 2D image and uses `tiny_vit_21m_224` to classify into 6 targets. \n* And third model receives 1D raw time series waveform and uses `EegNet-1D` [here][2] to classify into 6 targets.\n\n\n\n\n\n# Pseudo Label Vote<10\nUsing our 3 single models above, we pseudo label the train data with `vote<10`. We set the new labels as `new_label = 0.1 * old_label + 0.3 * model1 + 0.3 * model2 + 0.3 * model3`.\n\n# Build One Mega Model and Train with Vote>=10 plus Pseudo Vote<10\nWe load our 3 pretrained models above and remove the 3 heads. We combine them into one model, concatenate the final embeddings and add 1 new head. Then we train with 2x copies of `vote>=10` data concatenated with 1x copy of `vote<10 with pseudo` data. (Note the 2D spectrogram model above actually trains with both 10 sec spectrograms and 50 spectrograms in one image as depicted below):\n\n\n\n# Single Model Performance\n## ==> 🎉 CV = 0.246, Public LB = 0.243, Private LB = 0.290 🎉\n\n# More Details\nTo successfully accomplish the above and achieve **single model Gold Medal** Hooray!, there are many small details. Here are some:\n* Preprocess all raw waveform with MNE library \n\n from mne.filter import filter_data, notch_filter\n sample = data.T[[0,4,5,6, 11,15,16,17, 0,1,2,3, 11,12,13,14]]\\\n - data.T[[4,5,6,7, 15,16,17,18, 1,2,3,7, 12,13,14,18]]\n sample = notch_filter(sample.astype('float64'), 200, 60, n_jobs=32, verbose='ERROR')\n sample = filter_data(sample.astype('float64'), 200, 0.5, 40, n_jobs=32, verbose='ERROR') \n sample = np.clip(sample,-500,500)\n sample = np.nan_to_num(sample, nan=0)\n\n* Use all 107k rows of train data (instead of only 1 row per eeg_id).\n* Read all eeg parquets into CPU RAM and create spectrograms inside dataloader with `torchaudio.transforms.MelSpectrogram` and crop with `eeg_label_offset_seconds`.\n\n import torchaudio.transforms as T\n # 50 SECOND SPEC\n make_spec1 = T.MelSpectrogram(n_fft=2048, win_length=1280, hop_length=19,\n f_min=0,f_max=20,sample_rate=200,n_mels=64).to(device)\n # MIDDLE 10 SECOND SPEC\n make_spec2 = T.MelSpectrogram(n_fft=1280, win_length=32, hop_length=4,\n f_min=0,f_max=20,sample_rate=200,n_mels=64).to(device)\n\n* Each epoch train with 1 sample from each unique `eeg_id`. But randomly select 1 sample from all `eeg_id` samples with `train = train.loc[train.eeg_id == EEG_ID].sample(1)`.\n\n* Apply data augmentations to raw waveform (not spectrogram) inside dataloader before creating spectrogram.\n* Use data augmentation \"brain-right-left flip\", \"brain-temporal-parasagittal-flip\", \"waveform-crop-rescale\", \"waveform-invert\", \"frequency-dropout\"\n\n* Use `tiny_vit_21m_512` for 2D spectrograms and `tiny_vit_21m_224` for 2D wave plots.\n* Use Nischay EegNet-1D [here][2] for 1D raw wave data model.\n* Tune LR and learning schedule of each model individually using my discussion post [here][3]\n* Cross validate with train `vote>=10` and observe perfect correlation between CV and LB.\n\n# Inference Code Published\nMy single model inference notebook is published [here][4]. This single model achieves **Gold 14th place**. My final submission is an ensemble of this model with some other models and achieves **Gold 8th place**.\n\nThe code describes the PyTorch model details, the preprocess details, PyTorch data loader details, and data augmentation details. Enjoy!\n\n[1]: https://github.com/bdsp-core/IIIC-SPaRCNet/blob/main/IIIC_Classification-Supplemental.pdf \n[2]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/471666\n[3]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/488083\n[4]: https://www.kaggle.com/code/cdeotte/single-model-gold-solution-cv-0-24-lb-0-24", "2744045": "Using plots is an intriguing idea. I never saw it before.\n\nGlad you ended in top 10 despite sharing so much. Congrats!\n", "2747902": "Hi Chris! 🎉**Congratulations** on your gold medal, and thank you for your selfless sharing.\n\nAs a novice, I am now confused about choosing PyTorch or TensorFlow. Many baselines were produced during the beginning of competition based on TensorFlow (including your [WaveNet Starter](https://www.kaggle.com/code/cdeotte/wavenet-starter-lb-0-52), [EfficientNetB0 Starter](https://www.kaggle.com/code/cdeotte/efficientnetb0-starter-lb-0-43)), but in the end most gold medals' solutions used PyTorch. I wanna ask if there is any reason?\n\nIt seems that PyTorch has a more complete API for implementing many custom aspects: \n\n- lr schedule (Cosine annealing is manually implemented in the *Train Scheduler* section of your [EfficientNetB0 Starter](https://www.kaggle.com/code/cdeotte/efficientnetb0-starter-lb-0-43#Train-Scheduler), but it can be simply implemented through `torch.optim.lr_scheduler.CosineAnnealingLR` in PyTorch)\n- Adding sample weights (PyTorch is easier than the TF)\n- Use `entmax` instead of `softmax` (referring to the [1st place solution](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/492560), I did not find a direct way to implement it in TF)\n- And many more...\n\nIn this competition, I also spent much time solving the problem of different TF versions. Version 2.13 in Kaggle Notebook and my local version 2.15 are incompatible with *.h5* weight files🥲. \n\nTo sum up, TF seems far less useful than PyTorch. Is this the current situation or is it just my bias haha. Based on this, can you give any suggestions on the use of these two tools?", "2744226": "\"Second model receives 2D matplotlib plots of the montage waveforms converted into a 2D image\" 🤯 mind blown", "2745920": "**UPDATE** I published my single model inference notebook [here][1]. It achieves 🎉 CV = 0.246, Public LB = 0.243, Private LB = 0.290 🎉 It describes the PyTorch model details, the preprocess details, PyTorch data loader details, and data augmentation details. Enjoy!\n\n[1]: https://www.kaggle.com/code/cdeotte/single-model-gold-solution-cv-0-24-lb-0-24", "2744085": "Hi Chris, Congratulations! I do have to say that my brian has exploded : from this \"The task in this competition is to predict annotator opinions. By reading the host's paper here, we learn that there are 119 overall annotators in train data and 20 expert annotators in test data. The paper shows the average Seizure prediction from 119 annotators is 18.8% whereas the 20 experts are less likely to classify as Seizure and their average is 1.5%.\n\nWow, this is a big difference therefore we need to train our models with expert opinions instead of overall opinions. We can locate the expert annotators in the train data with the filter expert annotators = train.loc[train.vote_count>=10].\"\n\nI tried reading the original paper and I totally missed that there was a difference amongst annotators - I thought it was all radiology fellows. Furthermore, I had no idea that the experts were those with a vote count of more than 10. Why isn't this on the main page ? Is this what all Kaggle competitions are like? ", "2796609": "Thank you for your sharings throughout the competition.", "2782804": "@cdeotte Hi, thanks a lot for this post, I highly appreciate the work you have done here.\nI have the following question.\nAs I understand from this post `tiny_vit_21m_224 `\nuses that 10seconds data as an input, but it is already pretrained model. How can I find that model that only uses 10seconds data and train separately ? ", "2758405": "I ran into some issues where data enhancement became a bottleneck for the training network. I used some data enhancement methods to increase network generalization, which resulted in my cpu running at 100% while my gpu was only using 10%. How can data enhancement also be accelerated using Gpus?", "2755002": "As described comment [here](https://www.kaggle.com/code/nischaydnk/lightning-1d-eegnet-training-pipeline-hbs/comments#2643122), the eeg net 1D model seems to have a GRU layer which treats the batch dimension as sequence dimension because the layer is instantiated with the default setting batch_first=False.\nSo there are interactions between samples in batches and I'm confused to interpret this model architecture.\nThis seemed bug for me at first glance but not modified here.\n\nHere is the result of the one mega model from the [solution notebook](https://www.kaggle.com/code/cdeotte/single-model-gold-solution-cv-0-24-lb-0-24)\n\n\n\nTo test there are interactions with samples in a batch, I duplicated the test record and got slightly different result from above.\n\n\n\nNext I set the parameter batch_first=True, and get the same output for the duplicated row.\n\n\n\nI wonder there is an intent behind not setting batch_first=True or this is just a bug.", "2754948": "thank you for sharing this, very helpful", "2752717": "Thank you for your sharings.\nI have a question about \"Then we train with 2x copies of vote>=10 data concatenated with 1x copy of vote<10 with pseudo data.\".\nwhat is the meaning of copies ?? \nI don't understand what exactly it is.", "2751453": "thank you for sharing this, very helpful ", "2748258": "Very helpful", "2746120": "the graphs is looking little dangerous and attracting me to know more!!\ncongratulations for the gold @cdeotte !!", "2745937": "Very helpful!!", "2745732": "Thanks for sharing this solution it will be very helpful to us.", "2745654": "Great and helpful work👌🏻 congrats☘️", "2744907": "Great work on HMS - Harmful Brain Activity Classification! Your approach was clear and effective. Have you considered [specific suggestion] for even better results?", "2744788": "Hi Chris! Congrats with the gold, and thank you for your early stages posts and especially 50s spectrogramms, those were very handy! \n\nI also used 1D -> 2D plot with the same motivation to show the model ~ same stuff as were shown to annotators, and I also was surprised it works, a nice trick indeed.\n\nHave you measured the pseudolabels impact? I found PLs to improve the public score just a bit (on ~0.001 order) on multiple occasions and decided to trust it, but did not do any hold out validation so could not tell more precisely, that would be interesting to see if it helped you more.", "2744412": "I believe ViT here refers to visual transformers? I had started training one from scratch, should have known better to rely on pre-trained solutions, very nice!", "2744254": "Congrats @cdeotte on achieve 8th Place.", "2752510": "Thank you for your sharings throughout the competition.\n\nDid you use larger models instead of tiny-vit and get worse results, or is there a specific reason for using smaller models?", "2745444": "Thank you for sharing your process selflessly from beginning to end.\nYou are a true benevolent person who is admired by kagglers.\nTHX ! ! ", "2744670": "Congratulations on securing 8th place in this competition. Thanks for sharing your solution details. ", "2744421": "Great work, your sharing early in the game make this game much more interesting and helped so much people to join.\n Could you help share your cv results of your first step 3 models? ", "2744344": "Great solution Chris! I also tried to use similar graphs implemented through matplotlib at the very beginning, but gave up on this idea because I couldn't get good results, and all the articles I saw said that this is not a good solution.", "2744261": "What post processing did you use? You aid you used some in a past yesterday.", "2824203": "Hi Chris ... Congratulations on your goldmedal! ", "2782824": "", "2756059": "Thanks for sharing!", "2753171": "Thank you for sharing", "2746716": "Thank you for sharing your process", "2768884": "Thank you for your sharings." } }