![\"Pipeline\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7951864%2Faf2152ac5a9a165a53b5e9c562a373e2%2FLabel-Refine-Pipeline.png?generation=1712843769135040&alt=media\")
This was my first time trying the EEG related task, and I found it to be a highly valuable learning experience. Thanks to Harvard Medical School and Kaggle for hosting this fun competition.
\nThe input data is duplicate by eeg_id and votes columns, and final obtain 20183 samples in train data.
\n = pd.read_csv()\n = df.columns[-:]\n = df.drop_duplicates(subset=[] + list(TARGETS))\nI split this train data into 5-fold by using GroupKFold method with expert_consensus and patient_id label.
\nIn my solution, the input data include eeg signal (1D) and spectrogram (2D). For the spectrogram input, I refer rafaelzimmermann1's notebook to convert eeg to spectrogram, and final combine the exist kaggle spectrogram and converted spectorgram as model's input.
\nThere are three cv models for ensemble:
\nFinal, I ensemble the prediction results of three cv models by: 0.34 * Model1 + 0.32 * Model2 + 0.34 * Model3
\nI adpated the two-stage training strategy refers to the seanbearden's notebook. And this strategy bring significant public LB metric imporvement in WaveNet and EfficientNet models (for pre-validation), and I find the cv metric for fine data is seem strong correlation with public LB metric:
\n| Model | \nPublic LB | \n
|---|---|
| WaveNet | \n0.46 | \n
| WaveNet (two-stage) | \n0.39 | \n
| EfficientNet-B0 | \n0.42 | \n
| EfficientNet-B0 (two-stage) | \n0.34 | \n
After two-stage training strategy, I also tried the label refine training strategy according to rafaelzimmermann1's notebook. I tested this strategy and get improvement of local cv metric:
\n| Model | \nCV | \nPublic LB | \n
|---|---|---|
| EfficientNet-B1 | \n0.3102 | \n-- | \n
| EfficientNet-B0 (label-refine) | \n0.3058 | \n-- | \n
| WaveNet + EfficientNet-B1 | \n0.2903 | \n-- | \n
| WaveNet + EfficientNet-B0 (label-refine) | \n0.2686 | \n0.28 | \n
I used the CutMix Augmentation during model training, and this trick could bring some improvement of cv metric:
\n| Model | \nCV | \nPublic LB | \n
|---|---|---|
| EfficientNet-B1 | \n0.3130 | \n0.30 | \n
| EfficientNet-B0 (label-refine) | \n0.3102 | \n-- | \n
In the label refine training pipeline, I set the iteration equals to 1 for models except for EfficientNet-B1 model. For the EfficientNet-B1 model, I set the iteration equals to 2 for better performance (just a little).
\nThe final training pipeline can be concluded:
Finally, In order to avoid the possible data drift in two-stage training mode, I selected the teacher model (trained with pseudo-label coarse data and fine data, public LB: 0.29) and the student model (finetuned with pseudo-label fine data, public LB: 0.27) as final models.
", "messages": [ { "id": 2746849, "postDate": "2024-04-11T14:08:33.017Z", "content": "This was my first time trying the EEG related task, and I found it to be a highly valuable learning experience. Thanks to Harvard Medical School and Kaggle for hosting this fun competition.
\nThe input data is duplicate by eeg_id and votes columns, and final obtain 20183 samples in train data.
\n = pd.read_csv()\n = df.columns[-:]\n = df.drop_duplicates(subset=[] + list(TARGETS))\nI split this train data into 5-fold by using GroupKFold method with expert_consensus and patient_id label.
\nIn my solution, the input data include eeg signal (1D) and spectrogram (2D). For the spectrogram input, I refer rafaelzimmermann1's notebook to convert eeg to spectrogram, and final combine the exist kaggle spectrogram and converted spectorgram as model's input.
\nThere are three cv models for ensemble:
\nFinal, I ensemble the prediction results of three cv models by: 0.34 * Model1 + 0.32 * Model2 + 0.34 * Model3
\nI adpated the two-stage training strategy refers to the seanbearden's notebook. And this strategy bring significant public LB metric imporvement in WaveNet and EfficientNet models (for pre-validation), and I find the cv metric for fine data is seem strong correlation with public LB metric:
\n| Model | \nPublic LB | \n
|---|---|
| WaveNet | \n0.46 | \n
| WaveNet (two-stage) | \n0.39 | \n
| EfficientNet-B0 | \n0.42 | \n
| EfficientNet-B0 (two-stage) | \n0.34 | \n
After two-stage training strategy, I also tried the label refine training strategy according to rafaelzimmermann1's notebook. I tested this strategy and get improvement of local cv metric:
\n| Model | \nCV | \nPublic LB | \n
|---|---|---|
| EfficientNet-B1 | \n0.3102 | \n-- | \n
| EfficientNet-B0 (label-refine) | \n0.3058 | \n-- | \n
| WaveNet + EfficientNet-B1 | \n0.2903 | \n-- | \n
| WaveNet + EfficientNet-B0 (label-refine) | \n0.2686 | \n0.28 | \n
I used the CutMix Augmentation during model training, and this trick could bring some improvement of cv metric:
\n| Model | \nCV | \nPublic LB | \n
|---|---|---|
| EfficientNet-B1 | \n0.3130 | \n0.30 | \n
| EfficientNet-B0 (label-refine) | \n0.3102 | \n-- | \n
In the label refine training pipeline, I set the iteration equals to 1 for models except for EfficientNet-B1 model. For the EfficientNet-B1 model, I set the iteration equals to 2 for better performance (just a little).
\nThe final training pipeline can be concluded:
Finally, In order to avoid the possible data drift in two-stage training mode, I selected the teacher model (trained with pseudo-label coarse data and fine data, public LB: 0.29) and the student model (finetuned with pseudo-label fine data, public LB: 0.27) as final models.
", "rawMarkdown": "This was my first time trying the EEG related task, and I found it to be a highly valuable learning experience. Thanks to Harvard Medical School and Kaggle for hosting this fun competition.\n\n## Data\n### Train Data\nThe input data is duplicate by eeg_id and votes columns, and final obtain 20183 samples in train data. \n```\ndf = pd.read_csv('train.csv')\nTARGETS = df.columns[-6:]\ntrain_df = df.drop_duplicates(subset=['eeg_id'] + list(TARGETS))\n```\nI split this train data into 5-fold by using GroupKFold method with expert_consensus and patient_id label.\n\n### Multi-Modal Input Data\nIn my solution, the input data include eeg signal (1D) and spectrogram (2D). For the spectrogram input, I refer [rafaelzimmermann1's notebook](https://www.kaggle.com/code/rafaelzimmermann1/no-ensemble-new-spectrograms-label-refine) to convert eeg to spectrogram, and final combine the exist kaggle spectrogram and converted spectorgram as model's input.\n\n## Model\nThere are three cv models for ensemble:\n+ Model1: EfficientNet-B1 with spectrogram data\nI refer [minhsienweng's notebook](https://www.kaggle.com/code/minhsienweng/train-infer-efficientnetb0-starter) to build and train the model, and I adopt EfficientNet-B1 model for better performance.\n+ Model2: ResNet-1D + EfficientNet-B1 with eeg and spectrogram data\nI refer [nischaydnk's notebook](https://www.kaggle.com/code/nischaydnk/training-multimodal-1d-2d-approach-eegs) to build multi-modal model for eeg and spectrogram data.\n+ Model3: WaveNet + EfficientNet-B1 with eeg and spectrogram data\nBase on the ResNet-1D + EfficientNet-B1 multi-modal model architecture, I replace the ResNet-1D with WaveNet (which refers [abaojiang's notebook](https://www.kaggle.com/code/abaojiang/lb-0-46-dilatedinception-wavenet-training)) model for better processing power for eeg signals.\n\nFinal, I ensemble the prediction results of three cv models by: 0.34 * Model1 + 0.32 * Model2 + 0.34 * Model3\n\n## Key Training Strategies\n### Two-stage training\nI adpated the two-stage training strategy refers to the [seanbearden's notebook](https://www.kaggle.com/code/seanbearden/effnetb0-2-pop-model-train-twice-lb-0-39). And this strategy bring significant public LB metric imporvement in WaveNet and EfficientNet models (for pre-validation), and I find the cv metric for fine data is seem strong correlation with public LB metric:\n| Model | Public LB |\n|-----------------|:--------:|\n| WaveNet | 0.46 |\n| WaveNet (two-stage) | 0.39 |\n| EfficientNet-B0 | 0.42 |\n| EfficientNet-B0 (two-stage) | 0.34 |\n\n### Label refine training\nAfter two-stage training strategy, I also tried the label refine training strategy according to [rafaelzimmermann1's notebook](https://www.kaggle.com/code/rafaelzimmermann1/no-ensemble-new-spectrograms-label-refine). I tested this strategy and get improvement of local cv metric:\n\n| Model | CV | Public LB |\n|-----------------|:--------:|:--------:|\n| EfficientNet-B1 | 0.3102 | -- |\n| EfficientNet-B0 (label-refine) | 0.3058 | -- |\n| WaveNet + EfficientNet-B1 | 0.2903 | -- |\n| WaveNet + EfficientNet-B0 (label-refine) | 0.2686 | 0.28 |\n\n### Other tricks\n#### CutMix Augmentation\nI used the [CutMix](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/479446) Augmentation during model training, and this trick could bring some improvement of cv metric:\n\n| Model | CV | Public LB |\n|-----------------|:--------:|:--------:|\n| EfficientNet-B1 | 0.3130 | 0.30 |\n| EfficientNet-B0 (label-refine) | 0.3102 | -- |\n\n#### Iteration Steps ####\nIn the label refine training pipeline, I set the iteration equals to 1 for models except for EfficientNet-B1 model. For the EfficientNet-B1 model, I set the iteration equals to 2 for better performance (just a little).\n\n### Final training pipeline\nThe final training pipeline can be concluded: \n\n#### Final Submission ###\nFinally, In order to avoid the possible data drift in two-stage training mode, I selected the teacher model (trained with pseudo-label coarse data and fine data, public LB: 0.29) and the student model (finetuned with pseudo-label fine data, public LB: 0.27) as final models.", "votes": 9 } ], "comments": [], "raw_markdown_by_id": { "2746849": "This was my first time trying the EEG related task, and I found it to be a highly valuable learning experience. Thanks to Harvard Medical School and Kaggle for hosting this fun competition.\n\n## Data\n### Train Data\nThe input data is duplicate by eeg_id and votes columns, and final obtain 20183 samples in train data. \n```\ndf = pd.read_csv('train.csv')\nTARGETS = df.columns[-6:]\ntrain_df = df.drop_duplicates(subset=['eeg_id'] + list(TARGETS))\n```\nI split this train data into 5-fold by using GroupKFold method with expert_consensus and patient_id label.\n\n### Multi-Modal Input Data\nIn my solution, the input data include eeg signal (1D) and spectrogram (2D). For the spectrogram input, I refer [rafaelzimmermann1's notebook](https://www.kaggle.com/code/rafaelzimmermann1/no-ensemble-new-spectrograms-label-refine) to convert eeg to spectrogram, and final combine the exist kaggle spectrogram and converted spectorgram as model's input.\n\n## Model\nThere are three cv models for ensemble:\n+ Model1: EfficientNet-B1 with spectrogram data\nI refer [minhsienweng's notebook](https://www.kaggle.com/code/minhsienweng/train-infer-efficientnetb0-starter) to build and train the model, and I adopt EfficientNet-B1 model for better performance.\n+ Model2: ResNet-1D + EfficientNet-B1 with eeg and spectrogram data\nI refer [nischaydnk's notebook](https://www.kaggle.com/code/nischaydnk/training-multimodal-1d-2d-approach-eegs) to build multi-modal model for eeg and spectrogram data.\n+ Model3: WaveNet + EfficientNet-B1 with eeg and spectrogram data\nBase on the ResNet-1D + EfficientNet-B1 multi-modal model architecture, I replace the ResNet-1D with WaveNet (which refers [abaojiang's notebook](https://www.kaggle.com/code/abaojiang/lb-0-46-dilatedinception-wavenet-training)) model for better processing power for eeg signals.\n\nFinal, I ensemble the prediction results of three cv models by: 0.34 * Model1 + 0.32 * Model2 + 0.34 * Model3\n\n## Key Training Strategies\n### Two-stage training\nI adpated the two-stage training strategy refers to the [seanbearden's notebook](https://www.kaggle.com/code/seanbearden/effnetb0-2-pop-model-train-twice-lb-0-39). And this strategy bring significant public LB metric imporvement in WaveNet and EfficientNet models (for pre-validation), and I find the cv metric for fine data is seem strong correlation with public LB metric:\n| Model | Public LB |\n|-----------------|:--------:|\n| WaveNet | 0.46 |\n| WaveNet (two-stage) | 0.39 |\n| EfficientNet-B0 | 0.42 |\n| EfficientNet-B0 (two-stage) | 0.34 |\n\n### Label refine training\nAfter two-stage training strategy, I also tried the label refine training strategy according to [rafaelzimmermann1's notebook](https://www.kaggle.com/code/rafaelzimmermann1/no-ensemble-new-spectrograms-label-refine). I tested this strategy and get improvement of local cv metric:\n\n| Model | CV | Public LB |\n|-----------------|:--------:|:--------:|\n| EfficientNet-B1 | 0.3102 | -- |\n| EfficientNet-B0 (label-refine) | 0.3058 | -- |\n| WaveNet + EfficientNet-B1 | 0.2903 | -- |\n| WaveNet + EfficientNet-B0 (label-refine) | 0.2686 | 0.28 |\n\n### Other tricks\n#### CutMix Augmentation\nI used the [CutMix](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/479446) Augmentation during model training, and this trick could bring some improvement of cv metric:\n\n| Model | CV | Public LB |\n|-----------------|:--------:|:--------:|\n| EfficientNet-B1 | 0.3130 | 0.30 |\n| EfficientNet-B0 (label-refine) | 0.3102 | -- |\n\n#### Iteration Steps ####\nIn the label refine training pipeline, I set the iteration equals to 1 for models except for EfficientNet-B1 model. For the EfficientNet-B1 model, I set the iteration equals to 2 for better performance (just a little).\n\n### Final training pipeline\nThe final training pipeline can be concluded: \n\n#### Final Submission ###\nFinally, In order to avoid the possible data drift in two-stage training mode, I selected the teacher model (trained with pseudo-label coarse data and fine data, public LB: 0.29) and the student model (finetuned with pseudo-label fine data, public LB: 0.27) as final models." } }