{
  "id": 492619,
  "title": "6th Place Solution for the HMS - Harmful Brain Activity Classification Competition",
  "url": "/competitions/hms-harmful-brain-activity-classification/writeups/meow-6th-place-solution-for-the-hms-harmful-brain",
  "author_name": "",
  "post_date": "2024-04-10T10:51:50.593Z",
  "votes": 40,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I can't believe that I can achive a solo gold medal with a few experiments, one expirement per day. I am so happy right now and I'm so grateful to everyone at Kaggle for all that you've shared. Thank you Chris Deotte for all the amazing things you've shared. Thank you seanbearden for two stage training.</p>\n<h1>1. Context</h1>\n<ul>\n<li>Business context: <a href=\"https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/overview\" target=\"_blank\">https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/overview</a></li>\n<li>Data context: <a href=\"https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/data\" target=\"_blank\">https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/data</a></li>\n</ul>\n<h1>2. Overview of the Approach</h1>\n<p>In early-stage, I experiment with Kaggle spectrogram, next to Kaggle + eeg spectrogram, next to Kaggle + eeg spectrogram + raw 50s eeg. Finally, I using multi-modal model with 4 input: kaggle spectrogram, eeg spectrogram, raw 50s eeg data, raw 10s eeg data. Net contain 4 separate backbone for each input. I use 50s(20s) raw eeg data to extract global feature, 10s raw eeg data to extract local feature. </p>\n<h2>2.1 Split fold</h2>\n<p>I use GroupKFold, group by patient_id.</p>\n<ul>\n<li>Split folds notebook: <a href=\"https://www.kaggle.com/code/quan0095/split-kfold-totalvote-hms/\" target=\"_blank\">https://www.kaggle.com/code/quan0095/split-kfold-totalvote-hms/</a></li>\n</ul>\n<h2>2.2 Preprocess</h2>\n<ul>\n<li>I find and remove eeg_id have nan percent &gt; 10%.</li>\n<li>Find Nan notebook: <a href=\"https://www.kaggle.com/code/quan0095/find-nan/\" target=\"_blank\">https://www.kaggle.com/code/quan0095/find-nan/</a></li>\n</ul>\n<h3>Kaggle spectrogram</h3>\n<p>I clip and log transform kaggle spectrogram:\n<code>spec_img = np.clip(spec_img,np.exp(-4),np.exp(8))</code>\n<code>spec_img = np.log(spec_img)</code>\n<code>spec_img = np.nan_to_num(spec_img, nan=0.0)</code>\nNormalize by using mean, standard.</p>\n<h3>Create eeg spectrogram</h3>\n<p>I use 4 chain with 4 features: \nFp1-F7, F7-T3, T3-T5, T5-O1\nFp1-F3, F3-C3, C3-P3, P3-O1\nFp2-F8, F8-T4, T4-T6, T6-O2\nFp2-F4, F4-C4, C4-P4, P4-O2</p>\n<h4>Version 1</h4>\n<ul>\n<li>I crop raw eeg segment by using eeg_label_offset_seconds. And use signal.spectrogram to create spectrogram. I concat 16 img shape (128, 142) to create 4x4 image with shape (512, 568).</li>\n<li>Normalize by log transform and divide by 2.0.</li>\n<li>Before feed to vit, I resize image to (518, 518, 1)\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2Fd3aa62505f039fa4f5f332e740490bb2%2FScreenshot%20from%202024-04-10%2015-36-02.png?generation=1712738515292787&amp;alt=media\" alt=\"\"></li>\n</ul>\n<h4>Version 2</h4>\n<ul>\n<li>Same with version 1, change nperseg from 70 to 39, to create large image with new shape (128,256). 4x4 image have new shape (512, 1024). </li>\n<li>Before feed to vit, I remain image size (512, 1024, 1)</li>\n</ul>\n<h4>Version 3</h4>\n<ul>\n<li>I average 4 feature in a chain. Concat 4 chain image to create 4x1 image have shape (512, 256)</li>\n<li>Before feed to vit, I resize image to (518, 518, 1)\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F9aef44118a233e166ee126c3969eea4a%2FScreenshot%20from%202024-04-10%2016-11-42.png?generation=1712740319706335&amp;alt=media\" alt=\"\"></li>\n<li>By experience, I have found version 3 is badder than version 1. But when ensemble, kl loss decrease slightly.</li>\n</ul>\n<h3>Create global raw eeg image</h3>\n<h4>Version 1</h4>\n<ul>\n<li>EEG_LENGTH=50</li>\n<li>I fill NaN by mean value</li>\n<li>I use butter filter with bandpass [0.5, 40]. New eeg clip by (-1024, 1024). I reshape eeg (4, 10000) to (4, 200, EEG_LENGTH). Next, I concat 4 image (200, EGG_LENGTH) to one image (200, 4xEGG_LENGTH). Finally, I concat 4 chain image to a single image with shape (200, 4x4xEGG_LENGTH)</li>\n<li>I normalize by divide to 104</li>\n<li>Before feed to vit, I resize image to (518, 518, 1)\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2Fa484cd44bbd98c339a8116dc9ad04d57%2FScreenshot%20from%202024-04-10%2016-20-10.png?generation=1712740826625485&amp;alt=media\" alt=\"\"></li>\n</ul>\n<h4>Version 2</h4>\n<ul>\n<li>Same as version 1, EEG_LENGTH=20</li>\n</ul>\n<h3>Create local raw eeg image</h3>\n<ul>\n<li>Same as version 1 of global raw eeg image, EEG_LENGTH=10</li>\n</ul>\n<h2>2.3 Augmentation</h2>\n<ul>\n<li>I use mixup with alpha=1.0. </li>\n<li>For kaggle and eeg spectrogram, I use A.XYMasking.</li>\n<li>For raw eeg data: I use custom augmentation: Random insert NaN to eeg raw data (both global raw and local raw)</li>\n</ul>\n<h2>2.4 Training Strategy</h2>\n<p>I use Adan optimizer with one cycle learning rate scheduler, model ema with decay=0.995, gradient checkpointing. LR=0.00017. Backbone: dinov2 vit family from timm library. For train, per each epoch, I random select an eeg_label_offset_seconds per each eeg_id. For evaluate, I using a first eeg_label_offset_seconds of an eeg_id.</p>\n<h3>Stage 1</h3>\n<p>Train with 5 epoch with data have number vote &lt; 10. </p>\n<h3>Stage 2</h3>\n<p>Train with 7 epoch with data have number vote &gt;= 10. </p>\n<h1>3. Details of the submission</h1>\n<h2>3.1 Test time augmentation</h2>\n<p>I use 3xTTA with 10s raw eeg input: shift-left 2s from center, center, shift-right 2s from center. 3xTTA increase a little Public score &lt; 0.01.</p>\n<h2>3.2 Details of ensemble</h2>\n<p><strong>Model 1</strong>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F5839a6e1f1eedf549c4997b60a4b25c7%2Fmodel_1.drawio.png?generation=1712727644167695&amp;alt=media\" alt=\"\">\n<strong>Model 2</strong>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F79176628f79aefb2f52b909939366f7b%2Fmodel_2.drawio.png?generation=1712727661965581&amp;alt=media\" alt=\"\">\n<strong>Model 3</strong>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F6ce068d0fb6a840e4315612048970460%2Fmodel_3.drawio.png?generation=1712727678141546&amp;alt=media\" alt=\"\">\n<strong>Model 4</strong>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F5c118134c590d7bf91073d82a2ef66ad%2Fmodel_4.drawio.png?generation=1712727693599176&amp;alt=media\" alt=\"\">\n<strong>Model 5</strong>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F104d5c9b925dc6f23338ed556ab22935%2Fmodel_5.drawio.png?generation=1712727710736406&amp;alt=media\" alt=\"\"></p>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>OOF</th>\n<th>OOF kaggle spec</th>\n<th>OOF eeg spec</th>\n<th>OOF global raw</th>\n<th>OOF local raw</th>\n<th>Public</th>\n<th>Private</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>model 1</td>\n<td>0.221329</td>\n<td>0.318149</td>\n<td>0.267640</td>\n<td>0.251836</td>\n<td>0.264803</td>\n<td>0.232427</td>\n<td>0.289764</td>\n</tr>\n<tr>\n<td>model 2</td>\n<td>0.217614</td>\n<td>0.325917</td>\n<td>0.266803</td>\n<td>0.240895</td>\n<td>0.264423</td>\n<td>0.229686</td>\n<td>0.287550</td>\n</tr>\n<tr>\n<td>model 3</td>\n<td>0.217881</td>\n<td>0.317720</td>\n<td>0.268293</td>\n<td>0.247168</td>\n<td>0.263158</td>\n<td>0.228330</td>\n<td>0.285105</td>\n</tr>\n<tr>\n<td>model 4</td>\n<td>0.218214</td>\n<td>0.315174</td>\n<td>0.264225</td>\n<td>0.250879</td>\n<td>0.263150</td>\n<td>0.229865</td>\n<td>0.290503</td>\n</tr>\n<tr>\n<td>model 5</td>\n<td>0.223236</td>\n<td>0.318931</td>\n<td>0.334042</td>\n<td>0.248258</td>\n<td>0.261953</td>\n<td>0.23025</td>\n<td>0.287976</td>\n</tr>\n<tr>\n<td>ensemble</td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td>0.22600</td>\n<td>0.283302</td>\n</tr>\n</tbody>\n</table>\n<p>The weaknesses of my solution is: my ensemble is not diversity. I can't use vit base, vit large for all backbone, because GPU RAM limit.</p>\n<h2>3.3 Submit strategy</h2>\n<p>Strategy 1: Best CV, train and validate on all data.\nStrategy 2: Best LB, train and validate on data with number voter &gt;= 10.</p>\n<h2>3.4 What didn't work</h2>\n<ul>\n<li>I have try Wavenet, 1DCNN-GRU, squeezeformer, eegnet, eegconformer with raw data, but not success compare with 2D Vision transformer.</li>\n<li>I have stack 4 chain to create an image with 4 channel, but not success with eeg spectrogram and raw eeg data.</li>\n<li>I have try normalize eeg spectrogram and raw eeg data by mean, std per channel.</li>\n<li>I have try butter filter with lowpass, and other range filters.</li>\n<li>I have try feed original size of raw eeg image to vit, but not success, kl loss is more higher. Finally, I must resize raw eeg image to (518, 518, 1). Large image size improve kl-loss.</li>\n<li>Data augmentation have little effect, except mixup.</li>\n</ul>\n<h1>Sources</h1>\n<ul>\n<li>Seanbearden, Two-stage training: <a href=\"https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477135\" target=\"_blank\">https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477135</a></li>\n<li>Chris Deotte, Understanding Competition Data: <a href=\"https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/468010\" target=\"_blank\">https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/468010</a></li>\n<li>Chris Deotte, EEG spectrogram: <a href=\"https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467877\" target=\"_blank\">https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467877</a></li>\n<li>Inference notebook: <a href=\"https://www.kaggle.com/code/quan0095/hms-best-public-final/\" target=\"_blank\">https://www.kaggle.com/code/quan0095/hms-best-public-final/</a></li>\n<li>Training source: <a href=\"https://github.com/quanvuhust/kaggle_hms\" target=\"_blank\">https://github.com/quanvuhust/kaggle_hms</a></li>\n</ul>",
  "messages": [
    {
      "id": "2745013",
      "postDate": "04/10/2024 09:51:05",
      "content": "<p>I can't believe that I can achive a solo gold medal with a few experiments, one expirement per day. I am so happy right now and I'm so grateful to everyone at Kaggle for all that you've shared. Thank you Chris Deotte for all the amazing things you've shared. Thank you seanbearden for two stage training.</p>\n<h1>1. Context</h1>\n<ul>\n<li>Business context: <a href=\"https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/overview\" target=\"_blank\">https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/overview</a></li>\n<li>Data context: <a href=\"https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/data\" target=\"_blank\">https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/data</a></li>\n</ul>\n<h1>2. Overview of the Approach</h1>\n<p>In early-stage, I experiment with Kaggle spectrogram, next to Kaggle + eeg spectrogram, next to Kaggle + eeg spectrogram + raw 50s eeg. Finally, I using multi-modal model with 4 input: kaggle spectrogram, eeg spectrogram, raw 50s eeg data, raw 10s eeg data. Net contain 4 separate backbone for each input. I use 50s(20s) raw eeg data to extract global feature, 10s raw eeg data to extract local feature. </p>\n<h2>2.1 Split fold</h2>\n<p>I use GroupKFold, group by patient_id.</p>\n<ul>\n<li>Split folds notebook: <a href=\"https://www.kaggle.com/code/quan0095/split-kfold-totalvote-hms/\" target=\"_blank\">https://www.kaggle.com/code/quan0095/split-kfold-totalvote-hms/</a></li>\n</ul>\n<h2>2.2 Preprocess</h2>\n<ul>\n<li>I find and remove eeg_id have nan percent &gt; 10%.</li>\n<li>Find Nan notebook: <a href=\"https://www.kaggle.com/code/quan0095/find-nan/\" target=\"_blank\">https://www.kaggle.com/code/quan0095/find-nan/</a></li>\n</ul>\n<h3>Kaggle spectrogram</h3>\n<p>I clip and log transform kaggle spectrogram:\n<code>spec_img = np.clip(spec_img,np.exp(-4),np.exp(8))</code>\n<code>spec_img = np.log(spec_img)</code>\n<code>spec_img = np.nan_to_num(spec_img, nan=0.0)</code>\nNormalize by using mean, standard.</p>\n<h3>Create eeg spectrogram</h3>\n<p>I use 4 chain with 4 features: \nFp1-F7, F7-T3, T3-T5, T5-O1\nFp1-F3, F3-C3, C3-P3, P3-O1\nFp2-F8, F8-T4, T4-T6, T6-O2\nFp2-F4, F4-C4, C4-P4, P4-O2</p>\n<h4>Version 1</h4>\n<ul>\n<li>I crop raw eeg segment by using eeg_label_offset_seconds. And use signal.spectrogram to create spectrogram. I concat 16 img shape (128, 142) to create 4x4 image with shape (512, 568).</li>\n<li>Normalize by log transform and divide by 2.0.</li>\n<li>Before feed to vit, I resize image to (518, 518, 1)\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2Fd3aa62505f039fa4f5f332e740490bb2%2FScreenshot%20from%202024-04-10%2015-36-02.png?generation=1712738515292787&amp;alt=media\" alt=\"\"></li>\n</ul>\n<h4>Version 2</h4>\n<ul>\n<li>Same with version 1, change nperseg from 70 to 39, to create large image with new shape (128,256). 4x4 image have new shape (512, 1024). </li>\n<li>Before feed to vit, I remain image size (512, 1024, 1)</li>\n</ul>\n<h4>Version 3</h4>\n<ul>\n<li>I average 4 feature in a chain. Concat 4 chain image to create 4x1 image have shape (512, 256)</li>\n<li>Before feed to vit, I resize image to (518, 518, 1)\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F9aef44118a233e166ee126c3969eea4a%2FScreenshot%20from%202024-04-10%2016-11-42.png?generation=1712740319706335&amp;alt=media\" alt=\"\"></li>\n<li>By experience, I have found version 3 is badder than version 1. But when ensemble, kl loss decrease slightly.</li>\n</ul>\n<h3>Create global raw eeg image</h3>\n<h4>Version 1</h4>\n<ul>\n<li>EEG_LENGTH=50</li>\n<li>I fill NaN by mean value</li>\n<li>I use butter filter with bandpass [0.5, 40]. New eeg clip by (-1024, 1024). I reshape eeg (4, 10000) to (4, 200, EEG_LENGTH). Next, I concat 4 image (200, EGG_LENGTH) to one image (200, 4xEGG_LENGTH). Finally, I concat 4 chain image to a single image with shape (200, 4x4xEGG_LENGTH)</li>\n<li>I normalize by divide to 104</li>\n<li>Before feed to vit, I resize image to (518, 518, 1)\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2Fa484cd44bbd98c339a8116dc9ad04d57%2FScreenshot%20from%202024-04-10%2016-20-10.png?generation=1712740826625485&amp;alt=media\" alt=\"\"></li>\n</ul>\n<h4>Version 2</h4>\n<ul>\n<li>Same as version 1, EEG_LENGTH=20</li>\n</ul>\n<h3>Create local raw eeg image</h3>\n<ul>\n<li>Same as version 1 of global raw eeg image, EEG_LENGTH=10</li>\n</ul>\n<h2>2.3 Augmentation</h2>\n<ul>\n<li>I use mixup with alpha=1.0. </li>\n<li>For kaggle and eeg spectrogram, I use A.XYMasking.</li>\n<li>For raw eeg data: I use custom augmentation: Random insert NaN to eeg raw data (both global raw and local raw)</li>\n</ul>\n<h2>2.4 Training Strategy</h2>\n<p>I use Adan optimizer with one cycle learning rate scheduler, model ema with decay=0.995, gradient checkpointing. LR=0.00017. Backbone: dinov2 vit family from timm library. For train, per each epoch, I random select an eeg_label_offset_seconds per each eeg_id. For evaluate, I using a first eeg_label_offset_seconds of an eeg_id.</p>\n<h3>Stage 1</h3>\n<p>Train with 5 epoch with data have number vote &lt; 10. </p>\n<h3>Stage 2</h3>\n<p>Train with 7 epoch with data have number vote &gt;= 10. </p>\n<h1>3. Details of the submission</h1>\n<h2>3.1 Test time augmentation</h2>\n<p>I use 3xTTA with 10s raw eeg input: shift-left 2s from center, center, shift-right 2s from center. 3xTTA increase a little Public score &lt; 0.01.</p>\n<h2>3.2 Details of ensemble</h2>\n<p><strong>Model 1</strong>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F5839a6e1f1eedf549c4997b60a4b25c7%2Fmodel_1.drawio.png?generation=1712727644167695&amp;alt=media\" alt=\"\">\n<strong>Model 2</strong>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F79176628f79aefb2f52b909939366f7b%2Fmodel_2.drawio.png?generation=1712727661965581&amp;alt=media\" alt=\"\">\n<strong>Model 3</strong>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F6ce068d0fb6a840e4315612048970460%2Fmodel_3.drawio.png?generation=1712727678141546&amp;alt=media\" alt=\"\">\n<strong>Model 4</strong>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F5c118134c590d7bf91073d82a2ef66ad%2Fmodel_4.drawio.png?generation=1712727693599176&amp;alt=media\" alt=\"\">\n<strong>Model 5</strong>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F104d5c9b925dc6f23338ed556ab22935%2Fmodel_5.drawio.png?generation=1712727710736406&amp;alt=media\" alt=\"\"></p>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>OOF</th>\n<th>OOF kaggle spec</th>\n<th>OOF eeg spec</th>\n<th>OOF global raw</th>\n<th>OOF local raw</th>\n<th>Public</th>\n<th>Private</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>model 1</td>\n<td>0.221329</td>\n<td>0.318149</td>\n<td>0.267640</td>\n<td>0.251836</td>\n<td>0.264803</td>\n<td>0.232427</td>\n<td>0.289764</td>\n</tr>\n<tr>\n<td>model 2</td>\n<td>0.217614</td>\n<td>0.325917</td>\n<td>0.266803</td>\n<td>0.240895</td>\n<td>0.264423</td>\n<td>0.229686</td>\n<td>0.287550</td>\n</tr>\n<tr>\n<td>model 3</td>\n<td>0.217881</td>\n<td>0.317720</td>\n<td>0.268293</td>\n<td>0.247168</td>\n<td>0.263158</td>\n<td>0.228330</td>\n<td>0.285105</td>\n</tr>\n<tr>\n<td>model 4</td>\n<td>0.218214</td>\n<td>0.315174</td>\n<td>0.264225</td>\n<td>0.250879</td>\n<td>0.263150</td>\n<td>0.229865</td>\n<td>0.290503</td>\n</tr>\n<tr>\n<td>model 5</td>\n<td>0.223236</td>\n<td>0.318931</td>\n<td>0.334042</td>\n<td>0.248258</td>\n<td>0.261953</td>\n<td>0.23025</td>\n<td>0.287976</td>\n</tr>\n<tr>\n<td>ensemble</td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td></td>\n<td>0.22600</td>\n<td>0.283302</td>\n</tr>\n</tbody>\n</table>\n<p>The weaknesses of my solution is: my ensemble is not diversity. I can't use vit base, vit large for all backbone, because GPU RAM limit.</p>\n<h2>3.3 Submit strategy</h2>\n<p>Strategy 1: Best CV, train and validate on all data.\nStrategy 2: Best LB, train and validate on data with number voter &gt;= 10.</p>\n<h2>3.4 What didn't work</h2>\n<ul>\n<li>I have try Wavenet, 1DCNN-GRU, squeezeformer, eegnet, eegconformer with raw data, but not success compare with 2D Vision transformer.</li>\n<li>I have stack 4 chain to create an image with 4 channel, but not success with eeg spectrogram and raw eeg data.</li>\n<li>I have try normalize eeg spectrogram and raw eeg data by mean, std per channel.</li>\n<li>I have try butter filter with lowpass, and other range filters.</li>\n<li>I have try feed original size of raw eeg image to vit, but not success, kl loss is more higher. Finally, I must resize raw eeg image to (518, 518, 1). Large image size improve kl-loss.</li>\n<li>Data augmentation have little effect, except mixup.</li>\n</ul>\n<h1>Sources</h1>\n<ul>\n<li>Seanbearden, Two-stage training: <a href=\"https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477135\" target=\"_blank\">https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477135</a></li>\n<li>Chris Deotte, Understanding Competition Data: <a href=\"https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/468010\" target=\"_blank\">https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/468010</a></li>\n<li>Chris Deotte, EEG spectrogram: <a href=\"https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467877\" target=\"_blank\">https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467877</a></li>\n<li>Inference notebook: <a href=\"https://www.kaggle.com/code/quan0095/hms-best-public-final/\" target=\"_blank\">https://www.kaggle.com/code/quan0095/hms-best-public-final/</a></li>\n<li>Training source: <a href=\"https://github.com/quanvuhust/kaggle_hms\" target=\"_blank\">https://github.com/quanvuhust/kaggle_hms</a></li>\n</ul>",
      "rawMarkdown": "I can't believe that I can achive a solo gold medal with a few experiments, one expirement per day. I am so happy right now and I'm so grateful to everyone at Kaggle for all that you've shared. Thank you Chris Deotte for all the amazing things you've shared. Thank you seanbearden for two stage training.\n# 1. Context\n- Business context: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/overview\n- Data context: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/data\n# 2. Overview of the Approach\nIn early-stage, I experiment with Kaggle spectrogram, next to Kaggle + eeg spectrogram, next to Kaggle + eeg spectrogram + raw 50s eeg. Finally, I using multi-modal model with 4 input: kaggle spectrogram, eeg spectrogram, raw 50s eeg data, raw 10s eeg data. Net contain 4 separate backbone for each input. I use 50s(20s) raw eeg data to extract global feature, 10s raw eeg data to extract local feature. \n## 2.1 Split fold\nI use GroupKFold, group by patient_id.\n- Split folds notebook: https://www.kaggle.com/code/quan0095/split-kfold-totalvote-hms/\n## 2.2 Preprocess\n- I find and remove eeg_id have nan percent > 10%.\n- Find Nan notebook: https://www.kaggle.com/code/quan0095/find-nan/\n### Kaggle spectrogram\nI clip and log transform kaggle spectrogram:\n`spec_img = np.clip(spec_img,np.exp(-4),np.exp(8)) `\n`spec_img = np.log(spec_img)`\n`spec_img = np.nan_to_num(spec_img, nan=0.0) `\nNormalize by using mean, standard.\n### Create eeg spectrogram\nI use 4 chain with 4 features: \nFp1-F7, F7-T3, T3-T5, T5-O1\nFp1-F3, F3-C3, C3-P3, P3-O1\nFp2-F8, F8-T4, T4-T6, T6-O2\nFp2-F4, F4-C4, C4-P4, P4-O2\n#### Version 1\n- I crop raw eeg segment by using eeg_label_offset_seconds. And use signal.spectrogram to create spectrogram. I concat 16 img shape (128, 142) to create 4x4 image with shape (512, 568).\n- Normalize by log transform and divide by 2.0.\n- Before feed to vit, I resize image to (518, 518, 1)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2Fd3aa62505f039fa4f5f332e740490bb2%2FScreenshot%20from%202024-04-10%2015-36-02.png?generation=1712738515292787&alt=media)\n#### Version 2\n- Same with version 1, change nperseg from 70 to 39, to create large image with new shape (128,256). 4x4 image have new shape (512, 1024). \n- Before feed to vit, I remain image size (512, 1024, 1)\n#### Version 3\n- I average 4 feature in a chain. Concat 4 chain image to create 4x1 image have shape (512, 256)\n- Before feed to vit, I resize image to (518, 518, 1)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F9aef44118a233e166ee126c3969eea4a%2FScreenshot%20from%202024-04-10%2016-11-42.png?generation=1712740319706335&alt=media)\n- By experience, I have found version 3 is badder than version 1. But when ensemble, kl loss decrease slightly.\n### Create global raw eeg image\n#### Version 1\n- EEG_LENGTH=50\n- I fill NaN by mean value\n- I use butter filter with bandpass [0.5, 40]. New eeg clip by (-1024, 1024). I reshape eeg (4, 10000) to (4, 200, EEG_LENGTH). Next, I concat 4 image (200, EGG_LENGTH) to one image (200, 4xEGG_LENGTH). Finally, I concat 4 chain image to a single image with shape (200, 4x4xEGG_LENGTH)\n- I normalize by divide to 104\n- Before feed to vit, I resize image to (518, 518, 1)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2Fa484cd44bbd98c339a8116dc9ad04d57%2FScreenshot%20from%202024-04-10%2016-20-10.png?generation=1712740826625485&alt=media)\n#### Version 2\n- Same as version 1, EEG_LENGTH=20\n### Create local raw eeg image\n- Same as version 1 of global raw eeg image, EEG_LENGTH=10\n## 2.3 Augmentation\n- I use mixup with alpha=1.0. \n- For kaggle and eeg spectrogram, I use A.XYMasking.\n- For raw eeg data: I use custom augmentation: Random insert NaN to eeg raw data (both global raw and local raw)\n## 2.4 Training Strategy\nI use Adan optimizer with one cycle learning rate scheduler, model ema with decay=0.995, gradient checkpointing. LR=0.00017. Backbone: dinov2 vit family from timm library. For train, per each epoch, I random select an eeg_label_offset_seconds per each eeg_id. For evaluate, I using a first eeg_label_offset_seconds of an eeg_id.\n### Stage 1\nTrain with 5 epoch with data have number vote < 10. \n### Stage 2\nTrain with 7 epoch with data have number vote >= 10. \n# 3. Details of the submission\n## 3.1 Test time augmentation\nI use 3xTTA with 10s raw eeg input: shift-left 2s from center, center, shift-right 2s from center. 3xTTA increase a little Public score < 0.01.\n## 3.2 Details of ensemble\n**Model 1**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F5839a6e1f1eedf549c4997b60a4b25c7%2Fmodel_1.drawio.png?generation=1712727644167695&alt=media)\n**Model 2**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F79176628f79aefb2f52b909939366f7b%2Fmodel_2.drawio.png?generation=1712727661965581&alt=media)\n**Model 3**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F6ce068d0fb6a840e4315612048970460%2Fmodel_3.drawio.png?generation=1712727678141546&alt=media)\n**Model 4**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F5c118134c590d7bf91073d82a2ef66ad%2Fmodel_4.drawio.png?generation=1712727693599176&alt=media)\n**Model 5**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F104d5c9b925dc6f23338ed556ab22935%2Fmodel_5.drawio.png?generation=1712727710736406&alt=media)\n| model |  OOF|OOF kaggle spec|OOF eeg spec|OOF global raw|OOF local raw| Public | Private |\n| --- | --- | --- | --- | --- | --- | --- |\n| model 1 | 0.221329 |0.318149  | 0.267640 |  0.251836| 0.264803 | 0.232427 | 0.289764|\n| model 2 | 0.217614 |0.325917  | 0.266803 |0.240895  |0.264423  |0.229686  |0.287550   |\n| model 3 | 0.217881 |0.317720  | 0.268293 |0.247168 | 0.263158|0.228330  | 0.285105 |\n| model 4 |0.218214  |  0.315174| 0.264225 | 0.250879 | 0.263150 |0.229865  | 0.290503 |\n| model 5 |0.223236  | 0.318931 | 0.334042 | 0.248258 | 0.261953 |0.23025  |0.287976  |\n| ensemble |  |  | |  |  |0.22600  |  0.283302 |\n\nThe weaknesses of my solution is: my ensemble is not diversity. I can't use vit base, vit large for all backbone, because GPU RAM limit.\n## 3.3 Submit strategy\nStrategy 1: Best CV, train and validate on all data.\nStrategy 2: Best LB, train and validate on data with number voter >= 10.\n## 3.4 What didn't work\n- I have try Wavenet, 1DCNN-GRU, squeezeformer, eegnet, eegconformer with raw data, but not success compare with 2D Vision transformer.\n- I have stack 4 chain to create an image with 4 channel, but not success with eeg spectrogram and raw eeg data.\n- I have try normalize eeg spectrogram and raw eeg data by mean, std per channel.\n- I have try butter filter with lowpass, and other range filters.\n- I have try feed original size of raw eeg image to vit, but not success, kl loss is more higher. Finally, I must resize raw eeg image to (518, 518, 1). Large image size improve kl-loss.\n- Data augmentation have little effect, except mixup.\n# Sources\n- Seanbearden, Two-stage training: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477135\n- Chris Deotte, Understanding Competition Data: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/468010\n- Chris Deotte, EEG spectrogram: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467877\n- Inference notebook: https://www.kaggle.com/code/quan0095/hms-best-public-final/\n- Training source: https://github.com/quanvuhust/kaggle_hms",
      "votes": null
    },
    {
      "id": "2745825",
      "postDate": "04/10/2024 21:25:15",
      "content": "<p>Well done, very impressive solo result!</p>",
      "rawMarkdown": "Well done, very impressive solo result!",
      "votes": null
    },
    {
      "id": "2746893",
      "postDate": "04/11/2024 14:28:22",
      "content": "<p>Thank you so much for the detailed model share, especially for the git and inference. Congratulations on your achievement :)</p>",
      "rawMarkdown": "Thank you so much for the detailed model share, especially for the git and inference. Congratulations on your achievement :)",
      "votes": null
    },
    {
      "id": "2747564",
      "postDate": "04/12/2024 01:03:11",
      "content": "<p>Thank you for sharing, I have learned a lot of knowledge🥳</p>",
      "rawMarkdown": "Thank you for sharing, I have learned a lot of knowledge🥳",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2745825,
      "author_name": "romainhardy",
      "author_url": "",
      "post_date": "04/10/2024 21:25:15",
      "content": "<p>Well done, very impressive solo result!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2746893,
      "author_name": "rafaelzimmermann1",
      "author_url": "",
      "post_date": "04/11/2024 14:28:22",
      "content": "<p>Thank you so much for the detailed model share, especially for the git and inference. Congratulations on your achievement :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2747564,
      "author_name": "roger92",
      "author_url": "",
      "post_date": "04/12/2024 01:03:11",
      "content": "<p>Thank you for sharing, I have learned a lot of knowledge🥳</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2745013": "I can't believe that I can achive a solo gold medal with a few experiments, one expirement per day. I am so happy right now and I'm so grateful to everyone at Kaggle for all that you've shared. Thank you Chris Deotte for all the amazing things you've shared. Thank you seanbearden for two stage training.\n# 1. Context\n- Business context: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/overview\n- Data context: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/data\n# 2. Overview of the Approach\nIn early-stage, I experiment with Kaggle spectrogram, next to Kaggle + eeg spectrogram, next to Kaggle + eeg spectrogram + raw 50s eeg. Finally, I using multi-modal model with 4 input: kaggle spectrogram, eeg spectrogram, raw 50s eeg data, raw 10s eeg data. Net contain 4 separate backbone for each input. I use 50s(20s) raw eeg data to extract global feature, 10s raw eeg data to extract local feature. \n## 2.1 Split fold\nI use GroupKFold, group by patient_id.\n- Split folds notebook: https://www.kaggle.com/code/quan0095/split-kfold-totalvote-hms/\n## 2.2 Preprocess\n- I find and remove eeg_id have nan percent > 10%.\n- Find Nan notebook: https://www.kaggle.com/code/quan0095/find-nan/\n### Kaggle spectrogram\nI clip and log transform kaggle spectrogram:\n`spec_img = np.clip(spec_img,np.exp(-4),np.exp(8)) `\n`spec_img = np.log(spec_img)`\n`spec_img = np.nan_to_num(spec_img, nan=0.0) `\nNormalize by using mean, standard.\n### Create eeg spectrogram\nI use 4 chain with 4 features: \nFp1-F7, F7-T3, T3-T5, T5-O1\nFp1-F3, F3-C3, C3-P3, P3-O1\nFp2-F8, F8-T4, T4-T6, T6-O2\nFp2-F4, F4-C4, C4-P4, P4-O2\n#### Version 1\n- I crop raw eeg segment by using eeg_label_offset_seconds. And use signal.spectrogram to create spectrogram. I concat 16 img shape (128, 142) to create 4x4 image with shape (512, 568).\n- Normalize by log transform and divide by 2.0.\n- Before feed to vit, I resize image to (518, 518, 1)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2Fd3aa62505f039fa4f5f332e740490bb2%2FScreenshot%20from%202024-04-10%2015-36-02.png?generation=1712738515292787&alt=media)\n#### Version 2\n- Same with version 1, change nperseg from 70 to 39, to create large image with new shape (128,256). 4x4 image have new shape (512, 1024). \n- Before feed to vit, I remain image size (512, 1024, 1)\n#### Version 3\n- I average 4 feature in a chain. Concat 4 chain image to create 4x1 image have shape (512, 256)\n- Before feed to vit, I resize image to (518, 518, 1)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F9aef44118a233e166ee126c3969eea4a%2FScreenshot%20from%202024-04-10%2016-11-42.png?generation=1712740319706335&alt=media)\n- By experience, I have found version 3 is badder than version 1. But when ensemble, kl loss decrease slightly.\n### Create global raw eeg image\n#### Version 1\n- EEG_LENGTH=50\n- I fill NaN by mean value\n- I use butter filter with bandpass [0.5, 40]. New eeg clip by (-1024, 1024). I reshape eeg (4, 10000) to (4, 200, EEG_LENGTH). Next, I concat 4 image (200, EGG_LENGTH) to one image (200, 4xEGG_LENGTH). Finally, I concat 4 chain image to a single image with shape (200, 4x4xEGG_LENGTH)\n- I normalize by divide to 104\n- Before feed to vit, I resize image to (518, 518, 1)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2Fa484cd44bbd98c339a8116dc9ad04d57%2FScreenshot%20from%202024-04-10%2016-20-10.png?generation=1712740826625485&alt=media)\n#### Version 2\n- Same as version 1, EEG_LENGTH=20\n### Create local raw eeg image\n- Same as version 1 of global raw eeg image, EEG_LENGTH=10\n## 2.3 Augmentation\n- I use mixup with alpha=1.0. \n- For kaggle and eeg spectrogram, I use A.XYMasking.\n- For raw eeg data: I use custom augmentation: Random insert NaN to eeg raw data (both global raw and local raw)\n## 2.4 Training Strategy\nI use Adan optimizer with one cycle learning rate scheduler, model ema with decay=0.995, gradient checkpointing. LR=0.00017. Backbone: dinov2 vit family from timm library. For train, per each epoch, I random select an eeg_label_offset_seconds per each eeg_id. For evaluate, I using a first eeg_label_offset_seconds of an eeg_id.\n### Stage 1\nTrain with 5 epoch with data have number vote < 10. \n### Stage 2\nTrain with 7 epoch with data have number vote >= 10. \n# 3. Details of the submission\n## 3.1 Test time augmentation\nI use 3xTTA with 10s raw eeg input: shift-left 2s from center, center, shift-right 2s from center. 3xTTA increase a little Public score < 0.01.\n## 3.2 Details of ensemble\n**Model 1**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F5839a6e1f1eedf549c4997b60a4b25c7%2Fmodel_1.drawio.png?generation=1712727644167695&alt=media)\n**Model 2**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F79176628f79aefb2f52b909939366f7b%2Fmodel_2.drawio.png?generation=1712727661965581&alt=media)\n**Model 3**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F6ce068d0fb6a840e4315612048970460%2Fmodel_3.drawio.png?generation=1712727678141546&alt=media)\n**Model 4**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F5c118134c590d7bf91073d82a2ef66ad%2Fmodel_4.drawio.png?generation=1712727693599176&alt=media)\n**Model 5**\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2832287%2F104d5c9b925dc6f23338ed556ab22935%2Fmodel_5.drawio.png?generation=1712727710736406&alt=media)\n| model |  OOF|OOF kaggle spec|OOF eeg spec|OOF global raw|OOF local raw| Public | Private |\n| --- | --- | --- | --- | --- | --- | --- |\n| model 1 | 0.221329 |0.318149  | 0.267640 |  0.251836| 0.264803 | 0.232427 | 0.289764|\n| model 2 | 0.217614 |0.325917  | 0.266803 |0.240895  |0.264423  |0.229686  |0.287550   |\n| model 3 | 0.217881 |0.317720  | 0.268293 |0.247168 | 0.263158|0.228330  | 0.285105 |\n| model 4 |0.218214  |  0.315174| 0.264225 | 0.250879 | 0.263150 |0.229865  | 0.290503 |\n| model 5 |0.223236  | 0.318931 | 0.334042 | 0.248258 | 0.261953 |0.23025  |0.287976  |\n| ensemble |  |  | |  |  |0.22600  |  0.283302 |\n\nThe weaknesses of my solution is: my ensemble is not diversity. I can't use vit base, vit large for all backbone, because GPU RAM limit.\n## 3.3 Submit strategy\nStrategy 1: Best CV, train and validate on all data.\nStrategy 2: Best LB, train and validate on data with number voter >= 10.\n## 3.4 What didn't work\n- I have try Wavenet, 1DCNN-GRU, squeezeformer, eegnet, eegconformer with raw data, but not success compare with 2D Vision transformer.\n- I have stack 4 chain to create an image with 4 channel, but not success with eeg spectrogram and raw eeg data.\n- I have try normalize eeg spectrogram and raw eeg data by mean, std per channel.\n- I have try butter filter with lowpass, and other range filters.\n- I have try feed original size of raw eeg image to vit, but not success, kl loss is more higher. Finally, I must resize raw eeg image to (518, 518, 1). Large image size improve kl-loss.\n- Data augmentation have little effect, except mixup.\n# Sources\n- Seanbearden, Two-stage training: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477135\n- Chris Deotte, Understanding Competition Data: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/468010\n- Chris Deotte, EEG spectrogram: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467877\n- Inference notebook: https://www.kaggle.com/code/quan0095/hms-best-public-final/\n- Training source: https://github.com/quanvuhust/kaggle_hms",
    "2745825": "Well done, very impressive solo result!",
    "2746893": "Thank you so much for the detailed model share, especially for the git and inference. Congratulations on your achievement :)",
    "2747564": "Thank you for sharing, I have learned a lot of knowledge🥳"
  },
  "source": "meta"
}