![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4712534%2Fc4e39641fa8ec13588b62065ae1e9b58%2FScreenshot%202024-02-10%20at%2010.40.36%20PM.png?generation=1707585942607444&alt=media\")
Hi again, sharing another baseline for everyone to play around with. I am able to get good results within couple of experiments.
\nIn continuation to my previous 1D ResNet based [approach](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/471666), this pipeline basically introduces 2D encoder (effcientnet-b0).
\nThe idea behind the approach is to utilise both Raw Signals as well as Kaggle/EEG signals based Spectrograms within a single architecture & train the entire network together. Also, I do not claim this solution to be superior than methods like separately finetuning 1D & 2D models and then ensemble, but this definitely does have potential to reach much higher scores.
\nTraining notebook: link
\nSubmission notebook: link
Basically, now we send two inputs within the architecture: Raw signals w/ Feature engineering from @cdeotte + concatenated Kaggle+EEG spectrograms , torch version shared here by @alejopaullier
\nEach input is separately processed by the 1D & 2D CNN based Encoders, followed by fully connected layer to make equi-dimensional features. These features are then normalized & used to calculate contrastive loss. Since we do not have any pariwise labels, we keep all the labels as 1 while calculating loss, this will basically encourage the hidden respresentahions from 1D and 2D models to more closer to one another.
\n`
\n = torch.nn.functional.normalize(embedding_1d, p=, dim=)\n = torch.nn.functional.normalize(embedding_2d, p=, dim=)\n\n\n = torch.es(embedding_1d.size()).to(self.device) \n = self.contrastive_loss(embedding_1d, embedding_2d, contrastive_target)\n\n = classification_loss + classification_loss1* + classification_loss2* + contrastive_loss* \nApart from this, we are also using KL Divergence Loss on multiple outputs generated from linear heads: concatenated features from 1D + 2D encoders, 1D CNN features, 2D CNN features. I just tried this to get higher weightage to task based loss function than contrastive loss & this worked better than just having single linear head for making predictions.
\nThis is the weightage I am giving to calculate the final predictions.
`
\n = outputs*. + y1*. + y2*.\n\n = nn.Softmax(dim=)(outputs)`\nThanks. Feel free to do more experiments to improve the results or share new ideas related to multimodal approaches in the thread.
", "messages": [ { "id": "2646248", "postDate": "02/10/2024 18:34:55", "content": "Hi again, sharing another baseline for everyone to play around with. I am able to get good results within couple of experiments.
\nIn continuation to my previous 1D ResNet based [approach](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/471666), this pipeline basically introduces 2D encoder (effcientnet-b0).
\nThe idea behind the approach is to utilise both Raw Signals as well as Kaggle/EEG signals based Spectrograms within a single architecture & train the entire network together. Also, I do not claim this solution to be superior than methods like separately finetuning 1D & 2D models and then ensemble, but this definitely does have potential to reach much higher scores.
\nTraining notebook: link
\nSubmission notebook: link
Basically, now we send two inputs within the architecture: Raw signals w/ Feature engineering from @cdeotte + concatenated Kaggle+EEG spectrograms , torch version shared here by @alejopaullier
\nEach input is separately processed by the 1D & 2D CNN based Encoders, followed by fully connected layer to make equi-dimensional features. These features are then normalized & used to calculate contrastive loss. Since we do not have any pariwise labels, we keep all the labels as 1 while calculating loss, this will basically encourage the hidden respresentahions from 1D and 2D models to more closer to one another.
\n`
\n = torch.nn.functional.normalize(embedding_1d, p=, dim=)\n = torch.nn.functional.normalize(embedding_2d, p=, dim=)\n\n\n = torch.es(embedding_1d.size()).to(self.device) \n = self.contrastive_loss(embedding_1d, embedding_2d, contrastive_target)\n\n = classification_loss + classification_loss1* + classification_loss2* + contrastive_loss* \nApart from this, we are also using KL Divergence Loss on multiple outputs generated from linear heads: concatenated features from 1D + 2D encoders, 1D CNN features, 2D CNN features. I just tried this to get higher weightage to task based loss function than contrastive loss & this worked better than just having single linear head for making predictions.
\nThis is the weightage I am giving to calculate the final predictions.
`
\n = outputs*. + y1*. + y2*.\n\n = nn.Softmax(dim=)(outputs)`\nThanks. Feel free to do more experiments to improve the results or share new ideas related to multimodal approaches in the thread.
", "rawMarkdown": "Hi again, sharing another baseline for everyone to play around with. I am able to get good results within couple of experiments. \n\nIn continuation to my previous **1D ResNet based [approach]**(https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/471666), this pipeline basically introduces 2D encoder (**effcientnet-b0**). \n \nThe idea behind the approach is to utilise both Raw Signals as well as Kaggle/EEG signals based Spectrograms within a single architecture & train the entire network together. Also, I do not claim this solution to be superior than methods like separately finetuning 1D & 2D models and then ensemble, but this definitely does have potential to reach much higher scores.\n\n**Training notebook:** [link](https://www.kaggle.com/nischaydnk/training-multimodal-1d-2d-approach-eegs) \n**Submission notebook:** [link](https://www.kaggle.com/nischaydnk/multimodal-1d-2d-eeg-approach-submission) \n\n\n## Architecture:\n\n\n\nBasically, now we send two inputs within the architecture: Raw signals w/ Feature engineering from @cdeotte + *concatenated Kaggle+EEG spectrograms* , torch version shared [here](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/472092) by @alejopaullier \n\nEach input is separately processed by the 1D & 2D CNN based Encoders, followed by fully connected layer to make equi-dimensional features. These features are then normalized & used to calculate contrastive loss. Since we do not have any pariwise labels, we keep all the labels as 1 while calculating loss, this will basically encourage the hidden respresentahions from 1D and 2D models to more closer to one another. \n\n` \n \n embedding_1d = torch.nn.functional.normalize(embedding_1d, p=2, dim=1)\n embedding_2d = torch.nn.functional.normalize(embedding_2d, p=2, dim=1)\n\n \n contrastive_target = torch.ones(embedding_1d.size(0)).to(self.device) # Assuming all pairs are similar\n contrastive_loss = self.contrastive_loss(embedding_1d, embedding_2d, contrastive_target)\n\n total_loss = classification_loss + classification_loss1*0.5 + classification_loss2*0.5 + contrastive_loss*0.5 # Aux losses`\n\nApart from this, we are also using KL Divergence Loss on multiple outputs generated from linear heads: concatenated features from 1D + 2D encoders, 1D CNN features, 2D CNN features. I just tried this to get higher weightage to task based loss function than contrastive loss & this worked better than just having single linear head for making predictions. \nThis is the weightage I am giving to calculate the final predictions.\n\n` \n\n outputs = outputs*0.5 + y1*0.25 + y2*0.25\n \n outputs = nn.Softmax(dim=1)(outputs)`\n\n\n#### **Few more additions in training strategy: **\n- Horizontal Flip Augmentation for Spectrograms\n- 8 channel Feature Engineering shared by Chris\n- Auxiliary Loss\n *total_loss = classification_loss + classification_loss1*0.5 + classification_loss2*0.5 + contrastive_loss*0.5*\n- Multiple outputs\n\nThanks. Feel free to do more experiments to improve the results or share new ideas related to multimodal approaches in the thread.", "votes": null }, { "id": "2648075", "postDate": "02/12/2024 04:36:13", "content": "Maybe this is a dumb question but I wonder if instead of trying to minimise contrastive loss here, one could get interesting results by trying to maximise contrastive loss, i.e., maybe it would help increase diversity of the information processed.
", "rawMarkdown": "Maybe this is a dumb question but I wonder if instead of trying to minimise contrastive loss here, one could get interesting results by trying to maximise contrastive loss, i.e., maybe it would help increase diversity of the information processed.", "votes": null }, { "id": "2649260", "postDate": "02/12/2024 17:41:10", "content": "Very nice!
", "rawMarkdown": "Very nice!", "votes": null }, { "id": "2649527", "postDate": "02/12/2024 22:52:41", "content": "Hey @nischaydnk very nice! How do you choose the weights for the auxiliary loss?
", "rawMarkdown": "Hey @nischaydnk very nice! How do you choose the weights for the auxiliary loss?", "votes": null }, { "id": "2650386", "postDate": "02/13/2024 12:35:26", "content": "It could be an option, if we penalize the loss function and use the contrastive loss as a regularization term to increase diversity, that might help with overfitting.
", "rawMarkdown": "It could be an option, if we penalize the loss function and use the contrastive loss as a regularization term to increase diversity, that might help with overfitting.", "votes": null }, { "id": "2651200", "postDate": "02/14/2024 02:59:56", "content": "Was anyone able to run the training notebook on Kaggle? I'm currently facing two issues:
\nNice work, I even have similar ideas in my scientific work!
", "rawMarkdown": "Nice work, I even have similar ideas in my scientific work!", "votes": null }, { "id": "2662543", "postDate": "02/22/2024 01:04:52", "content": "hi Have you tried this method and has the effect increased
", "rawMarkdown": "hi Have you tried this method and has the effect increased", "votes": null }, { "id": "2667259", "postDate": "02/25/2024 02:51:32", "content": "Hehe not yet. Been working on other stuff. Would like to see if anyone else has tried though.
", "rawMarkdown": "Hehe not yet. Been working on other stuff. Would like to see if anyone else has tried though.", "votes": null }, { "id": "3058956", "postDate": "11/30/2024 07:05:34", "content": "Very cool, thanks!
", "rawMarkdown": "Very cool, thanks!", "votes": null } ], "comments": [ { "id": 2648075, "author_name": "caelhasse", "author_url": "", "post_date": "02/12/2024 04:36:13", "content": "Maybe this is a dumb question but I wonder if instead of trying to minimise contrastive loss here, one could get interesting results by trying to maximise contrastive loss, i.e., maybe it would help increase diversity of the information processed.
", "votes": null, "replies": [ { "id": 2650386, "author_name": "nartaa", "author_url": "", "post_date": "02/13/2024 12:35:26", "content": "It could be an option, if we penalize the loss function and use the contrastive loss as a regularization term to increase diversity, that might help with overfitting.
", "votes": null, "replies": [] }, { "id": 2662543, "author_name": "chenboluo", "author_url": "", "post_date": "02/22/2024 01:04:52", "content": "hi Have you tried this method and has the effect increased
", "votes": null, "replies": [ { "id": 2667259, "author_name": "caelhasse", "author_url": "", "post_date": "02/25/2024 02:51:32", "content": "Hehe not yet. Been working on other stuff. Would like to see if anyone else has tried though.
", "votes": null, "replies": [] } ] } ] }, { "id": 2649260, "author_name": "", "author_url": "", "post_date": "02/12/2024 17:41:10", "content": "Very nice!
", "votes": null, "replies": [] }, { "id": 2649527, "author_name": "rdizzl3", "author_url": "", "post_date": "02/12/2024 22:52:41", "content": "Hey @nischaydnk very nice! How do you choose the weights for the auxiliary loss?
", "votes": null, "replies": [] }, { "id": 2651200, "author_name": "yantxx", "author_url": "", "post_date": "02/14/2024 02:59:56", "content": "Was anyone able to run the training notebook on Kaggle? I'm currently facing two issues:
\nNice work, I even have similar ideas in my scientific work!
", "votes": null, "replies": [] }, { "id": 3058956, "author_name": "guneevarora", "author_url": "", "post_date": "11/30/2024 07:05:34", "content": "Very cool, thanks!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2646248": "Hi again, sharing another baseline for everyone to play around with. I am able to get good results within couple of experiments. \n\nIn continuation to my previous **1D ResNet based [approach]**(https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/471666), this pipeline basically introduces 2D encoder (**effcientnet-b0**). \n \nThe idea behind the approach is to utilise both Raw Signals as well as Kaggle/EEG signals based Spectrograms within a single architecture & train the entire network together. Also, I do not claim this solution to be superior than methods like separately finetuning 1D & 2D models and then ensemble, but this definitely does have potential to reach much higher scores.\n\n**Training notebook:** [link](https://www.kaggle.com/nischaydnk/training-multimodal-1d-2d-approach-eegs) \n**Submission notebook:** [link](https://www.kaggle.com/nischaydnk/multimodal-1d-2d-eeg-approach-submission) \n\n\n## Architecture:\n\n\n\nBasically, now we send two inputs within the architecture: Raw signals w/ Feature engineering from @cdeotte + *concatenated Kaggle+EEG spectrograms* , torch version shared [here](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/472092) by @alejopaullier \n\nEach input is separately processed by the 1D & 2D CNN based Encoders, followed by fully connected layer to make equi-dimensional features. These features are then normalized & used to calculate contrastive loss. Since we do not have any pariwise labels, we keep all the labels as 1 while calculating loss, this will basically encourage the hidden respresentahions from 1D and 2D models to more closer to one another. \n\n` \n \n embedding_1d = torch.nn.functional.normalize(embedding_1d, p=2, dim=1)\n embedding_2d = torch.nn.functional.normalize(embedding_2d, p=2, dim=1)\n\n \n contrastive_target = torch.ones(embedding_1d.size(0)).to(self.device) # Assuming all pairs are similar\n contrastive_loss = self.contrastive_loss(embedding_1d, embedding_2d, contrastive_target)\n\n total_loss = classification_loss + classification_loss1*0.5 + classification_loss2*0.5 + contrastive_loss*0.5 # Aux losses`\n\nApart from this, we are also using KL Divergence Loss on multiple outputs generated from linear heads: concatenated features from 1D + 2D encoders, 1D CNN features, 2D CNN features. I just tried this to get higher weightage to task based loss function than contrastive loss & this worked better than just having single linear head for making predictions. \nThis is the weightage I am giving to calculate the final predictions.\n\n` \n\n outputs = outputs*0.5 + y1*0.25 + y2*0.25\n \n outputs = nn.Softmax(dim=1)(outputs)`\n\n\n#### **Few more additions in training strategy: **\n- Horizontal Flip Augmentation for Spectrograms\n- 8 channel Feature Engineering shared by Chris\n- Auxiliary Loss\n *total_loss = classification_loss + classification_loss1*0.5 + classification_loss2*0.5 + contrastive_loss*0.5*\n- Multiple outputs\n\nThanks. Feel free to do more experiments to improve the results or share new ideas related to multimodal approaches in the thread.", "2648075": "Maybe this is a dumb question but I wonder if instead of trying to minimise contrastive loss here, one could get interesting results by trying to maximise contrastive loss, i.e., maybe it would help increase diversity of the information processed.", "2649260": "Very nice!", "2649527": "Hey @nischaydnk very nice! How do you choose the weights for the auxiliary loss?", "2650386": "It could be an option, if we penalize the loss function and use the contrastive loss as a regularization term to increase diversity, that might help with overfitting.", "2651200": "Was anyone able to run the training notebook on Kaggle? I'm currently facing two issues:\n- When I update local paths to what I believe is the correct Kaggle path, they just don't work.\n- If the brain-eeg-spectrograms dataset is loaded, the notebook runs out of memory right at the beginning. Since it seems that this dataset is necessary for the training, I find myself unable to continue.", "2652001": "Nice work, I even have similar ideas in my scientific work!", "2662543": "hi Have you tried this method and has the effect increased", "2667259": "Hehe not yet. Been working on other stuff. Would like to see if anyone else has tried though.", "3058956": "Very cool, thanks!" }, "source": "meta" }