![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4495635%2F84e82be8053a16525be76ec7ae1c39c2%2FScreenshot%20from%202024-02-28%2023-51-16.png?generation=1709153551808393&alt=media\")
Greetings everyone,
\nI'm excited to share my Features+Head Starter, which achieves a competitive [LB 0.34]. This notebook is built upon the foundation laid by Chris in his remarkable work. Along side a great insight from @zijiangyang1116 in their post here about two stage training!
\nWith this starter notebook, we can train single models, save them, then submit them for thier individual LBs, then submit them with weighted ensemble. (All models included)
\n
Potential improvements:
\nLet me know if you come across any issues, or have any suggestions.
", "messages": [ { "id": "2612730", "postDate": "01/21/2024 15:27:33", "content": "Greetings everyone,
\nI'm excited to share my Features+Head Starter, which achieves a competitive [LB 0.34]. This notebook is built upon the foundation laid by Chris in his remarkable work. Along side a great insight from @zijiangyang1116 in their post here about two stage training!
\nWith this starter notebook, we can train single models, save them, then submit them for thier individual LBs, then submit them with weighted ensemble. (All models included)
\n
Potential improvements:
\nLet me know if you come across any issues, or have any suggestions.
", "rawMarkdown": "Greetings everyone,\n\nI'm excited to share my [Features+Head Starter](https://www.kaggle.com/code/nartaa/features-head-starter), which achieves a competitive [LB 0.34]. This notebook is built upon the foundation laid by Chris in his remarkable work. Along side a great insight from @zijiangyang1116 in their post [here](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477461) about two stage training!\n\nWith this starter notebook, we can train single models, save them, then submit them for thier individual LBs, then submit them with weighted ensemble. (All models included)\n\n\n\nPotential improvements:\n1. **Highlight the 10 seconds in the middle of the spectograms** since experts pay attention to the 10 seconds in the middle, I could bring that section of the image to the model's attention.\n2. **Use custom loss** I could use a weighted loss that calculates both Binary Cross Entropy and Kullback–Leibler Divergence.\n3. **Data Augmentation** use [MixUp](https://keras.io/examples/vision/mixup/), and Implement various data augmentation techniques, such as flipping, rotating the spectrogram images.\n4. **Weighted voting** Maybe we could have a weighted voting of some sort, so the more votes there is, the more confident we are in those votes and should give them more weight.\nExamples: (votes -> weighted probability)\n1- [2, 2, 0, 0, 0, 0] -> [0.3, 0.3, 0.1, 0.1, 0.1, 0.1]\n2- [10, 10, 0, 0, 0, 0] -> [0.5, 0.5, 0, 0, 0, 0]\nSome math formula is needed.\n5. **[Distributed Learning](https://keras.io/guides/distribution/)** I should implement custom training lopps to support multi-GPU and TPU, because freezing the weights for the backbone model doesn't seem to work with the fit method.\n6. **Stem for backbone** I could build a custom stem and replace part of the original stem in the backbone(reconnect the normalization layers from the original to the new added stem), this way a (IMG_SIZE,IMG_SIZE,8) shape can be passed and reduced to 3 channels with trainable weights.\n7. **Dimensionality Reduction with Autoencoders** I could train an autoencoder to learn the hidden representaion of the 8 channel images to a 3 channel images.\n8. **Sequential Model** Since EGG is a time series signal, I could transfer learn and fine tune a Transformer Encoder for the classification task.\n9. **Machine Learning for Classification:** With the head and feature models separated, we can explore employing machine learning techniques for classification, including random forest, gradient boost models, stacking, and blending.\n10. **[Freezing Backbone Weights](https://keras.io/examples/vision/image_classification_efficientnet_fine_tuning/):** Consider freezing most of the trainable weights of the EfficientNetB2 backbone, while selectively keeping specific layers trainable on the upper layers. This approach allows for prolonged training of the backend, focusing on layers with higher-level features.\n11. **Experimenting with Different Backbones:** Explore training and ensembling other backbone models to further enhance performance.\n12. **Remove Outliers and Noise** further investigation to preprocess the data, looking at the UMAP, it seems outliers exist that can be removed, also missing values (Nans) exist in spectrograms, and possibly EEG.\n13. **Reproduce EEG Spectrograms** I could recalculate EEG spectograms, fill NANs before any operation, remove log transformation and standardization and do that when generating data.\n14. **Input Dropout** Since the input image is a monotone that is repeated on the three channels, I could implement a random dropout in the generator, I shaould try dropout of complete channels, keep one since it has all the information, another approach is partial dropout from each channel.\n\nLet me know if you come across any issues, or have any suggestions.", "votes": null }, { "id": "2612765", "postDate": "01/21/2024 16:01:59", "content": "Would I be correct in my assumption that the only reason for the higher score is using votes instead of consensus? Or is there an additional reason for the improved accuracy?
\nAnyway, it is nice to see a higher score TF notebook. TY.
Hello! Why did you choose the P100 instead of the T4 x2? With these modifications to the code, how many hours were required for training? Chris's EfficientNet training was completed in 1 hour.
", "rawMarkdown": "Hello! Why did you choose the P100 instead of the T4 x2? With these modifications to the code, how many hours were required for training? Chris's EfficientNet training was completed in 1 hour.", "votes": null }, { "id": "2612863", "postDate": "01/21/2024 17:07:58", "content": "I had some issues utilizing the T4x2 which I couldn't resolve quickly, but I will probably try to use it again.
\nWith P100 it took 1.5 hours.
Using the votes reduced LB from 0.52 to 0.49
\nTraining the head separately reduced LB from 0.57 to 0.52
\nYW.
Would I be correct in my assumption that the only reason for the higher score is using votes instead of consensus? Or is there an additional reason for the improved accuracy?
\nAnyway, it is nice to see a higher score TF notebook. TY.
Using the votes reduced LB from 0.52 to 0.49
\nTraining the head separately reduced LB from 0.57 to 0.52
\nYW.
Hello! Why did you choose the P100 instead of the T4 x2? With these modifications to the code, how many hours were required for training? Chris's EfficientNet training was completed in 1 hour.
", "votes": null, "replies": [ { "id": 2612863, "author_name": "nartaa", "author_url": "", "post_date": "01/21/2024 17:07:58", "content": "I had some issues utilizing the T4x2 which I couldn't resolve quickly, but I will probably try to use it again.
\nWith P100 it took 1.5 hours.