I share two of my single \"possibly\" silver models.
\n\nTo my disappointment, I don't select these model as final model, \nso I missed the silver medal because I underestimated the local scores.
\nHowever, these models contributed to my final ensemble ones and brought me the bronze medal.
\nThese 2 models are inspired by TensorFlow Speech Recognition Challenge I joined in past.
bigru + attention + mix-up + hard voting (private LB 0.668 (same as 38th) / public LB 0.568)\nIt's similar model with Bruno Aquino's kenel, but 2 main different points.
\n\nmath\n\\tilde{x} = λx_i +(1−λ)x_j\n\\tilde{y} = λy_i +(1−λ)y_j\n\nIf \\tilde{y} >= 0.5 \\, I labeled the new sample as class 1. In addition, I implement it with PyTorch instead of Keras (I don't know what effect that brings the model performance. )
\n
SE-ResNet 50 + window (chunk) 40 step 20 summary feature + hard voting (private LB 0.663 (same as 53th) / public LB 0.587)\nThis model composed of 2 specific features.
\n\n2D-1D SE-Resnet 50\nI learned cnn works well for signal classification tasks in the above speech recognition \ncompetition. That drives me to adopt cnn models for the power line signals.
\nInstead of simple cnn models, I used SE-ResNet 50 expecting it to find more deep features.
\n(I also tried more simple CNN models with attention, but they are inferior to SE-ResNet in both public/private LB. )
To support signal inputs, I changed 3 points from original 2D SE-ResNet for image.
\n\nThese steps are inspired by Deep Learning With Convolutional Neural Networks for EEG Decoding and Visualization by Schirrmeister etc. and DeepSense: A Unified Deep Learning Framework for Time-Series Mobile Sensing Data Processing by Yao etc.\narXiv:1611.01942v2.
\n\nThe SE-ResNet also seems over-fitting to privte LB and it leaves improvements for the model parameters.
In conclusion, the both of them seem to include some hints but requires more improvements \nto obtain the high scores in both private and public LB.\nI'll share my codes in near feature.
", "messages": [ { "id": 496534, "postDate": "2019-03-22T09:59:15.873Z", "content": "I share two of my single \"possibly\" silver models.
\n\nTo my disappointment, I don't select these model as final model, \nso I missed the silver medal because I underestimated the local scores.
\nHowever, these models contributed to my final ensemble ones and brought me the bronze medal.
\nThese 2 models are inspired by TensorFlow Speech Recognition Challenge I joined in past.
bigru + attention + mix-up + hard voting (private LB 0.668 (same as 38th) / public LB 0.568)\nIt's similar model with Bruno Aquino's kenel, but 2 main different points.
\n\nmath\n\\tilde{x} = λx_i +(1−λ)x_j\n\\tilde{y} = λy_i +(1−λ)y_j\n\nIf \\tilde{y} >= 0.5 \\, I labeled the new sample as class 1. In addition, I implement it with PyTorch instead of Keras (I don't know what effect that brings the model performance. )
\n
SE-ResNet 50 + window (chunk) 40 step 20 summary feature + hard voting (private LB 0.663 (same as 53th) / public LB 0.587)\nThis model composed of 2 specific features.
\n\n2D-1D SE-Resnet 50\nI learned cnn works well for signal classification tasks in the above speech recognition \ncompetition. That drives me to adopt cnn models for the power line signals.
\nInstead of simple cnn models, I used SE-ResNet 50 expecting it to find more deep features.
\n(I also tried more simple CNN models with attention, but they are inferior to SE-ResNet in both public/private LB. )
To support signal inputs, I changed 3 points from original 2D SE-ResNet for image.
\n\nThese steps are inspired by Deep Learning With Convolutional Neural Networks for EEG Decoding and Visualization by Schirrmeister etc. and DeepSense: A Unified Deep Learning Framework for Time-Series Mobile Sensing Data Processing by Yao etc.\narXiv:1611.01942v2.
\n\nThe SE-ResNet also seems over-fitting to privte LB and it leaves improvements for the model parameters.
In conclusion, the both of them seem to include some hints but requires more improvements \nto obtain the high scores in both private and public LB.\nI'll share my codes in near feature.
", "rawMarkdown": "I share two of my single \"possibly\" silver models.\n\nTo my disappointment, I don't select these model as final model, \nso I missed the silver medal because I underestimated the local scores. \nHowever, these models contributed to my final ensemble ones and brought me the bronze medal. \nThese 2 models are inspired by [TensorFlow Speech Recognition Challenge](https://www.kaggle.com/c/tensorflow-speech-recognition-challenge) I joined in past. \n\n\n1. **bigru + attention + mix-up + hard voting (private LB 0.668 (same as 38th) / public LB 0.568)**\n It's similar model with [Bruno Aquino's kenel](https://www.kaggle.com/braquino/5-fold-lstm-attention-fully-commented-0-694), but 2 main different points. \n \n - *hard voting of each cv model's classification (threshold=0.5)*\n Instead of tuning the threshold with the averaged probabilities of those by cv models, \n the classes of the samples are decided by majority of the cv models with the fixed threshold 0.5. I guess this hard voting bring the model robustness. \n - *mix-up*\n This augmentation method is introduced by the paper [mixup: Beyond Empirical Risk Minimization](https://arxiv.org/abs/1710.09412). \n The augmentation generates a new sample from 2 training samples based of the adding. \n ```math\n \\tilde{x} = λx_i +(1−λ)x_j\n \\tilde{y} = λy_i +(1−λ)y_j\n ```\n If \\\\tilde{y} >= 0.5 \\\\, I labeled the new sample as class 1. \n I used 3 as the mix-up ratio, but I don't have enough time to tune the ratio. \n Please see the blog post [the detail](http://www.inference.vc/mixup-data-dependent-data- \n augmentation/) about mix-up. \n The mix-up improved private LB to 0.668 from 0.599 but worsened public LB to 0.568 from \n 0.632. Therefore, it requires more careful evaluation to know whether mix-up really works. \n \n In addition, I implement it with PyTorch instead of Keras (I don't know what effect that brings the model performance. ) \n