{ "id": 211315, "title": "[1st Place] Mel Spectrogram + Blended ResNets 🌋🔥", "url": "/competitions/predict-volcanic-eruptions-ingv-oe/discussion/211315", "author_name": "Jie Feng", "post_date": "2021-01-14T15:48:17.038000", "votes": 16, "comment_count": 8, "views": 0, "content": "

First of all, thanks to INGV and Kaggle for hosting this competition!
\nEven though it isn’t ranked, we had a lot of fun working with this dataset.

\n

1. Spectrograms

\n\n

2. Tabular datasets

\n\n

3. Final Ensemble

\n\n

Thank you for reading! Hope this has been helpful.

", "messages": [ { "id": 1153062, "postDate": "2021-01-14T15:48:17.037Z", "content": "

First of all, thanks to INGV and Kaggle for hosting this competition!
\nEven though it isn’t ranked, we had a lot of fun working with this dataset.

\n

1. Spectrograms

\n\n

2. Tabular datasets

\n\n

3. Final Ensemble

\n\n

Thank you for reading! Hope this has been helpful.

", "rawMarkdown": "\nFirst of all, thanks to INGV and Kaggle for hosting this competition!\nEven though it isn’t ranked, we had a lot of fun working with this dataset.\n \n**1. Spectrograms**\n- We used librosa mel spectrograms to generate 256x256 spectrograms\n- [Notebook Link](https://www.kaggle.com/guimingjiang/ingv-process-data-into-mel-spectrograms-256x256)\n- Lower frequencies are given more space in the spectrogram\n- 256x256 images were fed into CNNs: ResNext, SEResNet, ResNeSt and their outputs were ensembled\n- EfficientNets did not perform as well as ResNets in this data\n- Validation MAE for the well performing ones is less than 1e6\n- ResNet LB scores: 4100000 - 4700000\n - ResNeXt101 : 4121853\n - ResNeSt269e: 4357483\n - SeResNet152: 4737569 \n - Other ResNet models are also in this range\n \n**2. Tabular datasets**\n- Thanks to @carpediemamigo and @josemori for their wonderful notebooks\n- The inference from the tsfresh dataset and the tree model dataset from their notebooks were used in the final ensemble\n- We were not able to make Neural Networks e.g. TabNet/MLP work well on this dataset.\n \n**3. Final Ensemble**\n- Our final model was formed using a **weighted blend** of the following models:\n - ResNeXt101 ~ 50%\n - ResNest269e ~ 20%\n - Other ResNet-type models ~ 20%\n - Tree Models ~ 10%\n- We have found that ensembling sufficient models, even based on the same data processing provides sufficient stability in results (not much significant difference in LB scores by exchanging 1 or 2 models)\n- At the same time, ensembling sufficient models allows for good private LB stability.\n \nThank you for reading! Hope this has been helpful.", "votes": 16 }, { "id": 1175270, "postDate": "2021-01-29T03:54:53.937Z", "content": "

Very cool guys and very well done. How much time did the spectrogram generation take to run on both the train and test data?

", "rawMarkdown": "Very cool guys and very well done. How much time did the spectrogram generation take to run on both the train and test data? ", "votes": 1, "replies": [ { "id": 1176333, "postDate": "2021-01-29T15:27:26.123Z", "content": "

Thanks! As for the spectrogram, the generation took an hour to run.

", "rawMarkdown": "Thanks! As for the spectrogram, the generation took an hour to run.", "votes": 1 } ] }, { "id": 1157671, "postDate": "2021-01-18T04:54:10.633Z", "content": "

great work on the first place. i thought this was an interesting competition and data set and will learn from your solution.

", "rawMarkdown": "great work on the first place. i thought this was an interesting competition and data set and will learn from your solution.", "votes": 1 }, { "id": 1153106, "postDate": "2021-01-14T16:09:47.357Z", "content": "

Thanks INGV and Kaggle for this contest, and huge thanks to my teammate @jiefeng98 on working on this contest together

\n

Here's some other ideas we hoped to explore but didn't have the time/opportunity to:

\n

1. Training Augmentation

\n\n

2. Test-Time Augmentation

\n\n

Our initial experiments didn't really pan out for either but it could be a matter of further hyperparameter optimisation/methodology (e.g. weighted blend of TTA results)

\n

If anyone has managed to make this work, we hope you share your code!

", "rawMarkdown": "Thanks INGV and Kaggle for this contest, and huge thanks to my teammate @jiefeng98 on working on this contest together\n\nHere's some other ideas we hoped to explore but didn't have the time/opportunity to: \n \n**1. Training Augmentation**\n- Since some seismic sensors for some datasets are NA values, we can synthetically expand the dataset by blanking out individual columns (sensors) at test time and effectively get ~10x training data\n \n**2. Test-Time Augmentation**\n- Similar to the above, we can run test-time augmentation by running each train dataset through the model with individual columns blanked out\n\nOur initial experiments didn't really pan out for either but it could be a matter of further hyperparameter optimisation/methodology (e.g. weighted blend of TTA results)\n\nIf anyone has managed to make this work, we hope you share your code!", "votes": 2 }, { "id": 1230701, "postDate": "2021-03-08T11:24:01.303Z", "content": "

Great work!
\nI got the the best result combining channels 4,5,& 6 to one RGB image using mel-spectrograms and Efnet B5.

", "rawMarkdown": "Great work!\nI got the the best result combining channels 4,5,& 6 to one RGB image using mel-spectrograms and Efnet B5. " }, { "id": 1228095, "postDate": "2021-03-06T05:33:46.107Z", "content": "

Thanks for the solution! Could you share the hyperparameters of the ResNet training if you don't mind, I am trying to reproduce your result but I can't get such low validation MAE like you.

", "rawMarkdown": "Thanks for the solution! Could you share the hyperparameters of the ResNet training if you don't mind, I am trying to reproduce your result but I can't get such low validation MAE like you." }, { "id": 1181378, "postDate": "2021-02-01T20:50:38.490Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 1181818, "postDate": "2021-02-02T06:15:33.270Z", "content": "

Thanks!

\n

You're right - the main technique here is processing the initial time-series tabular data (the sensor values) into a spectrogram. Afterwards, we can treat this as a regression problem using image data as an input, and then we can fine-tune strong pre-trained ResNet models.

\n

In this case, we convert the 10 column table (10 sensors) into a 10-channel image, where each channel represents the spectrogram of that sensor. During inference, we will convert the input table into 10 spectrograms. Most ResNets are trained on 3-channel images, so small modifications are needed to the model.

", "rawMarkdown": "Thanks! \n\nYou're right - the main technique here is processing the initial time-series tabular data (the sensor values) into a spectrogram. Afterwards, we can treat this as a regression problem using image data as an input, and then we can fine-tune strong pre-trained ResNet models.\n\nIn this case, we convert the 10 column table (10 sensors) into a 10-channel image, where each channel represents the spectrogram of that sensor. During inference, we will convert the input table into 10 spectrograms. Most ResNets are trained on 3-channel images, so small modifications are needed to the model.\n", "votes": 1 } ] } ], "comments": [ { "id": 1175270, "author_name": "Matt S.", "author_url": "", "post_date": "2021-01-29T03:54:53.937000", "content": "

Very cool guys and very well done. How much time did the spectrogram generation take to run on both the train and test data?

", "votes": 1, "replies": [ { "id": 1176333, "author_name": "Jie Feng", "author_url": "", "post_date": "2021-01-29T15:27:26.123000", "content": "

Thanks! As for the spectrogram, the generation took an hour to run.

", "votes": 1, "replies": [] } ] }, { "id": 1157671, "author_name": "Dave E", "author_url": "", "post_date": "2021-01-18T04:54:10.633000", "content": "

great work on the first place. i thought this was an interesting competition and data set and will learn from your solution.

", "votes": 1, "replies": [] }, { "id": 1153106, "author_name": "Gui Ming Jiang", "author_url": "", "post_date": "2021-01-14T16:09:47.357000", "content": "

Thanks INGV and Kaggle for this contest, and huge thanks to my teammate @jiefeng98 on working on this contest together

\n

Here's some other ideas we hoped to explore but didn't have the time/opportunity to:

\n

1. Training Augmentation

\n\n

2. Test-Time Augmentation

\n\n

Our initial experiments didn't really pan out for either but it could be a matter of further hyperparameter optimisation/methodology (e.g. weighted blend of TTA results)

\n

If anyone has managed to make this work, we hope you share your code!

", "votes": 2, "replies": [] }, { "id": 1230701, "author_name": "Michael Markzon", "author_url": "", "post_date": "2021-03-08T11:24:01.303000", "content": "

Great work!
\nI got the the best result combining channels 4,5,& 6 to one RGB image using mel-spectrograms and Efnet B5.

", "votes": 0, "replies": [] }, { "id": 1228095, "author_name": "jojo", "author_url": "", "post_date": "2021-03-06T05:33:46.107000", "content": "

Thanks for the solution! Could you share the hyperparameters of the ResNet training if you don't mind, I am trying to reproduce your result but I can't get such low validation MAE like you.

", "votes": 0, "replies": [] }, { "id": 1181378, "author_name": "", "author_url": "", "post_date": "2021-02-01T20:50:38.490000", "content": "", "votes": 0, "replies": [ { "id": 1181818, "author_name": "Gui Ming Jiang", "author_url": "", "post_date": "2021-02-02T06:15:33.270000", "content": "

Thanks!

\n

You're right - the main technique here is processing the initial time-series tabular data (the sensor values) into a spectrogram. Afterwards, we can treat this as a regression problem using image data as an input, and then we can fine-tune strong pre-trained ResNet models.

\n

In this case, we convert the 10 column table (10 sensors) into a 10-channel image, where each channel represents the spectrogram of that sensor. During inference, we will convert the input table into 10 spectrograms. Most ResNets are trained on 3-channel images, so small modifications are needed to the model.

", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "1153062": "\nFirst of all, thanks to INGV and Kaggle for hosting this competition!\nEven though it isn’t ranked, we had a lot of fun working with this dataset.\n \n**1. Spectrograms**\n- We used librosa mel spectrograms to generate 256x256 spectrograms\n- [Notebook Link](https://www.kaggle.com/guimingjiang/ingv-process-data-into-mel-spectrograms-256x256)\n- Lower frequencies are given more space in the spectrogram\n- 256x256 images were fed into CNNs: ResNext, SEResNet, ResNeSt and their outputs were ensembled\n- EfficientNets did not perform as well as ResNets in this data\n- Validation MAE for the well performing ones is less than 1e6\n- ResNet LB scores: 4100000 - 4700000\n - ResNeXt101 : 4121853\n - ResNeSt269e: 4357483\n - SeResNet152: 4737569 \n - Other ResNet models are also in this range\n \n**2. Tabular datasets**\n- Thanks to @carpediemamigo and @josemori for their wonderful notebooks\n- The inference from the tsfresh dataset and the tree model dataset from their notebooks were used in the final ensemble\n- We were not able to make Neural Networks e.g. TabNet/MLP work well on this dataset.\n \n**3. Final Ensemble**\n- Our final model was formed using a **weighted blend** of the following models:\n - ResNeXt101 ~ 50%\n - ResNest269e ~ 20%\n - Other ResNet-type models ~ 20%\n - Tree Models ~ 10%\n- We have found that ensembling sufficient models, even based on the same data processing provides sufficient stability in results (not much significant difference in LB scores by exchanging 1 or 2 models)\n- At the same time, ensembling sufficient models allows for good private LB stability.\n \nThank you for reading! Hope this has been helpful.", "1175270": "Very cool guys and very well done. How much time did the spectrogram generation take to run on both the train and test data? ", "1157671": "great work on the first place. i thought this was an interesting competition and data set and will learn from your solution.", "1153106": "Thanks INGV and Kaggle for this contest, and huge thanks to my teammate @jiefeng98 on working on this contest together\n\nHere's some other ideas we hoped to explore but didn't have the time/opportunity to: \n \n**1. Training Augmentation**\n- Since some seismic sensors for some datasets are NA values, we can synthetically expand the dataset by blanking out individual columns (sensors) at test time and effectively get ~10x training data\n \n**2. Test-Time Augmentation**\n- Similar to the above, we can run test-time augmentation by running each train dataset through the model with individual columns blanked out\n\nOur initial experiments didn't really pan out for either but it could be a matter of further hyperparameter optimisation/methodology (e.g. weighted blend of TTA results)\n\nIf anyone has managed to make this work, we hope you share your code!", "1230701": "Great work!\nI got the the best result combining channels 4,5,& 6 to one RGB image using mel-spectrograms and Efnet B5. ", "1228095": "Thanks for the solution! Could you share the hyperparameters of the ResNet training if you don't mind, I am trying to reproduce your result but I can't get such low validation MAE like you.", "1181378": "" } }