![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F936560%2Fe204c8ee2954d4efa119c58b27b303f0%2FResume_Models.png?generation=1610035276254338&alt=media\")
First of all, I would like to thank to Kaggle and INGV (Istituto nazionale di geofisica e vulcanologia) National Institute of Geophysics and Volcanology for organizing this competition. I have learned a lot and really enjoyed the competition.
\nI attach the github link below in case you want to check our implementations:
\nhttps://github.com/EnricRovira/INGV-VolcanicEruptionPrediction
Finally i want to thank my teammate @ajcostarino, who provided nice ideas and efforts.
\nWe builded different base models. They are resumed on the table below. The best one scored 4.9 on public LB.
\n
The multi head attention model is builded on top of a wavenet head in order to reduce the size of the sequence, and later on whe build 5 heads of 300 depth each.
\nFor the MHA model we just perform Standar Scaling (Substract mean and divide by std).
\nFor spectograms models we builded different kind of spectograms (mel, stft, scipy spectogram) and feeded them to residual CNNs and worked really find.
\nIn order to regularize the NN we added a composite [weigh=4(MAE), weight=1(quantileLoss)]
\nWe decay the learning rate every 10 epochs by a 0.9 factor and added early stopping for a total of 100 epochs. Depending on the model the training times differ, the stft one took 12h to train but the tabular ones only took few minutes.
\nWe performed Stratified Cross Validation with 5 CV and save for each fold the best validation model.
\nLater we predicted with every fold and average the results.
We tried different techniques but what it worked better was Blending/Stacking with time test augmentations depending on the architecture. For CNN we add image augmentations and for raw signal ones we augmentated the signals (random noise, time shifting, sensor dropour and so on).
\nFinally we build XGB, CATBoost and Tabnet models on top of this stacking predictions and average them to perform our final submission which scored 4.421.208 on private leaderboard
\nGranular fields images
\nClustering
\nTime Series valdiation
As some competitios released on their solutions we should have reblended the last step and added more tabular data.
\nI hope the explanation was understandable and helpful.
", "messages": [ { "id": "1142842", "postDate": "01/07/2021 16:16:20", "content": "First of all, I would like to thank to Kaggle and INGV (Istituto nazionale di geofisica e vulcanologia) National Institute of Geophysics and Volcanology for organizing this competition. I have learned a lot and really enjoyed the competition.
\nI attach the github link below in case you want to check our implementations:
\nhttps://github.com/EnricRovira/INGV-VolcanicEruptionPrediction
Finally i want to thank my teammate @ajcostarino, who provided nice ideas and efforts.
\nWe builded different base models. They are resumed on the table below. The best one scored 4.9 on public LB.
\n
The multi head attention model is builded on top of a wavenet head in order to reduce the size of the sequence, and later on whe build 5 heads of 300 depth each.
\nFor the MHA model we just perform Standar Scaling (Substract mean and divide by std).
\nFor spectograms models we builded different kind of spectograms (mel, stft, scipy spectogram) and feeded them to residual CNNs and worked really find.
\nIn order to regularize the NN we added a composite [weigh=4(MAE), weight=1(quantileLoss)]
\nWe decay the learning rate every 10 epochs by a 0.9 factor and added early stopping for a total of 100 epochs. Depending on the model the training times differ, the stft one took 12h to train but the tabular ones only took few minutes.
\nWe performed Stratified Cross Validation with 5 CV and save for each fold the best validation model.
\nLater we predicted with every fold and average the results.
We tried different techniques but what it worked better was Blending/Stacking with time test augmentations depending on the architecture. For CNN we add image augmentations and for raw signal ones we augmentated the signals (random noise, time shifting, sensor dropour and so on).
\nFinally we build XGB, CATBoost and Tabnet models on top of this stacking predictions and average them to perform our final submission which scored 4.421.208 on private leaderboard
\nGranular fields images
\nClustering
\nTime Series valdiation
As some competitios released on their solutions we should have reblended the last step and added more tabular data.
\nI hope the explanation was understandable and helpful.
", "rawMarkdown": "First of all, I would like to thank to Kaggle and INGV (Istituto nazionale di geofisica e vulcanologia) National Institute of Geophysics and Volcanology for organizing this competition. I have learned a lot and really enjoyed the competition.\n\nI attach the github link below in case you want to check our implementations:\n[https://github.com/EnricRovira/INGV-VolcanicEruptionPrediction](url)\n\nFinally i want to thank my teammate @ajcostarino, who provided nice ideas and efforts.\n\n\n# 1. Base Models\n\nWe builded different base models. They are resumed on the table below. The best one scored 4.9 on public LB.\n\n\n\n\nThe multi head attention model is builded on top of a wavenet head in order to reduce the size of the sequence, and later on whe build 5 heads of 300 depth each.\n\n\n# 2. Preprocessing\n\nFor the MHA model we just perform Standar Scaling (Substract mean and divide by std). \n\nFor spectograms models we builded different kind of spectograms (mel, stft, scipy spectogram) and feeded them to residual CNNs and worked really find.\n\n### 2.1 Models training\n\nIn order to regularize the NN we added a composite [weigh=4(MAE), weight=1(quantileLoss)]\n\nWe decay the learning rate every 10 epochs by a 0.9 factor and added early stopping for a total of 100 epochs. Depending on the model the training times differ, the stft one took 12h to train but the tabular ones only took few minutes.\n\n\n# 3 Validation\n\nWe performed Stratified Cross Validation with 5 CV and save for each fold the best validation model.\nLater we predicted with every fold and average the results.\n\n\n# 4. Stacking\n\nWe tried different techniques but what it worked better was Blending/Stacking with time test augmentations depending on the architecture. For CNN we add image augmentations and for raw signal ones we augmentated the signals (random noise, time shifting, sensor dropour and so on).\n\nFinally we build XGB, CATBoost and Tabnet models on top of this stacking predictions and average them to perform our final submission which scored 4.421.208 on private leaderboard\n\n\n# 5. Things we tried that didnt work\n\nGranular fields images\nClustering \nTime Series valdiation\n\n\n# 6. Improvements\n\nAs some competitios released on their solutions we should have reblended the last step and added more tabular data.\n\nI hope the explanation was understandable and helpful.", "votes": null }, { "id": "1143373", "postDate": "01/07/2021 21:38:33", "content": "Hello,
\nCongrats on the great results and thanks for sharing your code and solution!
\nI also experimented with CNN and STFFT (feel free to check my notebook about it)). Could you please elaborate on your decisions for your CNN structure ?
\nUnfortunately I did not have access to a GPU so I sticked to a 1D version to be able to train it. On which system did you train your model ?
\nAlso, did you notice any difference in the results when using the different spectograms?
", "rawMarkdown": "Hello, \n\nCongrats on the great results and thanks for sharing your code and solution!\n\nI also experimented with CNN and STFFT (feel free to check my [notebook](https://www.kaggle.com/adriencossa/stfft-and-1d-cnn) about it)). Could you please elaborate on your decisions for your CNN structure ? \n\nUnfortunately I did not have access to a GPU so I sticked to a 1D version to be able to train it. On which system did you train your model ?\n\nAlso, did you notice any difference in the results when using the different spectograms?", "votes": null }, { "id": "1144635", "postDate": "01/08/2021 15:27:03", "content": "Great Model new to data science all of this is a great help
", "rawMarkdown": "Great Model new to data science all of this is a great help", "votes": null }, { "id": "1148084", "postDate": "01/10/2021 21:57:55", "content": "You can check the github repo for any further details, if you still have more doubts dont hesitate to reply it
", "rawMarkdown": "You can check the github repo for any further details, if you still have more doubts dont hesitate to reply it", "votes": null }, { "id": "1155348", "postDate": "01/16/2021 11:15:18", "content": "Good one, Appreciated. Upvoted
", "rawMarkdown": "Good one, Appreciated. Upvoted", "votes": null }, { "id": "1309235", "postDate": "05/15/2021 19:01:55", "content": "Hello!
\nCongrats for your place in this competition and thank you for sharing your solution.
I'm trying to understand your solution and I've started with the spectrograms and the ResNet model (model_big). I've accomplished the spectrograms dataset construction, but here I don't really get it. I have an error: \"Requested crop size (118, 225) is larger than the image size (40, 118)\".
\nI understand that this model is separated in two files (03_BuildSpectograms_v0.2 and 03_SpectogramModel_Big_v0.5). Is there one more file to complete this model or that's all?
Thank you!!
", "rawMarkdown": "Hello! \nCongrats for your place in this competition and thank you for sharing your solution.\n\nI'm trying to understand your solution and I've started with the spectrograms and the ResNet model (model_big). I've accomplished the spectrograms dataset construction, but here I don't really get it. I have an error: \"Requested crop size (118, 225) is larger than the image size (40, 118)\". \nI understand that this model is separated in two files (03_BuildSpectograms_v0.2 and 03_SpectogramModel_Big_v0.5). Is there one more file to complete this model or that's all?\n\nThank you!!", "votes": null }, { "id": "1309248", "postDate": "05/15/2021 19:20:42", "content": "Hello,
\nThanks!!
\nUnfortunatelly both spectograms pipelines are on the same script \"DataPrep/GenerateSpectograms\" either small and big one, just changing some params and removing some comments you can manage to generate the big ones.
\nRegards
", "rawMarkdown": "Hello,\n\nThanks!!\n\nUnfortunatelly both spectograms pipelines are on the same script \"DataPrep/GenerateSpectograms\" either small and big one, just changing some params and removing some comments you can manage to generate the big ones.\n\nRegards", "votes": null }, { "id": "1319833", "postDate": "05/23/2021 14:28:51", "content": "Just to make sure, that's how it should look like? (the spectrograms for the model big) Is it ok to have that padding on the right side?
\nhttps://drive.google.com/drive/folders/15UXM6cewuK80VwyT3YStX9T260S2AVIf?usp=sharing
\n(I put the image on drive because I couldn't upload it here)
Thanks!!
", "rawMarkdown": "Just to make sure, that's how it should look like? (the spectrograms for the model big) Is it ok to have that padding on the right side?\nhttps://drive.google.com/drive/folders/15UXM6cewuK80VwyT3YStX9T260S2AVIf?usp=sharing\n(I put the image on drive because I couldn't upload it here)\n\nThanks!!", "votes": null }, { "id": "1319994", "postDate": "05/23/2021 16:41:37", "content": "Nope,
\nThe x axis represents time, and on this case time (seq len) has always the same size.
\nPadding should be on case you need it on the y axis
", "rawMarkdown": "Nope,\n\nThe x axis represents time, and on this case time (seq len) has always the same size.\n\nPadding should be on case you need it on the y axis", "votes": null }, { "id": "1322230", "postDate": "05/25/2021 09:40:20", "content": "And the extracted mfccs must have the same size as the spectrograms (128, 235). Actually, you feed the mfccs to the network and not the mel spectrograms, right?
\nAt mfccs, the y axis is number of mfccs extracted, so I think it is better to set mfccs=40, than to add padding to the y axis, isn't it? Then for the x axis, the time, theoretically it should be 235 as in the case of spectrograms, because the sampling rate is 100 Hz and the length of the file is also 60000. So it should be 600 samples, as in the case of the mel specs. I don't understand why for mfccs the time axis is only 118 and not 235…
", "rawMarkdown": "And the extracted mfccs must have the same size as the spectrograms (128, 235). Actually, you feed the mfccs to the network and not the mel spectrograms, right?\n\nAt mfccs, the y axis is number of mfccs extracted, so I think it is better to set mfccs=40, than to add padding to the y axis, isn't it? Then for the x axis, the time, theoretically it should be 235 as in the case of spectrograms, because the sampling rate is 100 Hz and the length of the file is also 60000. So it should be 600 samples, as in the case of the mel specs. I don't understand why for mfccs the time axis is only 118 and not 235...", "votes": null }, { "id": "1322951", "postDate": "05/25/2021 20:18:16", "content": "Depending of the network, the small one is for the mfcc and the big one is for the spectograms, the performance was nearly the same.
", "rawMarkdown": "Depending of the network, the small one is for the mfcc and the big one is for the spectograms, the performance was nearly the same.", "votes": null } ], "comments": [ { "id": 1143373, "author_name": "adriencossa", "author_url": "", "post_date": "01/07/2021 21:38:33", "content": "Hello,
\nCongrats on the great results and thanks for sharing your code and solution!
\nI also experimented with CNN and STFFT (feel free to check my notebook about it)). Could you please elaborate on your decisions for your CNN structure ?
\nUnfortunately I did not have access to a GPU so I sticked to a 1D version to be able to train it. On which system did you train your model ?
\nAlso, did you notice any difference in the results when using the different spectograms?
", "votes": null, "replies": [ { "id": 1148084, "author_name": "enric1296", "author_url": "", "post_date": "01/10/2021 21:57:55", "content": "You can check the github repo for any further details, if you still have more doubts dont hesitate to reply it
", "votes": null, "replies": [] } ] }, { "id": 1144635, "author_name": "achyutanantsingh", "author_url": "", "post_date": "01/08/2021 15:27:03", "content": "Great Model new to data science all of this is a great help
", "votes": null, "replies": [] }, { "id": 1155348, "author_name": "mragpavank", "author_url": "", "post_date": "01/16/2021 11:15:18", "content": "Good one, Appreciated. Upvoted
", "votes": null, "replies": [] }, { "id": 1309235, "author_name": "andreeatodor", "author_url": "", "post_date": "05/15/2021 19:01:55", "content": "Hello!
\nCongrats for your place in this competition and thank you for sharing your solution.
I'm trying to understand your solution and I've started with the spectrograms and the ResNet model (model_big). I've accomplished the spectrograms dataset construction, but here I don't really get it. I have an error: \"Requested crop size (118, 225) is larger than the image size (40, 118)\".
\nI understand that this model is separated in two files (03_BuildSpectograms_v0.2 and 03_SpectogramModel_Big_v0.5). Is there one more file to complete this model or that's all?
Thank you!!
", "votes": null, "replies": [ { "id": 1309248, "author_name": "enric1296", "author_url": "", "post_date": "05/15/2021 19:20:42", "content": "Hello,
\nThanks!!
\nUnfortunatelly both spectograms pipelines are on the same script \"DataPrep/GenerateSpectograms\" either small and big one, just changing some params and removing some comments you can manage to generate the big ones.
\nRegards
", "votes": null, "replies": [] }, { "id": 1319833, "author_name": "andreeatodor", "author_url": "", "post_date": "05/23/2021 14:28:51", "content": "Just to make sure, that's how it should look like? (the spectrograms for the model big) Is it ok to have that padding on the right side?
\nhttps://drive.google.com/drive/folders/15UXM6cewuK80VwyT3YStX9T260S2AVIf?usp=sharing
\n(I put the image on drive because I couldn't upload it here)
Thanks!!
", "votes": null, "replies": [] }, { "id": 1319994, "author_name": "enric1296", "author_url": "", "post_date": "05/23/2021 16:41:37", "content": "Nope,
\nThe x axis represents time, and on this case time (seq len) has always the same size.
\nPadding should be on case you need it on the y axis
", "votes": null, "replies": [] }, { "id": 1322230, "author_name": "andreeatodor", "author_url": "", "post_date": "05/25/2021 09:40:20", "content": "And the extracted mfccs must have the same size as the spectrograms (128, 235). Actually, you feed the mfccs to the network and not the mel spectrograms, right?
\nAt mfccs, the y axis is number of mfccs extracted, so I think it is better to set mfccs=40, than to add padding to the y axis, isn't it? Then for the x axis, the time, theoretically it should be 235 as in the case of spectrograms, because the sampling rate is 100 Hz and the length of the file is also 60000. So it should be 600 samples, as in the case of the mel specs. I don't understand why for mfccs the time axis is only 118 and not 235…
", "votes": null, "replies": [] }, { "id": 1322951, "author_name": "enric1296", "author_url": "", "post_date": "05/25/2021 20:18:16", "content": "Depending of the network, the small one is for the mfcc and the big one is for the spectograms, the performance was nearly the same.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1142842": "First of all, I would like to thank to Kaggle and INGV (Istituto nazionale di geofisica e vulcanologia) National Institute of Geophysics and Volcanology for organizing this competition. I have learned a lot and really enjoyed the competition.\n\nI attach the github link below in case you want to check our implementations:\n[https://github.com/EnricRovira/INGV-VolcanicEruptionPrediction](url)\n\nFinally i want to thank my teammate @ajcostarino, who provided nice ideas and efforts.\n\n\n# 1. Base Models\n\nWe builded different base models. They are resumed on the table below. The best one scored 4.9 on public LB.\n\n\n\n\nThe multi head attention model is builded on top of a wavenet head in order to reduce the size of the sequence, and later on whe build 5 heads of 300 depth each.\n\n\n# 2. Preprocessing\n\nFor the MHA model we just perform Standar Scaling (Substract mean and divide by std). \n\nFor spectograms models we builded different kind of spectograms (mel, stft, scipy spectogram) and feeded them to residual CNNs and worked really find.\n\n### 2.1 Models training\n\nIn order to regularize the NN we added a composite [weigh=4(MAE), weight=1(quantileLoss)]\n\nWe decay the learning rate every 10 epochs by a 0.9 factor and added early stopping for a total of 100 epochs. Depending on the model the training times differ, the stft one took 12h to train but the tabular ones only took few minutes.\n\n\n# 3 Validation\n\nWe performed Stratified Cross Validation with 5 CV and save for each fold the best validation model.\nLater we predicted with every fold and average the results.\n\n\n# 4. Stacking\n\nWe tried different techniques but what it worked better was Blending/Stacking with time test augmentations depending on the architecture. For CNN we add image augmentations and for raw signal ones we augmentated the signals (random noise, time shifting, sensor dropour and so on).\n\nFinally we build XGB, CATBoost and Tabnet models on top of this stacking predictions and average them to perform our final submission which scored 4.421.208 on private leaderboard\n\n\n# 5. Things we tried that didnt work\n\nGranular fields images\nClustering \nTime Series valdiation\n\n\n# 6. Improvements\n\nAs some competitios released on their solutions we should have reblended the last step and added more tabular data.\n\nI hope the explanation was understandable and helpful.", "1143373": "Hello, \n\nCongrats on the great results and thanks for sharing your code and solution!\n\nI also experimented with CNN and STFFT (feel free to check my [notebook](https://www.kaggle.com/adriencossa/stfft-and-1d-cnn) about it)). Could you please elaborate on your decisions for your CNN structure ? \n\nUnfortunately I did not have access to a GPU so I sticked to a 1D version to be able to train it. On which system did you train your model ?\n\nAlso, did you notice any difference in the results when using the different spectograms?", "1144635": "Great Model new to data science all of this is a great help", "1148084": "You can check the github repo for any further details, if you still have more doubts dont hesitate to reply it", "1155348": "Good one, Appreciated. Upvoted", "1309235": "Hello! \nCongrats for your place in this competition and thank you for sharing your solution.\n\nI'm trying to understand your solution and I've started with the spectrograms and the ResNet model (model_big). I've accomplished the spectrograms dataset construction, but here I don't really get it. I have an error: \"Requested crop size (118, 225) is larger than the image size (40, 118)\". \nI understand that this model is separated in two files (03_BuildSpectograms_v0.2 and 03_SpectogramModel_Big_v0.5). Is there one more file to complete this model or that's all?\n\nThank you!!", "1309248": "Hello,\n\nThanks!!\n\nUnfortunatelly both spectograms pipelines are on the same script \"DataPrep/GenerateSpectograms\" either small and big one, just changing some params and removing some comments you can manage to generate the big ones.\n\nRegards", "1319833": "Just to make sure, that's how it should look like? (the spectrograms for the model big) Is it ok to have that padding on the right side?\nhttps://drive.google.com/drive/folders/15UXM6cewuK80VwyT3YStX9T260S2AVIf?usp=sharing\n(I put the image on drive because I couldn't upload it here)\n\nThanks!!", "1319994": "Nope,\n\nThe x axis represents time, and on this case time (seq len) has always the same size.\n\nPadding should be on case you need it on the y axis", "1322230": "And the extracted mfccs must have the same size as the spectrograms (128, 235). Actually, you feed the mfccs to the network and not the mel spectrograms, right?\n\nAt mfccs, the y axis is number of mfccs extracted, so I think it is better to set mfccs=40, than to add padding to the y axis, isn't it? Then for the x axis, the time, theoretically it should be 235 as in the case of spectrograms, because the sampling rate is 100 Hz and the length of the file is also 60000. So it should be 600 samples, as in the case of the mel specs. I don't understand why for mfccs the time axis is only 118 and not 235...", "1322951": "Depending of the network, the small one is for the mfcc and the big one is for the spectograms, the performance was nearly the same." }, "source": "meta" }