First, thanks to Kaggle and INGV for hosting this interesting and challenging competition. Second, congrats to all the teams that managed to get into the top places. Also, thanks to the people who published their baseline approaches, especially to @carpediemamigo and @ajcostarino for their notebooks that helped me a lot to improve my scores. In the following, I would like to share my solution to the problem.
\n1) Feature engineering
\nFirst, I generated features for each sensor representing basic descriptive statistics, i.e. mean, variance, quantiles etc., in the time and frequency domain (Fourier transform of the signal). Then, I added some other features (Mel-frequency cepstral coefficients, quantiles of standard deviations calculated on rolling windows) which were motivated by the winning solution of the LANL Earthquake Prediction competition. Lastly, I used @carpediemamigo's 7730 features generated using tsfresh. Thus, I ended up having ~8000 features overall to choose from.
2) Validation and feature selection
\nI used a simple shuffled 5-fold CV for local validation. As many other competitors pointed out, there was a huge discrepancy between CV and LB. This was driven by the fact that the training and test sets were quite different. As the two datasets had similar number of data points and the public test set was ~50% of the entire test set, this competition was an example where we could strongly trust the LB score. This is also backed by the fact that there was very little shake-up in the leaderboard.
Despite that, I tried to rely on the CV when making a modelling decision because: i) we have a limited number of submissions a day, ii) improvements in CV translated to improvements in LB, especially in the beginning. I chose the features via stepwise forward selection based on CV error. Following the intuition from the LANL competition's winner solutions, I tried to use a small number of features, I ended up using ~100 in each of my models.
\n2) 1st level models
\nThough I tried out many different models, my best solution uses three models in the 1st level: LightGBM, XGBoost and NN. LightGBM resulted in the best performance among these models. I saved the out-of-fold (OOF) predictions along with the test predictions for the 2nd levels stacking models, and I used fixed validation folds. I aimed for models that have a similar good performance but their predictions are not that correlated, so these models usually use different sets of features as well.
2) 2nd level models
\nI created two 2nd level stacking models, the basis of these models was the 1st level LightGBM model.
2) Average blending
\nFinally, I averaged the predictions of the two 2nd level models. Each had around ~4.5M error but they had a correlation of ~0.95, so the error dropped to ~4.38M after the averaging.
Visualization of my modelling approach is depicted below. GitHub repo with the codes can be found here.
", "messages": [ { "id": "1144521", "postDate": "01/08/2021 14:01:37", "content": "First, thanks to Kaggle and INGV for hosting this interesting and challenging competition. Second, congrats to all the teams that managed to get into the top places. Also, thanks to the people who published their baseline approaches, especially to @carpediemamigo and @ajcostarino for their notebooks that helped me a lot to improve my scores. In the following, I would like to share my solution to the problem.
\n1) Feature engineering
\nFirst, I generated features for each sensor representing basic descriptive statistics, i.e. mean, variance, quantiles etc., in the time and frequency domain (Fourier transform of the signal). Then, I added some other features (Mel-frequency cepstral coefficients, quantiles of standard deviations calculated on rolling windows) which were motivated by the winning solution of the LANL Earthquake Prediction competition. Lastly, I used @carpediemamigo's 7730 features generated using tsfresh. Thus, I ended up having ~8000 features overall to choose from.
2) Validation and feature selection
\nI used a simple shuffled 5-fold CV for local validation. As many other competitors pointed out, there was a huge discrepancy between CV and LB. This was driven by the fact that the training and test sets were quite different. As the two datasets had similar number of data points and the public test set was ~50% of the entire test set, this competition was an example where we could strongly trust the LB score. This is also backed by the fact that there was very little shake-up in the leaderboard.
Despite that, I tried to rely on the CV when making a modelling decision because: i) we have a limited number of submissions a day, ii) improvements in CV translated to improvements in LB, especially in the beginning. I chose the features via stepwise forward selection based on CV error. Following the intuition from the LANL competition's winner solutions, I tried to use a small number of features, I ended up using ~100 in each of my models.
\n2) 1st level models
\nThough I tried out many different models, my best solution uses three models in the 1st level: LightGBM, XGBoost and NN. LightGBM resulted in the best performance among these models. I saved the out-of-fold (OOF) predictions along with the test predictions for the 2nd levels stacking models, and I used fixed validation folds. I aimed for models that have a similar good performance but their predictions are not that correlated, so these models usually use different sets of features as well.
2) 2nd level models
\nI created two 2nd level stacking models, the basis of these models was the 1st level LightGBM model.
2) Average blending
\nFinally, I averaged the predictions of the two 2nd level models. Each had around ~4.5M error but they had a correlation of ~0.95, so the error dropped to ~4.38M after the averaging.
Visualization of my modelling approach is depicted below. GitHub repo with the codes can be found here.
", "rawMarkdown": "First, thanks to Kaggle and INGV for hosting this interesting and challenging competition. Second, congrats to all the teams that managed to get into the top places. Also, thanks to the people who published their baseline approaches, especially to @carpediemamigo and @ajcostarino for their notebooks that helped me a lot to improve my scores. In the following, I would like to share my solution to the problem.\n\n**1) Feature engineering**\nFirst, I generated features for each sensor representing basic descriptive statistics, i.e. mean, variance, quantiles etc., in the time and frequency domain (Fourier transform of the signal). Then, I added some other features (Mel-frequency cepstral coefficients, quantiles of standard deviations calculated on rolling windows) which were motivated by the [winning solution](https://www.kaggle.com/dkaraflos/1-geomean-nn-and-6featlgbm-2-259-private-lb) of the [LANL Earthquake Prediction](https://www.kaggle.com/c/LANL-Earthquake-Prediction) competition. Lastly, I used @carpediemamigo's 7730 features generated using `tsfresh`. Thus, I ended up having ~8000 features overall to choose from. \n\n**2) Validation and feature selection**\nI used a simple shuffled 5-fold CV for local validation. As many other competitors pointed out, there was a huge discrepancy between CV and LB. This was driven by the fact that the training and test sets were quite different. As the two datasets had similar number of data points and the public test set was ~50% of the entire test set, this competition was an example where we could strongly trust the LB score. This is also backed by the fact that there was very little shake-up in the leaderboard. \n\nDespite that, I tried to rely on the CV when making a modelling decision because: i) we have a limited number of submissions a day, ii) improvements in CV translated to improvements in LB, especially in the beginning. I chose the features via stepwise forward selection based on CV error. Following the intuition from the LANL competition's winner solutions, I tried to use a small number of features, I ended up using ~100 in each of my models.\n\n**2) 1st level models**\nThough I tried out many different models, my best solution uses three models in the 1st level: LightGBM, XGBoost and NN. LightGBM resulted in the best performance among these models. I saved the out-of-fold (OOF) predictions along with the test predictions for the 2nd levels stacking models, and I used fixed validation folds. I aimed for models that have a similar good performance but their predictions are not that correlated, so these models usually use different sets of features as well. \n\n**2) 2nd level models**\nI created two 2nd level stacking models, the basis of these models was the 1st level LightGBM model.\n- adding the OOF predictions of LightGBM and XGBoost to the 1st level LightGBM model\n- adding the OOF predictions of LightGBM and NN to the 1st level LightGBM model\n\n**2) Average blending**\nFinally, I averaged the predictions of the two 2nd level models. Each had around ~4.5M error but they had a correlation of ~0.95, so the error dropped to ~4.38M after the averaging.\n\nVisualization of my modelling approach is depicted below. GitHub repo with the codes can be found [here](https://github.com/leventelippenszky/INGV-Volcanic-Eruption-Prediction).", "votes": null }, { "id": "1144529", "postDate": "01/08/2021 14:05:37", "content": "Does anyone know how to upload an image to the post? I tried 'Insert Image' but it doesn't work.
", "rawMarkdown": "Does anyone know how to upload an image to the post? I tried 'Insert Image' but it doesn't work.", "votes": null } ], "comments": [ { "id": 1144529, "author_name": "leventelippenszky", "author_url": "", "post_date": "01/08/2021 14:05:37", "content": "Does anyone know how to upload an image to the post? I tried 'Insert Image' but it doesn't work.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1144521": "First, thanks to Kaggle and INGV for hosting this interesting and challenging competition. Second, congrats to all the teams that managed to get into the top places. Also, thanks to the people who published their baseline approaches, especially to @carpediemamigo and @ajcostarino for their notebooks that helped me a lot to improve my scores. In the following, I would like to share my solution to the problem.\n\n**1) Feature engineering**\nFirst, I generated features for each sensor representing basic descriptive statistics, i.e. mean, variance, quantiles etc., in the time and frequency domain (Fourier transform of the signal). Then, I added some other features (Mel-frequency cepstral coefficients, quantiles of standard deviations calculated on rolling windows) which were motivated by the [winning solution](https://www.kaggle.com/dkaraflos/1-geomean-nn-and-6featlgbm-2-259-private-lb) of the [LANL Earthquake Prediction](https://www.kaggle.com/c/LANL-Earthquake-Prediction) competition. Lastly, I used @carpediemamigo's 7730 features generated using `tsfresh`. Thus, I ended up having ~8000 features overall to choose from. \n\n**2) Validation and feature selection**\nI used a simple shuffled 5-fold CV for local validation. As many other competitors pointed out, there was a huge discrepancy between CV and LB. This was driven by the fact that the training and test sets were quite different. As the two datasets had similar number of data points and the public test set was ~50% of the entire test set, this competition was an example where we could strongly trust the LB score. This is also backed by the fact that there was very little shake-up in the leaderboard. \n\nDespite that, I tried to rely on the CV when making a modelling decision because: i) we have a limited number of submissions a day, ii) improvements in CV translated to improvements in LB, especially in the beginning. I chose the features via stepwise forward selection based on CV error. Following the intuition from the LANL competition's winner solutions, I tried to use a small number of features, I ended up using ~100 in each of my models.\n\n**2) 1st level models**\nThough I tried out many different models, my best solution uses three models in the 1st level: LightGBM, XGBoost and NN. LightGBM resulted in the best performance among these models. I saved the out-of-fold (OOF) predictions along with the test predictions for the 2nd levels stacking models, and I used fixed validation folds. I aimed for models that have a similar good performance but their predictions are not that correlated, so these models usually use different sets of features as well. \n\n**2) 2nd level models**\nI created two 2nd level stacking models, the basis of these models was the 1st level LightGBM model.\n- adding the OOF predictions of LightGBM and XGBoost to the 1st level LightGBM model\n- adding the OOF predictions of LightGBM and NN to the 1st level LightGBM model\n\n**2) Average blending**\nFinally, I averaged the predictions of the two 2nd level models. Each had around ~4.5M error but they had a correlation of ~0.95, so the error dropped to ~4.38M after the averaging.\n\nVisualization of my modelling approach is depicted below. GitHub repo with the codes can be found [here](https://github.com/leventelippenszky/INGV-Volcanic-Eruption-Prediction).", "1144529": "Does anyone know how to upload an image to the post? I tried 'Insert Image' but it doesn't work." }, "source": "meta" }