When I first started this competition, the first thing I did was look at how many samples exist, what their distribution was, and then setup my CV. I split the data into 6 buckets, and have been using four of those to build 4-fold CV models and kept the remaining two buckets for stacking. I had just been doing OOF blending for ensembling. But since I have ~48 trained models now, I tried to give the holdout stacker a go.
\nThe results were…… not what I was expecting.
\nI've tried a variety of model types for the stacker: GPR, LogisticRegression, Linear, Ridge, Lasso, Elastic, SVR (RBF+Linear), MultinomialNB, RF, SGDR. Basically everything under the sun except a GBDT and a NNet. No matter what ML model I use to stack the holdout predictions, the model's validation performance has never beaten a linear combination of weights derived from a straight scipy.optimize.minimize. How is that possible??
\nI base my understanding of stacking on @kazanova's great Kaggle coursera series. I believe stacking with a ML model (particularly, one that has more than one parameter per base-model's prediction) is that it should be superior because it can detect \"zones\" where one model preforms good and prefer it those regions; and then prefer other models in regions where their results are better.
\nBest Stack:
\nRandom Minimize Model Weights:
\nThe best stacker model was cumlSGD. No other models tested came even close (example CVs ~840, ~800, 720, 0.5!, etc).
\nDid I shoot myself in the foot? Is better to spend the next full week re-training all of my models on the full data rather than keeping 33% holdout for a stacker that is worthless? Has any one else experimented with stacking their models yet? I'm running experiments now with GBDT and NNet models to finish off the experiment set..
", "messages": [ { "id": "1493437", "postDate": "08/27/2021 22:27:03", "content": "When I first started this competition, the first thing I did was look at how many samples exist, what their distribution was, and then setup my CV. I split the data into 6 buckets, and have been using four of those to build 4-fold CV models and kept the remaining two buckets for stacking. I had just been doing OOF blending for ensembling. But since I have ~48 trained models now, I tried to give the holdout stacker a go.
\nThe results were…… not what I was expecting.
\nI've tried a variety of model types for the stacker: GPR, LogisticRegression, Linear, Ridge, Lasso, Elastic, SVR (RBF+Linear), MultinomialNB, RF, SGDR. Basically everything under the sun except a GBDT and a NNet. No matter what ML model I use to stack the holdout predictions, the model's validation performance has never beaten a linear combination of weights derived from a straight scipy.optimize.minimize. How is that possible??
\nI base my understanding of stacking on @kazanova's great Kaggle coursera series. I believe stacking with a ML model (particularly, one that has more than one parameter per base-model's prediction) is that it should be superior because it can detect \"zones\" where one model preforms good and prefer it those regions; and then prefer other models in regions where their results are better.
\nBest Stack:
\nRandom Minimize Model Weights:
\nThe best stacker model was cumlSGD. No other models tested came even close (example CVs ~840, ~800, 720, 0.5!, etc).
\nDid I shoot myself in the foot? Is better to spend the next full week re-training all of my models on the full data rather than keeping 33% holdout for a stacker that is worthless? Has any one else experimented with stacking their models yet? I'm running experiments now with GBDT and NNet models to finish off the experiment set..
", "rawMarkdown": "When I first started this competition, the first thing I did was look at how many samples exist, what their distribution was, and then setup my CV. I split the data into 6 buckets, and have been using four of those to build 4-fold CV models and kept the remaining two buckets for stacking. I had just been doing OOF blending for ensembling. But since I have ~48 trained models now, I tried to give the holdout stacker a go. \n\nThe results were...... not what I was expecting.\n\nI've tried a variety of model types for the stacker: GPR, LogisticRegression, Linear, Ridge, Lasso, Elastic, SVR (RBF+Linear), MultinomialNB, RF, SGDR. Basically everything under the sun except a GBDT and a NNet. No matter what ML model I use to stack the holdout predictions, the model's validation performance has never beaten a linear combination of weights derived from a straight scipy.optimize.minimize. How is that possible??\n\nI base my understanding of stacking on @kazanova's great Kaggle coursera series. I believe stacking with a ML model (particularly, one that has more than one parameter per base-model's prediction) is that it should be superior because it can detect \"zones\" where one model preforms good and prefer it those regions; and then prefer other models in regions where their results are better.\n\nBest Stack:\n- Val: 0.874972 (again, this is trained+validated on 2/6th of the data, split in two folds)\n- LB: 0.8767\n\nRandom Minimize Model Weights:\n- Val 0.875204\n- LB: 0.8768\n\nThe best stacker model was cumlSGD. No other models tested came even close (example CVs ~840, ~800, 720, 0.5!, etc).\n\nDid I shoot myself in the foot? Is better to spend the next full week re-training all of my models on the full data rather than keeping 33% holdout for a stacker that is worthless? Has any one else experimented with stacking their models yet? I'm running experiments now with GBDT and NNet models to finish off the experiment set..", "votes": null }, { "id": "1493457", "postDate": "08/27/2021 23:00:10", "content": "From nearly experiments I did, it looked like the dataset was nearly homogeneous: either working on the whole dataset or on 10% of it, would lead to the same conclusions. Hence, similarly to you, I opted to train my models on 90% of the dataset and kept the rest as an holdhout I would use for stacking. As for now, I have trained around 15 models with holdout scores around 0.861-0.872. I used them to feed a second layer. As stackers, so far, I tried logistic regressions, MLP, LGBM and none of them were able to outperform the score I could reach with first layer models (both on holdout score and LB)….
\nAs you do, I currently reconsider my training methodology and wonder if I shouldn't train my model on 100% of the dataset …
", "rawMarkdown": "From nearly experiments I did, it looked like the dataset was nearly homogeneous: either working on the whole dataset or on 10% of it, would lead to the same conclusions. Hence, similarly to you, I opted to train my models on 90% of the dataset and kept the rest as an holdhout I would use for stacking. As for now, I have trained around 15 models with holdout scores around 0.861-0.872. I used them to feed a second layer. As stackers, so far, I tried logistic regressions, MLP, LGBM and none of them were able to outperform the score I could reach with first layer models (both on holdout score and LB)....\n\nAs you do, I currently reconsider my training methodology and wonder if I shouldn't train my model on 100% of the dataset ...", "votes": null }, { "id": "1493478", "postDate": "08/27/2021 23:44:14", "content": "Thank you for confirming my suspicion.
\nIt might be beneficial to look at the top ~300 worst FP and top ~300 worst FN by prediction error. If all models are essentially failing on the same samples, that conclusively answers why stacking only produces modest gains VS just training on more data.
\nLooking at model correlations, they're all very similar (0.96-0.99999), so I guess they're all essentially learning the same thing. Some GW are just easier for all models to see (?), other GWs are more difficult.
\nNow if only we could learn the mapping to make the hard ones to see one's easier…
", "rawMarkdown": "Thank you for confirming my suspicion.\n\nIt might be beneficial to look at the top ~300 worst FP and top ~300 worst FN by prediction error. If all models are essentially failing on the same samples, that conclusively answers why stacking only produces modest gains VS just training on more data.\n\nLooking at model correlations, they're all very similar (0.96-0.99999), so I guess they're all essentially learning the same thing. Some GW are just easier for all models to see (?), other GWs are more difficult. \n\nNow if only we could learn the mapping to make the hard ones to see one's easier...", "votes": null }, { "id": "1494498", "postDate": "08/28/2021 17:53:56", "content": "@cpmpml stated a week ago that their best model was a single model and that doing ensembling for ROC AUC competitions does not work well often.
", "rawMarkdown": "cpmpml stated a week ago that their best model was a single model and that doing ensembling for ROC AUC competitions does not work well often.", "votes": null }, { "id": "1494581", "postDate": "08/28/2021 19:44:22", "content": "Not exactly. I said our best model achieved 0.880x and that ensembling does not help much with roc-auc in general.
", "rawMarkdown": "Not exactly. I said our best model achieved 0.880x and that ensembling does not help much with roc-auc in general.", "votes": null }, { "id": "1494587", "postDate": "08/28/2021 19:56:46", "content": "I recall @felipebihaiek, I was in that conversation :-). OOF Ensembling gives me a little boost—but it's pennies to the dollar compared to my base models. Still worth it at current LB. Stacking, however, hasn't provided me with any return whatsoever, so now I'm burning cycles re-running experiments which is undesired.
", "rawMarkdown": "I recall @felipebihaiek, I was in that conversation :-). OOF Ensembling gives me a little boost—but it's pennies to the dollar compared to my base models. Still worth it at current LB. Stacking, however, hasn't provided me with any return whatsoever, so now I'm burning cycles re-running experiments which is undesired.", "votes": null }, { "id": "1495016", "postDate": "08/29/2021 08:41:03", "content": "Hi Thanks for sharing amazing information..
\ni think it is doubt question, but i wonder that why do you using ML model rather than deep learning?
\nAs i think that this problem is so deep and complicated, So deep model will be answer.
And moreover, why are you using only 4-cv without 2 holdout?
\ni guess amounts of train data is significant in ml field.
Thanks.
", "rawMarkdown": "Hi Thanks for sharing amazing information..\ni think it is doubt question, but i wonder that why do you using ML model rather than deep learning?\nAs i think that this problem is so deep and complicated, So deep model will be answer.\n\nAnd moreover, why are you using only 4-cv without 2 holdout?\ni guess amounts of train data is significant in ml field.\n\nThanks.", "votes": null }, { "id": "1495231", "postDate": "08/29/2021 11:55:01", "content": "the 2nd layer stack should be one that optimizes AUC directly …. maybe that will work
", "rawMarkdown": "the 2nd layer stack should be one that optimizes AUC directly .... maybe that will work", "votes": null }, { "id": "1496313", "postDate": "08/30/2021 09:12:37", "content": "Thanks for sharing amazing information.. =))
", "rawMarkdown": "Thanks for sharing amazing information.. =))", "votes": null }, { "id": "1561012", "postDate": "10/27/2021 09:09:27", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
From nearly experiments I did, it looked like the dataset was nearly homogeneous: either working on the whole dataset or on 10% of it, would lead to the same conclusions. Hence, similarly to you, I opted to train my models on 90% of the dataset and kept the rest as an holdhout I would use for stacking. As for now, I have trained around 15 models with holdout scores around 0.861-0.872. I used them to feed a second layer. As stackers, so far, I tried logistic regressions, MLP, LGBM and none of them were able to outperform the score I could reach with first layer models (both on holdout score and LB)….
\nAs you do, I currently reconsider my training methodology and wonder if I shouldn't train my model on 100% of the dataset …
", "votes": null, "replies": [ { "id": 1493478, "author_name": "authman", "author_url": "", "post_date": "08/27/2021 23:44:14", "content": "Thank you for confirming my suspicion.
\nIt might be beneficial to look at the top ~300 worst FP and top ~300 worst FN by prediction error. If all models are essentially failing on the same samples, that conclusively answers why stacking only produces modest gains VS just training on more data.
\nLooking at model correlations, they're all very similar (0.96-0.99999), so I guess they're all essentially learning the same thing. Some GW are just easier for all models to see (?), other GWs are more difficult.
\nNow if only we could learn the mapping to make the hard ones to see one's easier…
", "votes": null, "replies": [] } ] }, { "id": 1494498, "author_name": "felipebihaiek", "author_url": "", "post_date": "08/28/2021 17:53:56", "content": "@cpmpml stated a week ago that their best model was a single model and that doing ensembling for ROC AUC competitions does not work well often.
", "votes": null, "replies": [ { "id": 1494581, "author_name": "cpmpml", "author_url": "", "post_date": "08/28/2021 19:44:22", "content": "Not exactly. I said our best model achieved 0.880x and that ensembling does not help much with roc-auc in general.
", "votes": null, "replies": [] }, { "id": 1494587, "author_name": "authman", "author_url": "", "post_date": "08/28/2021 19:56:46", "content": "I recall @felipebihaiek, I was in that conversation :-). OOF Ensembling gives me a little boost—but it's pennies to the dollar compared to my base models. Still worth it at current LB. Stacking, however, hasn't provided me with any return whatsoever, so now I'm burning cycles re-running experiments which is undesired.
", "votes": null, "replies": [] } ] }, { "id": 1495016, "author_name": "leemop", "author_url": "", "post_date": "08/29/2021 08:41:03", "content": "Hi Thanks for sharing amazing information..
\ni think it is doubt question, but i wonder that why do you using ML model rather than deep learning?
\nAs i think that this problem is so deep and complicated, So deep model will be answer.
And moreover, why are you using only 4-cv without 2 holdout?
\ni guess amounts of train data is significant in ml field.
Thanks.
", "votes": null, "replies": [] }, { "id": 1495231, "author_name": "hengck23", "author_url": "", "post_date": "08/29/2021 11:55:01", "content": "the 2nd layer stack should be one that optimizes AUC directly …. maybe that will work
", "votes": null, "replies": [] }, { "id": 1496313, "author_name": "firefliesqn", "author_url": "", "post_date": "08/30/2021 09:12:37", "content": "Thanks for sharing amazing information.. =))
", "votes": null, "replies": [] }, { "id": 1561012, "author_name": "zerafachris", "author_url": "", "post_date": "10/27/2021 09:09:27", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris