I am curious what people's experience has been with blending models to improve their score. I am at a point where (without feature engineering which in hindsight would've been a better use of my time) my LGBM and NN models are as tuned as they're going to be. They are individually around 1.000 and blend to 0.951. I've got an SVM model that scored 1.139 so I thought I'd give it a whirl and add it (ended up at 0.960). There is still room to improve the SVM model, which is my current focus.
\n\nMy question is - how close does the 'worst' model need to be to the pack to add value to the blend?
", "messages": [ { "id": "439635", "postDate": "12/16/2018 01:05:07", "content": "I am curious what people's experience has been with blending models to improve their score. I am at a point where (without feature engineering which in hindsight would've been a better use of my time) my LGBM and NN models are as tuned as they're going to be. They are individually around 1.000 and blend to 0.951. I've got an SVM model that scored 1.139 so I thought I'd give it a whirl and add it (ended up at 0.960). There is still room to improve the SVM model, which is my current focus.
\n\nMy question is - how close does the 'worst' model need to be to the pack to add value to the blend?
", "rawMarkdown": "I am curious what people's experience has been with blending models to improve their score. I am at a point where (without feature engineering which in hindsight would've been a better use of my time) my LGBM and NN models are as tuned as they're going to be. They are individually around 1.000 and blend to 0.951. I've got an SVM model that scored 1.139 so I thought I'd give it a whirl and add it (ended up at 0.960). There is still room to improve the SVM model, which is my current focus.\n\nMy question is - how close does the 'worst' model need to be to the pack to add value to the blend?", "votes": null }, { "id": "439740", "postDate": "12/16/2018 08:37:00", "content": "Improvement from blending is a function of their individual performance and correlation between them. So most probably your SVM model uses similar features with your NN and SVMs are mathematically neural networks. Check correlation of your predictions from different models, then decide your weights in blending.
", "rawMarkdown": "Improvement from blending is a function of their individual performance and correlation between them. So most probably your SVM model uses similar features with your NN and SVMs are mathematically neural networks. Check correlation of your predictions from different models, then decide your weights in blending.", "votes": null }, { "id": "439757", "postDate": "12/16/2018 09:25:50", "content": "My experience with has not been so good. I blended two models\nxgb model with public lb 0.986\nlgbm model with public lb 1.000\nand blend scored only 0.991 on lb. I am using blend them all kernel as reference, any hints on what could be wrong in my blending.
", "rawMarkdown": "My experience with has not been so good. I blended two models\nxgb model with public lb 0.986\nlgbm model with public lb 1.000\nand blend scored only 0.991 on lb. I am using blend them all kernel as reference, any hints on what could be wrong in my blending.", "votes": null }, { "id": "439760", "postDate": "12/16/2018 09:44:29", "content": "I think those are very similar models so I’m not sure blending will gain a lot.
", "rawMarkdown": "I think those are very similar models so I’m not sure blending will gain a lot.", "votes": null }, { "id": "439925", "postDate": "12/16/2018 16:50:43", "content": "Thanks for sharing. I now have 31 hours to create a decent NN model and blend it in with my LGBM :)
", "rawMarkdown": "Thanks for sharing. I now have 31 hours to create a decent NN model and blend it in with my LGBM :)", "votes": null }, { "id": "439952", "postDate": "12/16/2018 18:00:36", "content": "This is doable, good luck!
", "rawMarkdown": "This is doable, good luck!", "votes": null }, { "id": "439970", "postDate": "12/16/2018 18:36:01", "content": "Ya that must be the reason. Then, I should move to nn model but I am afraid I am out of time. But I will give it a shot.
", "rawMarkdown": "Ya that must be the reason. Then, I should move to nn model but I am afraid I am out of time. But I will give it a shot.", "votes": null }, { "id": "439992", "postDate": "12/16/2018 20:21:51", "content": "I am blending three different models, one based in your contribution( thanks a lot,Jim), the second one an approach where i have separated extra and galactic objects ,each one with a different parametrization lightgbm, after that i have joined both predictions in an unique dataframe, and the third one a neural network in an unique model .\nBy now Lb 0.963 and still training two models with new features for tomorrow submission, until the last day working hard ;-), hoping to improve the score a little, at least to obtain bronze medal, that is being very expensive , in this my first competition.\nI would like to express my appreciation and give thanks to everyone that has participated in kernels and discussions. It has been great to share time with you. I wish you the best. Happy Christmas!!!
", "rawMarkdown": "I am blending three different models, one based in your contribution( thanks a lot,Jim), the second one an approach where i have separated extra and galactic objects ,each one with a different parametrization lightgbm, after that i have joined both predictions in an unique dataframe, and the third one a neural network in an unique model .\nBy now Lb 0.963 and still training two models with new features for tomorrow submission, until the last day working hard ;-), hoping to improve the score a little, at least to obtain bronze medal, that is being very expensive , in this my first competition.\nI would like to express my appreciation and give thanks to everyone that has participated in kernels and discussions. It has been great to share time with you. I wish you the best. Happy Christmas!!!", "votes": null }, { "id": "440003", "postDate": "12/16/2018 20:57:04", "content": "Maybe i don't fully understand, but when you say that you worked on and NN and LGBM models without feature engineering, how are you able to use the data without any feature engineering? Particularlly given that the time series data is so disperate.
", "rawMarkdown": "Maybe i don't fully understand, but when you say that you worked on and NN and LGBM models without feature engineering, how are you able to use the data without any feature engineering? Particularlly given that the time series data is so disperate.", "votes": null }, { "id": "440007", "postDate": "12/16/2018 21:24:46", "content": "I’m not saying I did no feature Engineering. I did a little in my SomethingDifferent kernel. I did a little more beyond that. I also used the features from Chia-ta Tsais kernel.
\n\nI had some really cool features developed for the training set however I didn’t think I could process test with the computational resources available. In hindsight I wish I had tried.
", "rawMarkdown": "I’m not saying I did no feature Engineering. I did a little in my SomethingDifferent kernel. I did a little more beyond that. I also used the features from Chia-ta Tsais kernel.\n\nI had some really cool features developed for the training set however I didn’t think I could process test with the computational resources available. In hindsight I wish I had tried.", "votes": null }, { "id": "440008", "postDate": "12/16/2018 21:25:35", "content": "Without feature engineering? No, i haven't said that. I am looking for new features to improve my last models until tomorrow because if base models are better their blending will obtain best result , at least if they are not correlated.
", "rawMarkdown": "Without feature engineering? No, i haven't said that. I am looking for new features to improve my last models until tomorrow because if base models are better their blending will obtain best result , at least if they are not correlated.", "votes": null }, { "id": "440038", "postDate": "12/16/2018 23:37:00", "content": "Just started with this. Did a linear blend of a 1.034 and a 1.014 model, which got me to 0.958. Considering the diversity of the two models, I feel some more juice could be extracted with a shallow NNet, DT, or Logistic ensembler instead. What's the best way to go about doing that? My understanding is:
\n\nIs that right?
", "rawMarkdown": "Just started with this. Did a linear blend of a 1.034 and a 1.014 model, which got me to 0.958. Considering the diversity of the two models, I feel some more juice could be extracted with a shallow NNet, DT, or Logistic ensembler instead. What's the best way to go about doing that? My understanding is:\n\n- Take multiclass OOF train predictions of both models (14+14=28)\n- Split into desired # of folds, where each fold need not necessarily be the same as the underlying lvl-1 model\n- Train the stacker model\n- Take multiclass submission predictions of both models (14+14=28)\n- Average the Fold predictions of the stacker model ran in inference on the above data\n- Fancy Class99 calculation\n- Profit?\n\nIs that right?", "votes": null }, { "id": "440046", "postDate": "12/17/2018 00:18:50", "content": "\n\n\nIs that right?
\n
I'm afraid there is only one way to knpw: do it and submit ;) Is that right?But what you say seems reasonable at least. It is not what we do, hence I can't be sure, but still, it looks reasonable.
", "rawMarkdown": "> Is that right?\n\nI'm afraid there is only one way to knpw: do it and submit ;) Is that right?But what you say seems reasonable at least. It is not what we do, hence I can't be sure, but still, it looks reasonable.", "votes": null }, { "id": "440047", "postDate": "12/17/2018 00:25:08", "content": "I tried but I got a 12.3 LB score, ROFL
\n\nI'm praying it was due to my model \"source-control\" failing (e.g. wrong train oof predictions, oof preds being overwritten by a different run, etc), and I'm last-ditch re-running on my models tonight; but wanted to be sure on the procedure side. Looking forward to hearing your notes after the comp!
\n\nGetting to the end of a competition is like getting to the end of a series you just solo binged watched. No idea what to do next in life (if there isn't another comp lined up), and feeling guilty about all the time spent neglecting one's significant other / kids / etc.
", "rawMarkdown": "I tried but I got a 12.3 LB score, ROFL\n\nI'm praying it was due to my model \"source-control\" failing (e.g. wrong train oof predictions, oof preds being overwritten by a different run, etc), and I'm last-ditch re-running on my models tonight; but wanted to be sure on the procedure side. Looking forward to hearing your notes after the comp!\n\nGetting to the end of a competition is like getting to the end of a series you just solo binged watched. No idea what to do next in life (if there isn't another comp lined up), and feeling guilty about all the time spent neglecting one's significant other / kids / etc.", "votes": null }, { "id": "440278", "postDate": "12/17/2018 10:21:22", "content": "How long approximately SVM is taking for training ?
", "rawMarkdown": "How long approximately SVM is taking for training ?", "votes": null }, { "id": "440565", "postDate": "12/17/2018 17:54:33", "content": "Ughhh I jacked up. This is what sleep coding does. What I was doing wrong:
\n\nTrain data is split into 5 folds, and 5 copies of \"ModelA\" are CV'd on this data. The OOF predictions of train are therefore actually 20% slices of the traindata predicted using a single model each. If I then take all 5 models and infer the entire test set with each model—averaging the results, what I end up with is a single 100% slice that is a non-weighted blend of all 5 models. This test set prediction and oof train prediction pair cannot be used as a base model in stacking because they're essentially different models, hence the 12.x LB explosion.
\n\nWhat I should have done is, from the get go, partition the training data => {trainpart,holdoutpart}. Then do as above, e.g. 5fold CV on the train_part and run inference on both holdoutpart as well as the full test submission using all 5 folds averaged. The stacking model would then be trained on the 5 holdoutparts and would then be applied to the submission data that has already been ran through the same models.
\n\nTimes up on this competition so it looks like my only choices for ensembling are linear blending of different models and creating more bags of my faster to train ones.
", "rawMarkdown": "Ughhh I jacked up. This is what sleep coding does. What I was doing wrong:\n\nTrain data is split into 5 folds, and 5 copies of \"ModelA\" are CV'd on this data. The OOF predictions of train are therefore actually 20% slices of the traindata predicted using a _single_ model each. If I then take all 5 models and infer the entire test set with each model—averaging the results, what I end up with is a single 100% slice that is a non-weighted blend of all 5 models. This test set prediction and oof train prediction pair cannot be used as a base model in stacking because they're essentially different models, hence the 12.x LB explosion.\n\nWhat I should have done is, from the get go, partition the training data => {trainpart,holdoutpart}. _Then_ do as above, e.g. 5fold CV on the train_part and run inference on _both_ holdoutpart as well as the full test submission using all 5 folds averaged. The stacking model would then be trained on the 5 holdoutparts and would then be applied to the submission data that has already been ran through the same models.\n\nTimes up on this competition so it looks like my only choices for ensembling are linear blending of different models and creating more bags of my faster to train ones.", "votes": null }, { "id": "440581", "postDate": "12/17/2018 18:15:00", "content": "SVM took a lot of resources. I went at it in two different ways. I built individual classifiers for each class (is 52, is not 52) and I also used the SVM multiclass classifier. The multiclass timed out if I used more than 5 folds. In order to predict test I had to split the data first by fold and then by rows (1.2million, 1.2million, rest).
\n\nI also had to apply a custom function before the scores were decent.
", "rawMarkdown": "SVM took a lot of resources. I went at it in two different ways. I built individual classifiers for each class (is 52, is not 52) and I also used the SVM multiclass classifier. The multiclass timed out if I used more than 5 folds. In order to predict test I had to split the data first by fold and then by rows (1.2million, 1.2million, rest).\n\nI also had to apply a custom function before the scores were decent.", "votes": null } ], "comments": [ { "id": 439740, "author_name": "aerdem4", "author_url": "", "post_date": "12/16/2018 08:37:00", "content": "Improvement from blending is a function of their individual performance and correlation between them. So most probably your SVM model uses similar features with your NN and SVMs are mathematically neural networks. Check correlation of your predictions from different models, then decide your weights in blending.
", "votes": null, "replies": [] }, { "id": 439757, "author_name": "aashish7936", "author_url": "", "post_date": "12/16/2018 09:25:50", "content": "My experience with has not been so good. I blended two models\nxgb model with public lb 0.986\nlgbm model with public lb 1.000\nand blend scored only 0.991 on lb. I am using blend them all kernel as reference, any hints on what could be wrong in my blending.
", "votes": null, "replies": [ { "id": 439760, "author_name": "jimpsull", "author_url": "", "post_date": "12/16/2018 09:44:29", "content": "I think those are very similar models so I’m not sure blending will gain a lot.
", "votes": null, "replies": [] }, { "id": 439970, "author_name": "aashish7936", "author_url": "", "post_date": "12/16/2018 18:36:01", "content": "Ya that must be the reason. Then, I should move to nn model but I am afraid I am out of time. But I will give it a shot.
", "votes": null, "replies": [] } ] }, { "id": 439925, "author_name": "sdoria", "author_url": "", "post_date": "12/16/2018 16:50:43", "content": "Thanks for sharing. I now have 31 hours to create a decent NN model and blend it in with my LGBM :)
", "votes": null, "replies": [ { "id": 439952, "author_name": "cpmpml", "author_url": "", "post_date": "12/16/2018 18:00:36", "content": "This is doable, good luck!
", "votes": null, "replies": [] } ] }, { "id": 439992, "author_name": "joxemi", "author_url": "", "post_date": "12/16/2018 20:21:51", "content": "I am blending three different models, one based in your contribution( thanks a lot,Jim), the second one an approach where i have separated extra and galactic objects ,each one with a different parametrization lightgbm, after that i have joined both predictions in an unique dataframe, and the third one a neural network in an unique model .\nBy now Lb 0.963 and still training two models with new features for tomorrow submission, until the last day working hard ;-), hoping to improve the score a little, at least to obtain bronze medal, that is being very expensive , in this my first competition.\nI would like to express my appreciation and give thanks to everyone that has participated in kernels and discussions. It has been great to share time with you. I wish you the best. Happy Christmas!!!
", "votes": null, "replies": [] }, { "id": 440003, "author_name": "molenr", "author_url": "", "post_date": "12/16/2018 20:57:04", "content": "Maybe i don't fully understand, but when you say that you worked on and NN and LGBM models without feature engineering, how are you able to use the data without any feature engineering? Particularlly given that the time series data is so disperate.
", "votes": null, "replies": [ { "id": 440007, "author_name": "jimpsull", "author_url": "", "post_date": "12/16/2018 21:24:46", "content": "I’m not saying I did no feature Engineering. I did a little in my SomethingDifferent kernel. I did a little more beyond that. I also used the features from Chia-ta Tsais kernel.
\n\nI had some really cool features developed for the training set however I didn’t think I could process test with the computational resources available. In hindsight I wish I had tried.
", "votes": null, "replies": [] }, { "id": 440008, "author_name": "joxemi", "author_url": "", "post_date": "12/16/2018 21:25:35", "content": "Without feature engineering? No, i haven't said that. I am looking for new features to improve my last models until tomorrow because if base models are better their blending will obtain best result , at least if they are not correlated.
", "votes": null, "replies": [] } ] }, { "id": 440038, "author_name": "authman", "author_url": "", "post_date": "12/16/2018 23:37:00", "content": "Just started with this. Did a linear blend of a 1.034 and a 1.014 model, which got me to 0.958. Considering the diversity of the two models, I feel some more juice could be extracted with a shallow NNet, DT, or Logistic ensembler instead. What's the best way to go about doing that? My understanding is:
\n\nIs that right?
", "votes": null, "replies": [ { "id": 440046, "author_name": "cpmpml", "author_url": "", "post_date": "12/17/2018 00:18:50", "content": "\n\n\nIs that right?
\n
I'm afraid there is only one way to knpw: do it and submit ;) Is that right?But what you say seems reasonable at least. It is not what we do, hence I can't be sure, but still, it looks reasonable.
", "votes": null, "replies": [] }, { "id": 440047, "author_name": "authman", "author_url": "", "post_date": "12/17/2018 00:25:08", "content": "I tried but I got a 12.3 LB score, ROFL
\n\nI'm praying it was due to my model \"source-control\" failing (e.g. wrong train oof predictions, oof preds being overwritten by a different run, etc), and I'm last-ditch re-running on my models tonight; but wanted to be sure on the procedure side. Looking forward to hearing your notes after the comp!
\n\nGetting to the end of a competition is like getting to the end of a series you just solo binged watched. No idea what to do next in life (if there isn't another comp lined up), and feeling guilty about all the time spent neglecting one's significant other / kids / etc.
", "votes": null, "replies": [] }, { "id": 440565, "author_name": "authman", "author_url": "", "post_date": "12/17/2018 17:54:33", "content": "Ughhh I jacked up. This is what sleep coding does. What I was doing wrong:
\n\nTrain data is split into 5 folds, and 5 copies of \"ModelA\" are CV'd on this data. The OOF predictions of train are therefore actually 20% slices of the traindata predicted using a single model each. If I then take all 5 models and infer the entire test set with each model—averaging the results, what I end up with is a single 100% slice that is a non-weighted blend of all 5 models. This test set prediction and oof train prediction pair cannot be used as a base model in stacking because they're essentially different models, hence the 12.x LB explosion.
\n\nWhat I should have done is, from the get go, partition the training data => {trainpart,holdoutpart}. Then do as above, e.g. 5fold CV on the train_part and run inference on both holdoutpart as well as the full test submission using all 5 folds averaged. The stacking model would then be trained on the 5 holdoutparts and would then be applied to the submission data that has already been ran through the same models.
\n\nTimes up on this competition so it looks like my only choices for ensembling are linear blending of different models and creating more bags of my faster to train ones.
", "votes": null, "replies": [] } ] }, { "id": 440278, "author_name": "mks2192", "author_url": "", "post_date": "12/17/2018 10:21:22", "content": "How long approximately SVM is taking for training ?
", "votes": null, "replies": [ { "id": 440581, "author_name": "jimpsull", "author_url": "", "post_date": "12/17/2018 18:15:00", "content": "SVM took a lot of resources. I went at it in two different ways. I built individual classifiers for each class (is 52, is not 52) and I also used the SVM multiclass classifier. The multiclass timed out if I used more than 5 folds. In order to predict test I had to split the data first by fold and then by rows (1.2million, 1.2million, rest).
\n\nI also had to apply a custom function before the scores were decent.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "439635": "I am curious what people's experience has been with blending models to improve their score. I am at a point where (without feature engineering which in hindsight would've been a better use of my time) my LGBM and NN models are as tuned as they're going to be. They are individually around 1.000 and blend to 0.951. I've got an SVM model that scored 1.139 so I thought I'd give it a whirl and add it (ended up at 0.960). There is still room to improve the SVM model, which is my current focus.\n\nMy question is - how close does the 'worst' model need to be to the pack to add value to the blend?", "439740": "Improvement from blending is a function of their individual performance and correlation between them. So most probably your SVM model uses similar features with your NN and SVMs are mathematically neural networks. Check correlation of your predictions from different models, then decide your weights in blending.", "439757": "My experience with has not been so good. I blended two models\nxgb model with public lb 0.986\nlgbm model with public lb 1.000\nand blend scored only 0.991 on lb. I am using blend them all kernel as reference, any hints on what could be wrong in my blending.", "439760": "I think those are very similar models so I’m not sure blending will gain a lot.", "439925": "Thanks for sharing. I now have 31 hours to create a decent NN model and blend it in with my LGBM :)", "439952": "This is doable, good luck!", "439970": "Ya that must be the reason. Then, I should move to nn model but I am afraid I am out of time. But I will give it a shot.", "439992": "I am blending three different models, one based in your contribution( thanks a lot,Jim), the second one an approach where i have separated extra and galactic objects ,each one with a different parametrization lightgbm, after that i have joined both predictions in an unique dataframe, and the third one a neural network in an unique model .\nBy now Lb 0.963 and still training two models with new features for tomorrow submission, until the last day working hard ;-), hoping to improve the score a little, at least to obtain bronze medal, that is being very expensive , in this my first competition.\nI would like to express my appreciation and give thanks to everyone that has participated in kernels and discussions. It has been great to share time with you. I wish you the best. Happy Christmas!!!", "440003": "Maybe i don't fully understand, but when you say that you worked on and NN and LGBM models without feature engineering, how are you able to use the data without any feature engineering? Particularlly given that the time series data is so disperate.", "440007": "I’m not saying I did no feature Engineering. I did a little in my SomethingDifferent kernel. I did a little more beyond that. I also used the features from Chia-ta Tsais kernel.\n\nI had some really cool features developed for the training set however I didn’t think I could process test with the computational resources available. In hindsight I wish I had tried.", "440008": "Without feature engineering? No, i haven't said that. I am looking for new features to improve my last models until tomorrow because if base models are better their blending will obtain best result , at least if they are not correlated.", "440038": "Just started with this. Did a linear blend of a 1.034 and a 1.014 model, which got me to 0.958. Considering the diversity of the two models, I feel some more juice could be extracted with a shallow NNet, DT, or Logistic ensembler instead. What's the best way to go about doing that? My understanding is:\n\n- Take multiclass OOF train predictions of both models (14+14=28)\n- Split into desired # of folds, where each fold need not necessarily be the same as the underlying lvl-1 model\n- Train the stacker model\n- Take multiclass submission predictions of both models (14+14=28)\n- Average the Fold predictions of the stacker model ran in inference on the above data\n- Fancy Class99 calculation\n- Profit?\n\nIs that right?", "440046": "> Is that right?\n\nI'm afraid there is only one way to knpw: do it and submit ;) Is that right?But what you say seems reasonable at least. It is not what we do, hence I can't be sure, but still, it looks reasonable.", "440047": "I tried but I got a 12.3 LB score, ROFL\n\nI'm praying it was due to my model \"source-control\" failing (e.g. wrong train oof predictions, oof preds being overwritten by a different run, etc), and I'm last-ditch re-running on my models tonight; but wanted to be sure on the procedure side. Looking forward to hearing your notes after the comp!\n\nGetting to the end of a competition is like getting to the end of a series you just solo binged watched. No idea what to do next in life (if there isn't another comp lined up), and feeling guilty about all the time spent neglecting one's significant other / kids / etc.", "440278": "How long approximately SVM is taking for training ?", "440565": "Ughhh I jacked up. This is what sleep coding does. What I was doing wrong:\n\nTrain data is split into 5 folds, and 5 copies of \"ModelA\" are CV'd on this data. The OOF predictions of train are therefore actually 20% slices of the traindata predicted using a _single_ model each. If I then take all 5 models and infer the entire test set with each model—averaging the results, what I end up with is a single 100% slice that is a non-weighted blend of all 5 models. This test set prediction and oof train prediction pair cannot be used as a base model in stacking because they're essentially different models, hence the 12.x LB explosion.\n\nWhat I should have done is, from the get go, partition the training data => {trainpart,holdoutpart}. _Then_ do as above, e.g. 5fold CV on the train_part and run inference on _both_ holdoutpart as well as the full test submission using all 5 folds averaged. The stacking model would then be trained on the 5 holdoutparts and would then be applied to the submission data that has already been ran through the same models.\n\nTimes up on this competition so it looks like my only choices for ensembling are linear blending of different models and creating more bags of my faster to train ones.", "440581": "SVM took a lot of resources. I went at it in two different ways. I built individual classifiers for each class (is 52, is not 52) and I also used the SVM multiclass classifier. The multiclass timed out if I used more than 5 folds. In order to predict test I had to split the data first by fold and then by rows (1.2million, 1.2million, rest).\n\nI also had to apply a custom function before the scores were decent." }, "source": "meta" }