Has anyone else had success using ensemble? I know a few people were talking about how ensemble decreased their score.\nFor me, ensembling has given me a LB boost of 0.012 relative to the best model in the ensemble.\nPlease share your ensemble results below!
", "messages": [ { "id": "594213", "postDate": "08/07/2019 17:48:29", "content": "Has anyone else had success using ensemble? I know a few people were talking about how ensemble decreased their score.\nFor me, ensembling has given me a LB boost of 0.012 relative to the best model in the ensemble.\nPlease share your ensemble results below!
", "rawMarkdown": "Has anyone else had success using ensemble? I know a few people were talking about how ensemble decreased their score.\nFor me, ensembling has given me a LB boost of 0.012 relative to the best model in the ensemble.\nPlease share your ensemble results below!", "votes": null }, { "id": "607414", "postDate": "08/25/2019 07:42:15", "content": "3 models scoring [one at ~0.79, two at ~0.78], boosted to 0799. Weird enough replacing one of the 0.78 model with a worse performing 0.75 bumped it up to 0.8. If you don't mind answering, what were the individual model scores?
", "rawMarkdown": "3 models scoring [one at ~0.79, two at ~0.78], boosted to 0799. Weird enough replacing one of the 0.78 model with a worse performing 0.75 bumped it up to 0.8. If you don't mind answering, what were the individual model scores?", "votes": null }, { "id": "615639", "postDate": "09/02/2019 07:44:29", "content": "@joonl04 what about ensembling regression models? Any ideas on that?
", "rawMarkdown": "joonl04 what about ensembling regression models? Any ideas on that?", "votes": null }, { "id": "615646", "postDate": "09/02/2019 07:52:06", "content": "@yousof9 all my models are regression models :) the models used in ensemble only vary by preprocessing and architecture. I have added 3 more models since I made that comment, all scoring around ~0.79
", "rawMarkdown": "yousof9 all my models are regression models :) the models used in ensemble only vary by preprocessing and architecture. I have added 3 more models since I made that comment, all scoring around ~0.79", "votes": null }, { "id": "615678", "postDate": "09/02/2019 08:37:27", "content": "@joonl04 How did you ensemble regression models? I am confused on what would be the best way to do it. Simple averaging doesn't seem like a great idea. 1 and 4 gives 2.5 average. What does that even mean?
", "rawMarkdown": "joonl04 How did you ensemble regression models? I am confused on what would be the best way to do it. Simple averaging doesn't seem like a great idea. 1 and 4 gives 2.5 average. What does that even mean?", "votes": null }, { "id": "615726", "postDate": "09/02/2019 09:47:24", "content": "I share my ensemble evidence and perspective : I believe ensemble in regression makes sense.
\n\nFirst of all, base models should be consistent in prediction, i.e. though two models may predict different severity levels, but their predictions should almost always within +/- 1 level. In this case, it makes sense to ensemble in a soft prediction ... In our cases, this condition is satisfied for all high-performance models.
\n\nNormally in our cases, ensemble is most effective when one model is \"unconfidence\".
\n\none uncertain case of one model, i.e. soft prediction of 2.46 , indicating \"unconfidence\" whether an eye should be class 2 or class 3 (with threshold 2.5), will be helped by another model which is more confident e.g. predicting 3.04 ... So the average is 2.75 and the final prediction is class 3.
\n\nThese base models are of comparable performances, and quite diverse (wrt training process), so ensemble makes sense in my opinion.
", "rawMarkdown": "I share my ensemble evidence and perspective : I believe ensemble in regression makes sense.\n\nFirst of all, base models should be consistent in prediction, i.e. though two models may predict different severity levels, but their predictions should almost always within +/- 1 level. In this case, it makes sense to ensemble in a soft prediction ... In our cases, this condition is satisfied for all high-performance models.\n\nNormally in our cases, **ensemble is most effective when one model is \"unconfidence\"**.\n\none uncertain case of one model, i.e. soft prediction of 2.46 , indicating \"unconfidence\" whether an eye should be class 2 or class 3 (with threshold 2.5), will be helped by another model which is more confident e.g. predicting 3.04 ... So the average is 2.75 and the final prediction is class 3. \n\nThese base models are of comparable performances, and quite diverse (wrt training process), so ensemble makes sense in my opinion.", "votes": null }, { "id": "615762", "postDate": "09/02/2019 10:56:33", "content": "@ratthachat Do you cap soft predictions at 4 and 0? Otherwise they might be some cases where you get a soft prediction of 9, swaying all other predictions..
", "rawMarkdown": "ratthachat Do you cap soft predictions at 4 and 0? Otherwise they might be some cases where you get a soft prediction of 9, swaying all other predictions..", "votes": null }, { "id": "615779", "postDate": "09/02/2019 11:22:18", "content": "I use ordinal regression where the output always in [0,4]. My teammate's model use standard regression and he clip the output to [0,4]. The two methods can combine smoothly. ;)
", "rawMarkdown": "I use ordinal regression where the output always in [0,4]. My teammate's model use standard regression and he clip the output to [0,4]. The two methods can combine smoothly. ;)", "votes": null }, { "id": "616263", "postDate": "09/02/2019 23:14:49", "content": "Cool. I'll give it a go. Thank you. =)
", "rawMarkdown": "Cool. I'll give it a go. Thank you. =)", "votes": null }, { "id": "616364", "postDate": "09/03/2019 03:56:54", "content": "I think you should average before thresholding.
", "rawMarkdown": "I think you should average before thresholding.", "votes": null } ], "comments": [ { "id": 607414, "author_name": "joonl04", "author_url": "", "post_date": "08/25/2019 07:42:15", "content": "3 models scoring [one at ~0.79, two at ~0.78], boosted to 0799. Weird enough replacing one of the 0.78 model with a worse performing 0.75 bumped it up to 0.8. If you don't mind answering, what were the individual model scores?
", "votes": null, "replies": [ { "id": 615639, "author_name": "yousof9", "author_url": "", "post_date": "09/02/2019 07:44:29", "content": "@joonl04 what about ensembling regression models? Any ideas on that?
", "votes": null, "replies": [] }, { "id": 615646, "author_name": "joonl04", "author_url": "", "post_date": "09/02/2019 07:52:06", "content": "@yousof9 all my models are regression models :) the models used in ensemble only vary by preprocessing and architecture. I have added 3 more models since I made that comment, all scoring around ~0.79
", "votes": null, "replies": [] }, { "id": 615678, "author_name": "yousof9", "author_url": "", "post_date": "09/02/2019 08:37:27", "content": "@joonl04 How did you ensemble regression models? I am confused on what would be the best way to do it. Simple averaging doesn't seem like a great idea. 1 and 4 gives 2.5 average. What does that even mean?
", "votes": null, "replies": [] }, { "id": 616364, "author_name": "vishnus", "author_url": "", "post_date": "09/03/2019 03:56:54", "content": "I think you should average before thresholding.
", "votes": null, "replies": [] } ] }, { "id": 615726, "author_name": "ratthachat", "author_url": "", "post_date": "09/02/2019 09:47:24", "content": "I share my ensemble evidence and perspective : I believe ensemble in regression makes sense.
\n\nFirst of all, base models should be consistent in prediction, i.e. though two models may predict different severity levels, but their predictions should almost always within +/- 1 level. In this case, it makes sense to ensemble in a soft prediction ... In our cases, this condition is satisfied for all high-performance models.
\n\nNormally in our cases, ensemble is most effective when one model is \"unconfidence\".
\n\none uncertain case of one model, i.e. soft prediction of 2.46 , indicating \"unconfidence\" whether an eye should be class 2 or class 3 (with threshold 2.5), will be helped by another model which is more confident e.g. predicting 3.04 ... So the average is 2.75 and the final prediction is class 3.
\n\nThese base models are of comparable performances, and quite diverse (wrt training process), so ensemble makes sense in my opinion.
", "votes": null, "replies": [ { "id": 615762, "author_name": "yousof9", "author_url": "", "post_date": "09/02/2019 10:56:33", "content": "@ratthachat Do you cap soft predictions at 4 and 0? Otherwise they might be some cases where you get a soft prediction of 9, swaying all other predictions..
", "votes": null, "replies": [] }, { "id": 615779, "author_name": "ratthachat", "author_url": "", "post_date": "09/02/2019 11:22:18", "content": "I use ordinal regression where the output always in [0,4]. My teammate's model use standard regression and he clip the output to [0,4]. The two methods can combine smoothly. ;)
", "votes": null, "replies": [] }, { "id": 616263, "author_name": "yousof9", "author_url": "", "post_date": "09/02/2019 23:14:49", "content": "Cool. I'll give it a go. Thank you. =)
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "594213": "Has anyone else had success using ensemble? I know a few people were talking about how ensemble decreased their score.\nFor me, ensembling has given me a LB boost of 0.012 relative to the best model in the ensemble.\nPlease share your ensemble results below!", "607414": "3 models scoring [one at ~0.79, two at ~0.78], boosted to 0799. Weird enough replacing one of the 0.78 model with a worse performing 0.75 bumped it up to 0.8. If you don't mind answering, what were the individual model scores?", "615639": "joonl04 what about ensembling regression models? Any ideas on that?", "615646": "yousof9 all my models are regression models :) the models used in ensemble only vary by preprocessing and architecture. I have added 3 more models since I made that comment, all scoring around ~0.79", "615678": "joonl04 How did you ensemble regression models? I am confused on what would be the best way to do it. Simple averaging doesn't seem like a great idea. 1 and 4 gives 2.5 average. What does that even mean?", "615726": "I share my ensemble evidence and perspective : I believe ensemble in regression makes sense.\n\nFirst of all, base models should be consistent in prediction, i.e. though two models may predict different severity levels, but their predictions should almost always within +/- 1 level. In this case, it makes sense to ensemble in a soft prediction ... In our cases, this condition is satisfied for all high-performance models.\n\nNormally in our cases, **ensemble is most effective when one model is \"unconfidence\"**.\n\none uncertain case of one model, i.e. soft prediction of 2.46 , indicating \"unconfidence\" whether an eye should be class 2 or class 3 (with threshold 2.5), will be helped by another model which is more confident e.g. predicting 3.04 ... So the average is 2.75 and the final prediction is class 3. \n\nThese base models are of comparable performances, and quite diverse (wrt training process), so ensemble makes sense in my opinion.", "615762": "ratthachat Do you cap soft predictions at 4 and 0? Otherwise they might be some cases where you get a soft prediction of 9, swaying all other predictions..", "615779": "I use ordinal regression where the output always in [0,4]. My teammate's model use standard regression and he clip the output to [0,4]. The two methods can combine smoothly. ;)", "616263": "Cool. I'll give it a go. Thank you. =)", "616364": "I think you should average before thresholding." }, "source": "meta" }