{ "id": 103873, "title": "Best ensemble method ?", "url": "/competitions/aptos2019-blindness-detection/discussion/103873", "author_name": "", "post_date": "2019-08-12T15:45:55.341065300Z", "votes": 7, "comment_count": 22, "views": 0, "content": "

I was exploring different methods of ensembles. \n1. Ensemble average \n\n sum_pred = model_pred + model1_pred +model2_pred\n prediction = sum_pred / 3\n\n2. Horizontal voting ensemble (https://machinelearningmastery.com/horizontal-voting-ensemble/)\n3. Weighted ensemble\nnew_pred = (model_preds *0.4) + (model1_preds*0.6)

\n\n

Which one do you think is the best and the most appropriate for this competition?\n(I tried weighted ensemble and didn't get much of a boost to my score.)

", "messages": [ { "id": "597644", "postDate": "08/12/2019 15:45:55", "content": "

I was exploring different methods of ensembles. \n1. Ensemble average \n\n sum_pred = model_pred + model1_pred +model2_pred\n prediction = sum_pred / 3\n\n2. Horizontal voting ensemble (https://machinelearningmastery.com/horizontal-voting-ensemble/)\n3. Weighted ensemble\nnew_pred = (model_preds *0.4) + (model1_preds*0.6)

\n\n

Which one do you think is the best and the most appropriate for this competition?\n(I tried weighted ensemble and didn't get much of a boost to my score.)

", "rawMarkdown": "I was exploring different methods of ensembles. \n1. Ensemble average \n```\n sum_pred = model_pred + model1_pred +model2_pred\n prediction = sum_pred / 3\n```\n2. Horizontal voting ensemble (https://machinelearningmastery.com/horizontal-voting-ensemble/)\n3. Weighted ensemble\n`new_pred = (model_preds *0.4) + (model1_preds*0.6)`\n\nWhich one do you think is the best and the most appropriate for this competition?\n(I tried weighted ensemble and didn't get much of a boost to my score.)", "votes": null }, { "id": "598311", "postDate": "08/13/2019 12:57:47", "content": "

Hi Kyrylo,

\n\n

Have you thought about stacking by building another model (for example a random forest or GBM) on top of your predictions? Out of the three you mentioned I think a weighted ensemble works well since optimizing Quadratic Weighted Kappa lends itself well to regression. After that you can optimize the rounding thresholds. For more information about rounding thresholds check out the \"Evaluation\" section of this kernel:\nhttps://www.kaggle.com/carlolepelaars/efficientnetb3-with-keras-aptos-2019

\n\n

There is a great video by Marios Michailidis where he explains model stacking and provides a framework for it:\nhttps://youtu.be/9Vk1rXLhG48

\n\n

Hope this helps!

", "rawMarkdown": "Hi Kyrylo, \n\nHave you thought about stacking by building another model (for example a random forest or GBM) on top of your predictions? Out of the three you mentioned I think a weighted ensemble works well since optimizing Quadratic Weighted Kappa lends itself well to regression. After that you can optimize the rounding thresholds. For more information about rounding thresholds check out the \"Evaluation\" section of this kernel:\nhttps://www.kaggle.com/carlolepelaars/efficientnetb3-with-keras-aptos-2019\n\nThere is a great video by Marios Michailidis where he explains model stacking and provides a framework for it:\nhttps://youtu.be/9Vk1rXLhG48\n\nHope this helps!", "votes": null }, { "id": "598324", "postDate": "08/13/2019 13:12:55", "content": "

Whatever works best for you :) There is never one clear thing to recommend with blending imho.

", "rawMarkdown": "Whatever works best for you :) There is never one clear thing to recommend with blending imho.", "votes": null }, { "id": "598381", "postDate": "08/13/2019 14:33:31", "content": "

Thanks !

", "rawMarkdown": "Thanks !", "votes": null }, { "id": "598421", "postDate": "08/13/2019 14:57:18", "content": "

You're welcome!

", "rawMarkdown": "You're welcome!", "votes": null }, { "id": "598431", "postDate": "08/13/2019 15:10:27", "content": "

I use following blender , It gives me really good boost. Hope it will help you as well =)

", "rawMarkdown": "I use following [blender](https://www.vitamix.com/us/en_us/Shop/Certified-Reconditioned-Standard?cid=ppc-Evergreen-Google-SmartShoppingFeed&coupon=07-0063&ef_id=Cj0KCQjwv8nqBRDGARIsAHfR9wCLYgH23f4IKcmCOO3VHkGcO6Tm_L4zgnJPNKkN9Cq5roEtTQSNHcgaAtd8EALw_wcB:G:s&s_kwcid=AL!6700!3!354251602010!!!u!760203275654!&gclid=Cj0KCQjwv8nqBRDGARIsAHfR9wCLYgH23f4IKcmCOO3VHkGcO6Tm_L4zgnJPNKkN9Cq5roEtTQSNHcgaAtd8EALw_wcB) , It gives me really good boost. Hope it will help you as well =)", "votes": null }, { "id": "598440", "postDate": "08/13/2019 15:24:03", "content": "

https://www.youtube.com/watch?v=lAl28d6tbko

", "rawMarkdown": "https://www.youtube.com/watch?v=lAl28d6tbko", "votes": null }, { "id": "598441", "postDate": "08/13/2019 15:24:44", "content": "

Hope you didn't lie....

", "rawMarkdown": "Hope you didn't lie....", "votes": null }, { "id": "598464", "postDate": "08/13/2019 15:49:51", "content": "

Will It Blend?

", "rawMarkdown": "Will It Blend?", "votes": null }, { "id": "598466", "postDate": "08/13/2019 15:52:25", "content": "

Unfortunately, i think that @drhabib provided a better answer ...

", "rawMarkdown": "Unfortunately, i think that @drhabib provided a better answer ...", "votes": null }, { "id": "598497", "postDate": "08/13/2019 16:39:51", "content": "

Haha!! Too bad blender is not open source!

\n\n

One last suggestion: You can also try pystacknet for ensembling:\nhttps://github.com/h2oai/pystacknet

", "rawMarkdown": "Haha!! Too bad [blender](https://www.vitamix.com/us/en_us/Shop/Certified-Reconditioned-Standard?cid=ppc-Evergreen-Google-SmartShoppingFeed&coupon=07-0063&ef_id=Cj0KCQjwv8nqBRDGARIsAHfR9wCLYgH23f4IKcmCOO3VHkGcO6Tm_L4zgnJPNKkN9Cq5roEtTQSNHcgaAtd8EALw_wcB:G:s&s_kwcid=AL!6700!3!354251602010!!!u!760203275654!&gclid=Cj0KCQjwv8nqBRDGARIsAHfR9wCLYgH23f4IKcmCOO3VHkGcO6Tm_L4zgnJPNKkN9Cq5roEtTQSNHcgaAtd8EALw_wcB) is not open source!\n\nOne last suggestion: You can also try [pystacknet](https://github.com/h2oai/pystacknet) for ensembling:\nhttps://github.com/h2oai/pystacknet", "votes": null }, { "id": "598575", "postDate": "08/13/2019 18:33:16", "content": "

Thanks again!!

", "rawMarkdown": "Thanks again!!", "votes": null }, { "id": "598592", "postDate": "08/13/2019 18:55:24", "content": "

๐Ÿ˜€ Good luck with stacking!

", "rawMarkdown": "๐Ÿ˜€ Good luck with stacking!", "votes": null }, { "id": "598769", "postDate": "08/14/2019 02:31:15", "content": "

๐Ÿ˜‚ Best blending from @drhabib + @philippsinger (the price is a bit expensive though)

\n\n

PS. Regarding the topic, I havenโ€™t tried ensembling different models yet. But for K-Folds, just simple average works for me.

", "rawMarkdown": "๐Ÿ˜‚ Best blending from @drhabib + @philippsinger (the price is a bit expensive though)\n\nPS. Regarding the topic, I havenโ€™t tried ensembling different models yet. But for K-Folds, just simple average works for me.", "votes": null }, { "id": "598807", "postDate": "08/14/2019 03:53:55", "content": "

Well i can say ensembling definitely helped me. Try everything and see which works out. Be careful and try not to overfit the public LB as some people believe there will be quite a bit of mixup when the private scores are released(I dont really know tho)

", "rawMarkdown": "Well i can say ensembling definitely helped me. Try everything and see which works out. Be careful and try not to overfit the public LB as some people believe there will be quite a bit of mixup when the private scores are released(I dont really know tho)", "votes": null }, { "id": "598896", "postDate": "08/14/2019 07:47:08", "content": "

It's worth investing into a good blend in order to win this competition.

", "rawMarkdown": "It's worth investing into a good blend in order to win this competition.", "votes": null }, { "id": "599018", "postDate": "08/14/2019 11:45:00", "content": "

Simply use 5-fold I get 5 models, then I ensemble them with average weight, it can bring me round 0.7-0.8 improve

", "rawMarkdown": "Simply use 5-fold I get 5 models, then I ensemble them with average weight, it can bring me round 0.7-0.8 improve", "votes": null }, { "id": "599182", "postDate": "08/14/2019 16:14:32", "content": "

@haofanwang are you doing all the computations in kaggle kernels?

", "rawMarkdown": "haofanwang are you doing all the computations in kaggle kernels?", "votes": null }, { "id": "599216", "postDate": "08/14/2019 17:05:02", "content": "

I'm training the model locally, and just do inference in kernel.

", "rawMarkdown": "I'm training the model locally, and just do inference in kernel.", "votes": null }, { "id": "599234", "postDate": "08/14/2019 17:37:05", "content": "

So, you trained 5 models locally and then just get average weight on kernel?

", "rawMarkdown": "So, you trained 5 models locally and then just get average weight on kernel?", "votes": null }, { "id": "599246", "postDate": "08/14/2019 18:07:29", "content": "

Yes.

", "rawMarkdown": "Yes.", "votes": null }, { "id": "599250", "postDate": "08/14/2019 18:12:05", "content": "

Actually I just prepare to do as this, now I just do inference on locally for a fast submit.

", "rawMarkdown": "Actually I just prepare to do as this, now I just do inference on locally for a fast submit.", "votes": null }, { "id": "599497", "postDate": "08/15/2019 03:39:43", "content": "

joke

", "rawMarkdown": "joke", "votes": null } ], "comments": [ { "id": 598311, "author_name": "carlolepelaars", "author_url": "", "post_date": "08/13/2019 12:57:47", "content": "

Hi Kyrylo,

\n\n

Have you thought about stacking by building another model (for example a random forest or GBM) on top of your predictions? Out of the three you mentioned I think a weighted ensemble works well since optimizing Quadratic Weighted Kappa lends itself well to regression. After that you can optimize the rounding thresholds. For more information about rounding thresholds check out the \"Evaluation\" section of this kernel:\nhttps://www.kaggle.com/carlolepelaars/efficientnetb3-with-keras-aptos-2019

\n\n

There is a great video by Marios Michailidis where he explains model stacking and provides a framework for it:\nhttps://youtu.be/9Vk1rXLhG48

\n\n

Hope this helps!

", "votes": null, "replies": [ { "id": 598381, "author_name": "kir486680", "author_url": "", "post_date": "08/13/2019 14:33:31", "content": "

Thanks !

", "votes": null, "replies": [] }, { "id": 598421, "author_name": "carlolepelaars", "author_url": "", "post_date": "08/13/2019 14:57:18", "content": "

You're welcome!

", "votes": null, "replies": [] }, { "id": 598466, "author_name": "kir486680", "author_url": "", "post_date": "08/13/2019 15:52:25", "content": "

Unfortunately, i think that @drhabib provided a better answer ...

", "votes": null, "replies": [] }, { "id": 598497, "author_name": "carlolepelaars", "author_url": "", "post_date": "08/13/2019 16:39:51", "content": "

Haha!! Too bad blender is not open source!

\n\n

One last suggestion: You can also try pystacknet for ensembling:\nhttps://github.com/h2oai/pystacknet

", "votes": null, "replies": [] }, { "id": 598575, "author_name": "kir486680", "author_url": "", "post_date": "08/13/2019 18:33:16", "content": "

Thanks again!!

", "votes": null, "replies": [] }, { "id": 598592, "author_name": "carlolepelaars", "author_url": "", "post_date": "08/13/2019 18:55:24", "content": "

๐Ÿ˜€ Good luck with stacking!

", "votes": null, "replies": [] } ] }, { "id": 598324, "author_name": "philippsinger", "author_url": "", "post_date": "08/13/2019 13:12:55", "content": "

Whatever works best for you :) There is never one clear thing to recommend with blending imho.

", "votes": null, "replies": [] }, { "id": 598431, "author_name": "drhabib", "author_url": "", "post_date": "08/13/2019 15:10:27", "content": "

I use following blender , It gives me really good boost. Hope it will help you as well =)

", "votes": null, "replies": [ { "id": 598440, "author_name": "philippsinger", "author_url": "", "post_date": "08/13/2019 15:24:03", "content": "

https://www.youtube.com/watch?v=lAl28d6tbko

", "votes": null, "replies": [] }, { "id": 598441, "author_name": "kir486680", "author_url": "", "post_date": "08/13/2019 15:24:44", "content": "

Hope you didn't lie....

", "votes": null, "replies": [] }, { "id": 598464, "author_name": "kir486680", "author_url": "", "post_date": "08/13/2019 15:49:51", "content": "

Will It Blend?

", "votes": null, "replies": [] }, { "id": 598769, "author_name": "ratthachat", "author_url": "", "post_date": "08/14/2019 02:31:15", "content": "

๐Ÿ˜‚ Best blending from @drhabib + @philippsinger (the price is a bit expensive though)

\n\n

PS. Regarding the topic, I havenโ€™t tried ensembling different models yet. But for K-Folds, just simple average works for me.

", "votes": null, "replies": [] }, { "id": 598896, "author_name": "kir486680", "author_url": "", "post_date": "08/14/2019 07:47:08", "content": "

It's worth investing into a good blend in order to win this competition.

", "votes": null, "replies": [] }, { "id": 599497, "author_name": "jiangkun2", "author_url": "", "post_date": "08/15/2019 03:39:43", "content": "

joke

", "votes": null, "replies": [] } ] }, { "id": 598807, "author_name": "sidhanthholalkere", "author_url": "", "post_date": "08/14/2019 03:53:55", "content": "

Well i can say ensembling definitely helped me. Try everything and see which works out. Be careful and try not to overfit the public LB as some people believe there will be quite a bit of mixup when the private scores are released(I dont really know tho)

", "votes": null, "replies": [] }, { "id": 599018, "author_name": "haofanwang", "author_url": "", "post_date": "08/14/2019 11:45:00", "content": "

Simply use 5-fold I get 5 models, then I ensemble them with average weight, it can bring me round 0.7-0.8 improve

", "votes": null, "replies": [ { "id": 599182, "author_name": "prashantkikani", "author_url": "", "post_date": "08/14/2019 16:14:32", "content": "

@haofanwang are you doing all the computations in kaggle kernels?

", "votes": null, "replies": [] }, { "id": 599216, "author_name": "haofanwang", "author_url": "", "post_date": "08/14/2019 17:05:02", "content": "

I'm training the model locally, and just do inference in kernel.

", "votes": null, "replies": [] }, { "id": 599234, "author_name": "kir486680", "author_url": "", "post_date": "08/14/2019 17:37:05", "content": "

So, you trained 5 models locally and then just get average weight on kernel?

", "votes": null, "replies": [] }, { "id": 599246, "author_name": "haofanwang", "author_url": "", "post_date": "08/14/2019 18:07:29", "content": "

Yes.

", "votes": null, "replies": [] }, { "id": 599250, "author_name": "haofanwang", "author_url": "", "post_date": "08/14/2019 18:12:05", "content": "

Actually I just prepare to do as this, now I just do inference on locally for a fast submit.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "597644": "I was exploring different methods of ensembles. \n1. Ensemble average \n```\n sum_pred = model_pred + model1_pred +model2_pred\n prediction = sum_pred / 3\n```\n2. Horizontal voting ensemble (https://machinelearningmastery.com/horizontal-voting-ensemble/)\n3. Weighted ensemble\n`new_pred = (model_preds *0.4) + (model1_preds*0.6)`\n\nWhich one do you think is the best and the most appropriate for this competition?\n(I tried weighted ensemble and didn't get much of a boost to my score.)", "598311": "Hi Kyrylo, \n\nHave you thought about stacking by building another model (for example a random forest or GBM) on top of your predictions? Out of the three you mentioned I think a weighted ensemble works well since optimizing Quadratic Weighted Kappa lends itself well to regression. After that you can optimize the rounding thresholds. For more information about rounding thresholds check out the \"Evaluation\" section of this kernel:\nhttps://www.kaggle.com/carlolepelaars/efficientnetb3-with-keras-aptos-2019\n\nThere is a great video by Marios Michailidis where he explains model stacking and provides a framework for it:\nhttps://youtu.be/9Vk1rXLhG48\n\nHope this helps!", "598324": "Whatever works best for you :) There is never one clear thing to recommend with blending imho.", "598381": "Thanks !", "598421": "You're welcome!", "598431": "I use following [blender](https://www.vitamix.com/us/en_us/Shop/Certified-Reconditioned-Standard?cid=ppc-Evergreen-Google-SmartShoppingFeed&coupon=07-0063&ef_id=Cj0KCQjwv8nqBRDGARIsAHfR9wCLYgH23f4IKcmCOO3VHkGcO6Tm_L4zgnJPNKkN9Cq5roEtTQSNHcgaAtd8EALw_wcB:G:s&s_kwcid=AL!6700!3!354251602010!!!u!760203275654!&gclid=Cj0KCQjwv8nqBRDGARIsAHfR9wCLYgH23f4IKcmCOO3VHkGcO6Tm_L4zgnJPNKkN9Cq5roEtTQSNHcgaAtd8EALw_wcB) , It gives me really good boost. Hope it will help you as well =)", "598440": "https://www.youtube.com/watch?v=lAl28d6tbko", "598441": "Hope you didn't lie....", "598464": "Will It Blend?", "598466": "Unfortunately, i think that @drhabib provided a better answer ...", "598497": "Haha!! Too bad [blender](https://www.vitamix.com/us/en_us/Shop/Certified-Reconditioned-Standard?cid=ppc-Evergreen-Google-SmartShoppingFeed&coupon=07-0063&ef_id=Cj0KCQjwv8nqBRDGARIsAHfR9wCLYgH23f4IKcmCOO3VHkGcO6Tm_L4zgnJPNKkN9Cq5roEtTQSNHcgaAtd8EALw_wcB:G:s&s_kwcid=AL!6700!3!354251602010!!!u!760203275654!&gclid=Cj0KCQjwv8nqBRDGARIsAHfR9wCLYgH23f4IKcmCOO3VHkGcO6Tm_L4zgnJPNKkN9Cq5roEtTQSNHcgaAtd8EALw_wcB) is not open source!\n\nOne last suggestion: You can also try [pystacknet](https://github.com/h2oai/pystacknet) for ensembling:\nhttps://github.com/h2oai/pystacknet", "598575": "Thanks again!!", "598592": "๐Ÿ˜€ Good luck with stacking!", "598769": "๐Ÿ˜‚ Best blending from @drhabib + @philippsinger (the price is a bit expensive though)\n\nPS. Regarding the topic, I havenโ€™t tried ensembling different models yet. But for K-Folds, just simple average works for me.", "598807": "Well i can say ensembling definitely helped me. Try everything and see which works out. Be careful and try not to overfit the public LB as some people believe there will be quite a bit of mixup when the private scores are released(I dont really know tho)", "598896": "It's worth investing into a good blend in order to win this competition.", "599018": "Simply use 5-fold I get 5 models, then I ensemble them with average weight, it can bring me round 0.7-0.8 improve", "599182": "haofanwang are you doing all the computations in kaggle kernels?", "599216": "I'm training the model locally, and just do inference in kernel.", "599234": "So, you trained 5 models locally and then just get average weight on kernel?", "599246": "Yes.", "599250": "Actually I just prepare to do as this, now I just do inference on locally for a fast submit.", "599497": "joke" }, "source": "meta" }