No means to offend anyone, just want to have a better understanding of ensemble especially with rank if possible.
\nDays ealier, I saw some notebook suggest doing rank blend, and I followed and did on my own model.
\nI have 10 models now, and cv from 0.9 to 0.93 and lb from 0.93 to 0.95 each approximately, I use the bayesian Opitimization (from the public notebook) to find the best weight on my oof to maximize my auc. It seems that the every time I use the rank instead the origin, it will boost my cv for about 0.005. But the lb tend to decrease.
\nSo I am a little bit confused by the method. Since our score is based on the auc, our job should try to make the malignant in the test set be the highest 200-300 value(not sure whether it's right, correct me if I 'm wrong please),so imagine if I use 10 models and 7 of them are actually predicting right (predict value really in the highest 200 -300 value )on one malignant, is it more wise to use the rank or just mix them directly or any other better idea please?
(For 10 models, suppose each has a 90% to recognize one maligant
\n0.9^10 = 0.3486
\n0.9^9 0.110 = 0.3874
\n0.9^80.1^210C2 = 0.1937
\n0.9^70.1^310C3 = 0.0574
\nstop here, if we can make the above correct, it should be 0.98711 which is definitely competitive)
No means to offend anyone, just want to have a better understanding of ensemble especially with rank if possible.
\nDays ealier, I saw some notebook suggest doing rank blend, and I followed and did on my own model.
\nI have 10 models now, and cv from 0.9 to 0.93 and lb from 0.93 to 0.95 each approximately, I use the bayesian Opitimization (from the public notebook) to find the best weight on my oof to maximize my auc. It seems that the every time I use the rank instead the origin, it will boost my cv for about 0.005. But the lb tend to decrease.
\nSo I am a little bit confused by the method. Since our score is based on the auc, our job should try to make the malignant in the test set be the highest 200-300 value(not sure whether it's right, correct me if I 'm wrong please),so imagine if I use 10 models and 7 of them are actually predicting right (predict value really in the highest 200 -300 value )on one malignant, is it more wise to use the rank or just mix them directly or any other better idea please?
(For 10 models, suppose each has a 90% to recognize one maligant
\n0.9^10 = 0.3486
\n0.9^9 0.110 = 0.3874
\n0.9^80.1^210C2 = 0.1937
\n0.9^70.1^310C3 = 0.0574
\nstop here, if we can make the above correct, it should be 0.98711 which is definitely competitive)
For me, blending using something like Bayesian Opitimization only causes overfitting.
\nIf you see winner solutions in other comps, you will find simple average, unweighted gmean or rank average on several best single models are good enough.
Appreciate, I will probably make that change
", "rawMarkdown": "Appreciate, I will probably make that change", "votes": null }, { "id": "972558", "postDate": "08/16/2020 16:29:36", "content": "Different models train to a different level of confidence - I have 3 models which have different means and standard deviations of predicted probabilities. Blending them using \"rank\" would be a better way than a simple average blend.
\nThere are very few positive samples in this competition - Hence it would be incorrect to draw any conclusions from the various ensemble results. Predicting even a single \"1\" incorrectly would drop score as much as 0.0064. Given that there at least 700 ranks on leaderboard in a range of 0.006 (0.9600 is ranked@800 and 0.9660 is ranked@120) -it means shakeup is inevitable.
\nSometimes certain ensembles just work - but in general \"certain ways\" of doing ensembles are more right - If using different kinds of models, \"rank ensembles\" are more right in my view - at least in this competition.
", "rawMarkdown": "Different models train to a different level of confidence - I have 3 models which have different means and standard deviations of predicted probabilities. Blending them using \"rank\" would be a better way than a simple average blend.\n\nThere are very few positive samples in this competition - Hence it would be incorrect to draw any conclusions from the various ensemble results. Predicting even a single \"1\" incorrectly would drop score as much as 0.0064. Given that there at least 700 ranks on leaderboard in a range of 0.006 (0.9600 is ranked@800 and 0.9660 is ranked@120) -it means shakeup is inevitable. \n\nSometimes certain ensembles just work - but in general \"certain ways\" of doing ensembles are more right - If using different kinds of models, \"rank ensembles\" are more right in my view - at least in this competition.", "votes": null }, { "id": "972573", "postDate": "08/16/2020 16:40:46", "content": "Appreciate
", "rawMarkdown": "Appreciate", "votes": null }, { "id": "972583", "postDate": "08/16/2020 16:54:25", "content": "How about using a meta model for ensembling, with predictions from different models as input ?
", "rawMarkdown": "How about using a meta model for ensembling, with predictions from different models as input ?", "votes": null }, { "id": "972589", "postDate": "08/16/2020 16:57:30", "content": "Do you mean stacking?
", "rawMarkdown": "Do you mean stacking?", "votes": null }, { "id": "972639", "postDate": "08/16/2020 17:40:46", "content": "For me doesn't work also. Ranked blend is good for different scale models.
", "rawMarkdown": "For me doesn't work also. Ranked blend is good for different scale models.", "votes": null } ], "comments": [ { "id": 972438, "author_name": "vicioussong", "author_url": "", "post_date": "08/16/2020 14:55:49", "content": "For me, blending using something like Bayesian Opitimization only causes overfitting.
\nIf you see winner solutions in other comps, you will find simple average, unweighted gmean or rank average on several best single models are good enough.
Appreciate, I will probably make that change
", "votes": null, "replies": [] } ] }, { "id": 972558, "author_name": "watzisname", "author_url": "", "post_date": "08/16/2020 16:29:36", "content": "Different models train to a different level of confidence - I have 3 models which have different means and standard deviations of predicted probabilities. Blending them using \"rank\" would be a better way than a simple average blend.
\nThere are very few positive samples in this competition - Hence it would be incorrect to draw any conclusions from the various ensemble results. Predicting even a single \"1\" incorrectly would drop score as much as 0.0064. Given that there at least 700 ranks on leaderboard in a range of 0.006 (0.9600 is ranked@800 and 0.9660 is ranked@120) -it means shakeup is inevitable.
\nSometimes certain ensembles just work - but in general \"certain ways\" of doing ensembles are more right - If using different kinds of models, \"rank ensembles\" are more right in my view - at least in this competition.
", "votes": null, "replies": [ { "id": 972573, "author_name": "shenjiaxjtlu", "author_url": "", "post_date": "08/16/2020 16:40:46", "content": "Appreciate
", "votes": null, "replies": [] } ] }, { "id": 972583, "author_name": "ankitsajwan", "author_url": "", "post_date": "08/16/2020 16:54:25", "content": "How about using a meta model for ensembling, with predictions from different models as input ?
", "votes": null, "replies": [ { "id": 972589, "author_name": "shenjiaxjtlu", "author_url": "", "post_date": "08/16/2020 16:57:30", "content": "Do you mean stacking?
", "votes": null, "replies": [] } ] }, { "id": 972639, "author_name": "bfishh", "author_url": "", "post_date": "08/16/2020 17:40:46", "content": "For me doesn't work also. Ranked blend is good for different scale models.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "972399": "No means to offend anyone, just want to have a better understanding of ensemble especially with rank if possible. \nDays ealier, I saw some notebook suggest doing rank blend, and I followed and did on my own model.\nI have 10 models now, and cv from 0.9 to 0.93 and lb from 0.93 to 0.95 each approximately, I use the bayesian Opitimization (from the public notebook) to find the best weight on my oof to maximize my auc. It seems that the every time I use the rank instead the origin, it will boost my cv for about 0.005. But the lb tend to decrease.\nSo I am a little bit confused by the method. Since our score is based on the auc, our job should try to make the malignant in the test set be the highest 200-300 value(not sure whether it's right, correct me if I 'm wrong please),so imagine if I use 10 models and 7 of them are actually predicting right (predict value really in the highest 200 -300 value )on one malignant, is it more wise to use the rank or just mix them directly or any other better idea please? \n\n(For 10 models, suppose each has a 90% to recognize one maligant\n0.9^10 = 0.3486\n0.9^9 *0.1*10 = 0.3874\n0.9^8*0.1^2*10C2 = 0.1937\n0.9^7*0.1^3*10C3 = 0.0574\nstop here, if we can make the above correct, it should be 0.98711 which is definitely competitive)", "972438": "For me, blending using something like Bayesian Opitimization only causes overfitting.\nIf you see winner solutions in other comps, you will find simple average, unweighted gmean or rank average on several best single models are good enough.", "972460": "Appreciate, I will probably make that change", "972558": "Different models train to a different level of confidence - I have 3 models which have different means and standard deviations of predicted probabilities. Blending them using \"rank\" would be a better way than a simple average blend.\n\nThere are very few positive samples in this competition - Hence it would be incorrect to draw any conclusions from the various ensemble results. Predicting even a single \"1\" incorrectly would drop score as much as 0.0064. Given that there at least 700 ranks on leaderboard in a range of 0.006 (0.9600 is ranked@800 and 0.9660 is ranked@120) -it means shakeup is inevitable. \n\nSometimes certain ensembles just work - but in general \"certain ways\" of doing ensembles are more right - If using different kinds of models, \"rank ensembles\" are more right in my view - at least in this competition.", "972573": "Appreciate", "972583": "How about using a meta model for ensembling, with predictions from different models as input ?", "972589": "Do you mean stacking?", "972639": "For me doesn't work also. Ranked blend is good for different scale models." }, "source": "meta" }