{ "id": 337525, "title": "Optuna to find optimal weights", "url": "/competitions/amex-default-prediction/discussion/337525", "author_name": "vivekjan14", "post_date": "2022-07-16T13:34:35.198000", "votes": 19, "comment_count": 14, "views": 0, "content": "

Hi, Does optuna work well to find optimal weights for ensembling? Something like this:

\n
def objective(trial):\n  w1 = trial.suggest_float(\"w1\",-10,10)\n  w2 = trial.suggest_float(\"w2\",-10,10)\n  w3 = trial.suggest_float(\"w3\",-10,10)\n  w4 = trial.suggest_float(\"w4\",-10,10)\n  w5 = trial.suggest_float(\"w5\",-10,10)\n  val_preds = (w1*train[\"goss\"] + w2*train[\"xgboost\"] + w3*train[\"gbdt\"] + w4*train[\"dart\"]+w5*train[\"cat_boost\"])/(w1+w2+w3+w4+w5)\n  metric = amex_metric(train[CFG.target],val_preds)\n  return metric\nstudy = optuna.create_study(direction=\"maximize\")\nstudy.optimize(objective,n_trials=2000)\n
\n

Or is there a better way to do it? Thanks.

", "messages": [ { "id": 1857899, "postDate": "2022-07-16T13:34:35.200Z", "content": "

Hi, Does optuna work well to find optimal weights for ensembling? Something like this:

\n
def objective(trial):\n  w1 = trial.suggest_float(\"w1\",-10,10)\n  w2 = trial.suggest_float(\"w2\",-10,10)\n  w3 = trial.suggest_float(\"w3\",-10,10)\n  w4 = trial.suggest_float(\"w4\",-10,10)\n  w5 = trial.suggest_float(\"w5\",-10,10)\n  val_preds = (w1*train[\"goss\"] + w2*train[\"xgboost\"] + w3*train[\"gbdt\"] + w4*train[\"dart\"]+w5*train[\"cat_boost\"])/(w1+w2+w3+w4+w5)\n  metric = amex_metric(train[CFG.target],val_preds)\n  return metric\nstudy = optuna.create_study(direction=\"maximize\")\nstudy.optimize(objective,n_trials=2000)\n
\n

Or is there a better way to do it? Thanks.

", "rawMarkdown": "Hi, Does optuna work well to find optimal weights for ensembling? Something like this:\n\n```\ndef objective(trial):\n w1 = trial.suggest_float(\"w1\",-10,10)\n w2 = trial.suggest_float(\"w2\",-10,10)\n w3 = trial.suggest_float(\"w3\",-10,10)\n w4 = trial.suggest_float(\"w4\",-10,10)\n w5 = trial.suggest_float(\"w5\",-10,10)\n val_preds = (w1*train[\"goss\"] + w2*train[\"xgboost\"] + w3*train[\"gbdt\"] + w4*train[\"dart\"]+w5*train[\"cat_boost\"])/(w1+w2+w3+w4+w5)\n metric = amex_metric(train[CFG.target],val_preds)\n return metric\nstudy = optuna.create_study(direction=\"maximize\")\nstudy.optimize(objective,n_trials=2000)\n```\nOr is there a better way to do it? Thanks.", "votes": 19 }, { "id": 1868891, "postDate": "2022-07-24T09:58:38.483Z", "content": "

@tilii7 @aquatic @xiaowangiiiii @vivekjan14 Thank you for challenging my statement about negative weights! I've thought about it again and now agree that negative weights are not necessarily bad.

", "rawMarkdown": "@tilii7 @aquatic @xiaowangiiiii @vivekjan14 Thank you for challenging my statement about negative weights! I've thought about it again and now agree that negative weights are not necessarily bad.", "votes": 5 }, { "id": 1858802, "postDate": "2022-07-17T08:09:28.643Z", "content": "

In theory optuna should be able to find any function, but that only holds given a large enough number of iterations. I don't think you would get an optimal solution with 2000 trials.

\n

Besides, both @aquatic and @ambrosm have given you solutions that are more stable numerically and are much more likely to converge to an optimal solution.

\n

There is one suggestion that is not consistent with my experience.

\n
\n

I'd strongly recommend, however, that you use only nonnegative weights. If you get negative weights in an ensemble, something has gone wrong.

\n
\n

I used to subscribe to this point of view, but not any more. I find it that allowing negative weights often leads to better solution, and in fact I have a notebook from a different competition where the same solution proposed by @ambrosm has been employed, but with negative weights allowed.

\n

https://www.kaggle.com/code/tilii7/cross-validation-weighted-linear-blending-errors

\n

The blend of three models from that notebook ends up looking like this:

\n
 Your final model:\n -0.430584 * model-1\n 0.952081 * model-2\n 0.478912 * model-3\n
\n

There was a constraint in that notebook that all weights sum to 1, which I also think is unnecessary. In fact, if you choose to employ what @aquatic suggested and set fit_intercept=True, it is reasonable to expect that some of the weights will end up being negative. This is how one of my blends looks like when using logistic regression to find weights:

\n
 Your final model:\n  [-4.07375987] + ( 1.02083256 * model-1 ) + ( 3.41645157 * model-2 ) + ( 1.29123727 * model-3 ) + ( 1.81433148 * model-4 ) + ( -0.25400735 * model-5 ) + ( 0.80850964 * model-6 ) + ( 0.25919084 * model-7 ) + ( -0.04247169 * model-8 )\n
\n

The first number in that equation is a fixed value for intercept.

", "rawMarkdown": "In theory optuna should be able to find any function, but that only holds given a large enough number of iterations. I don't think you would get an optimal solution with 2000 trials.\n\nBesides, both @aquatic and @ambrosm have given you solutions that are more stable numerically and are much more likely to converge to an optimal solution.\n\nThere is one suggestion that is not consistent with my experience.\n\n> I'd strongly recommend, however, that you use only nonnegative weights. If you get negative weights in an ensemble, something has gone wrong.\n\nI used to subscribe to this point of view, but not any more. I find it that allowing negative weights often leads to better solution, and in fact I have a notebook from a different competition where the same solution proposed by @ambrosm has been employed, but with negative weights allowed.\n\nhttps://www.kaggle.com/code/tilii7/cross-validation-weighted-linear-blending-errors\n\nThe blend of three models from that notebook ends up looking like this:\n\n```\n Your final model:\n -0.430584 * model-1\n 0.952081 * model-2\n 0.478912 * model-3\n```\n\nThere was a constraint in that notebook that all weights sum to 1, which I also think is unnecessary. In fact, if you choose to employ what @aquatic suggested and set `fit_intercept=True`, it is reasonable to expect that some of the weights will end up being negative. This is how one of my blends looks like when using logistic regression to find weights:\n\n```\n Your final model:\n [-4.07375987] + ( 1.02083256 * model-1 ) + ( 3.41645157 * model-2 ) + ( 1.29123727 * model-3 ) + ( 1.81433148 * model-4 ) + ( -0.25400735 * model-5 ) + ( 0.80850964 * model-6 ) + ( 0.25919084 * model-7 ) + ( -0.04247169 * model-8 )\n```\n\nThe first number in that equation is a fixed value for intercept.", "votes": 4, "replies": [ { "id": 1867476, "postDate": "2022-07-23T08:33:36.430Z", "content": "

Hi, scipy.optimize.minimize requires starting weights. In my model, using optuna for 5000 trials and then using the weights from optuna as starting weights for scipy.optimize has given much better results in comparison to taking starting weights from a uniform distribution.

", "rawMarkdown": "Hi, scipy.optimize.minimize requires starting weights. In my model, using optuna for 5000 trials and then using the weights from optuna as starting weights for scipy.optimize has given much better results in comparison to taking starting weights from a uniform distribution.", "votes": 1 }, { "id": 1867492, "postDate": "2022-07-23T08:55:28.713Z", "content": "

Yes, starting weights from the uniform distribution can be difficult to overcome. Sometimes it is worth trying to start all weights being even.

\n
starting_values = [1/len(starting_test_models)]*len(starting_test_models)\n
", "rawMarkdown": "Yes, starting weights from the uniform distribution can be difficult to overcome. Sometimes it is worth trying to start all weights being even.\n\n starting_values = [1/len(starting_test_models)]*len(starting_test_models)\n", "votes": 1 } ] }, { "id": 1858417, "postDate": "2022-07-16T22:16:00.613Z", "content": "

Personally I just do stacking with logistic regression on out-of-fold prediction log odds and find it to work well (as it has in previous competitions). Regularization is nice, and I've never had any issue with negative weights. I don't feel there's anything theoretically or practically wrong with them even if it seems unintuitive. Have never quite understood why people sometimes prefer Optuna over this -- would love to hear why if anyone has a strong opinion.

", "rawMarkdown": "Personally I just do stacking with logistic regression on out-of-fold prediction log odds and find it to work well (as it has in previous competitions). Regularization is nice, and I've never had any issue with negative weights. I don't feel there's anything theoretically or practically wrong with them even if it seems unintuitive. Have never quite understood why people sometimes prefer Optuna over this -- would love to hear why if anyone has a strong opinion.", "votes": 4, "replies": [ { "id": 1860043, "postDate": "2022-07-18T06:17:43.173Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1859258, "postDate": "2022-07-17T13:57:15.080Z", "content": "

Hi @vivekjan14, long time ago I've tried this approach in a tabular playground competition and worked quite well.

", "rawMarkdown": "Hi @vivekjan14, long time ago I've tried this approach in a tabular playground competition and worked quite well.", "votes": 1, "replies": [ { "id": 1860049, "postDate": "2022-07-18T06:20:28.757Z", "content": "

Thank you. I wonder how it compares with using scipy.

", "rawMarkdown": "Thank you. I wonder how it compares with using scipy.", "votes": 1 } ] }, { "id": 1858015, "postDate": "2022-07-16T15:26:27.293Z", "content": "

Hello @vivekjan14 , I will recommend to train a model to find the optimal parameters based on the OOF prediction.

", "rawMarkdown": "Hello @vivekjan14 , I will recommend to train a model to find the optimal parameters based on the OOF prediction.", "votes": 1 }, { "id": 1858313, "postDate": "2022-07-16T19:52:55.410Z", "content": "

Hi @vivekjan14 I use scipy.optimize.minimize to find the best ensemble weights, but I don't know whether its result differs much from the Optuna result. I suggest that you try both and compare the scores.

\n

I'd strongly recommend, however, that you use only nonnegative weights. If you get negative weights in an ensemble, something has gone wrong.

", "rawMarkdown": "Hi @vivekjan14 I use scipy.optimize.minimize to find the best ensemble weights, but I don't know whether its result differs much from the Optuna result. I suggest that you try both and compare the scores.\n\nI'd strongly recommend, however, that you use only nonnegative weights. If you get negative weights in an ensemble, something has gone wrong.", "votes": 2, "replies": [ { "id": 1859004, "postDate": "2022-07-17T10:36:12.397Z", "content": "

Thanks…Ill try that out.

", "rawMarkdown": "Thanks...Ill try that out. ", "votes": 2 }, { "id": 1862909, "postDate": "2022-07-20T04:44:29.950Z", "content": "

I'm curious as to why negative weights are problematic.
\nI guess you think that the distribution of predicted outcomes is consistent across all models and that's why it doesn't work. Right?

", "rawMarkdown": "I'm curious as to why negative weights are problematic.\nI guess you think that the distribution of predicted outcomes is consistent across all models and that's why it doesn't work. Right?", "votes": 1 }, { "id": 1867461, "postDate": "2022-07-23T08:19:27.500Z", "content": "

Hi I just wanted to let you know that setting optuna for 5000 trials just to find initial weights for scipy and then optimizing with those initial weights using scipy.optimize is giving really good results. However, I don't think negative weights have any bad effect on the ensemble CV. Thanks for your advice!

", "rawMarkdown": "Hi I just wanted to let you know that setting optuna for 5000 trials just to find initial weights for scipy and then optimizing with those initial weights using scipy.optimize is giving really good results. However, I don't think negative weights have any bad effect on the ensemble CV. Thanks for your advice!", "votes": 1 } ] }, { "id": 1857988, "postDate": "2022-07-16T15:06:47.947Z", "content": "

Optuna is used for hyperparameter optimization, and does give good results

", "rawMarkdown": "Optuna is used for hyperparameter optimization, and does give good results" } ], "comments": [ { "id": 1868891, "author_name": "AmbrosM", "author_url": "", "post_date": "2022-07-24T09:58:38.483000", "content": "

@tilii7 @aquatic @xiaowangiiiii @vivekjan14 Thank you for challenging my statement about negative weights! I've thought about it again and now agree that negative weights are not necessarily bad.

", "votes": 5, "replies": [] }, { "id": 1858802, "author_name": "Tilii", "author_url": "", "post_date": "2022-07-17T08:09:28.643000", "content": "

In theory optuna should be able to find any function, but that only holds given a large enough number of iterations. I don't think you would get an optimal solution with 2000 trials.

\n

Besides, both @aquatic and @ambrosm have given you solutions that are more stable numerically and are much more likely to converge to an optimal solution.

\n

There is one suggestion that is not consistent with my experience.

\n
\n

I'd strongly recommend, however, that you use only nonnegative weights. If you get negative weights in an ensemble, something has gone wrong.

\n
\n

I used to subscribe to this point of view, but not any more. I find it that allowing negative weights often leads to better solution, and in fact I have a notebook from a different competition where the same solution proposed by @ambrosm has been employed, but with negative weights allowed.

\n

https://www.kaggle.com/code/tilii7/cross-validation-weighted-linear-blending-errors

\n

The blend of three models from that notebook ends up looking like this:

\n
 Your final model:\n -0.430584 * model-1\n 0.952081 * model-2\n 0.478912 * model-3\n
\n

There was a constraint in that notebook that all weights sum to 1, which I also think is unnecessary. In fact, if you choose to employ what @aquatic suggested and set fit_intercept=True, it is reasonable to expect that some of the weights will end up being negative. This is how one of my blends looks like when using logistic regression to find weights:

\n
 Your final model:\n  [-4.07375987] + ( 1.02083256 * model-1 ) + ( 3.41645157 * model-2 ) + ( 1.29123727 * model-3 ) + ( 1.81433148 * model-4 ) + ( -0.25400735 * model-5 ) + ( 0.80850964 * model-6 ) + ( 0.25919084 * model-7 ) + ( -0.04247169 * model-8 )\n
\n

The first number in that equation is a fixed value for intercept.

", "votes": 4, "replies": [ { "id": 1867476, "author_name": "vivekjan14", "author_url": "", "post_date": "2022-07-23T08:33:36.430000", "content": "

Hi, scipy.optimize.minimize requires starting weights. In my model, using optuna for 5000 trials and then using the weights from optuna as starting weights for scipy.optimize has given much better results in comparison to taking starting weights from a uniform distribution.

", "votes": 1, "replies": [] }, { "id": 1867492, "author_name": "Tilii", "author_url": "", "post_date": "2022-07-23T08:55:28.713000", "content": "

Yes, starting weights from the uniform distribution can be difficult to overcome. Sometimes it is worth trying to start all weights being even.

\n
starting_values = [1/len(starting_test_models)]*len(starting_test_models)\n
", "votes": 1, "replies": [] } ] }, { "id": 1858417, "author_name": "Joe Eddy", "author_url": "", "post_date": "2022-07-16T22:16:00.613000", "content": "

Personally I just do stacking with logistic regression on out-of-fold prediction log odds and find it to work well (as it has in previous competitions). Regularization is nice, and I've never had any issue with negative weights. I don't feel there's anything theoretically or practically wrong with them even if it seems unintuitive. Have never quite understood why people sometimes prefer Optuna over this -- would love to hear why if anyone has a strong opinion.

", "votes": 4, "replies": [ { "id": 1860043, "author_name": "", "author_url": "", "post_date": "2022-07-18T06:17:43.173000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1859258, "author_name": "Maximiliano Diaz Battan", "author_url": "", "post_date": "2022-07-17T13:57:15.080000", "content": "

Hi @vivekjan14, long time ago I've tried this approach in a tabular playground competition and worked quite well.

", "votes": 1, "replies": [ { "id": 1860049, "author_name": "vivekjan14", "author_url": "", "post_date": "2022-07-18T06:20:28.757000", "content": "

Thank you. I wonder how it compares with using scipy.

", "votes": 1, "replies": [] } ] }, { "id": 1858015, "author_name": "C4rl05/V", "author_url": "", "post_date": "2022-07-16T15:26:27.293000", "content": "

Hello @vivekjan14 , I will recommend to train a model to find the optimal parameters based on the OOF prediction.

", "votes": 1, "replies": [] }, { "id": 1858313, "author_name": "AmbrosM", "author_url": "", "post_date": "2022-07-16T19:52:55.410000", "content": "

Hi @vivekjan14 I use scipy.optimize.minimize to find the best ensemble weights, but I don't know whether its result differs much from the Optuna result. I suggest that you try both and compare the scores.

\n

I'd strongly recommend, however, that you use only nonnegative weights. If you get negative weights in an ensemble, something has gone wrong.

", "votes": 2, "replies": [ { "id": 1859004, "author_name": "vivekjan14", "author_url": "", "post_date": "2022-07-17T10:36:12.397000", "content": "

Thanks…Ill try that out.

", "votes": 2, "replies": [] }, { "id": 1862909, "author_name": "Mingjie Wang", "author_url": "", "post_date": "2022-07-20T04:44:29.950000", "content": "

I'm curious as to why negative weights are problematic.
\nI guess you think that the distribution of predicted outcomes is consistent across all models and that's why it doesn't work. Right?

", "votes": 1, "replies": [] }, { "id": 1867461, "author_name": "vivekjan14", "author_url": "", "post_date": "2022-07-23T08:19:27.500000", "content": "

Hi I just wanted to let you know that setting optuna for 5000 trials just to find initial weights for scipy and then optimizing with those initial weights using scipy.optimize is giving really good results. However, I don't think negative weights have any bad effect on the ensemble CV. Thanks for your advice!

", "votes": 1, "replies": [] } ] }, { "id": 1857988, "author_name": "Harsh Bansal", "author_url": "", "post_date": "2022-07-16T15:06:47.947000", "content": "

Optuna is used for hyperparameter optimization, and does give good results

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1857899": "Hi, Does optuna work well to find optimal weights for ensembling? Something like this:\n\n```\ndef objective(trial):\n w1 = trial.suggest_float(\"w1\",-10,10)\n w2 = trial.suggest_float(\"w2\",-10,10)\n w3 = trial.suggest_float(\"w3\",-10,10)\n w4 = trial.suggest_float(\"w4\",-10,10)\n w5 = trial.suggest_float(\"w5\",-10,10)\n val_preds = (w1*train[\"goss\"] + w2*train[\"xgboost\"] + w3*train[\"gbdt\"] + w4*train[\"dart\"]+w5*train[\"cat_boost\"])/(w1+w2+w3+w4+w5)\n metric = amex_metric(train[CFG.target],val_preds)\n return metric\nstudy = optuna.create_study(direction=\"maximize\")\nstudy.optimize(objective,n_trials=2000)\n```\nOr is there a better way to do it? Thanks.", "1868891": "@tilii7 @aquatic @xiaowangiiiii @vivekjan14 Thank you for challenging my statement about negative weights! I've thought about it again and now agree that negative weights are not necessarily bad.", "1858802": "In theory optuna should be able to find any function, but that only holds given a large enough number of iterations. I don't think you would get an optimal solution with 2000 trials.\n\nBesides, both @aquatic and @ambrosm have given you solutions that are more stable numerically and are much more likely to converge to an optimal solution.\n\nThere is one suggestion that is not consistent with my experience.\n\n> I'd strongly recommend, however, that you use only nonnegative weights. If you get negative weights in an ensemble, something has gone wrong.\n\nI used to subscribe to this point of view, but not any more. I find it that allowing negative weights often leads to better solution, and in fact I have a notebook from a different competition where the same solution proposed by @ambrosm has been employed, but with negative weights allowed.\n\nhttps://www.kaggle.com/code/tilii7/cross-validation-weighted-linear-blending-errors\n\nThe blend of three models from that notebook ends up looking like this:\n\n```\n Your final model:\n -0.430584 * model-1\n 0.952081 * model-2\n 0.478912 * model-3\n```\n\nThere was a constraint in that notebook that all weights sum to 1, which I also think is unnecessary. In fact, if you choose to employ what @aquatic suggested and set `fit_intercept=True`, it is reasonable to expect that some of the weights will end up being negative. This is how one of my blends looks like when using logistic regression to find weights:\n\n```\n Your final model:\n [-4.07375987] + ( 1.02083256 * model-1 ) + ( 3.41645157 * model-2 ) + ( 1.29123727 * model-3 ) + ( 1.81433148 * model-4 ) + ( -0.25400735 * model-5 ) + ( 0.80850964 * model-6 ) + ( 0.25919084 * model-7 ) + ( -0.04247169 * model-8 )\n```\n\nThe first number in that equation is a fixed value for intercept.", "1858417": "Personally I just do stacking with logistic regression on out-of-fold prediction log odds and find it to work well (as it has in previous competitions). Regularization is nice, and I've never had any issue with negative weights. I don't feel there's anything theoretically or practically wrong with them even if it seems unintuitive. Have never quite understood why people sometimes prefer Optuna over this -- would love to hear why if anyone has a strong opinion.", "1859258": "Hi @vivekjan14, long time ago I've tried this approach in a tabular playground competition and worked quite well.", "1858015": "Hello @vivekjan14 , I will recommend to train a model to find the optimal parameters based on the OOF prediction.", "1858313": "Hi @vivekjan14 I use scipy.optimize.minimize to find the best ensemble weights, but I don't know whether its result differs much from the Optuna result. I suggest that you try both and compare the scores.\n\nI'd strongly recommend, however, that you use only nonnegative weights. If you get negative weights in an ensemble, something has gone wrong.", "1857988": "Optuna is used for hyperparameter optimization, and does give good results" } }