![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F225499%2Fbdee903e1dc7294046400744aed2852a%2Fslice_plot.png?generation=1733391524963030&alt=media\")
During this competition one of the things that I focused on was hyperparameter optimisation. While analysing the hyperparameters of the trials I came across an interesting statistical phenomenon called the optimiser’s curse. Basically what it says is that when there is noise in the data the optimiser can overestimate the value or the importance of certain hyperparameters by sheer luck. One way to mitigate this is not to optimise the hyperparameters that are not correlated with the validation score because adding them introduces extra randomness.
Plot below could be used to study the relationship between the individual hyperparameters and the validation score.
\n
learning_rate shows a clear relationship with the objective value.lambda_1 and lambda_2 one can observe some cluster forming at the top right region (although a weak one). And given that there is quit a bit of noise in the data I would choose not to leave these out.Based on these observations one can argue to just optimise leaning_rate, lambda_1, lambda_2.
Let me know what you think.
\nThis is a nice youtube link where this phenomenon is explained.
", "messages": [ { "id": "3064159", "postDate": "12/05/2024 09:44:29", "content": "During this competition one of the things that I focused on was hyperparameter optimisation. While analysing the hyperparameters of the trials I came across an interesting statistical phenomenon called the optimiser’s curse. Basically what it says is that when there is noise in the data the optimiser can overestimate the value or the importance of certain hyperparameters by sheer luck. One way to mitigate this is not to optimise the hyperparameters that are not correlated with the validation score because adding them introduces extra randomness.
Plot below could be used to study the relationship between the individual hyperparameters and the validation score.
\n
learning_rate shows a clear relationship with the objective value.lambda_1 and lambda_2 one can observe some cluster forming at the top right region (although a weak one). And given that there is quit a bit of noise in the data I would choose not to leave these out.Based on these observations one can argue to just optimise leaning_rate, lambda_1, lambda_2.
Let me know what you think.
\nThis is a nice youtube link where this phenomenon is explained.
", "rawMarkdown": "During this competition one of the things that I focused on was hyperparameter optimisation. While analysing the hyperparameters of the trials I came across an interesting statistical phenomenon called `the optimiser’s curse`. Basically what it says is that when there is noise in the data the optimiser can overestimate the value or the importance of certain hyperparameters by sheer luck. One way to mitigate this is not to optimise the hyperparameters that are not correlated with the validation score because adding them introduces extra randomness. \n\nPlot below could be used to study the relationship between the individual hyperparameters and the validation score.\n\n\n\n- The `learning_rate` shows a clear relationship with the objective value.\n- For `lambda_1` and `lambda_2` one can observe some cluster forming at the top right region (although a weak one). And given that there is quit a bit of noise in the data I would choose not to leave these out.\n- The rest of the hyperparameters do not really correlate with the objective function.\n\nBased on these observations one can argue to just optimise `leaning_rate`, `lambda_1`, `lambda_2`. \n\nLet me know what you think.\n\nThis is a nice [youtube](https://www.youtube.com/watch?v=vC9sAD-ymhk&t=181s) link where this phenomenon is explained.", "votes": null }, { "id": "3065411", "postDate": "12/06/2024 19:04:16", "content": "Hi @wti200,
\nThank you for the link. Nice explanation.
\nWith your plots, I see a relation between objective function and feature fraction and bagging fraction too.
\nAnd with this small dataset, it's interesting to refit the 5 or 10 best trials with severel seeds to unsure the best trial is not a lucky one .
\nIn the following exmaple, let's imagine we have detected the best trial by using optuna with random_seed 42 (in my kfolds for example). Then I have sorted trials by score :
\n| seed | \n42 | \n
|---|---|
| trial n°1 | \n.49 | \n
| trial n°2 | \n.48 | \n
| trial n°3 | \n.475 | \n
| trial n°4 | \n.47 | \n
| trial n°5 | \n.46 | \n
Next, I do some other CV with those 5 trials, but with 2 other seeds (65 and 88) :
\n| seed | \n42 | \n65 | \n88 | \n
|---|---|---|---|
| trial n°1 | \n.49 | \n.45 | \n.40 | \n
| trial n°2 | \n.48 | \n.475 | \n.49 | \n
| trial n°3 | \n.475 | \n.46 | \n.5 | \n
| trial n°4 | \n.47 | \n.4 | \n.43 | \n
| trial n°5 | \n.46 | \n.4 | \n.5 | \n
We can see that trial n°2 was not the best with random state 42, but is better than trial n°1 for those 2 other random states.
\nAnd CV score with trial n°2 has a lower variance than with trial n°1 : in this case, I will choose hyperparams of trial n°2 !
@adaubas interesting, thanks for sharing. Optimal hyperparameters aside, do the overall scatterplots between each hyperparam vs. objective function follow the same pattern as the ones produced by @wti200, even after experimenting with different seeds?
", "rawMarkdown": "adaubas interesting, thanks for sharing. Optimal hyperparameters aside, do the overall scatterplots between each hyperparam vs. objective function follow the same pattern as the ones produced by @wti200, even after experimenting with different seeds?", "votes": null }, { "id": "3065499", "postDate": "12/06/2024 22:29:14", "content": "I tested different seeds after generating the resulting scatterplots from the optuna trials.
\nAnd no, my scatterplots don't have same patterns as bellow : for example, I set n_estimators using early stopping, not with optuna.
off topic, this is a good point I have just been thinking about with this small dataset, whether if one should set n_estimators with early stopping and get virtually better cv results or set it through optuna and be sure to not be overfitting the cv…
", "rawMarkdown": "off topic, this is a good point I have just been thinking about with this small dataset, whether if one should set **n_estimators** with early stopping and get virtually better cv results or set it through optuna and be sure to not be overfitting the cv…", "votes": null }, { "id": "3065742", "postDate": "12/07/2024 08:28:23", "content": "Hi @dib,
\nTake a look to a recent discussion about \"early stopping or not\" in another competition a few days ago : https://www.kaggle.com/competitions/playground-series-s4e12/discussion/549788
I believe there is no better way than early stopping to estimate the best n_estimators… Even in our current competition ; that's my opinion.
", "rawMarkdown": "Hi @dib,\nTake a look to a recent discussion about \"early stopping or not\" in another competition a few days ago : https://www.kaggle.com/competitions/playground-series-s4e12/discussion/549788\n\nI believe there is no better way than early stopping to estimate the best n_estimators... Even in our current competition ; that's my opinion.", "votes": null }, { "id": "3066043", "postDate": "12/07/2024 15:29:52", "content": "very good discussion post, thanks for sharing @adaubas. I agree that cv+es might be the best option most of the times, it is just that this time with these small dataset and metric based on hard labels makes me rethink it. Let's see if the future winners of the competition disclose how they set n_estimators
", "rawMarkdown": "very good discussion post, thanks for sharing @adaubas. I agree that cv+es might be the best option most of the times, it is just that this time with these small dataset and metric based on hard labels makes me rethink it. Let's see if the future winners of the competition disclose how they set n_estimators", "votes": null }, { "id": "3067053", "postDate": "12/08/2024 19:56:31", "content": "Thanks for pointing out that youtube video! The noise in hyperparameter fitting can 9ndeed make it an unsatisfying process…
\nfyi, I've been fitting with XGB models using a very limited set of hyperparameters: colsample_bytree, learning_rate, max_depth, n_estimators, and subsample. Doing a 2d grid search of learning_rate and n_estimators, and keeping the others fixed. To see and reduce variation in this process, the same grids were done on 5 shuffled XGB models and combined. There's a clear band of the highest test scores along the higher-learning-rates-with-fewer-estimators diagonal.
", "rawMarkdown": "Thanks for pointing out that youtube video! The noise in hyperparameter fitting can 9ndeed make it an unsatisfying process...\n\nfyi, I've been fitting with XGB models using a very limited set of hyperparameters: colsample_bytree, learning_rate, max_depth, n_estimators, and subsample. Doing a 2d grid search of learning_rate and n_estimators, and keeping the others fixed. To see and reduce variation in this process, the same grids were done on 5 shuffled XGB models and combined. There's a clear band of the highest test scores along the higher-learning-rates-with-fewer-estimators diagonal.", "votes": null }, { "id": "3068876", "postDate": "12/10/2024 18:48:59", "content": "Hi @adaubas, sorry for the late response.
\nMy hunch is that the random seed is just another hyperparameter that does not correlate with the objective function. And optimising it comes with the risk of overfitting. I do believe that generating predictions with multiple seeds can introduce additional variance in the individual models, which may improve the overall ensemble performance.
", "rawMarkdown": "Hi @adaubas, sorry for the late response. \n\nMy hunch is that the random seed is just another hyperparameter that does not correlate with the objective function. And optimising it comes with the risk of overfitting. I do believe that generating predictions with multiple seeds can introduce additional variance in the individual models, which may improve the overall ensemble performance.", "votes": null }, { "id": "3068879", "postDate": "12/10/2024 18:49:52", "content": "@adaubas thanks for the link!
", "rawMarkdown": "adaubas thanks for the link!", "votes": null } ], "comments": [ { "id": 3065411, "author_name": "adaubas", "author_url": "", "post_date": "12/06/2024 19:04:16", "content": "Hi @wti200,
\nThank you for the link. Nice explanation.
\nWith your plots, I see a relation between objective function and feature fraction and bagging fraction too.
\nAnd with this small dataset, it's interesting to refit the 5 or 10 best trials with severel seeds to unsure the best trial is not a lucky one .
\nIn the following exmaple, let's imagine we have detected the best trial by using optuna with random_seed 42 (in my kfolds for example). Then I have sorted trials by score :
\n| seed | \n42 | \n
|---|---|
| trial n°1 | \n.49 | \n
| trial n°2 | \n.48 | \n
| trial n°3 | \n.475 | \n
| trial n°4 | \n.47 | \n
| trial n°5 | \n.46 | \n
Next, I do some other CV with those 5 trials, but with 2 other seeds (65 and 88) :
\n| seed | \n42 | \n65 | \n88 | \n
|---|---|---|---|
| trial n°1 | \n.49 | \n.45 | \n.40 | \n
| trial n°2 | \n.48 | \n.475 | \n.49 | \n
| trial n°3 | \n.475 | \n.46 | \n.5 | \n
| trial n°4 | \n.47 | \n.4 | \n.43 | \n
| trial n°5 | \n.46 | \n.4 | \n.5 | \n
We can see that trial n°2 was not the best with random state 42, but is better than trial n°1 for those 2 other random states.
\nAnd CV score with trial n°2 has a lower variance than with trial n°1 : in this case, I will choose hyperparams of trial n°2 !
@adaubas interesting, thanks for sharing. Optimal hyperparameters aside, do the overall scatterplots between each hyperparam vs. objective function follow the same pattern as the ones produced by @wti200, even after experimenting with different seeds?
", "votes": null, "replies": [ { "id": 3065499, "author_name": "adaubas", "author_url": "", "post_date": "12/06/2024 22:29:14", "content": "I tested different seeds after generating the resulting scatterplots from the optuna trials.
\nAnd no, my scatterplots don't have same patterns as bellow : for example, I set n_estimators using early stopping, not with optuna.
off topic, this is a good point I have just been thinking about with this small dataset, whether if one should set n_estimators with early stopping and get virtually better cv results or set it through optuna and be sure to not be overfitting the cv…
", "votes": null, "replies": [ { "id": 3065742, "author_name": "adaubas", "author_url": "", "post_date": "12/07/2024 08:28:23", "content": "Hi @dib,
\nTake a look to a recent discussion about \"early stopping or not\" in another competition a few days ago : https://www.kaggle.com/competitions/playground-series-s4e12/discussion/549788
I believe there is no better way than early stopping to estimate the best n_estimators… Even in our current competition ; that's my opinion.
", "votes": null, "replies": [ { "id": 3066043, "author_name": "diegoiglesias", "author_url": "", "post_date": "12/07/2024 15:29:52", "content": "very good discussion post, thanks for sharing @adaubas. I agree that cv+es might be the best option most of the times, it is just that this time with these small dataset and metric based on hard labels makes me rethink it. Let's see if the future winners of the competition disclose how they set n_estimators
", "votes": null, "replies": [] }, { "id": 3068879, "author_name": "wti200", "author_url": "", "post_date": "12/10/2024 18:49:52", "content": "@adaubas thanks for the link!
", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 3068876, "author_name": "wti200", "author_url": "", "post_date": "12/10/2024 18:48:59", "content": "Hi @adaubas, sorry for the late response.
\nMy hunch is that the random seed is just another hyperparameter that does not correlate with the objective function. And optimising it comes with the risk of overfitting. I do believe that generating predictions with multiple seeds can introduce additional variance in the individual models, which may improve the overall ensemble performance.
", "votes": null, "replies": [] } ] }, { "id": 3067053, "author_name": "dan3dewey", "author_url": "", "post_date": "12/08/2024 19:56:31", "content": "Thanks for pointing out that youtube video! The noise in hyperparameter fitting can 9ndeed make it an unsatisfying process…
\nfyi, I've been fitting with XGB models using a very limited set of hyperparameters: colsample_bytree, learning_rate, max_depth, n_estimators, and subsample. Doing a 2d grid search of learning_rate and n_estimators, and keeping the others fixed. To see and reduce variation in this process, the same grids were done on 5 shuffled XGB models and combined. There's a clear band of the highest test scores along the higher-learning-rates-with-fewer-estimators diagonal.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3064159": "During this competition one of the things that I focused on was hyperparameter optimisation. While analysing the hyperparameters of the trials I came across an interesting statistical phenomenon called `the optimiser’s curse`. Basically what it says is that when there is noise in the data the optimiser can overestimate the value or the importance of certain hyperparameters by sheer luck. One way to mitigate this is not to optimise the hyperparameters that are not correlated with the validation score because adding them introduces extra randomness. \n\nPlot below could be used to study the relationship between the individual hyperparameters and the validation score.\n\n\n\n- The `learning_rate` shows a clear relationship with the objective value.\n- For `lambda_1` and `lambda_2` one can observe some cluster forming at the top right region (although a weak one). And given that there is quit a bit of noise in the data I would choose not to leave these out.\n- The rest of the hyperparameters do not really correlate with the objective function.\n\nBased on these observations one can argue to just optimise `leaning_rate`, `lambda_1`, `lambda_2`. \n\nLet me know what you think.\n\nThis is a nice [youtube](https://www.youtube.com/watch?v=vC9sAD-ymhk&t=181s) link where this phenomenon is explained.", "3065411": "Hi @wti200,\n\nThank you for the link. Nice explanation.\n\nWith your plots, I see a relation between objective function and feature fraction and bagging fraction too. \n\nAnd with this small dataset, it's interesting to refit the 5 or 10 best trials with severel seeds to unsure the best trial is not a lucky one . \n\n\nIn the following exmaple, let's imagine we have detected the best trial by using optuna with random_seed 42 (in my kfolds for example). Then I have sorted trials by score :\n| seed | 42 | \n| --- | --- |\n| trial n°1 | .49 | \n| trial n°2 | .48 | \n| trial n°3 | .475 | \n| trial n°4 | .47 | \n| trial n°5 | .46 | \n\nNext, I do some other CV with those 5 trials, but with 2 other seeds (65 and 88) : \n\n| seed | 42 | 65 | 88 |\n| --- | --- | --- | --- |\n| trial n°1 | .49 | .45 | .40 | \n| trial n°2 | .48 | .475 | .49 | \n| trial n°3 | .475 | .46 | .5 | \n| trial n°4 | .47 | .4 | .43 | \n| trial n°5 | .46 | .4 | .5 | \n\nWe can see that trial n°2 was not the best with random state 42, but is better than trial n°1 for those 2 other random states. \nAnd CV score with trial n°2 has a lower variance than with trial n°1 : in this case, I will choose hyperparams of trial n°2 !", "3065479": "adaubas interesting, thanks for sharing. Optimal hyperparameters aside, do the overall scatterplots between each hyperparam vs. objective function follow the same pattern as the ones produced by @wti200, even after experimenting with different seeds?", "3065499": "I tested different seeds *after* generating the resulting scatterplots from the optuna trials.\nAnd no, my scatterplots don't have same patterns as bellow : for example, I set **n_estimators** using early stopping, not with optuna.", "3065702": "off topic, this is a good point I have just been thinking about with this small dataset, whether if one should set **n_estimators** with early stopping and get virtually better cv results or set it through optuna and be sure to not be overfitting the cv…", "3065742": "Hi @dib,\nTake a look to a recent discussion about \"early stopping or not\" in another competition a few days ago : https://www.kaggle.com/competitions/playground-series-s4e12/discussion/549788\n\nI believe there is no better way than early stopping to estimate the best n_estimators... Even in our current competition ; that's my opinion.", "3066043": "very good discussion post, thanks for sharing @adaubas. I agree that cv+es might be the best option most of the times, it is just that this time with these small dataset and metric based on hard labels makes me rethink it. Let's see if the future winners of the competition disclose how they set n_estimators", "3067053": "Thanks for pointing out that youtube video! The noise in hyperparameter fitting can 9ndeed make it an unsatisfying process...\n\nfyi, I've been fitting with XGB models using a very limited set of hyperparameters: colsample_bytree, learning_rate, max_depth, n_estimators, and subsample. Doing a 2d grid search of learning_rate and n_estimators, and keeping the others fixed. To see and reduce variation in this process, the same grids were done on 5 shuffled XGB models and combined. There's a clear band of the highest test scores along the higher-learning-rates-with-fewer-estimators diagonal.", "3068876": "Hi @adaubas, sorry for the late response. \n\nMy hunch is that the random seed is just another hyperparameter that does not correlate with the objective function. And optimising it comes with the risk of overfitting. I do believe that generating predictions with multiple seeds can introduce additional variance in the individual models, which may improve the overall ensemble performance.", "3068879": "adaubas thanks for the link!" }, "source": "meta" }