{ "id": 367254, "title": "Let's share good boostings params, and some disappointment in Optuna ", "url": "/competitions/open-problems-multimodal/discussion/367254", "author_name": "Alexander Chervov", "post_date": "2022-11-19T21:12:39.215000", "votes": 3, "comment_count": 0, "views": 0, "content": "

Wanted to write such post for years, let me put laziness aside and finally do it:

\n

If you found some good params for Boostings - XGB, LGB or CarBoost - PLEASE SHARE - may be these params would be useful in other situations !

\n

Our turn:
\nFor LGB it seems the following params were good for us,
\non various feature sets, and for various targets (which is surprising by itself - same params good in many cases):

\n

{'n_estimators': 500, 'reg_alpha': 6.734991732483901, 'reg_lambda': 3.0151837674804667, 'colsample_bytree': 0.9, 'subsample': 1.0, 'max_depth': 6, 'learning_rate': 0.03814860248977263, 'num_leaves': 859, 'min_child_samples': 13}

\n

For XGB we used those ones:
\n XGB_PARAMETERS = { 'n_estimators': 225, 'max_depth': 6, 'learning_rate': 0.07190990240552594,
\n 'gamma': 2.0460152365496342, 'base_score': 0.5794691454636103, 'tree_method': 'gpu_hist',
\n 'predictor': 'gpu_predictor', 'max_leaves': 13, 'random_state': random.randint(0,10_000) # 123
\n }

\n

Some issue with the standard Optuna:
\nthese params were found with 200 trials of Optuna, but other launches with 400 … 4000(!) trials
\ngave WORSE params - see screenshot !
\nSo one cannot fully rely on Optuna in search of optimal params for LGB at least.

\n

We used standard Optuna tuner, may be one should use special one for LGB:
\n\"optuna.integration.lightgbm.LightGBMTuner\" by Kohei Ozaki, a Kaggle Grandmaster:
\nhttps://optuna.readthedocs.io/en/stable/reference/generated/optuna.integration.lightgbm.LightGBMTuner.html
\nHave any one tried it ?

\n

\"\"

\n

Details:
\nWe have found them on ONE Target (CD31), with 200 Optuna trials on 100 Features and train test split by day.
\n(Version 22 of the notebook: https://www.kaggle.com/code/alexandervc/mmscel-cv-modeling-with-playground-fold?scriptVersionId=109844122&cellId=28 )
\nBut later checked that they worked fine for ALL targets, different features sets and folds by all days:
\n(Checks were done in Versions: 14-38 of the notebook
\nhttps://www.kaggle.com/code/alexandervc/mmscel-cv-modeling-advanced#Set-the-model )

", "messages": [ { "id": 2036541, "postDate": "2022-11-19T21:12:39.217Z", "content": "

Wanted to write such post for years, let me put laziness aside and finally do it:

\n

If you found some good params for Boostings - XGB, LGB or CarBoost - PLEASE SHARE - may be these params would be useful in other situations !

\n

Our turn:
\nFor LGB it seems the following params were good for us,
\non various feature sets, and for various targets (which is surprising by itself - same params good in many cases):

\n

{'n_estimators': 500, 'reg_alpha': 6.734991732483901, 'reg_lambda': 3.0151837674804667, 'colsample_bytree': 0.9, 'subsample': 1.0, 'max_depth': 6, 'learning_rate': 0.03814860248977263, 'num_leaves': 859, 'min_child_samples': 13}

\n

For XGB we used those ones:
\n XGB_PARAMETERS = { 'n_estimators': 225, 'max_depth': 6, 'learning_rate': 0.07190990240552594,
\n 'gamma': 2.0460152365496342, 'base_score': 0.5794691454636103, 'tree_method': 'gpu_hist',
\n 'predictor': 'gpu_predictor', 'max_leaves': 13, 'random_state': random.randint(0,10_000) # 123
\n }

\n

Some issue with the standard Optuna:
\nthese params were found with 200 trials of Optuna, but other launches with 400 … 4000(!) trials
\ngave WORSE params - see screenshot !
\nSo one cannot fully rely on Optuna in search of optimal params for LGB at least.

\n

We used standard Optuna tuner, may be one should use special one for LGB:
\n\"optuna.integration.lightgbm.LightGBMTuner\" by Kohei Ozaki, a Kaggle Grandmaster:
\nhttps://optuna.readthedocs.io/en/stable/reference/generated/optuna.integration.lightgbm.LightGBMTuner.html
\nHave any one tried it ?

\n

\"\"

\n

Details:
\nWe have found them on ONE Target (CD31), with 200 Optuna trials on 100 Features and train test split by day.
\n(Version 22 of the notebook: https://www.kaggle.com/code/alexandervc/mmscel-cv-modeling-with-playground-fold?scriptVersionId=109844122&cellId=28 )
\nBut later checked that they worked fine for ALL targets, different features sets and folds by all days:
\n(Checks were done in Versions: 14-38 of the notebook
\nhttps://www.kaggle.com/code/alexandervc/mmscel-cv-modeling-advanced#Set-the-model )

", "rawMarkdown": "Wanted to write such post for years, let me put laziness aside and finally do it:\n\nIf you found some good params for Boostings - XGB, LGB or CarBoost - PLEASE SHARE - may be these params would be useful in other situations ! \n\nOur turn:\nFor LGB it seems the following params were good for us, \non various feature sets, and for various targets (which is surprising by itself - same params good in many cases):\n\n{'n_estimators': 500, 'reg_alpha': 6.734991732483901, 'reg_lambda': 3.0151837674804667, 'colsample_bytree': 0.9, 'subsample': 1.0, 'max_depth': 6, 'learning_rate': 0.03814860248977263, 'num_leaves': 859, 'min_child_samples': 13}\n\nFor XGB we used those ones: \n XGB_PARAMETERS = { 'n_estimators': 225, 'max_depth': 6, 'learning_rate': 0.07190990240552594,\n 'gamma': 2.0460152365496342, 'base_score': 0.5794691454636103, 'tree_method': 'gpu_hist', \n 'predictor': 'gpu_predictor', 'max_leaves': 13, 'random_state': random.randint(0,10_000) # 123\n } \n\nSome **issue with the standard Optuna**: \nthese params were found with 200 trials of Optuna, but other launches with 400 ... 4000(!) trials \ngave WORSE params - see screenshot ! \nSo one cannot fully rely on Optuna in search of optimal params for LGB at least.\n\nWe used standard Optuna tuner, may be one should use special one for LGB:\n\"optuna.integration.lightgbm.LightGBMTuner\" by Kohei Ozaki, a Kaggle Grandmaster:\nhttps://optuna.readthedocs.io/en/stable/reference/generated/optuna.integration.lightgbm.LightGBMTuner.html\n**Have any one tried it ?** \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2262596%2F023a3c23cffa0f1bbf3c9549192e78c2%2FOptuna%20LGB.png?generation=1668892703230167&alt=media) \n\n**Details:**\nWe have found them on ONE Target (CD31), with 200 Optuna trials on 100 Features and train test split by day. \n(Version 22 of the notebook: https://www.kaggle.com/code/alexandervc/mmscel-cv-modeling-with-playground-fold?scriptVersionId=109844122&cellId=28 )\nBut later checked that they worked fine for ALL targets, different features sets and folds by all days:\n(Checks were done in Versions: 14-38 of the notebook\nhttps://www.kaggle.com/code/alexandervc/mmscel-cv-modeling-advanced#Set-the-model )\n", "votes": 3 } ], "comments": [], "raw_markdown_by_id": { "2036541": "Wanted to write such post for years, let me put laziness aside and finally do it:\n\nIf you found some good params for Boostings - XGB, LGB or CarBoost - PLEASE SHARE - may be these params would be useful in other situations ! \n\nOur turn:\nFor LGB it seems the following params were good for us, \non various feature sets, and for various targets (which is surprising by itself - same params good in many cases):\n\n{'n_estimators': 500, 'reg_alpha': 6.734991732483901, 'reg_lambda': 3.0151837674804667, 'colsample_bytree': 0.9, 'subsample': 1.0, 'max_depth': 6, 'learning_rate': 0.03814860248977263, 'num_leaves': 859, 'min_child_samples': 13}\n\nFor XGB we used those ones: \n XGB_PARAMETERS = { 'n_estimators': 225, 'max_depth': 6, 'learning_rate': 0.07190990240552594,\n 'gamma': 2.0460152365496342, 'base_score': 0.5794691454636103, 'tree_method': 'gpu_hist', \n 'predictor': 'gpu_predictor', 'max_leaves': 13, 'random_state': random.randint(0,10_000) # 123\n } \n\nSome **issue with the standard Optuna**: \nthese params were found with 200 trials of Optuna, but other launches with 400 ... 4000(!) trials \ngave WORSE params - see screenshot ! \nSo one cannot fully rely on Optuna in search of optimal params for LGB at least.\n\nWe used standard Optuna tuner, may be one should use special one for LGB:\n\"optuna.integration.lightgbm.LightGBMTuner\" by Kohei Ozaki, a Kaggle Grandmaster:\nhttps://optuna.readthedocs.io/en/stable/reference/generated/optuna.integration.lightgbm.LightGBMTuner.html\n**Have any one tried it ?** \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2262596%2F023a3c23cffa0f1bbf3c9549192e78c2%2FOptuna%20LGB.png?generation=1668892703230167&alt=media) \n\n**Details:**\nWe have found them on ONE Target (CD31), with 200 Optuna trials on 100 Features and train test split by day. \n(Version 22 of the notebook: https://www.kaggle.com/code/alexandervc/mmscel-cv-modeling-with-playground-fold?scriptVersionId=109844122&cellId=28 )\nBut later checked that they worked fine for ALL targets, different features sets and folds by all days:\n(Checks were done in Versions: 14-38 of the notebook\nhttps://www.kaggle.com/code/alexandervc/mmscel-cv-modeling-advanced#Set-the-model )\n" } }