{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Hyperopt Made Simple!\n## Automated Parameter Tuning in One Function: quick_hyperopt()\n---\n![](https://i.imgur.com/JpaUi5T.png)\n\nParameter tuning can be a chore. Which parameters should be changed? And by how much? There are an almost infinite combination, so how can we find the best ones? Simple methods like GridSearch can manually run through some preset combinations but will not necessarily pick the best possible parameters, just the best in the set you provided. And this grid will have to be changed when you add new features or otherwise modify your data. A more nuanced approach is to use Bayesian optimisation, probabilistically selecting the optimum values for each parameter after every round of evaluation. \n\nThis kernel provides a single function for automated Bayesian hyperparameter optimisation with LightGBM, XGBoost and CatBoost via the Hyperopt package. Simply enter your data, target, and the type of model, and it will output a parameter dictionary you can then supply to your model-training kernels. While Hyperopt is a powerful tool, it can be difficult to implement and utilise. This function does the hard(-ish) work for you!\n\nI owe a great debt to Will Koehrsen for his [exhaustive kernel on automated parameter tuning](https://www.kaggle.com/willkoehrsen/automated-model-tuning). While I have streamlined, modified and generalised much of his work for easy use with different model frameworks, I strongly recommend reading his kernel from top to bottom. It contains a thorough examination of how Bayesian parameter tuning works better than random selection, along with a step-by-step guide on outputting the Hyperopt progress logs. If your aim is to truly understand the *how* and *why* of parameter tuning, his kernels are some of the best resources you'll find.\n\nThe basic form of the function in this kernel is as follows:\n\n`optimum_parameter_dictionary = quick_hyperopt(data, labels, 'model_type', NUM_EVALS)`\n\nwhere `'model_type'` is either `'lgbm'` for LightGBM, `'xgb'` for XGBoost or `'cb'` for CatBoost, and `NUM_EVALS` is the number of parameter optimisation rounds. `optimum_paramater_dictionary` is the parameter dictionary you can supply to the relevant model.\n\n# Usage Notes\n---\n\n* Parameter tuning can be a lengthy and intensive process. If the kernel is crashing or failing to finish, you may be using too much of your data; consider sampling it and running multiple kernels with each sample. You can then take the average/majority vote of each parameter that hyperopt selects. You can also reduce the number of evaluation rounds with the global parameter `NUM_EVALS`. \n\n* This function is written for regression tasks, but includes instructions on what must be modified if you are performing a classification task instead. I have tried to make this as simple as possible. By default it will use [ROC-AUC](http://gim.unmc.edu/dxtests/roc3.htm) as its objective metric for LightGBM/XGBoost and [Log Loss](https://www.kaggle.com/dansbecker/what-is-log-loss) for CatBoost.\n\n* For speed, when optimising a CatBoost model this function is designed to work with GPU. Some CatBoost parameters aren't yet written to work with GPU so I've left them out. LightGBM and XGBoost work quickly with Hyperopt on CPU alone but I've left options for activating them in the code - simply uncomment the relevant lines. For LightGBM you'll have to [compile the GPU version in your kernel by following the steps here.](https://www.kaggle.com/vinhnguyen/gpu-acceleration-for-lightgbm/) \n\n* You may want to include other parameters in the search space or remove others; hopefully the code is transparent enough for you to work out how to modify it yourself according to your needs. For example, you may want to use different evaluation metrics, learning rate ranges or model objectives.\n\n* The default objective metric for this function is Mean Absolute Error (MAE). For many regression tasks, Root Mean Squared Error (RMSE) is the preferred metric. Be sure to change this in the code below if required.\n\n# Data Preparation\n---\n\nYour data does not need any special preperation before using `quick_hyperopt()` beyond the requirements of the desired model (CatBoost for example may complain about NaNs and infinite values). Simply separate the features from the labels/targets. The training features can be either a Pandas DataFrame or 2-D NumPy array. The labels can be either a Pandas Series, a single-column Pandas DataFrame or a flattened 1-D NumPy array. "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd, numpy as np\ndata = pd.read_csv('../input/andrews-features/train_X.csv') #pandas DataFrame or numpy 2-D array\nlabels = pd.read_csv('../input/andrews-features/train_y.csv') #pandas Series, 1-column DataFrame or flattened numpy 1-D array","execution_count":1,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def col_replace(df, df2):\n    df_cols = df.columns.tolist()\n    df2_cols = df2.columns.tolist()\n    replace_cols = [x for x in df2_cols if x in df_cols]\n    add_cols = [x for x in df2_cols if x not in df_cols]\n    df[replace_cols] = df2[replace_cols]\n    df = pd.concat([df, df2[add_cols]], axis=1)\n    return(df.reset_index(drop=True))\n\ndata2 = pd.read_csv('../input/lanl-features/train_features_denoised.csv')\ndata = col_replace(data, data2).iloc[:-1, :].drop(['seg_id', 'target'], axis=1)","execution_count":2,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# quick_hyperopt()\n---\nCopy and paste the code below into your kernel and you're good to go! All questions, corrections and recommendations are welcome."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"#import required packages\nimport lightgbm as lgb\nimport xgboost as xgb\nimport catboost as cb\nimport gc\nfrom hyperopt import hp, tpe, Trials, STATUS_OK\nfrom hyperopt.fmin import fmin\nfrom hyperopt.pyll.stochastic import sample\n#optional but advised\nimport warnings\nwarnings.filterwarnings('ignore')\n\n#GLOBAL HYPEROPT PARAMETERS\nNUM_EVALS = 1000 #number of hyperopt evaluation rounds\nN_FOLDS = 5 #number of cross-validation folds on data in each evaluation round\n\n#LIGHTGBM PARAMETERS\nLGBM_MAX_LEAVES = 2**11 #maximum number of leaves per tree for LightGBM\nLGBM_MAX_DEPTH = 25 #maximum tree depth for LightGBM\nEVAL_METRIC_LGBM_REG = 'mae' #LightGBM regression metric. Note that 'rmse' is more commonly used \nEVAL_METRIC_LGBM_CLASS = 'auc'#LightGBM classification metric\n\n#XGBOOST PARAMETERS\nXGB_MAX_LEAVES = 2**12 #maximum number of leaves when using histogram splitting\nXGB_MAX_DEPTH = 25 #maximum tree depth for XGBoost\nEVAL_METRIC_XGB_REG = 'mae' #XGBoost regression metric\nEVAL_METRIC_XGB_CLASS = 'auc' #XGBoost classification metric\n\n#CATBOOST PARAMETERS\nCB_MAX_DEPTH = 8 #maximum tree depth in CatBoost\nOBJECTIVE_CB_REG = 'MAE' #CatBoost regression metric\nOBJECTIVE_CB_CLASS = 'Logloss' #CatBoost classification metric\n\n#OPTIONAL OUTPUT\nBEST_SCORE = 0\n\ndef quick_hyperopt(data, labels, package='lgbm', num_evals=NUM_EVALS, diagnostic=False):\n    \n    #==========\n    #LightGBM\n    #==========\n    \n    if package=='lgbm':\n        \n        print('Running {} rounds of LightGBM parameter optimisation:'.format(num_evals))\n        #clear space\n        gc.collect()\n        \n        integer_params = ['max_depth',\n                         'num_leaves',\n                          'max_bin',\n                         'min_data_in_leaf',\n                         'min_data_in_bin']\n        \n        def objective(space_params):\n            \n            #cast integer params from float to int\n            for param in integer_params:\n                space_params[param] = int(space_params[param])\n            \n            #extract nested conditional parameters\n            if space_params['boosting']['boosting'] == 'goss':\n                top_rate = space_params['boosting'].get('top_rate')\n                other_rate = space_params['boosting'].get('other_rate')\n                #0 <= top_rate + other_rate <= 1\n                top_rate = max(top_rate, 0)\n                top_rate = min(top_rate, 0.5)\n                other_rate = max(other_rate, 0)\n                other_rate = min(other_rate, 0.5)\n                space_params['top_rate'] = top_rate\n                space_params['other_rate'] = other_rate\n            \n            subsample = space_params['boosting'].get('subsample', 1.0)\n            space_params['boosting'] = space_params['boosting']['boosting']\n            space_params['subsample'] = subsample\n            \n            #for classification, set stratified=True and metrics=EVAL_METRIC_LGBM_CLASS\n            cv_results = lgb.cv(space_params, train, nfold = N_FOLDS, stratified=False,\n                                early_stopping_rounds=100, metrics=EVAL_METRIC_LGBM_REG, seed=42)\n            \n            best_loss = cv_results['l1-mean'][-1] #'l2-mean' for rmse\n            #for classification, comment out the line above and uncomment the line below:\n            #best_loss = 1 - cv_results['auc-mean'][-1]\n            #if necessary, replace 'auc-mean' with '[your-preferred-metric]-mean'\n            return{'loss':best_loss, 'status': STATUS_OK }\n        \n        train = lgb.Dataset(data, labels)\n                \n        #integer and string parameters, used with hp.choice()\n        boosting_list = [{'boosting': 'gbdt',\n                          'subsample': hp.uniform('subsample', 0.5, 1)},\n                         {'boosting': 'goss',\n                          'subsample': 1.0,\n                         'top_rate': hp.uniform('top_rate', 0, 0.5),\n                         'other_rate': hp.uniform('other_rate', 0, 0.5)}] #if including 'dart', make sure to set 'n_estimators'\n        metric_list = ['MAE', 'RMSE'] \n        #for classification comment out the line above and uncomment the line below\n        #metric_list = ['auc'] #modify as required for other classification metrics\n        objective_list_reg = ['huber', 'gamma', 'fair', 'tweedie']\n        objective_list_class = ['binary', 'cross_entropy']\n        #for classification set objective_list = objective_list_class\n        objective_list = objective_list_reg\n\n        space ={'boosting' : hp.choice('boosting', boosting_list),\n                'num_leaves' : hp.quniform('num_leaves', 2, LGBM_MAX_LEAVES, 1),\n                'max_depth': hp.quniform('max_depth', 2, LGBM_MAX_DEPTH, 1),\n                'max_bin': hp.quniform('max_bin', 32, 255, 1),\n                'min_data_in_leaf': hp.quniform('min_data_in_leaf', 1, 256, 1),\n                'min_data_in_bin': hp.quniform('min_data_in_bin', 1, 256, 1),\n                'min_gain_to_split' : hp.quniform('min_gain_to_split', 0.1, 5, 0.01),\n                'lambda_l1' : hp.uniform('lambda_l1', 0, 5),\n                'lambda_l2' : hp.uniform('lambda_l2', 0, 5),\n                'learning_rate' : hp.loguniform('learning_rate', np.log(0.005), np.log(0.2)),\n                'metric' : hp.choice('metric', metric_list),\n                'objective' : hp.choice('objective', objective_list),\n                'feature_fraction' : hp.quniform('feature_fraction', 0.5, 1, 0.01),\n                'bagging_fraction' : hp.quniform('bagging_fraction', 0.5, 1, 0.01)\n            }\n        \n        #optional: activate GPU for LightGBM\n        #follow compilation steps here:\n        #https://www.kaggle.com/vinhnguyen/gpu-acceleration-for-lightgbm/\n        #then uncomment lines below:\n        #space['device'] = 'gpu'\n        #space['gpu_platform_id'] = 0,\n        #space['gpu_device_id'] =  0\n\n        trials = Trials()\n        best = fmin(fn=objective,\n                    space=space,\n                    algo=tpe.suggest,\n                    max_evals=num_evals, \n                    trials=trials)\n                \n        #fmin() will return the index of values chosen from the lists/arrays in 'space'\n        #to obtain actual values, index values are used to subset the original lists/arrays\n        best['boosting'] = boosting_list[best['boosting']]['boosting']#nested dict, index twice\n        best['metric'] = metric_list[best['metric']]\n        best['objective'] = objective_list[best['objective']]\n                \n        #cast floats of integer params to int\n        for param in integer_params:\n            best[param] = int(best[param])\n        \n        print('{' + '\\n'.join('{}: {}'.format(k, v) for k, v in best.items()) + '}')\n        if diagnostic:\n            return(best, trials)\n        else:\n            return(best)\n    \n    #==========\n    #XGBoost\n    #==========\n    \n    if package=='xgb':\n        \n        print('Running {} rounds of XGBoost parameter optimisation:'.format(num_evals))\n        #clear space\n        gc.collect()\n        \n        integer_params = ['max_depth']\n        \n        def objective(space_params):\n            \n            for param in integer_params:\n                space_params[param] = int(space_params[param])\n                \n            #extract multiple nested tree_method conditional parameters\n            #libera te tutemet ex inferis\n            if space_params['tree_method']['tree_method'] == 'hist':\n                max_bin = space_params['tree_method'].get('max_bin')\n                space_params['max_bin'] = int(max_bin)\n                if space_params['tree_method']['grow_policy']['grow_policy']['grow_policy'] == 'depthwise':\n                    grow_policy = space_params['tree_method'].get('grow_policy').get('grow_policy').get('grow_policy')\n                    space_params['grow_policy'] = grow_policy\n                    space_params['tree_method'] = 'hist'\n                else:\n                    max_leaves = space_params['tree_method']['grow_policy']['grow_policy'].get('max_leaves')\n                    space_params['grow_policy'] = 'lossguide'\n                    space_params['max_leaves'] = int(max_leaves)\n                    space_params['tree_method'] = 'hist'\n            else:\n                space_params['tree_method'] = space_params['tree_method'].get('tree_method')\n                \n            #for classification replace EVAL_METRIC_XGB_REG with EVAL_METRIC_XGB_CLASS\n            cv_results = xgb.cv(space_params, train, nfold=N_FOLDS, metrics=[EVAL_METRIC_XGB_REG],\n                             early_stopping_rounds=100, stratified=False, seed=42)\n            \n            best_loss = cv_results['test-mae-mean'].iloc[-1] #or 'test-rmse-mean' if using RMSE\n            #for classification, comment out the line above and uncomment the line below:\n            #best_loss = 1 - cv_results['test-auc-mean'].iloc[-1]\n            #if necessary, replace 'test-auc-mean' with 'test-[your-preferred-metric]-mean'\n            return{'loss':best_loss, 'status': STATUS_OK }\n        \n        train = xgb.DMatrix(data, labels)\n        \n        #integer and string parameters, used with hp.choice()\n        boosting_list = ['gbtree', 'gblinear'] #if including 'dart', make sure to set 'n_estimators'\n        metric_list = ['MAE', 'RMSE'] \n        #for classification comment out the line above and uncomment the line below\n        #metric_list = ['auc']\n        #modify as required for other classification metrics classification\n        \n        tree_method = [{'tree_method' : 'exact'},\n               {'tree_method' : 'approx'},\n               {'tree_method' : 'hist',\n                'max_bin': hp.quniform('max_bin', 2**3, 2**7, 1),\n                'grow_policy' : {'grow_policy': {'grow_policy':'depthwise'},\n                                'grow_policy' : {'grow_policy':'lossguide',\n                                                  'max_leaves': hp.quniform('max_leaves', 32, XGB_MAX_LEAVES, 1)}}}]\n        \n        #if using GPU, replace 'exact' with 'gpu_exact' and 'hist' with\n        #'gpu_hist' in the nested dictionary above\n        \n        objective_list_reg = ['reg:linear', 'reg:gamma', 'reg:tweedie']\n        objective_list_class = ['reg:logistic', 'binary:logistic']\n        #for classification change line below to 'objective_list = objective_list_class'\n        objective_list = objective_list_reg\n        \n        space ={'boosting' : hp.choice('boosting', boosting_list),\n                'tree_method' : hp.choice('tree_method', tree_method),\n                'max_depth': hp.quniform('max_depth', 2, XGB_MAX_DEPTH, 1),\n                'reg_alpha' : hp.uniform('reg_alpha', 0, 5),\n                'reg_lambda' : hp.uniform('reg_lambda', 0, 5),\n                'min_child_weight' : hp.uniform('min_child_weight', 0, 5),\n                'gamma' : hp.uniform('gamma', 0, 5),\n                'learning_rate' : hp.loguniform('learning_rate', np.log(0.005), np.log(0.2)),\n                'eval_metric' : hp.choice('eval_metric', metric_list),\n                'objective' : hp.choice('objective', objective_list),\n                'colsample_bytree' : hp.quniform('colsample_bytree', 0.1, 1, 0.01),\n                'colsample_bynode' : hp.quniform('colsample_bynode', 0.1, 1, 0.01),\n                'colsample_bylevel' : hp.quniform('colsample_bylevel', 0.1, 1, 0.01),\n                'subsample' : hp.quniform('subsample', 0.5, 1, 0.05),\n                'nthread' : -1\n            }\n        \n        trials = Trials()\n        best = fmin(fn=objective,\n                    space=space,\n                    algo=tpe.suggest,\n                    max_evals=num_evals, \n                    trials=trials)\n        \n        best['tree_method'] = tree_method[best['tree_method']]['tree_method']\n        best['boosting'] = boosting_list[best['boosting']]\n        best['eval_metric'] = metric_list[best['eval_metric']]\n        best['objective'] = objective_list[best['objective']]\n        \n        #cast floats of integer params to int\n        for param in integer_params:\n            best[param] = int(best[param])\n        if 'max_leaves' in best:\n            best['max_leaves'] = int(best['max_leaves'])\n        if 'max_bin' in best:\n            best['max_bin'] = int(best['max_bin'])\n        \n        print('{' + '\\n'.join('{}: {}'.format(k, v) for k, v in best.items()) + '}')\n        \n        if diagnostic:\n            return(best, trials)\n        else:\n            return(best)\n    \n    #==========\n    #CatBoost\n    #==========\n    \n    if package=='cb':\n        \n        print('Running {} rounds of CatBoost parameter optimisation:'.format(num_evals))\n        \n        #clear memory \n        gc.collect()\n            \n        integer_params = ['depth',\n                          #'one_hot_max_size', #for categorical data\n                          'min_data_in_leaf',\n                          'max_bin']\n        \n        def objective(space_params):\n                        \n            #cast integer params from float to int\n            for param in integer_params:\n                space_params[param] = int(space_params[param])\n                \n            #extract nested conditional parameters\n            if space_params['bootstrap_type']['bootstrap_type'] == 'Bayesian':\n                bagging_temp = space_params['bootstrap_type'].get('bagging_temperature')\n                space_params['bagging_temperature'] = bagging_temp\n                \n            if space_params['grow_policy']['grow_policy'] == 'LossGuide':\n                max_leaves = space_params['grow_policy'].get('max_leaves')\n                space_params['max_leaves'] = int(max_leaves)\n                \n            space_params['bootstrap_type'] = space_params['bootstrap_type']['bootstrap_type']\n            space_params['grow_policy'] = space_params['grow_policy']['grow_policy']\n                           \n            #random_strength cannot be < 0\n            space_params['random_strength'] = max(space_params['random_strength'], 0)\n            #fold_len_multiplier cannot be < 1\n            space_params['fold_len_multiplier'] = max(space_params['fold_len_multiplier'], 1)\n                       \n            #for classification set stratified=True\n            cv_results = cb.cv(train, space_params, fold_count=N_FOLDS, \n                             early_stopping_rounds=25, stratified=False, partition_random_seed=42)\n           \n            best_loss = cv_results['test-MAE-mean'].iloc[-1] #'test-RMSE-mean' for RMSE\n            #for classification, comment out the line above and uncomment the line below:\n            #best_loss = cv_results['test-Logloss-mean'].iloc[-1]\n            #if necessary, replace 'test-Logloss-mean' with 'test-[your-preferred-metric]-mean'\n            \n            return{'loss':best_loss, 'status': STATUS_OK}\n        \n        train = cb.Pool(data, labels.astype('float32'))\n        \n        #integer and string parameters, used with hp.choice()\n        bootstrap_type = [{'bootstrap_type':'Poisson'}, \n                           {'bootstrap_type':'Bayesian',\n                            'bagging_temperature' : hp.loguniform('bagging_temperature', np.log(1), np.log(50))},\n                          {'bootstrap_type':'Bernoulli'}] \n        LEB = ['No', 'AnyImprovement', 'Armijo'] #remove 'Armijo' if not using GPU\n        #score_function = ['Correlation', 'L2', 'NewtonCorrelation', 'NewtonL2']\n        grow_policy = [{'grow_policy':'SymmetricTree'},\n                       {'grow_policy':'Depthwise'},\n                       {'grow_policy':'Lossguide',\n                        'max_leaves': hp.quniform('max_leaves', 2, 32, 1)}]\n        eval_metric_list_reg = ['MAE', 'RMSE', 'Poisson']\n        eval_metric_list_class = ['Logloss', 'AUC', 'F1']\n        #for classification change line below to 'eval_metric_list = eval_metric_list_class'\n        eval_metric_list = eval_metric_list_reg\n                \n        space ={'depth': hp.quniform('depth', 2, CB_MAX_DEPTH, 1),\n                'max_bin' : hp.quniform('max_bin', 1, 32, 1), #if using CPU just set this to 254\n                'l2_leaf_reg' : hp.uniform('l2_leaf_reg', 0, 5),\n                'min_data_in_leaf' : hp.quniform('min_data_in_leaf', 1, 50, 1),\n                'random_strength' : hp.loguniform('random_strength', np.log(0.005), np.log(5)),\n                #'one_hot_max_size' : hp.quniform('one_hot_max_size', 2, 16, 1), #uncomment if using categorical features\n                'bootstrap_type' : hp.choice('bootstrap_type', bootstrap_type),\n                'learning_rate' : hp.uniform('learning_rate', 0.05, 0.25),\n                'eval_metric' : hp.choice('eval_metric', eval_metric_list),\n                'objective' : OBJECTIVE_CB_REG,\n                #'score_function' : hp.choice('score_function', score_function), #crashes kernel - reason unknown\n                'leaf_estimation_backtracking' : hp.choice('leaf_estimation_backtracking', LEB),\n                'grow_policy': hp.choice('grow_policy', grow_policy),\n                #'colsample_bylevel' : hp.quniform('colsample_bylevel', 0.1, 1, 0.01),# CPU only\n                'fold_len_multiplier' : hp.loguniform('fold_len_multiplier', np.log(1.01), np.log(2.5)),\n                'od_type' : 'Iter',\n                'od_wait' : 25,\n                'task_type' : 'GPU',\n                'verbose' : 0\n            }\n        \n        #optional: run CatBoost without GPU\n        #uncomment line below\n        #space['task_type'] = 'CPU'\n            \n        trials = Trials()\n        best = fmin(fn=objective,\n                    space=space,\n                    algo=tpe.suggest,\n                    max_evals=num_evals, \n                    trials=trials)\n        \n        #unpack nested dicts first\n        best['bootstrap_type'] = bootstrap_type[best['bootstrap_type']]['bootstrap_type']\n        best['grow_policy'] = grow_policy[best['grow_policy']]['grow_policy']\n        best['eval_metric'] = eval_metric_list[best['eval_metric']]\n        \n        #best['score_function'] = score_function[best['score_function']] \n        #best['leaf_estimation_method'] = LEM[best['leaf_estimation_method']] #CPU only\n        best['leaf_estimation_backtracking'] = LEB[best['leaf_estimation_backtracking']]        \n        \n        #cast floats of integer params to int\n        for param in integer_params:\n            best[param] = int(best[param])\n        if 'max_leaves' in best:\n            best['max_leaves'] = int(best['max_leaves'])\n        \n        print('{' + '\\n'.join('{}: {}'.format(k, v) for k, v in best.items()) + '}')\n        \n        if diagnostic:\n            return(best, trials)\n        else:\n            return(best)\n    \n    else:\n        print('Package not recognised. Please use \"lgbm\" for LightGBM, \"xgb\" for XGBoost or \"cb\" for CatBoost.')              ","execution_count":3,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Examples\n---\nHere are some example usages. For reference, the data in this kernel is approximately 4200 rows by 136 columns. Participants of the LANL Earthquake Prediction challenge will know this dataset as \"Andrew's Features\". Simply run the function `quick_hyperopt()` with your training data & labels along with the string parameter for your model framework. \n\n---\n# LightGBM\n![](https://i.imgur.com/Jqw0FGz.jpg)\n\n\nLightGBM works quickly with Hyperopt and you can easily expect to run thousands of evaluations, conditional on the size of your dataset. The default parameter space I've defined covers a very broad range and will likely not need modifying unless you're running a classification task. Instructions for doing so are in the code.\n\nThe `num_leaves` parameter should be, at a maximum, equal to `2**max_depth`. In my experience this is overkill for the vast majority of cases when `max_depth` is greater than twelve and will quickly lead to overfitting. I've manually set it to `2**11` but you can easily alter it with the constant `LGBM_MAX_LEAVES` for your own purposes. Set it too high and it'll quickly exhaust the kernel memory - you can increase your available RAM by disabling the GPU disabled if desired.\n\nFor speed I haven't specified `'dart'` as one of the boosting options. If you want to include it, bear in mind that DART boosting cannot use early stopping so you'll have to specify `n_estimators` instead."},{"metadata":{"trusted":true},"cell_type":"code","source":"#obtain optimised parameter dictionary\nlgbm_params = quick_hyperopt(data, labels, 'lgbm', 2500)","execution_count":null,"outputs":[{"output_type":"stream","text":"Running 1000 rounds of LightGBM parameter optimisation:\n  4%|▍         | 40/1000 [03:50<1:02:32,  3.91s/it, best loss: 2.0360643634166644]","name":"stdout"}]},{"metadata":{},"cell_type":"markdown","source":"The parameter dictionary can then be saved as a kernel output. Remember to use `.item()` when loading it in a new kernel."},{"metadata":{"trusted":true},"cell_type":"code","source":"#open with: lgbm_params = np.load('../input/YOUR_FILEPATH/lgbm_params.npy').item()\nnp.save('lgbm_params.npy', lgbm_params)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---\n# XGBoost\n![](https://raw.githubusercontent.com/dmlc/dmlc.github.io/master/img/logo-m/xgboost.png)\n\nMy experience is that XGBoost takes longer than LightGBM to reach its optimum parameters. With less than 1000 evaluation rounds it can struggle to beat its default parameters! While it does eventually perform better than its defaults, to save time you might want to identify a set of parameters that already works well and then run a more intensive optimisation in a smaller range around them.\n\nHowever, XGBoost works quickly with Hyperopt and the parameter space I've set up should cover most use cases. Again, `'dart'` has been left out for speed.\n\nThis kernel only demonstrates LightGBM parameter tuning; you can view the XGBoost equivalent in earlier versions.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#xgb_params = quick_hyperopt(data, labels, 'xgb', 2000)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And save the dictionary to kernel output:"},{"metadata":{"trusted":true},"cell_type":"code","source":"#np.save('xgb_params.npy', xgb_params)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---\n# CatBoost \n![](https://siliconangle.com/wp-content/blogs.dir/1/files/2017/07/Yandex-CatBoost.png)\nCatBoost is more complicated when it comes to parameter tuning. The main issue is that defining too broad a parameter space will make it impractically slow or request too much space from the RAM, leading to a kernel crash. Some optimisation parameters work with the GPU, some do not. Frustratingly, the parameter `'rsm'`, alias `'colsample_bylevel'` which can dramatically speed up training, does not work with the GPU. I've tried to make the initial code as appropriate for general use as possible, but you will find you need to manually configure this process more than you would for LightGBM or XGBoost in accordance with the size and complexity of your data. Here are some official guidelines for CatBoost parameter tuning:\n\n* [Parameter tuning guidelines](https://catboost.ai/docs/concepts/parameter-tuning.html)\n* [Speeding up training](https://catboost.ai/docs/concepts/speed-up-training.html)\n* [Full CatBoost training parameter list](https://catboost.ai/docs/concepts/python-reference_train.html)\n* [CatBoost cross-validation guide](https://catboost.ai/docs/concepts/python-reference_cv.html)\n\nIf this process is working too slowly, here are some measures you can try:\n\n* Instead of using early stopping via `od_type` and `od_wait`, set a fixed number of trees using `iterations`. You may want to use a range of larger values for `learning_rate` when doing this, as low values may prevent model convergence.\n* minimise the range of `border_count`/`max_bin`. If training on GPU, the CatBoost docs do not recommend going over 32.\n* decrease the range of `max_leaves`. The CatBoost docs do not recommend going over 64. For speed, the current limit is 32.\n* `max_depth` increases the CV time exponentially - decreasing its maximum value in the code is probably the most important single change for accelerating cross-validation. The CatBoost docs state there is infrequently any benefit from values over 10. For the sake of speed in these example cases, I've set the maximum to 8. If you wish to adjust this, change the value of `CB_MAX_DEPTH`.\n\nThis implementation does not include any categorical features. If these are present in your training data you will have to add the 'cat_features' argument manually, and uncomment the 'one_hot_max_size' parameter in `quick_hyperopt()` if desired.\n\nNote that `eval_metric` is the metric that determines early stopping and it is not necessarily the loss metric/objective.\n\nThis kernel only demonstrates LightGBM parameter tuning; you can view the CatBoost equivalent in earlier versions."},{"metadata":{"trusted":true},"cell_type":"code","source":"#cb_params = quick_hyperopt(data, labels, 'cb', 100)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Again, when training is complete, save the dictionary to kernel output:"},{"metadata":{"trusted":true},"cell_type":"code","source":"#np.save('cb_params.npy', cb_params)#","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Diagnostic Mode\n---\nIf you wish to examine the individual evaluations, for example to see if there are any particular factors slowing down cross-validation, set `diagnostic=True` in `quick_hyperopt()`. This will output a tuple of the best iteration parameter dictionary, along with a Hyperopt trial list object of every round."},{"metadata":{"trusted":true},"cell_type":"code","source":"example_params, example_trials = quick_hyperopt(data, labels, 'lgbm', 5, diagnostic=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To examine the nth round, extract the trial dictionary with `your_trial_output.trials[n]`. The `misc['vals']` entry in this dictionary will display the parameters tried during that round. Each entry also records the start and end time for that iteration. "},{"metadata":{"trusted":true},"cell_type":"code","source":"example_trials.trials[3]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# LANL Earthquake Prediction\n---\nAs an experiment, let's compare the CV and LB scores obtained from the `quick_hyperopt()` parameters, and those from [this popular LANL Earthquake Prediction kernel.](https://www.kaggle.com/artgor/earthquakes-fe-more-features-and-samples)\n\nThe new and expanded Andrew's Features are used in this version."},{"metadata":{"trusted":true},"cell_type":"code","source":"lanl_params = {'num_leaves': 128,\n          'min_data_in_leaf': 79,\n          'objective': 'gamma',\n          'max_depth': -1,\n          'learning_rate': 0.01,\n          \"boosting\": \"gbdt\",\n          \"bagging_freq\": 5,\n          \"bagging_fraction\": 0.9,\n          \"bagging_seed\": 11,\n          \"metric\": 'mae',\n          \"verbosity\": -1,\n          'reg_alpha': 0.1302650970728192,\n          'reg_lambda': 0.3603427518866501,\n          'feature_fraction': 0.1\n         }","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import KFold\nfrom sklearn.metrics import mean_absolute_error\n\nsub_1 = pd.read_csv('../input/LANL-Earthquake-Prediction/sample_submission.csv')\nsub_2 = pd.read_csv('../input/LANL-Earthquake-Prediction/sample_submission.csv')\ntest = pd.read_csv('../input/andrews-features/test_X.csv')\ntest2 = pd.read_csv('../input/lanl-features/test_features_denoised.csv')\ntest = col_replace(test, test2).drop(['seg_id', 'target'], axis=1)\n\npreds1 = np.zeros(len(sub_1))\npreds2 = np.zeros(len(sub_2))\n\nMAE_val1 = 0\nMAE_val2 = 0\n\nn_fold = 5\nfolds = KFold(n_splits=n_fold, shuffle=True, random_state=42)\nX = data\ny = labels.time_to_failure.values\n\nfor train_idx, valid_idx in folds.split(y):\n    X_train, X_valid = X.iloc[train_idx], X.iloc[valid_idx]\n    y_train, y_valid = y[train_idx], y[valid_idx]\n        \n    model1 = lgb.LGBMRegressor(**lanl_params, n_estimators = 50000, n_jobs = -1, eval_metric='mae')\n    model1.fit(X_train, y_train,eval_set=[(X_train, y_train), (X_valid, y_valid)],\n              verbose=0, early_stopping_rounds=200)\n    val_preds1 = model1.predict(X_valid)\n    MAE_val1 += mean_absolute_error(y_valid, val_preds1)/n_fold\n    preds1 += model1.predict(test)/n_fold\n    \n    model2 = lgb.LGBMRegressor(**lgbm_params, n_estimators = 50000, n_jobs = -1)\n    model2.fit(X_train, y_train,eval_set=[(X_train, y_train), (X_valid, y_valid)],\n              verbose=0, early_stopping_rounds=200)\n    val_preds2 = model2.predict(X_valid)\n    MAE_val2 += mean_absolute_error(y_valid, val_preds2)/n_fold\n    preds2 += model2.predict(test)/n_fold\n    \nsub_1['time_to_failure'] = preds1\nsub_2['time_to_failure'] = preds2\nsub_1.to_csv('submission_lanl_params.csv', index=False)\nsub_2.to_csv('submission_hyperopt_params.csv', index=False)\n\nprint('CV score - public kernel parameters: {}'.format(MAE_val1))\nprint('CV score - quick_hyperopt parameters: {}'.format(MAE_val2))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The Hyperopt parameters clearly improve the cross-validation score, and the leaderboard scores for the previous kernel were as follows:\n\n* LANL kernel parameters: CV 2.047 LB 1.516\n* `quick_hyperopt` parameters: CV 2.030 LB **1.493**\n\n# Summary\n---\n\nIt's as simple as that! I've designed this function so that it will be 'plug-in-and-play' for the majority of regression cases. For LightGBM and XGBoost it will work without a GPU but for CatBoost some parameters will need to be removed from the optimisation process if you want to run on CPU alone. I don't recommend this since parameter tuning CatBoost with a GPU is already very slow. It is likely that the only parameters you will need to change are the number of optimisation rounds via `NUM_EVALS`, and the respective tree depths/maximum leaf counts for the different model frameworks. If you need to modify `quick_hyperopt()` for classification, instructions are in the code. Bear in mind that you may need to consider parameters that are unique to your data, such as `scale_pos_weight` if the classes are imbalanced. For other modifications please consult the Hyperopt docs, along with those for LightGBM, XGBoost and CatBoost:\n\n* [Hyperopt documentation](https://github.com/hyperopt/hyperopt/wiki/FMin)\n* [LightGBM documentation](https://lightgbm.readthedocs.io/en/latest/)\n* [XGBoost documentation](https://xgboost.readthedocs.io/en/latest/index.html)\n* [CatBoost documentation](https://catboost.ai/docs/)\n\nI hope this kernel will be of use to you. "}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}