{"cells":[{"metadata":{"_uuid":"760ffd240ce64ecb7e79aed42c6e96a8f55ab887","_cell_guid":"d91f91e7-77a8-487e-96ab-4a8660d25691"},"cell_type":"markdown","source":"## Bayesian Optimization - LightGBM"},{"metadata":{"_uuid":"e1b9d57b7a640194063b2d442ebaf2b0c1505c4c","_cell_guid":"e024538e-8d42-43a6-879f-803ee439849d"},"cell_type":"markdown","source":"Thanks to [NanoMathias's awesome notebook](https://www.kaggle.com/nanomathias/bayesian-optimization-of-xgboost-lb-0-9769/notebook), I got introduced to Scikit-Optimize and really felt the power of beyesian approach in parameter tuning.\n\nHowever, it seems like BayesSearchCV is still not capable of dealing with complex setups(e.g. specify categorical features in lightgbm)(Update: as Bai Xue has pointed out in comment section: BayesSearchCV is actually more flexible than I thought..)\n . After hours of exploring the package, here I'm going to shed some light on how to build a very flexible BayesSearch framework by using gp_minimize.\n"},{"metadata":{"collapsed":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":false},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport xgboost as xgb\nimport lightgbm as lgb\nimport gc\n\nfrom skopt.space import Real, Integer\nfrom skopt.utils import use_named_args\nimport itertools\nfrom sklearn.metrics import roc_auc_score\nfrom skopt import gp_minimize","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a237daafe8d38da0c196ccd13e7272047168e876","_cell_guid":"531ecd2a-d9df-44ec-85fd-ebfe4d70d6b6"},"cell_type":"markdown","source":"Instead of doing cross-validation, in this notebook I'm going to use a simple fixed training and testing set. Just for illustration purpose :)"},{"metadata":{"_kg_hide-input":false,"collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_kg_hide-output":false,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"code","source":"TRAINING_SIZE = 300000\nTEST_SIZE = 50000\n\n# Load data\ntrain = pd.read_csv(\n    '../input/train.csv', \n    skiprows=range(1,184903891-TRAINING_SIZE-TEST_SIZE), \n    nrows=TRAINING_SIZE,\n    parse_dates=['click_time']\n)\n\nval = pd.read_csv(\n    '../input/train.csv', \n    skiprows=range(1,184903891-TEST_SIZE), \n    nrows=TEST_SIZE,\n    parse_dates=['click_time']\n)\n\n# Split into X and y\ny_train = train['is_attributed']\ny_val = val['is_attributed']","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f308b3bc25f1eaa81cc13e3d57b8b98d34c7bffa","_cell_guid":"756bb2a5-0dee-4d90-9e41-0b7a9a7462b2"},"cell_type":"markdown","source":"Specify the parameter space we want to explore."},{"metadata":{"collapsed":true,"_uuid":"e83363e95806938192ff04c28fc9ab782cf8637c","_cell_guid":"40ee5aeb-1871-4f0d-94df-18470c83fba5","trusted":false},"cell_type":"code","source":"# from that dimension (`'log-uniform'` for the learning rate)\nspace  = [Integer(3, 10, name='max_depth'),\n          Integer(6, 30, name='num_leaves'),\n          Integer(50, 200, name='min_child_samples'),\n          Real(1, 400,  name='scale_pos_weight'),\n          Real(0.6, 0.9, name='subsample'),\n          Real(0.6, 0.9, name='colsample_bytree')\n         ]\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1748c3ea179a3e6a5361032c1a06d8fa93cc8196","_cell_guid":"f765f02f-d532-42d0-8d82-76ba697ff80f"},"cell_type":"markdown","source":"Below is the fun part. The function gp_minimize requires an objective function and what the function all needs is basically a metric we want to minimize. Of course, we can just use whatever training setup we have been using but just tweak it to return a AUC to minimize..(negative AUC)"},{"metadata":{"collapsed":true,"_uuid":"89fc544ec66b8c6793d92339c2306cdea02067fa","_cell_guid":"f22fdbcc-95f8-4556-8ef0-658e39c6a4b2","trusted":false},"cell_type":"code","source":"def objective(values):\n    \n\n    params = {'max_depth': values[0], \n          'num_leaves': values[1], \n          'min_child_samples': values[2], \n          'scale_pos_weight': values[3],\n            'subsample': values[4],\n            'colsample_bytree': values[5],\n             'metric':'auc',\n             'nthread': 8,\n             'boosting_type': 'gbdt',\n             'objective': 'binary',\n             'learning_rate':0.15,\n             'max_bin': 100,\n             'min_child_weight': 0,\n             'min_split_gain': 0,\n             'subsample_freq': 1}\n    \n\n    print('\\nNext set of params.....',params)\n    \n    feature_set = ['app','device','os','channel']\n    categorical = ['app','device','os','channel']\n    \n    \n    early_stopping_rounds = 50\n    num_boost_round       = 1000\n    \n        # Fit model on feature_set and calculate validation AUROC\n    xgtrain = lgb.Dataset(train[feature_set].values, label=y_train,feature_name=feature_set,\n                           categorical_feature=categorical)\n    xgvalid = lgb.Dataset(val[feature_set].values, label=y_val,feature_name=feature_set,\n                          categorical_feature=categorical)\n    \n    evals_results = {}\n    model_lgb     = lgb.train(params,xgtrain,valid_sets=[xgtrain, xgvalid], \n                              valid_names=['train','valid'], \n                               evals_result=evals_results, \n                               num_boost_round=num_boost_round,\n                                early_stopping_rounds=early_stopping_rounds,\n                               verbose_eval=None, feval=None)\n    \n    auc = -roc_auc_score(y_val, model_lgb.predict(val[model_lgb.feature_name()]))\n    \n    print('\\nAUROC.....',-auc,\".....iter.....\", model_lgb.current_iteration())\n    \n    gc.collect()\n    \n    return  auc","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"_uuid":"c9dc851b448edf18c85c5ecee344d2dfbdb72f67","_cell_guid":"743891e0-d0c6-4fb1-9cb5-4841aea8687c"},"cell_type":"markdown","source":"Alright, let's run the tuning process."},{"metadata":{"_uuid":"5916f40b386cbac26973d83bd7d2aad5aa86d39a","_cell_guid":"e9078725-44fc-4b29-82a5-67aa2c25b548","trusted":false,"collapsed":true},"cell_type":"code","source":"res_gp = gp_minimize(objective, space, n_calls=20,\n                     random_state=0,n_random_starts=10)\n\n\"Best score=%.4f\" % res_gp.fun","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"60d82f1eeb64d0db056109ee74cf718e49caa78b","_cell_guid":"012dc052-b124-4cc6-b250-1ba8e9effee9","trusted":false,"collapsed":true},"cell_type":"code","source":"from skopt.plots import plot_convergence\n\nplot_convergence(res_gp)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b92d086303e87f3bcfdd0c498ba6382de7194330","_cell_guid":"17b8f90c-c766-4819-99a1-873307f490b3"},"cell_type":"markdown","source":"The best AUC here is pretty weird but I guess that's all because I'm using a very arbitrary training and validation set :)"}],"metadata":{"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}