{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"},"cell_type":"markdown","source":"# Bayesian hyperparameter tuning of xgBoost\nI took the features generated in my [previous notebook](https://www.kaggle.com/nanomathias/feature-engineering-importance-testing) and tried to tune a xgBoost model to those features using bayesian optimization. I ran the code locally for a few days, after which it had found some good parameters which gave a score of 0.9769 (using all the features of the previous notebook) - this worked out well for me, since I do not have a lot of time to actively work on trying out different models etc, but letting a script like this run for a few days to find good parameters is easy :)"},{"metadata":{"_uuid":"8d84a68ab037f61ce6ce918fcc8e0e120b52ff5c","_cell_guid":"d157e67d-de73-49e8-8c06-f7912a102ef1"},"cell_type":"markdown","source":"## Example 1: xgBoost Parameter Tuning with Scikit-Optimize\nThe following code is exactly what I used to tune the parameters - only difference is that I ran with the last 20 million samples in training set, and used the features from [previous notebook](https://www.kaggle.com/nanomathias/feature-engineering-importance-testing) instead of the raw features as done here. First I'll load the needed libraries and data."},{"metadata":{"collapsed":true,"_uuid":"0861d5e64d73ee7de621d9289095a877f76369e5","_cell_guid":"db4170d4-695a-455a-9cb9-a3d6c2a99f21","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport xgboost as xgb\nimport lightgbm as lgb\nfrom skopt import BayesSearchCV\nfrom sklearn.model_selection import StratifiedKFold\n\n# SETTINGS - CHANGE THESE TO GET SOMETHING MEANINGFUL\nITERATIONS = 10 # 1000\nTRAINING_SIZE = 100000 # 20000000\nTEST_SIZE = 25000\n\n# Load data\nX = pd.read_csv(\n    '../input/train.csv', \n    skiprows=range(1,184903891-TRAINING_SIZE), \n    nrows=TRAINING_SIZE,\n    parse_dates=['click_time']\n)\n\n# Split into X and y\ny = X['is_attributed']\nX = X.drop(['click_time','is_attributed', 'attributed_time'], axis=1)","execution_count":1,"outputs":[]},{"metadata":{"_uuid":"93c4122474339eeecc093e9cbcc79d4f4ea2aa0c","_cell_guid":"48b0456d-399c-43b4-b8d3-83c73351e470"},"cell_type":"markdown","source":"To do the bayesian parameter tuning, I use the [BayesSearchCV](https://scikit-optimize.github.io/#skopt.BayesSearchCV) class of scikit-optimize. It works basically as a drop-in replacement for GridSearchCV and RandomSearchCV, but generally I get better results with it. In the following I define the BayesSearchCV object, and write a short convenience function that will be used during optimization to output current status of the tuning. Locally I have access to more cores and run with n_jobs=4 for the classifier, and n_jobs=6 for the BayesSearchCV object."},{"metadata":{"collapsed":true,"_uuid":"cdc63dc8bb9542c4ad02a308d7c3d984b09dad8e","_cell_guid":"f8217176-75c2-4819-85c9-95b5afa3a14f","trusted":true},"cell_type":"code","source":"# Classifier\nbayes_cv_tuner = BayesSearchCV(\n    estimator = xgb.XGBClassifier(\n        n_jobs = 1,\n        objective = 'binary:logistic',\n        eval_metric = 'auc',\n        silent=1,\n        tree_method='approx'\n    ),\n    search_spaces = {\n        'learning_rate': (0.01, 1.0, 'log-uniform'),\n        'min_child_weight': (0, 10),\n        'max_depth': (0, 50),\n        'max_delta_step': (0, 20),\n        'subsample': (0.01, 1.0, 'uniform'),\n        'colsample_bytree': (0.01, 1.0, 'uniform'),\n        'colsample_bylevel': (0.01, 1.0, 'uniform'),\n        'reg_lambda': (1e-9, 1000, 'log-uniform'),\n        'reg_alpha': (1e-9, 1.0, 'log-uniform'),\n        'gamma': (1e-9, 0.5, 'log-uniform'),\n        'min_child_weight': (0, 5),\n        'n_estimators': (50, 100),\n        'scale_pos_weight': (1e-6, 500, 'log-uniform')\n    },    \n    scoring = 'roc_auc',\n    cv = StratifiedKFold(\n        n_splits=3,\n        shuffle=True,\n        random_state=42\n    ),\n    n_jobs = 3,\n    n_iter = ITERATIONS,   \n    verbose = 0,\n    refit = True,\n    random_state = 42\n)\n\ndef status_print(optim_result):\n    \"\"\"Status callback durring bayesian hyperparameter search\"\"\"\n    \n    # Get all the models tested so far in DataFrame format\n    all_models = pd.DataFrame(bayes_cv_tuner.cv_results_)    \n    \n    # Get current parameters and the best parameters    \n    best_params = pd.Series(bayes_cv_tuner.best_params_)\n    print('Model #{}\\nBest ROC-AUC: {}\\nBest params: {}\\n'.format(\n        len(all_models),\n        np.round(bayes_cv_tuner.best_score_, 4),\n        bayes_cv_tuner.best_params_\n    ))\n    \n    # Save all model results\n    clf_name = bayes_cv_tuner.estimator.__class__.__name__\n    all_models.to_csv(clf_name+\"_cv_results.csv\")","execution_count":2,"outputs":[]},{"metadata":{"_uuid":"1a45b9722d17bdf4d97bc75fbf327311b21bd865","_cell_guid":"96cf894e-d153-45f5-829c-58bc427951b5"},"cell_type":"markdown","source":"Finally, let the parameter tuning run and wait for good results :)"},{"metadata":{"_uuid":"8f1a1c02de0c17f60ed834267aefdc6c28dd9b78","_cell_guid":"997763ce-92ed-463c-9f99-6d2f4af40b62","trusted":true},"cell_type":"code","source":"# Fit the model\nresult = bayes_cv_tuner.fit(X.values, y.values, callback=status_print)","execution_count":3,"outputs":[]},{"metadata":{"_uuid":"9dfa5be24aec747b6798424b8906e11486b4208c","_cell_guid":"df2220f9-a7f0-4554-826a-f59073afb9a6"},"cell_type":"markdown","source":"## Example 2: lightGBM Parameter Tuning with Scikit-Optimize\nI have not myself submitted any models run with lightGBM as of yet, but here is an example of how to run the parameter search with lightGBM instead of xgBoost"},{"metadata":{"_uuid":"784c2278a48af51e01adda8be4471f94d5bf83d8","_cell_guid":"3257fa29-0cde-45af-83c3-830730d20df2","trusted":true},"cell_type":"code","source":"# Classifier\nbayes_cv_tuner = BayesSearchCV(\n    estimator = lgb.LGBMRegressor(\n        objective='binary',\n        metric='auc',\n        n_jobs=1,\n        verbose=0\n    ),\n    search_spaces = {\n        'learning_rate': (0.01, 1.0, 'log-uniform'),\n        'num_leaves': (1, 100),      \n        'max_depth': (0, 50),\n        'min_child_samples': (0, 50),\n        'max_bin': (100, 1000),\n        'subsample': (0.01, 1.0, 'uniform'),\n        'subsample_freq': (0, 10),\n        'colsample_bytree': (0.01, 1.0, 'uniform'),\n        'min_child_weight': (0, 10),\n        'subsample_for_bin': (100000, 500000),\n        'reg_lambda': (1e-9, 1000, 'log-uniform'),\n        'reg_alpha': (1e-9, 1.0, 'log-uniform'),\n        'scale_pos_weight': (1e-6, 500, 'log-uniform'),\n        'n_estimators': (50, 100),\n    },    \n    scoring = 'roc_auc',\n    cv = StratifiedKFold(\n        n_splits=3,\n        shuffle=True,\n        random_state=42\n    ),\n    n_jobs = 3,\n    n_iter = ITERATIONS,   \n    verbose = 0,\n    refit = True,\n    random_state = 42\n)\n\n# Fit the model\nresult = bayes_cv_tuner.fit(X.values, y.values, callback=status_print)","execution_count":4,"outputs":[]},{"metadata":{"_uuid":"dbebf5d16038dea8154a2d85a8cd10b8f0971900"},"cell_type":"markdown","source":"## Example 3: Different cross-validators\nSome people have asked about CV strategy, and as seen I've just used the basic Stratified K-fold strategy; that was however mostly due to time constraints, and thus me not thinking that much about it. There are a lot of potentially better options, especially considering the temporal nature of this problem. Adding these are really easy using scikit-learn cross-validators; you just plug-n-play a new cross-validator into the `cv = ` options of BayesSearchCV. Examples could be a single train-test split, where we e.g. use one day for training, and one for testing (adjust accordingly):"},{"metadata":{"trusted":true,"_uuid":"13547a74b0781f10842a0d37e2f1147692e29752"},"cell_type":"code","source":"from sklearn.model_selection import PredefinedSplit\n\n# Training [index == -1], testing [index == 0])\ntest_fold = np.zeros(len(X))\ntest_fold[:(TRAINING_SIZE-TEST_SIZE)] = -1\ncv = PredefinedSplit(test_fold)\n\n# Check that we only have a single train-test split, and the size\ntrain_idx, test_idx = next(cv.split())\nprint(f\"Splits: {cv.get_n_splits()}, Train size: {len(train_idx)}, Test size: {len(test_idx)}\")","execution_count":6,"outputs":[]},{"metadata":{"_uuid":"7d8af2a9878653c53e9ad980b417c2326a0eede1"},"cell_type":"markdown","source":"Alternatively, we could want to use the [TimeSeriesSplit](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html#sklearn.model_selection.TimeSeriesSplit) cross-validator, which allows us to do several \"into the future folds\" for predictions"},{"metadata":{"trusted":true,"_uuid":"370cfec31720863437b0841cc9957e2b834483af"},"cell_type":"code","source":"from sklearn.model_selection import TimeSeriesSplit\n\n# Here we just do 3-fold timeseries CV\ncv = TimeSeriesSplit(max_train_size=None, n_splits=3)\n\n# Let us check the sizes of the folds. Note that you can keep train size constant with max_train_size if needed\nfor i, (train_index, test_index) in enumerate(cv.split(X)):\n    print(f\"Split {i+1} / {cv.get_n_splits()}:, Train size: {len(train_index)}, Test size: {len(test_index)}\")","execution_count":14,"outputs":[]},{"metadata":{"_uuid":"5b24d7d149418711a5bef90593e2bdf72d29a6a6","_cell_guid":"4fc94139-f487-4c60-9cd6-4c7f19048eea"},"cell_type":"markdown","source":"## Optimal xgBoost parameters\n![](http://)After a few days of running for xgBoost, it found the following optimal parameters. Again, note that these gave me a 0.9769 score on [these features](https://www.kaggle.com/nanomathias/feature-engineering-importance-testing) and not the raw features, by training on the entire training set."},{"metadata":{"collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"code","source":"{\n    'colsample_bylevel': 0.1,\n    'colsample_bytree': 1.0,\n    'gamma': 5.103973694670875e-08,\n    'learning_rate': 0.140626707498132,\n    'max_delta_step': 20,\n    'max_depth': 6,\n    'min_child_weight': 4,\n    'n_estimators': 100,\n    'reg_alpha': 1e-09,\n    'reg_lambda': 1000.0,\n    'scale_pos_weight': 499.99999999999994,\n    'subsample': 1.0\n}","execution_count":null,"outputs":[]}],"metadata":{"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}