{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Python Lightgbm DART Boosting Save Best Model","metadata":{"_uuid":"354b8e33-2647-4de5-9e5e-f97ed0b0fb90","_cell_guid":"e1c561ed-8fe4-4234-9b8e-6e59ead593b6","jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-09-07T05:36:20.999929Z","iopub.execute_input":"2022-09-07T05:36:21.000418Z","iopub.status.idle":"2022-09-07T05:36:21.024521Z","shell.execute_reply.started":"2022-09-07T05:36:21.000299Z","shell.execute_reply":"2022-09-07T05:36:21.023517Z"}}},{"cell_type":"markdown","source":"This is a related work to [this notebook that implemented the idea in R](https://www.kaggle.com/code/stautxie/lightgbm-dart-boosting-save-best-model/notebook). I want to thank \"The Devastator\" for suggesting to implement it for Python as well.\n\nThe objective of this notebook is to save the best model while running lightgbm with DART boosting. By default, the lightgbm implementation does not save the best model and it can be shown later in this notebook that the final output model from lightgbm with DART can be far worse than the optimal model found during the fitting.","metadata":{}},{"cell_type":"markdown","source":"As is shared in the [previous notebook](https://www.kaggle.com/code/stautxie/lightgbm-dart-boosting-save-best-model/notebook), while there are a few python implementations that save the best incumbent solution as pickled file, it is still more desirable to save this information in the RAM as \n\n1. Saving in RAM is much faster.\n2. Saving in RAM avoids frequent disk IO operations, which is both more time consuming and could shorten the life of the hard drive.","metadata":{}},{"cell_type":"markdown","source":"The idea of Python lightgbm DART callback is to use the Python function model_to_string(), which has similar functionality as R's save_model_to_string(). To this end, we develop the following class.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport lightgbm as lgb\nimport os\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.model_selection import train_test_split\nfrom random import seed","metadata":{"execution":{"iopub.status.busy":"2022-09-07T14:16:58.658632Z","iopub.execute_input":"2022-09-07T14:16:58.659850Z","iopub.status.idle":"2022-09-07T14:16:59.678146Z","shell.execute_reply.started":"2022-09-07T14:16:58.659747Z","shell.execute_reply":"2022-09-07T14:16:59.676839Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To follow the approach in our R notebook, we will also use the UCI Bank Marketing data set and convert all the categorical columns to numbers.","metadata":{}},{"cell_type":"code","source":"DAT_DIR = \"../input/bankbalanced\"\nbank_df = pd.read_csv(os.path.join(DAT_DIR, \"bank.csv\"))","metadata":{"execution":{"iopub.status.busy":"2022-09-07T14:16:59.686445Z","iopub.execute_input":"2022-09-07T14:16:59.686850Z","iopub.status.idle":"2022-09-07T14:16:59.724891Z","shell.execute_reply.started":"2022-09-07T14:16:59.686803Z","shell.execute_reply":"2022-09-07T14:16:59.723806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bank_df","metadata":{"execution":{"iopub.status.busy":"2022-09-07T14:16:59.726646Z","iopub.execute_input":"2022-09-07T14:16:59.726981Z","iopub.status.idle":"2022-09-07T14:16:59.759041Z","shell.execute_reply.started":"2022-09-07T14:16:59.726952Z","shell.execute_reply":"2022-09-07T14:16:59.757813Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cat_cols = ['job', 'marital', 'education', 'default','loan', 'housing', 'contact', 'month', 'poutcome', 'deposit']\nfor c in cat_cols:\n    le = LabelEncoder()\n    bank_df[c] = le.fit_transform(bank_df[c])","metadata":{"execution":{"iopub.status.busy":"2022-09-07T14:16:59.762773Z","iopub.execute_input":"2022-09-07T14:16:59.764006Z","iopub.status.idle":"2022-09-07T14:16:59.806620Z","shell.execute_reply.started":"2022-09-07T14:16:59.763954Z","shell.execute_reply":"2022-09-07T14:16:59.805442Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bank_df","metadata":{"execution":{"iopub.status.busy":"2022-09-07T14:16:59.808470Z","iopub.execute_input":"2022-09-07T14:16:59.808962Z","iopub.status.idle":"2022-09-07T14:16:59.833873Z","shell.execute_reply.started":"2022-09-07T14:16:59.808920Z","shell.execute_reply":"2022-09-07T14:16:59.832612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We then partition the training and testing data set to set up the lightgbm fitting.","metadata":{}},{"cell_type":"code","source":"X_cols = list(set(bank_df.columns.to_list()) - set(['default']))\ny_col = 'default'\nprint('X_cols = ', X_cols)\nprint('y_col = ', y_col)","metadata":{"execution":{"iopub.status.busy":"2022-09-07T14:16:59.835334Z","iopub.execute_input":"2022-09-07T14:16:59.836156Z","iopub.status.idle":"2022-09-07T14:16:59.842923Z","shell.execute_reply.started":"2022-09-07T14:16:59.836108Z","shell.execute_reply":"2022-09-07T14:16:59.842027Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X = bank_df.loc[:, X_cols]\ny = bank_df[y_col]\nprint(f'X shape = {X.shape}')\nprint(f'y shape = {len(y)}')","metadata":{"execution":{"iopub.status.busy":"2022-09-07T14:16:59.844165Z","iopub.execute_input":"2022-09-07T14:16:59.844589Z","iopub.status.idle":"2022-09-07T14:16:59.856331Z","shell.execute_reply.started":"2022-09-07T14:16:59.844517Z","shell.execute_reply":"2022-09-07T14:16:59.855065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tr_X, va_X, tr_y, va_y = train_test_split(X, y, test_size = 0.2, random_state=1234)\nprint(f'tr_X shape = {tr_X.shape}')\nprint(f'va_X shape = {va_X.shape}')\nprint(f'tr_y shape = {len(tr_y)}')\nprint(f'va_y shape = {len(va_y)}')","metadata":{"execution":{"iopub.status.busy":"2022-09-07T14:16:59.858020Z","iopub.execute_input":"2022-09-07T14:16:59.858928Z","iopub.status.idle":"2022-09-07T14:16:59.872249Z","shell.execute_reply.started":"2022-09-07T14:16:59.858877Z","shell.execute_reply":"2022-09-07T14:16:59.870860Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lgb_data_tr = lgb.Dataset(data=tr_X, label=tr_y)\nlgb_data_va = lgb.Dataset(data=va_X, label=va_y)","metadata":{"execution":{"iopub.status.busy":"2022-09-07T14:16:59.873704Z","iopub.execute_input":"2022-09-07T14:16:59.874415Z","iopub.status.idle":"2022-09-07T14:16:59.879094Z","shell.execute_reply.started":"2022-09-07T14:16:59.874379Z","shell.execute_reply":"2022-09-07T14:16:59.878000Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lgb_params = {'objective' : 'regression',\n              'boosting' : 'dart',\n              'metric'  : \"rmse\"}","metadata":{"execution":{"iopub.status.busy":"2022-09-07T14:16:59.880364Z","iopub.execute_input":"2022-09-07T14:16:59.881128Z","iopub.status.idle":"2022-09-07T14:16:59.891531Z","shell.execute_reply.started":"2022-09-07T14:16:59.881084Z","shell.execute_reply":"2022-09-07T14:16:59.890564Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mdl_lgb1 = lgb.train(params=lgb_params,\n                    train_set = lgb_data_tr,\n                    num_boost_round = 100,\n                    valid_sets = [lgb_data_tr, lgb_data_va],\n                    verbose_eval = 10,\n                    early_stopping_rounds = 10)","metadata":{"execution":{"iopub.status.busy":"2022-09-07T14:16:59.892827Z","iopub.execute_input":"2022-09-07T14:16:59.894012Z","iopub.status.idle":"2022-09-07T14:17:00.197126Z","shell.execute_reply.started":"2022-09-07T14:16:59.893964Z","shell.execute_reply":"2022-09-07T14:17:00.196193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"va_y_pred_unsaved = mdl_lgb1.predict(data=va_X)\nrmse_unsaved = np.sqrt(np.mean((va_y - va_y_pred_unsaved)**2))\nprint(f'RMSE for the unsaved lgb model: {rmse_unsaved}')","metadata":{"execution":{"iopub.status.busy":"2022-09-07T14:17:00.198424Z","iopub.execute_input":"2022-09-07T14:17:00.199427Z","iopub.status.idle":"2022-09-07T14:17:00.212948Z","shell.execute_reply.started":"2022-09-07T14:17:00.199385Z","shell.execute_reply":"2022-09-07T14:17:00.211964Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see from the warnings above, lightgbm does not allow early stopping while using DART boosting. This is primarily due to the fact that DART boosting will require lightgbm to revise the boosting trees periodically. This prohibits reconstructing the optimal tree when search stops. Consequently, prediction from the final model is far from optimal.","metadata":{"execution":{"iopub.status.busy":"2022-09-07T06:13:49.938071Z","iopub.execute_input":"2022-09-07T06:13:49.938485Z","iopub.status.idle":"2022-09-07T06:13:49.956847Z","shell.execute_reply.started":"2022-09-07T06:13:49.938445Z","shell.execute_reply":"2022-09-07T06:13:49.955574Z"}}},{"cell_type":"markdown","source":"To bypass this limit, we will create a customized class that stores the best model value and the model structure independently outside the booster.  This can be done via lightgbm's native callback functionality.","metadata":{}},{"cell_type":"markdown","source":"## Callback function","metadata":{}},{"cell_type":"markdown","source":"The class lgb_dart_cb contains the implementation. At each iteration, it compares the current model score with the best score and update the model accordingly. Here are a few notes:\n\n1. Parameter higher_is_better needs to be set to True for maximization, e.g., AUC.\n\n2. The first index of the env.evaluation_result_list within the __call__ method would need to be manually updated if necessary. See the comments in the code for details. While it is possible to parameterize the change, we decide to leave as is to save time.","metadata":{}},{"cell_type":"code","source":"class lgb_dart_cb:\n    def __init__(self, higher_is_better = False) -> None:\n        self.best_value = None\n        self.best_model = None\n        self.best_iteration = None\n        self.higher_is_better = higher_is_better\n        \n    def __call__(self, env) -> None:\n        #print(f'len(env.evaluation_result_list) = {len(env.evaluation_result_list)}')\n        \n        #**** Reset the first index of env.evaluation_result_list if the number of validation data set changes\n        #**** or the position of the validation score changes. \n        #**** Tip: uncomment the print() statement above to see the length of the list\n        score = env.evaluation_result_list[1][2]\n        #***************************\n        if self.higher_is_better: \n            sense = -1\n        else:\n            sense = 1\n            \n        if self.best_value is None or self.best_value * sense > score * sense:\n            print(f'**** Finding a better solution: iteration = {env.iteration}, value = {score}')\n            self.best_value = score\n            self.best_iteration = env.iteration + 1\n            self.best_model = env.model.model_to_string(num_iteration=env.iteration+1)\n        ","metadata":{"execution":{"iopub.status.busy":"2022-09-07T14:17:00.216676Z","iopub.execute_input":"2022-09-07T14:17:00.217631Z","iopub.status.idle":"2022-09-07T14:17:00.225706Z","shell.execute_reply.started":"2022-09-07T14:17:00.217592Z","shell.execute_reply":"2022-09-07T14:17:00.224691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dart_cb = lgb_dart_cb(higher_is_better = False)\nmdl_lgb = lgb.train(params=lgb_params,\n                    train_set = lgb_data_tr,\n                    num_boost_round = 100,\n                    valid_sets = [lgb_data_tr, lgb_data_va],\n                    verbose_eval = 10,\n                    early_stopping_rounds = 10, \n                   callbacks=[dart_cb])","metadata":{"execution":{"iopub.status.busy":"2022-09-07T14:17:00.227257Z","iopub.execute_input":"2022-09-07T14:17:00.227910Z","iopub.status.idle":"2022-09-07T14:17:00.595602Z","shell.execute_reply.started":"2022-09-07T14:17:00.227874Z","shell.execute_reply":"2022-09-07T14:17:00.594432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mdl_lgb.model_from_string(dart_cb.best_model)\n","metadata":{"execution":{"iopub.status.busy":"2022-09-07T14:17:00.599977Z","iopub.execute_input":"2022-09-07T14:17:00.601005Z","iopub.status.idle":"2022-09-07T14:17:00.622977Z","shell.execute_reply.started":"2022-09-07T14:17:00.600936Z","shell.execute_reply":"2022-09-07T14:17:00.621614Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"va_y_pred = mdl_lgb.predict(data=va_X)\nrmse = np.sqrt(np.mean((va_y - va_y_pred)**2))\nprint(f'rmse = {rmse}')","metadata":{"execution":{"iopub.status.busy":"2022-09-07T14:17:00.626989Z","iopub.execute_input":"2022-09-07T14:17:00.627754Z","iopub.status.idle":"2022-09-07T14:17:00.640438Z","shell.execute_reply.started":"2022-09-07T14:17:00.627704Z","shell.execute_reply":"2022-09-07T14:17:00.638616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can clearly see that the model saved by the customized callback function indeed produced the best score. This proves the validity of the implementation and concludes this notebook.","metadata":{}}]}