{"metadata":{"kernelspec":{"name":"ir","display_name":"R","language":"R"},"language_info":{"name":"R","codemirror_mode":"r","pygments_lexer":"r","mimetype":"text/x-r-source","file_extension":".r","version":"4.0.5"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Lightgbm DART Boosting save best model","metadata":{"_uuid":"051d70d956493feee0c6d64651c6a088724dca2a","_execution_state":"idle"}},{"cell_type":"markdown","source":"It is quite evident from multiple public notebooks (e.g., [this one](https://www.kaggle.com/code/ragnar123/amex-lgbm-dart-cv-0-7977), [this one](https://www.kaggle.com/code/thedevastator/lag-features-are-all-you-need), and [this one](https://www.kaggle.com/code/ragnar123/amex-lgbm-dart-cv-0-7963)) and discussions that DART boosting mode proves critical for achieving a good performance for this competition.  Meanwhile, several hacks have been improvised to address some of the challenges for DART boosting. Among them, the most important hack is to save the best model during each training fold.  ","metadata":{}},{"cell_type":"markdown","source":"By default, none of the popular boosting algorithms, e.g., xgboost, lightgbm, and catboost, allows early termination for DART boosting because the algorithms make changes to the ensemble trees during the training.  As model score fluctuates during the training, the final model when training ends may not be the best.  This leads to great interests to save the incumbent best models as training goes.","metadata":{}},{"cell_type":"markdown","source":"However, saving best model during DART boosting is not a trivial task particularly for R due to the following reasons:\n\n1. While both Python and R's implementations typically provide callback functionality, saving model (i.e., serialization) has its own challenges. More specifically, Python users make clever use of the pickle functionality to save the immediate solutions in binary format to disk without much IO overhead. Notable examples include [two implementations in this discussion thread](https://www.kaggle.com/competitions/amex-default-prediction/discussion/332575#1829172). However, R's equivalent serialization method saveRDS() and readRDS() do not work properly due to references to external C++ code. \n\n2. If we use the functions with R lightgbm implementation to dump the model in text files, it is also not desired. This is because saving model files in text format numurous times is prohibitively expensive, particularly as size of the ensemble trees grow.","metadata":{}},{"cell_type":"markdown","source":"In this notebook, we will develop a performant solution that relies on an undocumented R lightgbm function save_model_to_string() within the lgb.Booster class. To implement this idea, we also make use of the function closure to\n\n1. Ensure the save model always stays in the RAM. This reduces the IO time significantly at minimal increase of memory footprint.\n2. Retrieve the model after the training is completed with relatively low overhead.","metadata":{}},{"cell_type":"markdown","source":"The rest of the paper presents the implementation as well as uses the bank data set in the R lightgbm package to demonstrate the use.","metadata":{}},{"cell_type":"markdown","source":"## Baseline","metadata":{}},{"cell_type":"code","source":"library(lightgbm)\nlibrary(data.table)\nlibrary(magrittr)\nlibrary(MLmetrics)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T00:08:17.024918Z","iopub.execute_input":"2022-07-22T00:08:17.028124Z","iopub.status.idle":"2022-07-22T00:08:17.339876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data(bank)\nsetDT(bank)\n\ncat_cols <- names(bank)[unlist(lapply(bank, is.character))]\nbank[, (cat_cols):=lapply(.SD, function(x) as.numeric(factor(x))), .SDcols=cat_cols]\nhead(bank)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T00:09:18.959085Z","iopub.execute_input":"2022-07-22T00:09:18.961097Z","iopub.status.idle":"2022-07-22T00:09:19.029469Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"set.seed(1234)\ntest_ind <- sample(1:nrow(bank), size=800, replace=FALSE)\n\nlgb_data_tr <- lgb.Dataset(data=bank[-test_ind, -c(\"y\")] %>% as.matrix(),\n                           label=bank$y[-test_ind])\n\nlgb_data_va <- lgb.Dataset(data=bank[test_ind, -c(\"y\")] %>% as.matrix(),\n                           label=bank$y[test_ind])","metadata":{"execution":{"iopub.status.busy":"2022-07-22T00:09:30.408242Z","iopub.execute_input":"2022-07-22T00:09:30.410418Z","iopub.status.idle":"2022-07-22T00:09:30.446988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"While the label y was originally meant to be binary, we choose to treat it as a continuous variable so that lightgbm takes more iterations to solve the resultant regression problem.","metadata":{"execution":{"iopub.status.busy":"2022-06-28T03:40:19.004647Z","iopub.execute_input":"2022-06-28T03:40:19.008976Z","iopub.status.idle":"2022-06-28T03:40:19.113399Z"}}},{"cell_type":"code","source":"\nlgb_params <- list(objective='regression',\n                   boosting='dart',\n                   metric=\"rmse\")","metadata":{"execution":{"iopub.status.busy":"2022-07-22T00:10:44.696011Z","iopub.execute_input":"2022-07-22T00:10:44.697927Z","iopub.status.idle":"2022-07-22T00:10:44.715637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nmd_lgb <- lgb.train(params=lgb_params,\n                    data = lgb_data_tr,\n                    valids = list(test=lgb_data_va, train=lgb_data_tr),\n                    eval_freq = 10,\n                    nrounds = 100)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T00:11:37.234167Z","iopub.execute_input":"2022-07-22T00:11:37.235894Z","iopub.status.idle":"2022-07-22T00:11:37.606228Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_pred_va <- predict(md_lgb, data=bank[test_ind, -c(\"y\")] %>% as.matrix())\nRMSE_va <- RMSE(y_pred_va, bank$y[test_ind])\ncat(paste(\"RMSE = \", RMSE_va, \"\\n\", sep=\"\"))","metadata":{"execution":{"iopub.status.busy":"2022-07-22T00:14:20.961209Z","iopub.execute_input":"2022-07-22T00:14:20.963218Z","iopub.status.idle":"2022-07-22T00:14:20.998818Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, the model score on the testing population comes down to ~0.269 based on the iteration log above. However, the final model only has a score of 0.570. This shows that final model from the DART boosting can be significantly worse than the best model.","metadata":{}},{"cell_type":"markdown","source":"## Lightgbm DART Boosting Callback Function Implementation","metadata":{}},{"cell_type":"markdown","source":"Our implementation of the DART boosting callback is shown below. To simply the implementation, we choose to assume that only single evaluation metric is used. Otherwise, it is quite straightforward to follow the examples in the online [R lightgbm repository](https://github.com/microsoft/LightGBM/blob/master/R-package/R/callback.R) to implement for multiple metrics.","metadata":{}},{"cell_type":"markdown","source":"The function generator lgb_dart_callback() retains a closure, which includes variables best_score and best_model_str as well as function callback().  The latter is passed to lgb.train() so that the training algorithm knows who to call. Variable best_score saves the incumbent model score and higher_is_better parameter ensures the callback function can handle both metrics of minimization or maximization. Finally, we save the model in string within the best_model_str variable whenever we find a better model than the incumbent.","metadata":{}},{"cell_type":"code","source":"\nlgb_dart_callback <- function() {\n  best_score <- NULL\n  best_model_str <- NULL\n\n  callback <- function(env) {\n\n    if (!is.null(env$eval_list)) {\n      score <- env$eval_list[[2]]$value\n      higher_is_better <- ifelse(env$eval_list[[2]]$higher_better, 1, -1)\n \n      if (is.null(best_score) || higher_is_better * best_score < higher_is_better * score) {\n        cat(paste(\"****** Finding a better score: \", score, \", Current score = \", best_score, \n                  \", Iteration = \", env$iteration, \"\\n\", sep=\"\"))\n        best_score <<- score\n        best_model_str <<- env$model$save_model_to_string(num_iteration=env$iteration)\n      }\n    }\n\n  }\n  \n  attr(callback, \"call\") <- match.call()\n  attr(callback, \"name\") <- \"lgb_dart_callback\"\n  \n  return(callback)\n}","metadata":{"execution":{"iopub.status.busy":"2022-07-22T00:22:51.994835Z","iopub.execute_input":"2022-07-22T00:22:51.996897Z","iopub.status.idle":"2022-07-22T00:22:52.012199Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let us first create a function object of the callback, ","metadata":{}},{"cell_type":"code","source":"ld_cb <- lgb_dart_callback()","metadata":{"execution":{"iopub.status.busy":"2022-07-22T00:23:00.541228Z","iopub.execute_input":"2022-07-22T00:23:00.543154Z","iopub.status.idle":"2022-07-22T00:23:00.579949Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"md_lgb2 <- lgb.train(params=lgb_params,\n                    data = lgb_data_tr,\n                    valids = list(test=lgb_data_va, train=lgb_data_tr),\n                    nrounds = 100,\n                    eval_freq = 10,\n                    reset_data = TRUE,\n                    callbacks = list(ld_cb),\n                    serializable = TRUE)\n\ny_pred_va2 <- predict(md_lgb2, data=bank[test_ind, -c(\"y\")] %>% as.matrix())\n\nRMSE_va2 <- RMSE(y_pred_va2, bank$y[test_ind])\ncat(paste(\"RMSE = \", RMSE_va2, \"\\n\", sep=\"\"))","metadata":{"execution":{"iopub.status.busy":"2022-07-22T00:23:33.813827Z","iopub.execute_input":"2022-07-22T00:23:33.816891Z","iopub.status.idle":"2022-07-22T00:23:36.354661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The final model yields the final score as that in the previous section. This proves that we did not mess up the training process by passing the callback function.","metadata":{}},{"cell_type":"markdown","source":"Now, let us retrieve the saved model. Here, we use the function object lb_cb to find its closure (i.e., environment). This allows us to retrieve the best_model_str variable. We can probably build a more elegant solution with R6 Object-Orient Programming package. However, we are pretty happy with this simpler solution as code's complexity is fairly low.","metadata":{}},{"cell_type":"markdown","source":"Once the model string is retrieve, we can save it into a text file so that we can use R lightgbm's function lgb.load() to create the booster.","metadata":{}},{"cell_type":"code","source":"best_model_str <- get(\"best_model_str\", envir=environment(fun=ld_cb))\nfile_conn <- file(\"best_model.txt\")\nwriteLines(best_model_str, file_conn)\nclose(file_conn)","metadata":{"execution":{"iopub.status.busy":"2022-07-22T00:34:06.097873Z","iopub.execute_input":"2022-07-22T00:34:06.101294Z","iopub.status.idle":"2022-07-22T00:34:06.137543Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"md_lgb3 <- lgb.load(\"best_model.txt\")","metadata":{"execution":{"iopub.status.busy":"2022-07-22T00:34:30.079764Z","iopub.execute_input":"2022-07-22T00:34:30.081639Z","iopub.status.idle":"2022-07-22T00:34:30.122050Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_pred_va3 <- predict(md_lgb3, data=bank[test_ind, -c(\"y\")] %>% as.matrix())\n\nRMSE_va3 <- RMSE(y_pred_va3, bank$y[test_ind])\ncat(paste(\"RMSE = \", RMSE_va3, \"\\n\", sep=\"\"))","metadata":{"execution":{"iopub.status.busy":"2022-07-22T00:34:51.592304Z","iopub.execute_input":"2022-07-22T00:34:51.595144Z","iopub.status.idle":"2022-07-22T00:34:51.623546Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, the saved best model produces substantially improved model score than generated by the lgb.train() directly. This also demonstrates our implementation is performant.","metadata":{"execution":{"iopub.status.busy":"2022-06-28T03:44:32.761864Z","iopub.execute_input":"2022-06-28T03:44:32.764000Z","iopub.status.idle":"2022-06-28T03:44:32.828494Z"}}},{"cell_type":"markdown","source":"## Conclusions","metadata":{}},{"cell_type":"markdown","source":"In conclusion, we have developed an innovative solutions to save the immediate best solution for DART boosting within lightgbm framework.  This approach can be generalized to other boosting frameworks, like xgboost and catboost, provided similar procedures exist in these other packages with relatively low overheads. ","metadata":{}}]}