{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Extreme Gradient Boosting (XGBoost) Wrangling with Hyperparameters
","metadata":{}},{"cell_type":"markdown","source":"Extreme Gradient Boosting (XGBoost)
","metadata":{}},{"cell_type":"markdown","source":"# 🧾 1. Extreme Gradient Boosting (XGBoost)","metadata":{}},{"cell_type":"markdown","source":"XGBoost is an open-source software library which provides a regularizing gradient boosting framework for C++, Java, Python, R, Julia, Perl, and Scala. It works on Linux, Windows, and macOS. From the project description, it aims to provide a \"Scalable, Portable and Distributed Gradient Boosting Library\".\n\nExtreme Gradient Boosting Algorithm. Gradient boosting refers to **a class of ensemble machine learning algorithms that can be used for classification or regression predictive modeling problems**. **Ensembles are constructed from decision tree models**.\n\n**XGBoost is a more regularized form of Gradient Boosting**. XGBoost uses advanced regularization (L1 & L2), which improves model generalization capabilities. XGBoost delivers high performance as compared to Gradient Boosting. Its training is very fast and can be parallelized across clusters.\n\nExtreme Gradient Boosting (xgboost) is similar to gradient boosting framework but more efficient. It has both linear model solver and tree learning algorithms. So, what makes it fast is its capacity to do parallel computation on a single machine...","metadata":{}},{"cell_type":"markdown","source":"\n
![](\"https://cdn.educba.com/academy/wp-content/uploads/2019/06/XGBoost-Algorithm1.jpg\")
\n
Introduction to Hyperparameters
","metadata":{}},{"cell_type":"markdown","source":"# 🛠 2. Hyperparameters","metadata":{}},{"cell_type":"markdown","source":"### What is meant by hyperparameter tuning?\nIn machine learning, **hyperparameter optimization or tuning is the problem of choosing a set of optimal hyperparameters for a learning algorithm**. A hyperparameter is a parameter whose value is used to control the learning process. By contrast, the values of other parameters (typically node weights) are learned. The hyper-parameter tuning process is a tightrope walk to achieve a balance between underfitting and overfitting. Underfitting is when the machine learning model is unable to reduce the error for either the test or training set.\n\n### What is hyperparameter tuning example?\nSome examples of model hyperparameters include: The penalty in Logistic Regression Classifier i.e. L1 or L2 regularization. The learning rate for training a neural network. The C and sigma hyperparameters for support vector machines.\n\n### Why do we use hyperparameter tuning?\nHyperparameter tuning is an essential part of controlling the behavior of a machine learning model. If we don't correctly tune our hyperparameters, our estimated model parameters produce suboptimal results, as they don't minimize the loss function. This means our model makes more errors.","metadata":{}},{"cell_type":"markdown","source":"\n
![](\"https://blog.eduonix.com/wp-content/uploads/2018/06/Hyperparameter.jpg\")
\n
XGBoost Parameters - Classification
","metadata":{}},{"cell_type":"markdown","source":"# ⌛️ 3. XGBoost Hyperparameters for Classification","metadata":{}},{"cell_type":"markdown","source":"XGBoost Boosters
\n\n#### -> gbtree :\n - **gbtree** for **tree-based models**; \n \n#### -> gblinear :\n - **gblinear** for **linear models** to run at each iteration.\n\n#### -> dart :\n - **dart** is also a tree based model. \n - XGBoost mostly combines a huge number of regression trees with a small learning rate. In this situation, trees added early are significant and trees added late are unimportant. *Vinayak and Gilad-Bachrach* proposed a new method to add dropout techniques from the deep neural net community to boosted trees, and reported better results in some situations - it s called dart. It drops trees in order to solve the over-fitting. Trivial trees (to correct trivial errors) may be prevented. Because of the randomness introduced in the training, expect the following few differences: Training can be slower than gbtree because the random dropout prevents usage of the prediction buffer. The early stop might not be stable, due to the randomness.","metadata":{}},{"cell_type":"markdown","source":"General parameters
\nIt relates to which booster we are using to do boosting, commonly tree or linear model\n\n#### 1. booster [default=gbtree]\n - **gbtree** for **tree-based models**; **gblinear** for **linear models** to run at each iteration.\n \n#### 2. silent [default=0]:\n - **To activate Silent silent mode, set it to 1,meaning off; so, no messages will be printed.**\n\nIt’s generally a good idea, to keep it 0 as the messages might help in understanding the model; and how the metrics are going.\n\n#### 3. nthread [default to maximum number of threads available if not set]\nThis is used for parallel processing and number of cores in the system should be entered\nIf you wish to run on all cores, value should not be entered and algorithm will detect automatically\n","metadata":{}},{"cell_type":"markdown","source":"Booster parameters
\n\ndepend on which booster you have chosen. We will discuss about tree based boosters here.\n\n#### 1. eta [default=0.3]\n- Similiar to **learning_rate**; works with geeneralization.\n- **Typical final values to be used:** 0.01-0.2\n\n#### 2. min_child_weight [default=1]\n- Defines the **minimum sum of weights of all observations required in a child**. It is **used to control over-fitting**. Higher values prevent a model from learning - relations which might be highly specific to the particular sample selected for a tree.\n- Too high values can lead to under-fitting hence, it should be tuned using CV.\n\n#### 3. max_depth [default=6]\n- The **maximum depth of a tree**, it is also used **to control over-fitting** as higher depth will allow model to learn relations very specific to a particular sample. Should be tuned using CV.\n- **Typical values:** 3-10\n\n#### 4. max_leaf_nodes\n- The **maximum number of terminal nodes** or leaves in a tree.\n- Since binary trees are created, a depth of ‘n’ would produce a maximum of 2^n leaves. If this is defined, GBM will ignore max_depth.\n\n#### 5. gamma [default=0]\n- A node is split only when the resulting split gives a positive reduction in the loss function. **Gamma specifies the minimum loss reduction required for a split to occur.**\n- Makes the algorithm conservative. The values can vary depending on the loss function and should be tuned. **Can be used to control overfitting.**\n\n#### 6. max_delta_step [default=0]\n- In **maximum delta step we allow each tree’s weight estimation to be**. If the value is set to 0, it means there is no constraint. If it is set to a positive value, it can help making the update step more conservative.\n- This is generally not used.\n\n#### 7. subsample [default=1]\n- It denotes the **fraction of observations to be randomly samples for each tree**.\n- **Lower values make the algorithm more conservative and prevents overfitting** but too small values might lead to under-fitting.\n- **Typical values:** 0.5-1\n\n#### 8. colsample_bytree [default=1]\n- Denotes the **fraction of columns to be randomly samples for each tree**. Smaller colsample_bytree povides additional regulerization.\n- **Typical values:** 0.5-1\n\n#### 9. colsample_bylevel [default=1]\n- Denotes **the subsample ratio of columns for each split, in each level**.\n\n#### 10. alpha [default=0]\n- **L1 regularization term**.\n- L1 regularization forces the weights of uninformative features to be zero by substracting a small amount from the weight at each iteration and thus making the weight zero, eventually. It is also called regularization for simplicity. Applies on leaf weights (rather than feature weights); larger value mean more regularization. \n\n#### 11. lambda [default=1]\n- **L2 regularization term**.\n- This used to handle the regularization part of XGBoost. It should be explored to reduce overfitting.\n- L2 regularization acts like a force that removes a small percentage of weights at each iteration. Therefore, weights will never be equal to zero. L2 regularization penalizes (weight)² There is an additional parameter to tune the L2 regularization term which is called regularization rate (lambda).\n- Smoother than alpha. Also, applies on leaf weights.\n\n#### 12. scale_pos_weight [default=1]\n- A value **greater than 0 should be used** in case of high class imbalance as it helps in faster convergence.","metadata":{}},{"cell_type":"markdown","source":"Learning task parameters
\n\ndecide on the learning scenario. For example, regression tasks may use different parameters with ranking tasks.\n\n#### 1. objective [default=reg:linear]\n- This **defines the loss function to be minimized**. \n- Mostly used values are:\n - **reg:linear** - used for regressions\n - **reg:logistic** - used for classification, when you want the decision only, not the probability.\n - **binary:logistic** –logistic regression for binary classification, returns predicted probability (rather than decision class) \n - **multi:softmax** –multiclass classification using the softmax objective, returns predicted class (not probabilities); you also need to set an additional num_class (number of classes) parameter defining the number of unique classes\n - **multi:softprob** –same as softmax, but returns predicted probability of each data point belonging to each class.\n \n#### 2. eval_metric [ default according to objective ]\n- The metric to be used for validation data.\n- The default values are rmse for regression and error for classification.\n- **Typical values are:**\n - **rmse** – root mean square error\n - **mae** – mean absolute error\n - **logloss** – negative log-likelihood\n - **error** – Binary classification error rate (0.5 threshold)\n - **merror** – Multiclass classification error rate\n - **mlogloss** – Multiclass logloss\n - **auc** – Area under the curve\n \n#### 3. seed [default=0]\n- for reproducibility.","metadata":{}},{"cell_type":"markdown","source":"Command line parameters
\n\n- relate to behavior of CLI version of XGBoost.","metadata":{}},{"cell_type":"markdown","source":"Understanding Bias-Variance Tradeoff
","metadata":{}},{"cell_type":"markdown","source":"# 🧮 4. Understanding Bias-Variance Tradeoff","metadata":{}},{"cell_type":"markdown","source":"If you take a machine learning or statistics course, this is likely to be one of the most important concepts. When we allow the model to get more complicated (e.g. more depth), the model has better ability to fit the training data, resulting in a less biased model. However, such complicated model requires more data to fit.\n\nMost of parameters in XGBoost are about bias variance tradeoff. The best model should trade the model complexity with its predictive power carefully. Parameters Documentation will tell you whether each parameter will make the model more conservative or not. This can be used to help you turn the knob between complicated model and simple model.\n\n## More Resources :\n- [📋 Bias-Variance Tradeoff ➡️ with NumPy & Seaborn](https://www.kaggle.com/code/azminetoushikwasi/bias-variance-tradeoff-with-numpy-seaborn)\n- [Mastering Bias-Variance Tradeoff wih Polynomials](https://medium.com/@azmine_wasi/mastering-bias-variance-tradeoff-with-polynomials-part-02-29f9bb53bb26)","metadata":{}},{"cell_type":"markdown","source":"Approch to XGBoost Hyperparameter Tuning
","metadata":{}},{"cell_type":"markdown","source":"# 📑 6. Best approch to Hyper parameter tuning","metadata":{}},{"cell_type":"markdown","source":"Importing Necessary Modules and Functions
","metadata":{}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport os\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom scipy import stats\nimport warnings\nimport json\nfrom sklearn import manifold\n\nfrom sklearn.model_selection import train_test_split\n\nfrom sklearn.preprocessing import OrdinalEncoder\n\nimport xgboost as xgb\n\nfrom sklearn.model_selection import RandomizedSearchCV, GridSearchCV\nfrom sklearn.model_selection import StratifiedKFold\n\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.metrics import classification_report\nfrom sklearn.model_selection import cross_val_score, KFold\nfrom sklearn.metrics import confusion_matrix\nfrom xgboost import plot_tree\n\nwarnings.filterwarnings('ignore')","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:43:52.891064Z","iopub.execute_input":"2022-08-18T14:43:52.891530Z","iopub.status.idle":"2022-08-18T14:43:53.841341Z","shell.execute_reply.started":"2022-08-18T14:43:52.891496Z","shell.execute_reply":"2022-08-18T14:43:53.840138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"warnings.filterwarnings('ignore')","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:43:53.843830Z","iopub.execute_input":"2022-08-18T14:43:53.844279Z","iopub.status.idle":"2022-08-18T14:43:53.849672Z","shell.execute_reply.started":"2022-08-18T14:43:53.844235Z","shell.execute_reply":"2022-08-18T14:43:53.848532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def process_df(df):\n df=df.drop(['Name','PassengerId'],axis=1)\n df=df.dropna()\n \n target=df['Transported']\n df=df.drop(['Transported'],axis=1)\n target = target.astype(int)\n \n df['Cabin_1']= df['Cabin'].str[0]\n df['Cabin_2']= df['Cabin'].str[2]\n df['Cabin_3']= df['Cabin'].str[5]\n df=df.drop(['Cabin'],axis=1)\n \n \n # Create the training and test datasets\n X_train, X_test, y_train, y_test = train_test_split(df, \n target, \n test_size = 0.2, \n random_state=100,\n stratify=target)\n\n numaric_columns=list(df.select_dtypes(include=np.number).columns)\n print(\"Numaric columns (\"+str(len(numaric_columns))+\") :\",\", \".join(numaric_columns))\n \n cat_columns=df.select_dtypes(include=['object']).columns.tolist()\n print(\"Categorical columns (\"+str(len(cat_columns))+\") :\",\", \".join(cat_columns))\n \n \n X_train_n=X_train[numaric_columns]\n X_test_n=X_test[numaric_columns]\n \n X_train_c=X_train[cat_columns]\n X_test_c=X_test[cat_columns]\n \n \n encoder=OrdinalEncoder()\n X_train_c = encoder.fit_transform(X_train_c)\n X_train_c=pd.DataFrame(X_train_c)\n X_test_c = encoder.transform(X_test_c)\n X_test_c=pd.DataFrame(X_test_c)\n \n i=1\n for column in X_train_c:\n X_train_n[\"cat_\"+str(i)]=X_train_c[column]\n X_test_n[\"cat_\"+str(i)]=X_test_c[column]\n i=i+1\n \n #X_train=pd.concat([X_train_n,X_train_c],axis=1,ignore_index=True)\n #X_test=pd.concat([X_test_n,X_test_c],axis=1,ignore_index=True)\n \n X_train_n=X_train_n.fillna(X_train_n.mean())\n X_test_n=X_test_n.fillna(X_test_n.mean())\n \n \n return X_train_n, X_test_n, y_train, y_test","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:43:53.851475Z","iopub.execute_input":"2022-08-18T14:43:53.852144Z","iopub.status.idle":"2022-08-18T14:43:53.865986Z","shell.execute_reply.started":"2022-08-18T14:43:53.852107Z","shell.execute_reply":"2022-08-18T14:43:53.865006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Preparing the Data
","metadata":{}},{"cell_type":"code","source":"df= pd.read_csv(\"/kaggle/input/spaceship-titanic/train.csv\")\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:43:53.869136Z","iopub.execute_input":"2022-08-18T14:43:53.869768Z","iopub.status.idle":"2022-08-18T14:43:53.947289Z","shell.execute_reply.started":"2022-08-18T14:43:53.869719Z","shell.execute_reply":"2022-08-18T14:43:53.946008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train, X_test, y_train, y_test = process_df(df)\nprint(X_train.shape, X_test.shape, y_train.shape, y_test.shape)","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:43:53.948953Z","iopub.execute_input":"2022-08-18T14:43:53.949592Z","iopub.status.idle":"2022-08-18T14:43:54.044572Z","shell.execute_reply.started":"2022-08-18T14:43:53.949550Z","shell.execute_reply":"2022-08-18T14:43:54.043375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Preparing functions for making things easier later
","metadata":{}},{"cell_type":"markdown","source":"#### Base parameters : \n- booster = \"gbtree\", **as it is a classification problem**\n- objective = \"binary:logistic\", **as per the problem**\n- tree_method=\"gpu_hist\" **using GPU**","metadata":{}},{"cell_type":"code","source":"def xgb_helper(PARAMETERS,V_PARAM_NAME=False,V_PARAM_VALUES=False,BR=10):\n \n temp_dmatrix =xgb.DMatrix(data=X_train, label=y_train)\n \n if V_PARAM_VALUES==False:\n cv_results = xgb.cv(dtrain=temp_dmatrix, nfold=5,num_boost_round=BR,params=PARAMETERS, as_pandas=True, seed=123 )\n return cv_results\n \n else:\n results=[]\n \n for v_param_value in V_PARAM_VALUES:\n PARAMETERS[V_PARAM_NAME]=v_param_value\n cv_results = xgb.cv(dtrain=temp_dmatrix, nfold=5,num_boost_round=BR,params=PARAMETERS, as_pandas=True, seed=123)\n results.append((cv_results[\"train-auc-mean\"].tail().values[-1],cv_results[\"test-auc-mean\"].tail().values[-1]))\n \n data = list(zip(V_PARAM_VALUES, results))\n print(pd.DataFrame(data,columns=[V_PARAM_NAME,\"auc\"]))\n \n return cv_results","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:43:54.047241Z","iopub.execute_input":"2022-08-18T14:43:54.048550Z","iopub.status.idle":"2022-08-18T14:43:54.058642Z","shell.execute_reply.started":"2022-08-18T14:43:54.048498Z","shell.execute_reply":"2022-08-18T14:43:54.057389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create a general base model and evaluate performance
","metadata":{}},{"cell_type":"code","source":"PARAMETERS={\"objective\":'binary:logistic',\"eval_metric\":\"auc\"}\nxgb_helper(PARAMETERS)","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:43:54.060456Z","iopub.execute_input":"2022-08-18T14:43:54.061058Z","iopub.status.idle":"2022-08-18T14:43:57.161729Z","shell.execute_reply.started":"2022-08-18T14:43:54.061018Z","shell.execute_reply":"2022-08-18T14:43:57.160880Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Optimizing number of boosting rounds(as we will be using DMatrix from xgb)
","metadata":{}},{"cell_type":"code","source":"# Create the DMatrix: housing_dmatrix\nhousing_dmatrix =xgb.DMatrix(data=X_train, label=y_train)\n\n# Create the parameter dictionary for each tree: params \nparams = {\"objective\":\"binary:logistic\", \"max_depth\":5}\n\n# Create list of number of boosting rounds\nnum_rounds = [5, 10, 15, 20, 25]\n\n# Empty list to store final round rmse per XGBoost model\nfinal_rmse_per_round = []\n\n# Iterate over num_rounds and build one model per num_boost_round parameter\nfor curr_num_rounds in num_rounds:\n\n # Perform cross-validation: cv_results\n cv_results = xgb.cv(dtrain=housing_dmatrix, params=params, nfold=5, num_boost_round=curr_num_rounds, metrics=\"auc\", as_pandas=True, seed=123)\n \n # Append final round RMSE\n final_rmse_per_round.append(cv_results[\"test-auc-mean\"].tail().values[-1])\n\n# Print the resultant DataFrame\nnum_rounds_rmses = list(zip(num_rounds, final_rmse_per_round))\nprint(pd.DataFrame(num_rounds_rmses,columns=[\"num_boosting_rounds\",\"auc\"]))","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:43:57.165197Z","iopub.execute_input":"2022-08-18T14:43:57.167102Z","iopub.status.idle":"2022-08-18T14:43:58.425749Z","shell.execute_reply.started":"2022-08-18T14:43:57.167053Z","shell.execute_reply":"2022-08-18T14:43:58.424597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Pointers:\n- Taking num_boosting_rounds = 10; to avoid overfitting","metadata":{}},{"cell_type":"markdown","source":" 1. Choose the learning rate. Maybe you can start with a higher one. 0.5 - 0.1 is ok for starting in most cases.
","metadata":{}},{"cell_type":"code","source":"PARAMETERS={\"objective\":'binary:logistic',\"eval_metric\":\"auc\",\"learning_rate\": 0.5}\nxgb_helper(PARAMETERS)","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:43:58.430642Z","iopub.execute_input":"2022-08-18T14:43:58.431419Z","iopub.status.idle":"2022-08-18T14:43:58.747150Z","shell.execute_reply.started":"2022-08-18T14:43:58.431361Z","shell.execute_reply":"2022-08-18T14:43:58.746248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Pointers:\n- A good starting Classifier, with **train-auc-mean of 0.900422, test-auc-mean of 0.842716. Let;s keep tuning and reduce overfitting.**","metadata":{}},{"cell_type":"markdown","source":" 2. Using CV, tune max_depth and min_child_weight next.
","metadata":{}},{"cell_type":"markdown","source":" 2.1. Tuning max_depth.
\n\n**Tips:** Keep it around 3-10.","metadata":{}},{"cell_type":"code","source":"PARAMETERS={\"objective\":'binary:logistic',\"eval_metric\":\"auc\",\"learning_rate\": 0.5}\nV_PARAM_NAME=\"max_depth\"\nV_PARAM_VALUES=range(3,10,1)\n\ndata=xgb_helper(PARAMETERS,V_PARAM_NAME=V_PARAM_NAME,V_PARAM_VALUES=V_PARAM_VALUES);","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:43:58.753488Z","iopub.execute_input":"2022-08-18T14:43:58.754661Z","iopub.status.idle":"2022-08-18T14:44:00.861951Z","shell.execute_reply.started":"2022-08-18T14:43:58.754587Z","shell.execute_reply":"2022-08-18T14:44:00.860991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Pointers:\n- Taking max_depth 5, as per the score.","metadata":{}},{"cell_type":"markdown","source":"\n 2.2. Tuning min_child_weigh.
\n\n**Tips:** Keep it small for imbalanced datasets,good for balanced","metadata":{}},{"cell_type":"code","source":"PARAMETERS={\"objective\":'binary:logistic',\"eval_metric\":\"auc\",\"learning_rate\": 0.5,\"max_depth\":5}\nV_PARAM_NAME=\"min_child_weight\"\nV_PARAM_VALUES=range(0,5,1)\n\ndata=xgb_helper(PARAMETERS,V_PARAM_NAME=V_PARAM_NAME,V_PARAM_VALUES=V_PARAM_VALUES);","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:44:00.865978Z","iopub.execute_input":"2022-08-18T14:44:00.868505Z","iopub.status.idle":"2022-08-18T14:44:02.674480Z","shell.execute_reply.started":"2022-08-18T14:44:00.868470Z","shell.execute_reply":"2022-08-18T14:44:02.673581Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Pointers:\n- Taking min_child_weight 1, as per the score.","metadata":{}},{"cell_type":"markdown","source":" 3. Its time for gamma.
\n\n**Tips:** Keep it small like 0.1-0.2 forstarting. Will be tuned later.","metadata":{}},{"cell_type":"code","source":"PARAMETERS={\"objective\":'binary:logistic',\"eval_metric\":\"auc\",\"learning_rate\": 0.5,\"max_depth\":5,\"min_child_weight\":1}\nV_PARAM_NAME = \"gamma\"\nV_PARAM_VALUES = [0.1,0.2,0.5,1,1.5,2]\n\ndata=xgb_helper(PARAMETERS,V_PARAM_NAME=V_PARAM_NAME,V_PARAM_VALUES=V_PARAM_VALUES);","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:44:02.678462Z","iopub.execute_input":"2022-08-18T14:44:02.680536Z","iopub.status.idle":"2022-08-18T14:44:04.250619Z","shell.execute_reply.started":"2022-08-18T14:44:02.680496Z","shell.execute_reply":"2022-08-18T14:44:04.249664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Pointers:\n- Taking gamma t 1, as per the score.","metadata":{}},{"cell_type":"markdown","source":" 4. Tune subsample and colsample_bytree.
","metadata":{}},{"cell_type":"markdown","source":"\n 4.1. Tuning subsample.
\n\n**Tips:** Keep it small in range 0.5-0.9.","metadata":{}},{"cell_type":"code","source":"PARAMETERS={\"objective\":'binary:logistic',\"eval_metric\":\"auc\",\"learning_rate\": 0.5,\"max_depth\":5,\"min_child_weight\":1,\"gamma\":1}\nV_PARAM_NAME = \"subsample\"\nV_PARAM_VALUES = [.4,.5,.6,.7,.8,.9]\n\ndata=xgb_helper(PARAMETERS,V_PARAM_NAME=V_PARAM_NAME,V_PARAM_VALUES=V_PARAM_VALUES);","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:44:04.254975Z","iopub.execute_input":"2022-08-18T14:44:04.257269Z","iopub.status.idle":"2022-08-18T14:44:05.900547Z","shell.execute_reply.started":"2022-08-18T14:44:04.257232Z","shell.execute_reply":"2022-08-18T14:44:05.899611Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Pointers:\n- Taking 0.7","metadata":{}},{"cell_type":"markdown","source":"\n 4.2. Tune colsample_bytree.
\n\n**Tips:** Keep it small in range 0.5-0.9.","metadata":{}},{"cell_type":"code","source":"PARAMETERS={\"objective\":'binary:logistic',\"eval_metric\":\"auc\",\"learning_rate\": 0.5,\"max_depth\":5,\"min_child_weight\":1,\"gamma\":1,\"subsample\":0.7}\nV_PARAM_NAME = \"colsample_bytree\"\nV_PARAM_VALUES = [.4,.5,.6,.7,.8,.9]\n\ndata=xgb_helper(PARAMETERS,V_PARAM_NAME=V_PARAM_NAME,V_PARAM_VALUES=V_PARAM_VALUES);","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:44:05.905005Z","iopub.execute_input":"2022-08-18T14:44:05.907450Z","iopub.status.idle":"2022-08-18T14:44:07.309753Z","shell.execute_reply.started":"2022-08-18T14:44:05.907407Z","shell.execute_reply":"2022-08-18T14:44:07.308742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Pointers:\n- Taking 0.8","metadata":{}},{"cell_type":"markdown","source":"\n 4.3. Tune scale_pos_weight.
\n\n**Tips:** Based on class imbalance.","metadata":{}},{"cell_type":"code","source":"PARAMETERS={\"objective\":'binary:logistic',\"eval_metric\":\"auc\",\"learning_rate\": 0.5,\"max_depth\":5,\"min_child_weight\":1,\n \"gamma\":1,\"subsample\":0.7,\"colsample_bytree\":.8}\n\nV_PARAM_NAME = \"scale_pos_weight\"\nV_PARAM_VALUES = [.5,1,2]\n\ndata=xgb_helper(PARAMETERS,V_PARAM_NAME=V_PARAM_NAME,V_PARAM_VALUES=V_PARAM_VALUES);","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:44:07.314695Z","iopub.execute_input":"2022-08-18T14:44:07.317168Z","iopub.status.idle":"2022-08-18T14:44:08.076940Z","shell.execute_reply.started":"2022-08-18T14:44:07.317122Z","shell.execute_reply":"2022-08-18T14:44:08.075958Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Pointers:\n- Taking 1","metadata":{}},{"cell_type":"markdown","source":" 5. Tuning Regularization Parameters (alpha,lambda).
","metadata":{}},{"cell_type":"markdown","source":"\n 5.1. Tune alpha.
\n\n**Tips:** Based on class imbalance.","metadata":{}},{"cell_type":"code","source":"PARAMETERS={\"objective\":'binary:logistic',\"eval_metric\":\"auc\",\"learning_rate\": 0.5,\"max_depth\":5,\"min_child_weight\":1,\n \"gamma\":1,\"subsample\":0.7,\"colsample_bytree\":.8, \"scale_pos_weight\":1}\n\nV_PARAM_NAME = \"reg_alpha\"\nV_PARAM_VALUES = np.linspace(start=0.001, stop=1, num=20).tolist()\n\ndata=xgb_helper(PARAMETERS,V_PARAM_NAME=V_PARAM_NAME,V_PARAM_VALUES=V_PARAM_VALUES);","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:44:08.081172Z","iopub.execute_input":"2022-08-18T14:44:08.083570Z","iopub.status.idle":"2022-08-18T14:44:14.606667Z","shell.execute_reply.started":"2022-08-18T14:44:08.083531Z","shell.execute_reply":"2022-08-18T14:44:14.605524Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Pointers:\n- Taking .15","metadata":{}},{"cell_type":"markdown","source":"\n 5.2. Tune lambda.
\n\n**Tips:** Based on class imbalance.","metadata":{}},{"cell_type":"code","source":"PARAMETERS={\"objective\":'binary:logistic',\"eval_metric\":\"auc\",\"learning_rate\": 0.5,\"max_depth\":5,\"min_child_weight\":1,\n \"gamma\":1,\"subsample\":0.7,\"colsample_bytree\":.8, \"scale_pos_weight\":1,\"reg_alpha\":0.15}\n\nV_PARAM_NAME = \"reg_lambda\"\nV_PARAM_VALUES = np.linspace(start=0.001, stop=1, num=20).tolist()\n\ndata=xgb_helper(PARAMETERS,V_PARAM_NAME=V_PARAM_NAME,V_PARAM_VALUES=V_PARAM_VALUES);","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:44:14.611536Z","iopub.execute_input":"2022-08-18T14:44:14.613974Z","iopub.status.idle":"2022-08-18T14:44:19.586111Z","shell.execute_reply.started":"2022-08-18T14:44:14.613918Z","shell.execute_reply":"2022-08-18T14:44:19.585173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Pointers:\n- Taking 1","metadata":{}},{"cell_type":"markdown","source":" 6. Lastly, Reduce Learning Rate and add more trees
","metadata":{}},{"cell_type":"markdown","source":"\n 6.1. Reduce Learning rate.
","metadata":{}},{"cell_type":"code","source":"PARAMETERS={\"objective\":'binary:logistic',\"eval_metric\":\"auc\",\"max_depth\":5,\"min_child_weight\":1,\n \"gamma\":1,\"subsample\":0.7,\"colsample_bytree\":.8, \"scale_pos_weight\":1,\"reg_alpha\":0.15,\n \"reg_lambda\":1}\n\nV_PARAM_NAME = \"learning_rate\"\nV_PARAM_VALUES = np.linspace(start=0.01, stop=0.3, num=10).tolist()\n\ndata=xgb_helper(PARAMETERS,V_PARAM_NAME=V_PARAM_NAME,V_PARAM_VALUES=V_PARAM_VALUES);","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:44:19.588129Z","iopub.execute_input":"2022-08-18T14:44:19.588971Z","iopub.status.idle":"2022-08-18T14:44:22.069400Z","shell.execute_reply.started":"2022-08-18T14:44:19.588909Z","shell.execute_reply":"2022-08-18T14:44:22.068516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Pointers:\n- Taking .3","metadata":{}},{"cell_type":"markdown","source":" Full Model
","metadata":{}},{"cell_type":"code","source":"PARAMETERS={\"objective\":'binary:logistic',\"eval_metric\":\"auc\",\"max_depth\":5,\"min_child_weight\":1,\n \"gamma\":1,\"subsample\":0.7,\"colsample_bytree\":.8, \"scale_pos_weight\":1,\"reg_alpha\":0.15,\n \"reg_lambda\":1,\"learning_rate\": 0.3}\n\nclf = xgb.XGBClassifier( tree_method=\"gpu_hist\",objective=\"binary:logistic\",eval_metric=\"auc\",max_depth=5,min_child_weight=1,\n gamma=1,subsample=0.7,colsample_bytree=.8, scale_pos_weight=1,reg_alpha=0.15,\n reg_lambda=1,learning_rate= 0.3,n_estimators=800)\n\nclf.fit(X_train,y_train)\n\nclf.save_model(\"categorical-model.json\")","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:44:22.071175Z","iopub.execute_input":"2022-08-18T14:44:22.072104Z","iopub.status.idle":"2022-08-18T14:44:23.612953Z","shell.execute_reply.started":"2022-08-18T14:44:22.072055Z","shell.execute_reply":"2022-08-18T14:44:23.611686Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred = clf.predict(X_test)","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:47:16.114870Z","iopub.execute_input":"2022-08-18T14:47:16.115339Z","iopub.status.idle":"2022-08-18T14:47:16.137160Z","shell.execute_reply.started":"2022-08-18T14:47:16.115290Z","shell.execute_reply":"2022-08-18T14:47:16.136283Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import classification_report\nprint(classification_report(y_test, pred, target_names=[\"0\",\"1\"]))","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:49:02.962978Z","iopub.execute_input":"2022-08-18T14:49:02.963376Z","iopub.status.idle":"2022-08-18T14:49:02.979602Z","shell.execute_reply.started":"2022-08-18T14:49:02.963343Z","shell.execute_reply":"2022-08-18T14:49:02.978613Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import plot_roc_curve\nplot_roc_curve(clf, X_test, y_test)","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:50:16.142439Z","iopub.execute_input":"2022-08-18T14:50:16.143042Z","iopub.status.idle":"2022-08-18T14:50:16.461934Z","shell.execute_reply.started":"2022-08-18T14:50:16.142995Z","shell.execute_reply":"2022-08-18T14:50:16.460901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import roc_auc_score\nroc_auc_score(y_test,pred)","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:51:03.270112Z","iopub.execute_input":"2022-08-18T14:51:03.271081Z","iopub.status.idle":"2022-08-18T14:51:03.281271Z","shell.execute_reply.started":"2022-08-18T14:51:03.271042Z","shell.execute_reply":"2022-08-18T14:51:03.280092Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get a graph\ngraph = xgb.to_graphviz(clf, num_trees=1)\n# Or get a matplotlib axis\nax = xgb.plot_tree(clf, num_trees=1)\n# Get feature importances\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:46:54.944254Z","iopub.execute_input":"2022-08-18T14:46:54.944735Z","iopub.status.idle":"2022-08-18T14:46:56.669558Z","shell.execute_reply.started":"2022-08-18T14:46:54.944687Z","shell.execute_reply":"2022-08-18T14:46:56.668276Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"Which parameter does what?
","metadata":{}},{"cell_type":"markdown","source":"# 📒 5. Which parameter does what?\n\nControl Overfitting
\n\n\nWhen you observe high training accuracy, but low test accuracy, it is likely that you encountered overfitting problem.\nThere are in general two ways that you can control overfitting in XGBoost:\n- **The first way is to directly control model complexity.**\n - This includes **max_depth, min_child_weight** and **gamma**.\n \n \n- **The second way is to add randomness to make training robust to noise.**\n - This includes **subsample** and **colsample_bytree**.\n - You can also reduce stepsize **eta**. Remember to increase **num_round** when you do so.","metadata":{}},{"cell_type":"markdown","source":"Control Overfitting - Code Example : Method 1
","metadata":{}},{"cell_type":"markdown","source":"### Normal Condition","metadata":{}},{"cell_type":"code","source":"PARAMETERS={\"objective\":'binary:logistic',\"eval_metric\":\"auc\"}\nxgb_helper(PARAMETERS)","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:44:26.014058Z","iopub.execute_input":"2022-08-18T14:44:26.014585Z","iopub.status.idle":"2022-08-18T14:44:26.352529Z","shell.execute_reply.started":"2022-08-18T14:44:26.014535Z","shell.execute_reply":"2022-08-18T14:44:26.351300Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**So, the difference between train and test AUC score is about 0.05, aka 5% - which is quite higher.**\n\nLet's regularise.","metadata":{}},{"cell_type":"markdown","source":"### Overfitting Controlled","metadata":{}},{"cell_type":"code","source":"PARAMETERS={\"objective\":'binary:logistic',\"eval_metric\":\"auc\", \"max_depth\":2 , \"min_child_weight\":3, \"gamma\":2}\nxgb_helper(PARAMETERS)","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:44:26.354253Z","iopub.execute_input":"2022-08-18T14:44:26.355038Z","iopub.status.idle":"2022-08-18T14:44:26.512417Z","shell.execute_reply.started":"2022-08-18T14:44:26.354993Z","shell.execute_reply":"2022-08-18T14:44:26.511013Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**So, the difference between train and test AUC score is less than 0.01, aka 1% - which is quite better.**","metadata":{}},{"cell_type":"markdown","source":"Control Overfitting - Code Example : Method 2
","metadata":{}},{"cell_type":"code","source":"PARAMETERS={\"objective\":'binary:logistic',\"eval_metric\":\"auc\", \"subsample\":0.3,\"colsample_bytree\":0.3,\"eta\":.05}\nxgb_helper(PARAMETERS,25) #increasing num bossting round to 15","metadata":{"execution":{"iopub.status.busy":"2022-08-18T14:44:26.514561Z","iopub.execute_input":"2022-08-18T14:44:26.515614Z","iopub.status.idle":"2022-08-18T14:44:26.711473Z","shell.execute_reply.started":"2022-08-18T14:44:26.515560Z","shell.execute_reply":"2022-08-18T14:44:26.710094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**So, now the difference between train and test AUC score is less than 0.035, aka 3.5% - which is better than normal.**","metadata":{}},{"cell_type":"markdown","source":"Faster training performance
\n\nThere’s a parameter called **tree_method**, set it to **hist** or **gpu_hist** for faster computation.","metadata":{}},{"cell_type":"markdown","source":"Handle Imbalanced Dataset
\n\nFor common cases such as ads clickthrough log, the dataset is extremely imbalanced. This can affect the training of XGBoost model, and there are two ways to improve it.\n\n- **If you care only about the overall performance metric (AUC) of your prediction**\n - Balance the positive and negative weights via **scale_pos_weight**\n - Use AUC for evaluation\n \n \n- **If you care about predicting the right probability**\n - In such a case, you cannot re-balance the dataset\n - Set parameter **max_delta_step** to a finite number (say 1) to help convergence","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"# 📄 References\n- [XGBoost tips & tricks](https://www.kaggle.com/discussions/general/197466)\n- [Binary Classification: XGBoost Hyperparameter Tuning Scenarios](https://towardsdatascience.com/binary-classification-xgboost-hyperparameter-tuning-scenarios-by-non-exhaustive-grid-search-and-c261f4ce098d)\n- [Complete Guide to Parameter Tuning in XGBoost with codes in Python - Aarshay Jain](https://www.analyticsvidhya.com/blog/2016/03/complete-guide-parameter-tuning-xgboost-with-codes-python/)\n- [Notes on Parameter Tuning](https://xgboost.readthedocs.io/en/stable/tutorials/param_tuning.html)\n- [XGBoost Parameters](https://xgboost.readthedocs.io/en/stable/parameter.html)\n- [L1 and L2 Regularization — Explained | by Soner Yıldırım](https://towardsdatascience.com/l1-and-l2-regularization-explained-874c3b03f668)\n- [XGBoost | Wrestling with Hyperparameters](https://medium.com/@azmine_wasi/xgboost-wrestling-with-hyperparameters-detailed-guide-part-01-3ecc8280f02b)","metadata":{}},{"cell_type":"markdown","source":"Please UPVOTE if it helped!
","metadata":{}}]}