{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<b><h4>Feature Selection is the process of selecting a subset of features for Model Construction.</h4></b>\n\n<h4><b>There are some great benefits of doing feature selection in Kaggle Competitions.</b></h4><br>\n1. Shorter training times.\n\n2. Less memory usage.\n\n3. Simplification of models and data.\n\n4. Avoid curse of dimensionality.\n\n5. Avoid noisy and irrelevant features.\n\n<br>**The BorutaShap package, as the name suggests, combines the [Boruta feature selection algorithm](https://www.jstatsoft.org/article/view/v036i11) with the [SHAP](https://christophm.github.io/interpretable-ml-book/shap.html) technique.<br><br>**\n\n**This notebook demonstrates how to apply Boruta-SHAP Feature Selection on [Kaggle : American Express - Default Prediction](https://www.kaggle.com/competitions/amex-default-prediction).**\n\n**I used the data generated by this [notebook](https://www.kaggle.com/code/susnato/amex-data-preprocesing-feature-engineering) using this [dataset](https://www.kaggle.com/datasets/raddar/amex-data-integer-dtypes-parquet-format).**","metadata":{}},{"cell_type":"markdown","source":"<font color='red'>**<h2>If you like this notebook then please upvote</h2>**","metadata":{}},{"cell_type":"markdown","source":"## Install requirements","metadata":{}},{"cell_type":"code","source":"!pip install BorutaShap -q\n!pip install xgboost==1.6.1 -q","metadata":{"execution":{"iopub.status.busy":"2022-06-19T09:18:57.821651Z","iopub.execute_input":"2022-06-19T09:18:57.822068Z","iopub.status.idle":"2022-06-19T09:19:19.5782Z","shell.execute_reply.started":"2022-06-19T09:18:57.821987Z","shell.execute_reply":"2022-06-19T09:19:19.577194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Imports","metadata":{}},{"cell_type":"code","source":"import os\nimport gc\nimport glob\nimport tqdm\nimport numpy as np\nimport pandas as pd","metadata":{"execution":{"iopub.status.busy":"2022-06-19T09:19:19.579831Z","iopub.execute_input":"2022-06-19T09:19:19.580174Z","iopub.status.idle":"2022-06-19T09:19:19.587212Z","shell.execute_reply.started":"2022-06-19T09:19:19.580145Z","shell.execute_reply":"2022-06-19T09:19:19.585814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Load data","metadata":{}},{"cell_type":"code","source":"SEED = 42\n\ntrain_labels = pd.read_csv('../input/amex-default-prediction/train_labels.csv')\ntrain_labels['customer_ID'] = train_labels['customer_ID'].apply(lambda x: int(x[-16:], 16)).astype(np.int64)\ntrain_labels = train_labels.set_axis(train_labels['customer_ID'])\ntrain_labels = train_labels.drop(['customer_ID'], axis=1)\n\ntrain_pkls = sorted(glob.glob('../input/amex-data-preprocesing-feature-engineering/train_data_*'))\ntrain_y = sorted(glob.glob('../input/amex-data-preprocesing-feature-engineering/train_y_*.npy'))\ntest_pkls = sorted(glob.glob('../input/amex-data-preprocesing-feature-engineering/test_data_*'))","metadata":{"execution":{"iopub.status.busy":"2022-06-19T09:19:19.588343Z","iopub.execute_input":"2022-06-19T09:19:19.589186Z","iopub.status.idle":"2022-06-19T09:19:20.932245Z","shell.execute_reply.started":"2022-06-19T09:19:19.589132Z","shell.execute_reply":"2022-06-19T09:19:20.931458Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\ntrain_df = pd.read_pickle(train_pkls[0])\nprint(train_pkls[0])\nfor i in train_pkls[1:]:\n    print(i)\n    train_df = train_df.append(pd.read_pickle(i))\n    gc.collect()\n    \ny = train_labels.loc[train_df.index.values].values.astype(np.int8)\ntrain_df = train_df.drop(['D_64_1', 'D_66_0', 'D_68_0'], axis=1)\n\nX_train, X_val, y_train, y_val = train_test_split(train_df, y, \n                                                  stratify=y, test_size=0.25, \n                                                  random_state=42, shuffle=True)\nprint(X_train.shape,  y_train.shape, X_val.shape, y_val.shape)\ndel train_df, y\n_ = gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-19T09:19:20.934189Z","iopub.execute_input":"2022-06-19T09:19:20.935274Z","iopub.status.idle":"2022-06-19T09:19:46.647656Z","shell.execute_reply.started":"2022-06-19T09:19:20.93523Z","shell.execute_reply":"2022-06-19T09:19:46.645493Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Load and run the algorithm","metadata":{}},{"cell_type":"code","source":"import xgboost as xgb\nfrom BorutaShap import BorutaShap\n\nxgb_clf = xgb.XGBClassifier(\n        n_estimators = 1500,  \n        max_depth = 3,  \n        learning_rate = 0.05,  \n        subsample = 0.7,  \n        colsample_bytree = 0.5,  \n        min_child_weight = 5,  \n        reg_alpha = 0.5,  \n        reg_lambda = 2,  \n        tree_method ='gpu_hist',\n        predictor = 'gpu_predictor',\n        random_state = 42)\n\nFeature_Selector = BorutaShap(model = xgb_clf, \n                              importance_measure='shap', \n                              classification=True)\nFeature_Selector.fit(X=X_train.iloc[:75000], y=y_train[:75000], \n                     n_trials=50, random_state=SEED)\n\ndel xgb_clf\n_ = gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-19T09:19:46.649114Z","iopub.execute_input":"2022-06-19T09:19:46.649599Z","iopub.status.idle":"2022-06-19T09:44:11.865593Z","shell.execute_reply.started":"2022-06-19T09:19:46.649541Z","shell.execute_reply":"2022-06-19T09:44:11.864117Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Feature_Selector.plot(which_features='accepted', figsize=(16,12))","metadata":{"execution":{"iopub.status.busy":"2022-06-19T09:44:11.867198Z","iopub.execute_input":"2022-06-19T09:44:11.867539Z","iopub.status.idle":"2022-06-19T09:44:13.548821Z","shell.execute_reply.started":"2022-06-19T09:44:11.867505Z","shell.execute_reply":"2022-06-19T09:44:13.548066Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Feature_Selector.plot(which_features='tentative', figsize=(16,12))","metadata":{"execution":{"iopub.status.busy":"2022-06-19T09:44:13.550187Z","iopub.execute_input":"2022-06-19T09:44:13.550701Z","iopub.status.idle":"2022-06-19T09:44:14.109855Z","shell.execute_reply.started":"2022-06-19T09:44:13.550661Z","shell.execute_reply":"2022-06-19T09:44:14.109001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**If we have `tentative` features then we can either run the algorithm for more `trials` or we can use `Feature_Selector.TentativeRoughFix()`. It will sort out those `tentative` features into useful and not useful ones.**","metadata":{}},{"cell_type":"code","source":"Feature_Selector.TentativeRoughFix()","metadata":{"execution":{"iopub.status.busy":"2022-06-19T09:44:14.111012Z","iopub.execute_input":"2022-06-19T09:44:14.111392Z","iopub.status.idle":"2022-06-19T09:44:14.119662Z","shell.execute_reply.started":"2022-06-19T09:44:14.111361Z","shell.execute_reply":"2022-06-19T09:44:14.118866Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Now let's test new features","metadata":{}},{"cell_type":"markdown","source":"**But first we need to fit the model with all the data so that we can draw a baseline.**","metadata":{}},{"cell_type":"code","source":"# https://www.kaggle.com/competitions/amex-default-prediction/discussion/328020\ndef amex_metric(preds: np.ndarray, target: np.ndarray) -> float:\n    indices = np.argsort(preds)[::-1]\n    preds, target = preds[indices], target[indices]\n\n    weight = 20.0 - target * 19.0\n    cum_norm_weight = (weight / weight.sum()).cumsum()\n    four_pct_mask = cum_norm_weight <= 0.04\n    d = np.sum(target[four_pct_mask]) / np.sum(target)\n\n    weighted_target = target * weight\n    lorentz = (weighted_target / weighted_target.sum()).cumsum()\n    gini = ((lorentz - cum_norm_weight) * weight).sum()\n\n    n_pos = np.sum(target)\n    n_neg = target.shape[0] - n_pos\n    gini_max = 10 * n_neg * (n_pos + 20 * n_neg - 19) / (n_pos + 20 * n_neg)\n\n    g = gini / gini_max\n    return 0.5 * (g + d)","metadata":{"execution":{"iopub.status.busy":"2022-06-19T09:44:14.121032Z","iopub.execute_input":"2022-06-19T09:44:14.121552Z","iopub.status.idle":"2022-06-19T09:44:14.13159Z","shell.execute_reply.started":"2022-06-19T09:44:14.121514Z","shell.execute_reply":"2022-06-19T09:44:14.130695Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"xgb_clf_full = xgb.XGBClassifier(\n        n_estimators = 1500,  \n        max_depth = 3,  \n        learning_rate = 0.05,  \n        subsample = 0.7,  \n        colsample_bytree = 0.5,  \n        min_child_weight = 5,  \n        reg_alpha = 0.5,  \n        reg_lambda = 2,  \n        tree_method ='gpu_hist',\n        predictor = 'gpu_predictor',\n        random_state = SEED)\n\nxgb_clf_full.fit(X_train, y_train,\n                 eval_set=[(X_train, y_train), (X_val, y_val)],\n                 verbose=100)\n\nmetric_score_full = amex_metric(target=y_val.reshape(-1, ), \n                                preds=xgb_clf_full.predict_proba(X_val)[:, 1].reshape(-1, ))\nprint(\"The model scored : {} after using the whole set of features.\".format(metric_score_full))\n\ndel xgb_clf_full\n_ = gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-19T09:44:14.13422Z","iopub.execute_input":"2022-06-19T09:44:14.134587Z","iopub.status.idle":"2022-06-19T09:46:38.132191Z","shell.execute_reply.started":"2022-06-19T09:44:14.134551Z","shell.execute_reply":"2022-06-19T09:46:38.130544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Let's now train the model with only selected features**","metadata":{}},{"cell_type":"code","source":"selected_cols = Feature_Selector.accepted\nprint(\"There are only {} features selected\".format(len(selected_cols)))\n\nxgb_clf_selected = xgb.XGBClassifier(\n        n_estimators = 1500,  \n        max_depth = 3,  \n        learning_rate = 0.05,  \n        subsample = 0.7,  \n        colsample_bytree = 0.5,  \n        min_child_weight = 5,  \n        reg_alpha = 0.5,  \n        reg_lambda = 2,  \n        tree_method ='gpu_hist',\n        predictor = 'gpu_predictor',\n        random_state = SEED)\n\nxgb_clf_selected.fit(X_train[selected_cols], y_train,\n                 eval_set=[(X_train[selected_cols], y_train), \n                           (X_val[selected_cols], y_val)],\n                 verbose=100)\n\nmetric_score_selected = amex_metric(target=y_val.reshape(-1, ), \n                                    preds=xgb_clf_selected.predict_proba(X_val[selected_cols])[:, 1].reshape(-1, ))\nprint(\"The model scored : {} after using only : {} features.\".format(metric_score_selected, len(selected_cols)))\n\ndel xgb_clf_selected\n_ = gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-19T09:46:38.133579Z","iopub.execute_input":"2022-06-19T09:46:38.133981Z","iopub.status.idle":"2022-06-19T09:46:50.555845Z","shell.execute_reply.started":"2022-06-19T09:46:38.13394Z","shell.execute_reply":"2022-06-19T09:46:50.554279Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**We can get more performance by using more data.<br>\nI used Colab Pro to train the model on almost 300000 rows and here is the result.**","metadata":{}},{"cell_type":"code","source":"selected_cols_colab = np.load('../input/shapxgboost/accepted_features_XGBOOST.npy').tolist()\nprint(\"There are only {} features selected\".format(len(selected_cols_colab)))\n\nxgb_clf_selected_colab = xgb.XGBClassifier(\n        n_estimators = 1500,  \n        max_depth = 3,  \n        learning_rate = 0.05,  \n        subsample = 0.7,  \n        colsample_bytree = 0.5,  \n        min_child_weight = 5,  \n        reg_alpha = 0.5,  \n        reg_lambda = 2,  \n        tree_method ='gpu_hist',\n        predictor = 'gpu_predictor',\n        random_state = SEED)\n\nxgb_clf_selected_colab.fit(X_train[selected_cols_colab], y_train,\n                 eval_set=[(X_train[selected_cols_colab], y_train), \n                           (X_val[selected_cols_colab], y_val)],\n                 verbose=100)\n\nmetric_score_selected_colab = amex_metric(target=y_val.reshape(-1, ), \n                                    preds=xgb_clf_selected_colab.predict_proba(X_val[selected_cols_colab])[:, 1].reshape(-1, ))\nprint(\"The model scored : {} after using only : {} features.\".format(metric_score_selected_colab, len(selected_cols_colab)))\n\ndel xgb_clf_selected_colab\n_ = gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-19T09:53:56.184309Z","iopub.execute_input":"2022-06-19T09:53:56.184889Z","iopub.status.idle":"2022-06-19T09:54:17.149152Z","shell.execute_reply.started":"2022-06-19T09:53:56.184845Z","shell.execute_reply":"2022-06-19T09:54:17.148321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**As you can see, we are getting almost the same result with 116 features. And the previous result was not too far off despite having used very little data.**","metadata":{}},{"cell_type":"markdown","source":"## Conclusion\n\n* **Thie Features are for XGBOOST. But we can use any model as long as it has `fit` and `predict` method.**\n\n* **In this example we used a very thin slice of data and still managed to get almost the same result as before.**\n\n* **We can save a lot of memory by only using the `accepted` features. because Kaggle Notebook only allows us to load limited amount of features into memory.**\n\n* **As this competition goes on, more and more people will find useful and interesting features, we may use this approach then to find the relevent ones in those.**","metadata":{}},{"cell_type":"markdown","source":"## Reference\n\n* https://github.com/Ekeany/Boruta-Shap\n\n* https://medium.com/analytics-vidhya/is-this-the-best-feature-selection-algorithm-borutashap-8bc238aa1677\n\n* https://www.kaggle.com/code/carlmcbrideellis/feature-selection-using-the-boruta-shap-package/notebook","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}