{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Improving LightGBM model with Cleanlab for the *American Express - Default Prediction* competition\n\nThis notebook improves the LightGBM model from this [LightGBM Quickstart notebook](https://www.kaggle.com/code/ambrosm/amex-lightgbm-quickstart) using the [cleanlab](https://github.com/cleanlab/cleanlab/) library for data-centric AI. \n\n`cleanlab` improves any model by automatically removing datapoints inferred to contain errors from the model's training set. With under 5 extra lines of code, we can obtain a 1% reduction in error without changing any of the existing model, training, or data-processing code.\n\n| Model      | Public Score |\n| ----------- | ----------- |\n| LightGBM      | 0.785       |\n| LightGBM + `cleanlab`   | 0.787         |","metadata":{}},{"cell_type":"code","source":"!pip install git+https://github.com/cleanlab/cleanlab.git\n# Install latest version of cleanlab code (as of Jun 24, 2022); equivalent to installing from this commit:  !pip install git+https://github.com/cleanlab/cleanlab.git@4bd688f51c6d1135630e53dfeac2a9a223db03f3","metadata":{"execution":{"iopub.status.busy":"2022-06-24T17:27:23.423378Z","iopub.execute_input":"2022-06-24T17:27:23.423808Z","iopub.status.idle":"2022-06-24T17:28:03.924917Z","shell.execute_reply.started":"2022-06-24T17:27:23.423764Z","shell.execute_reply":"2022-06-24T17:28:03.923929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Import dependencies and set seeds:","metadata":{}},{"cell_type":"code","source":"import cleanlab\nfrom cleanlab.classification import CleanLearning\nfrom lightgbm import LGBMClassifier, log_evaluation\n\nimport numpy as np\nimport pandas as pd\nfrom IPython.display import display\nimport gc\nimport random\n\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.calibration import CalibrationDisplay\n\nSEED = 123  # for reproducibility\nnp.random.seed(SEED)\nrandom.seed(SEED)","metadata":{"execution":{"iopub.status.busy":"2022-06-24T17:34:22.872505Z","iopub.execute_input":"2022-06-24T17:34:22.872917Z","iopub.status.idle":"2022-06-24T17:34:24.610333Z","shell.execute_reply.started":"2022-06-24T17:34:22.872878Z","shell.execute_reply":"2022-06-24T17:34:24.609405Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Define metric for lightGBM model to evaluate. This metric code is taken from @yunchonggan's fast metric implementation: https://www.kaggle.com/competitions/amex-default-prediction/discussion/328020\n","metadata":{}},{"cell_type":"code","source":"def amex_metric(y_true: np.array, y_pred: np.array) -> float:\n\n    # count of positives and negatives\n    n_pos = y_true.sum()\n    n_neg = y_true.shape[0] - n_pos\n\n    # sorting by descring prediction values\n    indices = np.argsort(y_pred)[::-1]\n    preds, target = y_pred[indices], y_true[indices]\n\n    # filter the top 4% by cumulative row weights\n    weight = 20.0 - target * 19.0\n    cum_norm_weight = (weight / weight.sum()).cumsum()\n    four_pct_filter = cum_norm_weight <= 0.04\n\n    # default rate captured at 4%\n    d = target[four_pct_filter].sum() / n_pos\n\n    # weighted gini coefficient\n    lorentz = (target / n_pos).cumsum()\n    gini = ((lorentz - cum_norm_weight) * weight).sum()\n\n    # max weighted gini coefficient\n    gini_max = 10 * n_neg * (1 - 19 / (n_pos + 20 * n_neg))\n\n    # normalized weighted gini coefficient\n    g = gini / gini_max\n\n    return 0.5 * (g + d)\n\n\ndef lgb_amex_metric(y_true, y_pred):\n    \"\"\"The competition metric with lightgbm's calling convention\"\"\"\n    return ('amex', amex_metric(y_true, y_pred), True)","metadata":{"execution":{"iopub.status.busy":"2022-06-24T17:34:25.359764Z","iopub.execute_input":"2022-06-24T17:34:25.360147Z","iopub.status.idle":"2022-06-24T17:34:25.370623Z","shell.execute_reply.started":"2022-06-24T17:34:25.360118Z","shell.execute_reply":"2022-06-24T17:34:25.36983Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data Preprocessing\n\nWe just apply the same feature engineering steps as in [the original notebook](https://www.kaggle.com/code/ambrosm/amex-lightgbm-quickstart):","metadata":{}},{"cell_type":"code","source":"features_avg = ['B_1', 'B_2', 'B_3', 'B_4', 'B_5', 'B_6', 'B_8', 'B_9', 'B_10', 'B_11', 'B_12', 'B_13', 'B_14', 'B_15', 'B_16', 'B_17', 'B_18', 'B_19', 'B_20', 'B_21', 'B_22', 'B_23', 'B_24', 'B_25', 'B_28', 'B_29', 'B_30', 'B_32', 'B_33', 'B_37', 'B_38', 'B_39', 'B_40', 'B_41', 'B_42', 'D_39', 'D_41', 'D_42', 'D_43', 'D_44', 'D_45', 'D_46', 'D_47', 'D_48', 'D_50', 'D_51', 'D_53', 'D_54', 'D_55', 'D_58', 'D_59', 'D_60', 'D_61', 'D_62', 'D_65', 'D_66', 'D_69', 'D_70', 'D_71', 'D_72', 'D_73', 'D_74', 'D_75', 'D_76', 'D_77', 'D_78', 'D_80', 'D_82', 'D_84', 'D_86', 'D_91', 'D_92', 'D_94', 'D_96', 'D_103', 'D_104', 'D_108', 'D_112', 'D_113', 'D_114', 'D_115', 'D_117', 'D_118', 'D_119', 'D_120', 'D_121', 'D_122', 'D_123', 'D_124', 'D_125', 'D_126', 'D_128', 'D_129', 'D_131', 'D_132', 'D_133', 'D_134', 'D_135', 'D_136', 'D_140', 'D_141', 'D_142', 'D_144', 'D_145', 'P_2', 'P_3', 'P_4', 'R_1', 'R_2', 'R_3', 'R_7', 'R_8', 'R_9', 'R_10', 'R_11', 'R_14', 'R_15', 'R_16', 'R_17', 'R_20', 'R_21', 'R_22', 'R_24', 'R_26', 'R_27', 'S_3', 'S_5', 'S_6', 'S_7', 'S_9', 'S_11', 'S_12', 'S_13', 'S_15', 'S_16', 'S_18', 'S_22', 'S_23', 'S_25', 'S_26']\nfeatures_min = ['B_2', 'B_4', 'B_5', 'B_9', 'B_13', 'B_14', 'B_15', 'B_16', 'B_17', 'B_19', 'B_20', 'B_28', 'B_29', 'B_33', 'B_36', 'B_42', 'D_39', 'D_41', 'D_42', 'D_45', 'D_46', 'D_48', 'D_50', 'D_51', 'D_53', 'D_55', 'D_56', 'D_58', 'D_59', 'D_60', 'D_62', 'D_70', 'D_71', 'D_74', 'D_75', 'D_78', 'D_83', 'D_102', 'D_112', 'D_113', 'D_115', 'D_118', 'D_119', 'D_121', 'D_122', 'D_128', 'D_132', 'D_140', 'D_141', 'D_144', 'D_145', 'P_2', 'P_3', 'R_1', 'R_27', 'S_3', 'S_5', 'S_7', 'S_9', 'S_11', 'S_12', 'S_23', 'S_25']\nfeatures_max = ['B_1', 'B_2', 'B_3', 'B_4', 'B_5', 'B_6', 'B_7', 'B_8', 'B_9', 'B_10', 'B_12', 'B_13', 'B_14', 'B_15', 'B_16', 'B_17', 'B_18', 'B_19', 'B_21', 'B_23', 'B_24', 'B_25', 'B_29', 'B_30', 'B_33', 'B_37', 'B_38', 'B_39', 'B_40', 'B_42', 'D_39', 'D_41', 'D_42', 'D_43', 'D_44', 'D_45', 'D_46', 'D_47', 'D_48', 'D_49', 'D_50', 'D_52', 'D_55', 'D_56', 'D_58', 'D_59', 'D_60', 'D_61', 'D_63', 'D_64', 'D_65', 'D_70', 'D_71', 'D_72', 'D_73', 'D_74', 'D_76', 'D_77', 'D_78', 'D_80', 'D_82', 'D_84', 'D_91', 'D_102', 'D_105', 'D_107', 'D_110', 'D_111', 'D_112', 'D_115', 'D_116', 'D_117', 'D_118', 'D_119', 'D_121', 'D_122', 'D_123', 'D_124', 'D_125', 'D_126', 'D_128', 'D_131', 'D_132', 'D_133', 'D_134', 'D_135', 'D_136', 'D_138', 'D_140', 'D_141', 'D_142', 'D_144', 'D_145', 'P_2', 'P_3', 'P_4', 'R_1', 'R_3', 'R_5', 'R_6', 'R_7', 'R_8', 'R_10', 'R_11', 'R_14', 'R_17', 'R_20', 'R_26', 'R_27', 'S_3', 'S_5', 'S_7', 'S_8', 'S_11', 'S_12', 'S_13', 'S_15', 'S_16', 'S_22', 'S_23', 'S_24', 'S_25', 'S_26', 'S_27']\nfeatures_last = ['B_1', 'B_2', 'B_3', 'B_4', 'B_5', 'B_6', 'B_7', 'B_8', 'B_9', 'B_10', 'B_11', 'B_12', 'B_13', 'B_14', 'B_15', 'B_16', 'B_17', 'B_18', 'B_19', 'B_20', 'B_21', 'B_22', 'B_23', 'B_24', 'B_25', 'B_26', 'B_28', 'B_29', 'B_30', 'B_32', 'B_33', 'B_36', 'B_37', 'B_38', 'B_39', 'B_40', 'B_41', 'B_42', 'D_39', 'D_41', 'D_42', 'D_43', 'D_44', 'D_45', 'D_46', 'D_47', 'D_48', 'D_49', 'D_50', 'D_51', 'D_52', 'D_53', 'D_54', 'D_55', 'D_56', 'D_58', 'D_59', 'D_60', 'D_61', 'D_62', 'D_63', 'D_64', 'D_65', 'D_69', 'D_70', 'D_71', 'D_72', 'D_73', 'D_75', 'D_76', 'D_77', 'D_78', 'D_79', 'D_80', 'D_81', 'D_82', 'D_83', 'D_86', 'D_91', 'D_96', 'D_105', 'D_106', 'D_112', 'D_114', 'D_119', 'D_120', 'D_121', 'D_122', 'D_124', 'D_125', 'D_126', 'D_127', 'D_130', 'D_131', 'D_132', 'D_133', 'D_134', 'D_138', 'D_140', 'D_141', 'D_142', 'D_145', 'P_2', 'P_3', 'P_4', 'R_1', 'R_2', 'R_3', 'R_4', 'R_5', 'R_6', 'R_7', 'R_8', 'R_9', 'R_10', 'R_11', 'R_12', 'R_13', 'R_14', 'R_15', 'R_19', 'R_20', 'R_26', 'R_27', 'S_3', 'S_5', 'S_6', 'S_7', 'S_8', 'S_9', 'S_11', 'S_12', 'S_13', 'S_16', 'S_19', 'S_20', 'S_22', 'S_23', 'S_24', 'S_25', 'S_26', 'S_27']\nfor i in ['test', 'train']:\n    df = pd.read_parquet(f'../input/amex-data-integer-dtypes-parquet-format/{i}.parquet')\n    cid = pd.Categorical(df.pop('customer_ID'), ordered=True)\n    last = (cid != np.roll(cid, -1)) # mask for last statement of every customer\n    if 'target' in df.columns:\n        df.drop(columns=['target'], inplace=True)\n    gc.collect()\n    print('Read', i)\n    df_avg = (df\n              .groupby(cid)\n              .mean()[features_avg]\n              .rename(columns={f: f\"{f}_avg\" for f in features_avg})\n             )\n    gc.collect()\n    print('Computed avg', i)\n    df_min = (df\n              .groupby(cid)\n              .min()[features_min]\n              .rename(columns={f: f\"{f}_min\" for f in features_min})\n             )\n    gc.collect()\n    print('Computed min', i)\n    df_max = (df\n              .groupby(cid)\n              .max()[features_max]\n              .rename(columns={f: f\"{f}_max\" for f in features_max})\n             )\n    gc.collect()\n    print('Computed max', i)\n    df = (df.loc[last, features_last]\n          .rename(columns={f: f\"{f}_last\" for f in features_last})\n          .set_index(np.asarray(cid[last]))\n         )\n    gc.collect()\n    print('Computed last', i)\n    df = pd.concat([df, df_min, df_max, df_avg], axis=1)\n    if i == 'train': train = df\n    else: test = df\n    print(f\"{i} shape: {df.shape}\")\n    del df, df_avg, df_min, df_max, cid, last\n\ntarget = pd.read_csv('../input/amex-default-prediction/train_labels.csv').target.values\nprint(f\"target shape: {target.shape}\")","metadata":{"execution":{"iopub.status.busy":"2022-06-24T17:34:27.772556Z","iopub.execute_input":"2022-06-24T17:34:27.772921Z","iopub.status.idle":"2022-06-24T17:41:59.399853Z","shell.execute_reply.started":"2022-06-24T17:34:27.772894Z","shell.execute_reply":"2022-06-24T17:41:59.398905Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next split data into training/validation sets. In this notebook, we will only train the LightGBM model on the below training data. For a more competitive submission, you may want to train it on the merged training+validation data before submitting predictions from the resulting model.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\nfeatures = [f for f in train.columns if f != 'customer_ID' and f != 'target']\nprint(f\"{len(features)} features\")\n\nX_test = test[features]\nX_tr, X_va, y_tr, y_va = train_test_split(train[features], target, test_size=0.1, random_state=1, shuffle=True, stratify=target)\nprint(f\"Shape of training data: {X_tr.shape}\")","metadata":{"execution":{"iopub.status.busy":"2022-06-24T17:41:59.401761Z","iopub.execute_input":"2022-06-24T17:41:59.402397Z","iopub.status.idle":"2022-06-24T17:42:04.277522Z","shell.execute_reply.started":"2022-06-24T17:41:59.40236Z","shell.execute_reply":"2022-06-24T17:42:04.276362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Training models \n\nConstruct basic LightGBM model (that computes validation score every 20 boosting rounds):","metadata":{}},{"cell_type":"code","source":"def my_booster(random_state=SEED, n_estimators=200):\n    return LGBMClassifier(random_state=random_state, n_estimators=n_estimators)\n\nlgbm_kwargs = {'eval_set': [(X_va, y_va)],'eval_metric': [lgb_amex_metric], 'callbacks': [log_evaluation(20)]}\n","metadata":{"execution":{"iopub.status.busy":"2022-06-24T17:42:04.278756Z","iopub.execute_input":"2022-06-24T17:42:04.279103Z","iopub.status.idle":"2022-06-24T17:42:04.285916Z","shell.execute_reply.started":"2022-06-24T17:42:04.279072Z","shell.execute_reply":"2022-06-24T17:42:04.284907Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Fit and predict with base LightGBM model:","metadata":{}},{"cell_type":"code","source":"model = my_booster()\nmodel.fit(X_tr, y_tr, **lgbm_kwargs)\n\ny_va_pred_og = model.predict_proba(X_va, raw_score=True)\nscore_og = amex_metric(y_va, y_va_pred_og)\n\nn_trees = model.best_iteration_\nif n_trees is None: \n    n_trees = model.n_estimators\n    \ny_test_pred_og = model.predict_proba(X_test, raw_score=True)\nprint(f\"Base LightGBM model trained with {n_trees} trees.\")","metadata":{"execution":{"iopub.status.busy":"2022-06-24T17:42:04.288299Z","iopub.execute_input":"2022-06-24T17:42:04.289102Z","iopub.status.idle":"2022-06-24T17:43:51.615637Z","shell.execute_reply.started":"2022-06-24T17:42:04.289053Z","shell.execute_reply":"2022-06-24T17:43:51.614741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Add cleanlab to produce an improved version of the same model","metadata":{}},{"cell_type":"code","source":"model = my_booster()  # could be any classification model, not just LightGBM\ncl = CleanLearning(clf=model, verbose=True,\n                   find_label_issues_kwargs={\"frac_noise\": 0.2})\n\ncl.fit(X_tr, y_tr,clf_kwargs=lgbm_kwargs, sample_weight= np.ones((len(y_tr),)))\n\ny_va_pred_cl = cl.predict_proba(X_va, raw_score=True)\nscore_cl = amex_metric(y_va, y_va_pred_cl)\n\nn_trees_cl = cl.clf.best_iteration_  # Note we use cl.clf to access some attributes of base model\nif n_trees_cl is None: \n    n_trees_cl = cl.clf.n_estimators\n\ny_test_pred_cl = cl.predict_proba(X_test, raw_score=True)\nprint(f\"Cleanlab version of LightGBM model trained with {n_trees} trees.\")","metadata":{"execution":{"iopub.status.busy":"2022-06-24T17:43:51.616917Z","iopub.execute_input":"2022-06-24T17:43:51.617404Z","iopub.status.idle":"2022-06-24T17:52:24.055025Z","shell.execute_reply.started":"2022-06-24T17:43:51.617372Z","shell.execute_reply":"2022-06-24T17:52:24.053313Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Generating submission","metadata":{}},{"cell_type":"code","source":"predictions = y_test_pred_cl  # change to y_test_pred_og for predictions of base model\nsub = pd.DataFrame({'customer_ID': test.index, 'prediction': predictions})\nsub.to_csv('submission.csv', index=False)\ndisplay(sub)","metadata":{"execution":{"iopub.status.busy":"2022-06-24T17:52:24.057456Z","iopub.execute_input":"2022-06-24T17:52:24.057995Z","iopub.status.idle":"2022-06-24T17:52:29.687303Z","shell.execute_reply.started":"2022-06-24T17:52:24.057927Z","shell.execute_reply":"2022-06-24T17:52:29.686518Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Final Notes\n\nThe above LightGBM + `cleanlab` model was trained with default LightGBM hyperparameters. The `cleanlab` parameters can also be tuned to further improve overall performance. In practice, I find you can get much better results by manually inspecting the top issues `cleanlab` has identified rather than just automatically removing this data as is done in cleanlab's  `CleanLearning` approach. \n\nCleanlab may be especially useful for Gradient Boosting models (like LightGBM, XGBoost, or CatBoost), which are particularly  sensitive to noisy training data:  https://cs.cmu.edu/afs/cs/project/jair/pub/volume11/opitz99a-html/node14.html\n\nWhile this notebook used an LightGBM model, `cleanlab` can be used with any classifier. Feel free to experiment this with other models and let me know if you see an improvement!\n","metadata":{}}]}