{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# XGBoost Starter with NVTabular - LB 0.794\n\nThis notebook builds on the wonderful [XGBoost Starter - [0.793]](https://www.kaggle.com/code/cdeotte/xgboost-starter-0-793) by [Chris Deotte](https://www.kaggle.com/cdeotte).\n\nIn this notebook, we will use [NVTabular](https://github.com/NVIDIA-Merlin/NVTabular). It is a library designed specifically for processing tabular data that tighlty integrates with the rest of the [Merlin Framework](https://github.com/NVIDIA-Merlin/Merlin).\n\nNVTabular features very powerful data preprocessing techniques implemented using best practices straight from Kaggle (target encoding, for instance). It follows the `sklearn` `fit`, `transform` and `fit_transform` pattern, which allows us to write fewer lines of code using a familiar API.\n\nAlso, NVTabular leverages `dask_cudf` as it's backend -- this enables running calculations on arbitrarily large amounts of data that might not fit into our GPU RAM all at once (with some limitations -- there are certain operations that might still cause us OOM errors).\n\nLet's see if we will be able to put all this to good use in this competition! 🙂\n\nWe first need to install a bunch of things that are missing from the Kaggle kernel. The installation can take several minutes so do not be alarmed.","metadata":{}},{"cell_type":"markdown","source":"# Load Libraries","metadata":{}},{"cell_type":"code","source":"%%bash\n\nsudo apt update -y --fix-missing\nsudo apt install -y --no-install-recommends libexpat1-dev libsasl2-2 libssl-dev graphviz openssl protobuf-compiler software-properties-common libopenmpi-dev\nsudo apt autoremove -y\nsudo apt clean\nsudo rm -rf /var/lib/apt/lists/*\n\npip install betterproto graphviz pybind11 pydot pytest mpi4py\npip install nvidia-pyindex\npip install nvtabular","metadata":{"execution":{"iopub.status.busy":"2022-06-14T09:12:34.669777Z","iopub.execute_input":"2022-06-14T09:12:34.670497Z","iopub.status.idle":"2022-06-14T09:16:38.757887Z","shell.execute_reply.started":"2022-06-14T09:12:34.670093Z","shell.execute_reply":"2022-06-14T09:16:38.756825Z"},"_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!git clone https://github.com/NVIDIA-Merlin/NVTabular.git","metadata":{"execution":{"iopub.status.busy":"2022-06-14T09:16:38.759964Z","iopub.execute_input":"2022-06-14T09:16:38.760339Z","iopub.status.idle":"2022-06-14T09:17:12.603885Z","shell.execute_reply.started":"2022-06-14T09:16:38.760299Z","shell.execute_reply":"2022-06-14T09:17:12.602914Z"},"_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nos.chdir('NVTabular')\n\n!git fetch origin pull/1580/head:fix_dtype_discrepancy_after_groupby_aggs\n!git checkout fix_dtype_discrepancy_after_groupby_aggs\n!pip install -e .","metadata":{"execution":{"iopub.status.busy":"2022-06-14T09:17:12.605632Z","iopub.execute_input":"2022-06-14T09:17:12.606196Z","iopub.status.idle":"2022-06-14T09:18:15.084101Z","shell.execute_reply.started":"2022-06-14T09:17:12.606152Z","shell.execute_reply":"2022-06-14T09:18:15.083022Z"},"_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd, numpy as np # CPU libraries\nimport cupy, cudf # GPU libraries\nimport matplotlib.pyplot as plt, gc, os\nimport nvtabular as nvt\n\nprint('RAPIDS version',cudf.__version__)","metadata":{"execution":{"iopub.status.busy":"2022-06-14T09:18:15.086638Z","iopub.execute_input":"2022-06-14T09:18:15.087251Z","iopub.status.idle":"2022-06-14T09:18:17.484565Z","shell.execute_reply.started":"2022-06-14T09:18:15.087205Z","shell.execute_reply":"2022-06-14T09:18:17.482793Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# VERSION NAME FOR SAVED MODEL FILES\nVER = \"nvt_0\"\n\n# TRAIN RANDOM SEED\nSEED = 42\n\n# FILL NAN VALUE\nNAN_VALUE = -127 # will fit in int8\n\n# FOLDS PER MODEL\nFOLDS = 5","metadata":{"execution":{"iopub.status.busy":"2022-06-14T09:18:17.486445Z","iopub.execute_input":"2022-06-14T09:18:17.486849Z","iopub.status.idle":"2022-06-14T09:18:17.493682Z","shell.execute_reply.started":"2022-06-14T09:18:17.486807Z","shell.execute_reply":"2022-06-14T09:18:17.493038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Process and Feature Engineer Train Data\nWe will load @raddar Kaggle dataset from [here][1] with discussion [here][2]. Then we will engineer features suggested by @huseyincot in his notebooks [here][3] and [here][4]. We will use [NVTabular][6] (leveraging `cudf` by [RAPIDS][5]) and the GPU to create new features quickly.\n\n[1]: https://www.kaggle.com/datasets/raddar/amex-data-integer-dtypes-parquet-format\n[2]: https://www.kaggle.com/competitions/amex-default-prediction/discussion/328514\n[3]: https://www.kaggle.com/code/huseyincot/amex-catboost-0-793\n[4]: https://www.kaggle.com/code/huseyincot/amex-agg-data-how-it-created\n[5]: https://rapids.ai/\n[6]: https://github.com/NVIDIA-Merlin/NVTabular","metadata":{}},{"cell_type":"code","source":"def read_file(path = '', usecols = None):\n    if usecols is not None: df = cudf.read_parquet(path, columns=usecols)\n    else: df = cudf.read_parquet(path)\n    \n    df['customer_ID'] = df['customer_ID'].str[-16:].str.hex_to_int().astype('int64')\n    df.S_2 = cudf.to_datetime(df.S_2)\n    df = df.fillna(NAN_VALUE) \n    print('shape of data:', df.shape)\n    \n    return df\n\ntrain = read_file('../../input/amex-data-integer-dtypes-parquet-format/train.parquet')","metadata":{"execution":{"iopub.status.busy":"2022-06-14T09:18:17.495153Z","iopub.execute_input":"2022-06-14T09:18:17.495782Z","iopub.status.idle":"2022-06-14T09:18:41.139524Z","shell.execute_reply.started":"2022-06-14T09:18:17.495744Z","shell.execute_reply":"2022-06-14T09:18:41.138696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_cols = [c for c in list(train.columns) if c not in ['customer_ID','S_2']]\ncat_features = [\"B_30\",\"B_38\",\"D_114\",\"D_116\",\"D_117\",\"D_120\",\"D_126\",\"D_63\",\"D_64\",\"D_66\",\"D_68\"]\nnum_features = [col for col in all_cols if col not in cat_features]\n\ntargets = cudf.read_csv('../../input/amex-default-prediction/train_labels.csv')\ntargets['customer_ID'] = targets['customer_ID'].str[-16:].str.hex_to_int().astype('int64')\n\ntrain = train.merge(targets, on='customer_ID', how='left')","metadata":{"execution":{"iopub.status.busy":"2022-06-14T09:18:41.141132Z","iopub.execute_input":"2022-06-14T09:18:41.141647Z","iopub.status.idle":"2022-06-14T09:18:41.816874Z","shell.execute_reply.started":"2022-06-14T09:18:41.141608Z","shell.execute_reply":"2022-06-14T09:18:41.816045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = train.to_pandas()\nfig, axes = plt.subplots(4, 3, figsize=(15,8))\n\nfor col, ax in zip(cat_features, axes.flat):\n    vc = df[col].value_counts()\n    vc.reset_index(drop=True, inplace=True)\n    ax.bar(vc.index.astype(np.int32), vc.values)\n    ax.set_title('Frequency of ' + str(col))\n    ax.set_xlabel('Number frequency')\n    ax.set_ylabel('Frequency')\n\naxes.flatten()[-1].axis('off')    \nfig.tight_layout()","metadata":{"execution":{"iopub.status.busy":"2022-06-14T09:18:41.818236Z","iopub.execute_input":"2022-06-14T09:18:41.8186Z","iopub.status.idle":"2022-06-14T09:18:50.772506Z","shell.execute_reply.started":"2022-06-14T09:18:41.81856Z","shell.execute_reply":"2022-06-14T09:18:50.771728Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\nds = nvt.Dataset(train)\n\nds = nvt.Dataset(train, npartitions=5)\nds = ds.shuffle_by_keys(keys=['customer_ID'])\ntrain = ds.to_ddf()\ntrain = train.sort_values(['customer_ID', 'S_2'])\nds = nvt.Dataset(train)\n\nnum_aggs = num_features + ['customer_ID'] >> nvt.ops.Groupby(\n    'customer_ID',\n    aggs=['mean', 'std', 'min', 'max', 'last']\n)\n\ncat_aggs = cat_features + ['customer_ID'] >> nvt.ops.Groupby(\n    'customer_ID',\n    aggs=['count', 'last', 'nunique']\n)\n\nte = cat_features >> nvt.ops.TargetEncoding(\n    'target',\n    kfold=5\n) \nte = te + ['customer_ID'] >> nvt.ops.Groupby(\n    'customer_ID',\n    aggs=['mean', 'last']\n)\n\nout = num_aggs + cat_aggs + te >> nvt.ops.ReduceDtypeSize()\n\nwf = nvt.Workflow(out)\ntrain = wf.fit_transform(ds).compute()","metadata":{"execution":{"iopub.status.busy":"2022-06-14T09:18:50.773835Z","iopub.execute_input":"2022-06-14T09:18:50.774339Z","iopub.status.idle":"2022-06-14T09:19:51.160898Z","shell.execute_reply.started":"2022-06-14T09:18:50.7743Z","shell.execute_reply":"2022-06-14T09:19:51.159898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train.merge(targets, on='customer_ID', how='left')","metadata":{"execution":{"iopub.status.busy":"2022-06-14T09:19:51.164399Z","iopub.execute_input":"2022-06-14T09:19:51.164898Z","iopub.status.idle":"2022-06-14T09:19:51.451178Z","shell.execute_reply.started":"2022-06-14T09:19:51.164854Z","shell.execute_reply":"2022-06-14T09:19:51.450412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# the beauty of nvt.ops.ReduceDtypeSize! our dtypes got reduced to save memory... automagically!\ntrain.dtypes.value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-06-14T09:19:51.452982Z","iopub.execute_input":"2022-06-14T09:19:51.453393Z","iopub.status.idle":"2022-06-14T09:19:51.463893Z","shell.execute_reply.started":"2022-06-14T09:19:51.453339Z","shell.execute_reply":"2022-06-14T09:19:51.463164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train XGB\nWe will train using `DeviceQuantileDMatrix`. This has a very small GPU memory footprint.","metadata":{}},{"cell_type":"code","source":"# LOAD XGB LIBRARY\nfrom sklearn.model_selection import KFold\nimport xgboost as xgb\nprint('XGB Version',xgb.__version__)\n\n# XGB MODEL PARAMETERS\nxgb_parms = { \n    'max_depth':4, \n    'learning_rate':0.05, \n    'subsample':0.8,\n    'colsample_bytree':0.6, \n    'eval_metric':'logloss',\n    'objective':'binary:logistic',\n    'tree_method':'gpu_hist',\n    'predictor':'gpu_predictor',\n    'random_state':SEED\n}","metadata":{"execution":{"iopub.status.busy":"2022-06-14T09:19:51.465304Z","iopub.execute_input":"2022-06-14T09:19:51.46586Z","iopub.status.idle":"2022-06-14T09:19:51.556462Z","shell.execute_reply.started":"2022-06-14T09:19:51.46582Z","shell.execute_reply":"2022-06-14T09:19:51.555668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# NEEDED WITH DeviceQuantileDMatrix BELOW\nclass IterLoadForDMatrix(xgb.core.DataIter):\n    def __init__(self, df=None, features=None, target=None, batch_size=256*1024):\n        self.features = features\n        self.target = target\n        self.df = df\n        self.it = 0 # set iterator to 0\n        self.batch_size = batch_size\n        self.batches = int( np.ceil( len(df) / self.batch_size ) )\n        super().__init__()\n\n    def reset(self):\n        '''Reset the iterator'''\n        self.it = 0\n\n    def next(self, input_data):\n        '''Yield next batch of data.'''\n        if self.it == self.batches:\n            return 0 # Return 0 when there's no more batch.\n        \n        a = self.it * self.batch_size\n        b = min( (self.it + 1) * self.batch_size, len(self.df) )\n        dt = cudf.DataFrame(self.df.iloc[a:b])\n        input_data(data=dt[self.features], label=dt[self.target]) #, weight=dt['weight'])\n        self.it += 1\n        return 1","metadata":{"execution":{"iopub.status.busy":"2022-06-14T09:19:51.55754Z","iopub.execute_input":"2022-06-14T09:19:51.558319Z","iopub.status.idle":"2022-06-14T09:19:51.567863Z","shell.execute_reply.started":"2022-06-14T09:19:51.558283Z","shell.execute_reply":"2022-06-14T09:19:51.567081Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# https://www.kaggle.com/kyakovlev\n# https://www.kaggle.com/competitions/amex-default-prediction/discussion/327534\ndef amex_metric_mod(y_true, y_pred):\n\n    labels     = np.transpose(np.array([y_true, y_pred]))\n    labels     = labels[labels[:, 1].argsort()[::-1]]\n    weights    = np.where(labels[:,0]==0, 20, 1)\n    cut_vals   = labels[np.cumsum(weights) <= int(0.04 * np.sum(weights))]\n    top_four   = np.sum(cut_vals[:,0]) / np.sum(labels[:,0])\n\n    gini = [0,0]\n    for i in [1,0]:\n        labels         = np.transpose(np.array([y_true, y_pred]))\n        labels         = labels[labels[:, i].argsort()[::-1]]\n        weight         = np.where(labels[:,0]==0, 20, 1)\n        weight_random  = np.cumsum(weight / np.sum(weight))\n        total_pos      = np.sum(labels[:, 0] *  weight)\n        cum_pos_found  = np.cumsum(labels[:, 0] * weight)\n        lorentz        = cum_pos_found / total_pos\n        gini[i]        = np.sum((lorentz - weight_random) * weight)\n\n    return 0.5 * (gini[1]/gini[0] + top_four)","metadata":{"execution":{"iopub.status.busy":"2022-06-14T09:19:51.569278Z","iopub.execute_input":"2022-06-14T09:19:51.569714Z","iopub.status.idle":"2022-06-14T09:19:51.581809Z","shell.execute_reply.started":"2022-06-14T09:19:51.569677Z","shell.execute_reply":"2022-06-14T09:19:51.580929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"FEATURES = train.columns[1:-1]\n\nimportances = []\noof = []\ntrain = train.to_pandas() # free GPU memory\nTRAIN_SUBSAMPLE = 1.0\ngc.collect()\n\nskf = KFold(n_splits=FOLDS)\nfor fold,(train_idx, valid_idx) in enumerate(skf.split(\n            train, train.target )):\n    \n    # TRAIN WITH SUBSAMPLE OF TRAIN FOLD DATA\n    if TRAIN_SUBSAMPLE<1.0:\n        np.random.seed(SEED)\n        train_idx = np.random.choice(train_idx, \n                       int(len(train_idx)*TRAIN_SUBSAMPLE), replace=False)\n        np.random.seed(None)\n    \n    print('#'*25)\n    print('### Fold',fold+1)\n    print('### Train size',len(train_idx),'Valid size',len(valid_idx))\n    print(f'### Training with {int(TRAIN_SUBSAMPLE*100)}% fold data...')\n    print('#'*25)\n    \n    # TRAIN, VALID, TEST FOR FOLD K\n    Xy_train = IterLoadForDMatrix(train.loc[train_idx], FEATURES, 'target')\n    X_valid = train.loc[valid_idx, FEATURES]\n    y_valid = train.loc[valid_idx, 'target']\n    \n    dtrain = xgb.DeviceQuantileDMatrix(Xy_train, max_bin=256)\n    dvalid = xgb.DMatrix(data=X_valid, label=y_valid)\n    \n    # TRAIN MODEL FOLD K\n    model = xgb.train(xgb_parms, \n                dtrain=dtrain,\n                evals=[(dtrain,'train'),(dvalid,'valid')],\n                num_boost_round=9999,\n                early_stopping_rounds=100,\n                verbose_eval=100) \n    model.save_model(f'XGB_v{VER}_fold{fold}.xgb')\n    \n    # GET FEATURE IMPORTANCE FOR FOLD K\n    dd = model.get_score(importance_type='weight')\n    df = pd.DataFrame({'feature':dd.keys(),f'importance_{fold}':dd.values()})\n    importances.append(df)\n            \n    # INFER OOF FOLD K\n    oof_preds = model.predict(dvalid)\n    acc = amex_metric_mod(y_valid.values, oof_preds)\n    print('Kaggle Metric =',acc,'\\n')\n    \n    # SAVE OOF\n    df = train.loc[valid_idx, ['customer_ID','target'] ].copy()\n    df['oof_pred'] = oof_preds\n    oof.append( df )\n    \n    del dtrain, Xy_train, dd, df\n    del X_valid, y_valid, dvalid, model\n    _ = gc.collect()\n    \nprint('#'*25)\noof = pd.concat(oof,axis=0,ignore_index=True).set_index('customer_ID')\nacc = amex_metric_mod(oof.target.values, oof.oof_pred.values)\nprint('OVERALL CV Kaggle Metric =',acc)","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-06-14T09:19:51.583168Z","iopub.execute_input":"2022-06-14T09:19:51.583839Z","iopub.status.idle":"2022-06-14T09:29:27.565297Z","shell.execute_reply.started":"2022-06-14T09:19:51.583802Z","shell.execute_reply":"2022-06-14T09:29:27.564448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# CLEAN RAM\ndel train\n_ = gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-14T09:29:27.566528Z","iopub.execute_input":"2022-06-14T09:29:27.567508Z","iopub.status.idle":"2022-06-14T09:29:27.758387Z","shell.execute_reply.started":"2022-06-14T09:29:27.567468Z","shell.execute_reply":"2022-06-14T09:29:27.756452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Predict on test set","metadata":{}},{"cell_type":"code","source":"# CALCULATE SIZE OF EACH SEPARATE TEST PART\ndef get_rows(customers, test, NUM_PARTS = 4, verbose = ''):\n    chunk = len(customers)//NUM_PARTS\n    if verbose != '':\n        print(f'We will process {verbose} data as {NUM_PARTS} separate parts.')\n        print(f'There will be {chunk} customers in each part (except the last part).')\n        print('Below are number of rows in each part:')\n    rows = []\n\n    for k in range(NUM_PARTS):\n        if k==NUM_PARTS-1: cc = customers[k*chunk:]\n        else: cc = customers[k*chunk:(k+1)*chunk]\n        s = test.loc[test.customer_ID.isin(cc)].shape[0]\n        rows.append(s)\n    if verbose != '': print( rows )\n    return rows,chunk\n\n# COMPUTE SIZE OF 4 PARTS FOR TEST DATA\nNUM_PARTS = 4\nTEST_PATH = '../../input/amex-data-integer-dtypes-parquet-format/test.parquet'\n\nprint(f'Reading test data...')\ntest = read_file(path = TEST_PATH, usecols = ['customer_ID','S_2'])\ncustomers = test[['customer_ID']].drop_duplicates().sort_index().values.flatten()\nrows,num_cust = get_rows(customers, test[['customer_ID']], NUM_PARTS = NUM_PARTS, verbose = 'test')","metadata":{"execution":{"iopub.status.busy":"2022-06-14T09:30:31.631936Z","iopub.execute_input":"2022-06-14T09:30:31.632346Z","iopub.status.idle":"2022-06-14T09:30:34.09171Z","shell.execute_reply.started":"2022-06-14T09:30:31.632314Z","shell.execute_reply":"2022-06-14T09:30:34.090812Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# INFER TEST DATA IN PARTS\nskip_rows = 0\nskip_cust = 0\ntest_preds = []\n\nfor k in range(NUM_PARTS):\n    \n    # READ PART OF TEST DATA\n    print(f'\\nReading test data...')\n    test = read_file(path = TEST_PATH)\n    test = test.iloc[skip_rows:skip_rows+rows[k]]\n    skip_rows += rows[k]\n    print(f'=> Test part {k+1} has shape', test.shape )\n    \n    # PROCESS AND FEATURE ENGINEER PART OF TEST DATA\n    test['D_59'][test['D_59'] > 127] = -1\n    test['D_123'][test['D_123'] > 127] = -1\n\n    test['D_59'] = test['D_59'].astype(np.int8)\n    test['D_123'] = test['D_123'].astype(np.int8)\n    test['D_124'] = test['D_124'].astype(np.int16)\n    test['target'] = 0 # dummy variable to get our pipeline to run\n    test = wf.transform(nvt.Dataset(test)).compute()\n    test = test.set_index('customer_ID')\n    if k==NUM_PARTS-1: test = test.loc[customers[skip_cust:]]\n    else: test = test.loc[customers[skip_cust:skip_cust+num_cust]]\n    skip_cust += num_cust\n    \n    # TEST DATA FOR XGB\n    X_test = test[FEATURES]\n    dtest = xgb.DMatrix(data=X_test)\n    test = test[['P_2_mean']] # reduce memory\n    del X_test\n    gc.collect()\n\n    # INFER XGB MODELS ON TEST DATA\n    model = xgb.Booster()\n    model.load_model(f'XGB_v{VER}_fold0.xgb')\n    preds = model.predict(dtest)\n    for f in range(1,FOLDS):\n        model.load_model(f'XGB_v{VER}_fold{f}.xgb')\n        preds += model.predict(dtest)\n    preds /= FOLDS\n    test_preds.append(preds)\n\n    # CLEAN MEMORY\n    del dtest, model\n    _ = gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-14T09:37:19.308742Z","iopub.execute_input":"2022-06-14T09:37:19.309128Z","iopub.status.idle":"2022-06-14T09:39:19.939709Z","shell.execute_reply.started":"2022-06-14T09:37:19.309096Z","shell.execute_reply":"2022-06-14T09:39:19.938894Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# WRITE SUBMISSION FILE\ntest_preds = np.concatenate(test_preds)\ntest = cudf.DataFrame(index=customers,data={'prediction':test_preds})\nsub = cudf.read_csv('../../input/amex-default-prediction/sample_submission.csv')[['customer_ID']]\nsub['customer_ID_hash'] = sub['customer_ID'].str[-16:].str.hex_to_int().astype('int64')\nsub = sub.set_index('customer_ID_hash')\nsub = sub.merge(test[['prediction']], left_index=True, right_index=True, how='left')\nsub = sub.reset_index(drop=True)\n\n# DISPLAY PREDICTIONS\nsub.to_csv('../submission.csv', index=False)\nprint('Submission file shape is', sub.shape )\nsub.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-14T09:39:19.949019Z","iopub.execute_input":"2022-06-14T09:39:19.949947Z","iopub.status.idle":"2022-06-14T09:39:20.967529Z","shell.execute_reply.started":"2022-06-14T09:39:19.949846Z","shell.execute_reply":"2022-06-14T09:39:20.966698Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# PLOT PREDICTIONS\nplt.hist(sub.to_pandas().prediction, bins=100)\nplt.title('Test Predictions')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-14T09:39:20.969348Z","iopub.execute_input":"2022-06-14T09:39:20.970754Z","iopub.status.idle":"2022-06-14T09:39:21.654053Z","shell.execute_reply.started":"2022-06-14T09:39:20.970616Z","shell.execute_reply":"2022-06-14T09:39:21.653221Z"},"trusted":true},"execution_count":null,"outputs":[]}]}