{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Learning to rank\n\nIt is a subfield of machine learning often used to rank documents based on their relevance score. However, it is not that often used in Kaggle, so I decided to give it a shot for some proxy task in the [American Express](https://www.kaggle.com/competitions/amex-default-prediction) competition.\n\nThe data has temporal structure - each customer has N months of history observed (N = 1,...,13).\n\nWe are going to test the hypothesis, that there are time related features in the dataset, which have strong correlation with time. Before doing this I was interested if there was `time_since_customer` type of variables in the dataset. Spotting this by looking at the data is cumbersome. However, learning to rank models are perfect for this task!\n\n\n","metadata":{}},{"cell_type":"markdown","source":"# Model setup","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport xgboost as xgb\nfrom xgboost import plot_importance\nfrom sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-06-15T08:04:33.954489Z","iopub.execute_input":"2022-06-15T08:04:33.955601Z","iopub.status.idle":"2022-06-15T08:04:34.975902Z","shell.execute_reply.started":"2022-06-15T08:04:33.955492Z","shell.execute_reply":"2022-06-15T08:04:34.974528Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We are going to use the dataset I have cleaned up of the original data. More info in a kaggle [thread](https://www.kaggle.com/competitions/amex-default-prediction/discussion/328514).","metadata":{}},{"cell_type":"code","source":"train = pd.read_parquet('../input/amex-data-integer-dtypes-parquet-format/train.parquet')\n\n# train/val by customer_ID first symbol\nfi = {'0':0,'1':0,'2':0,'3':0,'4':1,'5':1,'6':1,'7':1,'8':2,'9':2,'a':2,'b':2,'c':3,'d':3,'e':3,'f':3}\ntrain['fold'] = train['customer_ID'].apply(lambda t: fi[t[0]])","metadata":{"execution":{"iopub.status.busy":"2022-06-15T08:04:34.978317Z","iopub.execute_input":"2022-06-15T08:04:34.978707Z","iopub.status.idle":"2022-06-15T08:05:00.997497Z","shell.execute_reply.started":"2022-06-15T08:04:34.978665Z","shell.execute_reply":"2022-06-15T08:05:00.996544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For simplicity, we will take the customers with the N=13 timeframe available.","metadata":{}},{"cell_type":"code","source":"train = train.loc[train.groupby('customer_ID')['customer_ID'].transform('count')==13].reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2022-06-15T08:05:00.998868Z","iopub.execute_input":"2022-06-15T08:05:00.999419Z","iopub.status.idle":"2022-06-15T08:05:05.884648Z","shell.execute_reply.started":"2022-06-15T08:05:00.999372Z","shell.execute_reply":"2022-06-15T08:05:05.883736Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we will construct rank based on the date field. The lower the rank - the closer we are to the last observation date for each customer.\n\nImportant: We are not going to use `S_2` as a feature! The idea is that a model should be able to approximate `S_2` based on other features (if there are time-related features of course).","metadata":{}},{"cell_type":"code","source":"train['rank'] = train.groupby('customer_ID')['S_2'].rank(ascending=False).astype(int)","metadata":{"execution":{"iopub.status.busy":"2022-06-15T08:05:05.886903Z","iopub.execute_input":"2022-06-15T08:05:05.887372Z","iopub.status.idle":"2022-06-15T08:05:18.211556Z","shell.execute_reply.started":"2022-06-15T08:05:05.887329Z","shell.execute_reply":"2022-06-15T08:05:18.210606Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training the model","metadata":{}},{"cell_type":"markdown","source":"Let's build a very simple `rank:pairwise` xgboost model. We are going to use GPU as it speeds up training at least 30x! More on that in [NVIDIA's blog](https://developer.nvidia.com/blog/learning-to-rank-with-xgboost-and-gpu/#:~:text=XGBoost%20is%20a%20widely%20used,descent%20using%20an%20objective%20function).","metadata":{}},{"cell_type":"code","source":"cols = [x for x in train.columns if x not in ('customer_ID','fold','S_2','rank')]\n\ntr = train.loc[train.fold == 0].reset_index(drop=True)\ndtrain = xgb.DMatrix(tr[cols], label=tr['rank'], group = tr.groupby('customer_ID').size().to_frame('size')['size'].to_numpy())\n\nva = train.loc[train.fold == 1].reset_index(drop=True)\ndvalid = xgb.DMatrix(va[cols], label=va['rank'], group = va.groupby('customer_ID').size().to_frame('size')['size'].to_numpy())\n\ndel train\n\nmodel = xgb.train({'tree_method': 'gpu_hist', \n                   'objective':'rank:pairwise', \n                   'subsample':1, \n                   'colsample_bytree':1}, \n            dtrain=dtrain,\n            evals=[(dtrain,'train'),\n                   (dvalid,'valid')],\n            num_boost_round=100,    \n            verbose_eval=100,                  \n            maximize=True)  \n\nplot_importance(model,max_num_features=10, grid=False)","metadata":{"execution":{"iopub.status.busy":"2022-06-15T08:05:18.213126Z","iopub.execute_input":"2022-06-15T08:05:18.213538Z","iopub.status.idle":"2022-06-15T08:05:48.328817Z","shell.execute_reply.started":"2022-06-15T08:05:18.2135Z","shell.execute_reply":"2022-06-15T08:05:48.328007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that three `D` (as for **D**elinquency) features stand out. We will investigate this further. But first let's see how accurate our very simple model is in predicting the time rank.","metadata":{}},{"cell_type":"code","source":"va['pred'] = model.predict(dvalid)\nva['rank_pred'] = va.groupby('customer_ID')['pred'].rank().astype(int)\n\ncm = confusion_matrix(va['rank'],va['rank_pred'],normalize='true')\ncmp = ConfusionMatrixDisplay(cm)\nfig, ax = plt.subplots(figsize=(10,10))\ncmp.plot(ax=ax)","metadata":{"execution":{"iopub.status.busy":"2022-06-15T08:05:48.330384Z","iopub.execute_input":"2022-06-15T08:05:48.330765Z","iopub.status.idle":"2022-06-15T08:05:51.905807Z","shell.execute_reply.started":"2022-06-15T08:05:48.330727Z","shell.execute_reply":"2022-06-15T08:05:51.904953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It seems the model is able to predict the rank correctly 40-50% of all instances. Quite good. But this rejects the hypothesis that we have a `time_since_customer` feature in the dataset - the accuracy should be 100% if the hypothesis was true.\n\nLet's dig in a bit deeper!","metadata":{}},{"cell_type":"markdown","source":"# Understanding `D_59`\n\nFeature importance indicates that there is something special about this feature. Let's display a few customers:\n","metadata":{}},{"cell_type":"code","source":"pd.set_option('display.max_rows', 100)\nva[['customer_ID','rank','rank_pred','D_59']].head(13*3)","metadata":{"execution":{"iopub.status.busy":"2022-06-15T08:05:51.906833Z","iopub.execute_input":"2022-06-15T08:05:51.907201Z","iopub.status.idle":"2022-06-15T08:05:52.269006Z","shell.execute_reply.started":"2022-06-15T08:05:51.907163Z","shell.execute_reply":"2022-06-15T08:05:52.268162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Quite interesting! Some customers have `D_59` time correlation (and almost perfect time rank prediction). But there are customers who do not have this behavior as `D_59` acts more like a constant. \n\nWe do know, that `D_59` feature is a delinquency feature. We can assume that `D_59` represents how many credit installments/months a customer has been late to repay on his/her credit card. However, we would see clients with `D_59 = 0` (no delinquencies) - which are not present in the dataset. Maybe the dataset was somehow stratified to only include previously delinquent costumers? There is a lot of room to speculate here.\n\nThere are 2 clear segments (`D_59` correlates / does not correlate with time) - this opens the opportunity to analyze these segments individually.","metadata":{}},{"cell_type":"markdown","source":"# Client segmentation","metadata":{}},{"cell_type":"code","source":"#create arbitrary segments based on min/max of D_59\ngg = va.loc[va.D_59!=-1].groupby('customer_ID')['D_59'].agg(('max','min')).reset_index()\ns1 = gg.loc[gg['max']-gg['min']>9].reset_index(drop=True)\ns2 = gg.loc[gg['max']-gg['min']<=9].reset_index(drop=True)\n\ns1.shape, s2.shape","metadata":{"execution":{"iopub.status.busy":"2022-06-15T08:05:52.2704Z","iopub.execute_input":"2022-06-15T08:05:52.270754Z","iopub.status.idle":"2022-06-15T08:05:53.036191Z","shell.execute_reply.started":"2022-06-15T08:05:52.270725Z","shell.execute_reply":"2022-06-15T08:05:53.035227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's see how the model confusion matrix looks like for these 2 segments separatelly.","metadata":{}},{"cell_type":"markdown","source":"### segment with temporal `D_59`:","metadata":{}},{"cell_type":"code","source":"seg = va.loc[va.customer_ID.isin(s1.customer_ID)]\ncm = confusion_matrix(seg['rank'],seg['rank_pred'],normalize='true')\ncmp = ConfusionMatrixDisplay(cm)\nfig, ax = plt.subplots(figsize=(10,10))\ncmp.plot(ax=ax)","metadata":{"execution":{"iopub.status.busy":"2022-06-15T08:05:53.037835Z","iopub.execute_input":"2022-06-15T08:05:53.038254Z","iopub.status.idle":"2022-06-15T08:05:54.860904Z","shell.execute_reply.started":"2022-06-15T08:05:53.038199Z","shell.execute_reply":"2022-06-15T08:05:54.860041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### segment without temporal `D_59`:","metadata":{}},{"cell_type":"code","source":"seg = va.loc[va.customer_ID.isin(s2.customer_ID)]\ncm = confusion_matrix(seg['rank'],seg['rank_pred'],normalize='true')\ncmp = ConfusionMatrixDisplay(cm)\nfig, ax = plt.subplots(figsize=(10,10))\ncmp.plot(ax=ax)","metadata":{"execution":{"iopub.status.busy":"2022-06-15T08:05:54.863383Z","iopub.execute_input":"2022-06-15T08:05:54.864024Z","iopub.status.idle":"2022-06-15T08:05:56.838158Z","shell.execute_reply.started":"2022-06-15T08:05:54.863982Z","shell.execute_reply":"2022-06-15T08:05:56.8373Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As expected, our model accuracy is 2-3x better for the segment showing temporal `D_59` behavior.","metadata":{}},{"cell_type":"markdown","source":"# End notes\n\nThere is a potential to segment clients based on their delinquency status. However, it is not clear if doing so gives any advantage in overall default risk modelling.","metadata":{}}]}