{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\n\nIn this work I'm going to do some exploratory data analysis for recent [American Express - Default Prediction](https://www.kaggle.com/competitions/amex-default-prediction) competition. It's still in progress and I'll try to update it whenever it's possible for me, I hope you find it useful!\n\n## Some Simple Information:\n### About the Task:\n- We're going to predict the *probability* of a future payment default, for each customer_ID.\n\n### About the Target:\n- On competition page, hosts added this: `The target binary variable is calculated by observing 18 months performance window after the latest credit card statement, and if the customer does not pay due amount in 120 days after their latest statement date it is considered a default event.`\n- Also the negative class has been subsampled by %5 to deal with imbalanced data, but still should be pretty imbalanced.\n\n### About the Metric:\n- Competition has it's own custom metric which details can be found [here](https://www.kaggle.com/competitions/amex-default-prediction/overview/evaluation).\n- Because of the downsampling with negative class, the negative labels are given a weight of 20.\n- Python code for the custom competition metric can be found [here](https://www.kaggle.com/code/inversion/amex-competition-metric-python).\n\n### About the Data:\n- Both train and test sets are pretty big, loading and working with them in limited memory environments should be points to take into account.\n- Features are anonymized and normalized. Anonymized features fell into these general categories:\n    * D_* = Delinquency variables\n    * S_* = Spend variables\n    * P_* = Payment variables\n    * B_* = Balance variables\n    * R_* = Risk variables\n- Data consists mostly of continious features but there are some categorical features given by hosts:\n    - `['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']`\n    - Besides these there are datetime, ID and label columns.","metadata":{}},{"cell_type":"markdown","source":"# Loading Packages","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom tqdm import tqdm\nimport matplotlib.pyplot as plt\nimport matplotlib.gridspec as gridspec\nimport seaborn as sns\n\nfrom sklearn.metrics import roc_auc_score, roc_curve\nfrom sklearn.model_selection import train_test_split\nimport lightgbm as lgb\n\nimport gc\n\n\norange_black = [\n    '#fdc029', '#df861d', '#FF6347', '#aa3d01', '#a30e15', '#800000', '#171820'\n]\n\n# Setting plot styling.\nplt.style.use('ggplot')\n\nplt.rcParams['figure.figsize'] = (18, 14)\nplt.rcParams['figure.dpi'] = 300\nplt.rcParams[\"axes.grid\"] = True\nplt.rcParams[\"grid.color\"] = orange_black[0]\nplt.rcParams[\"grid.alpha\"] = 0.5\nplt.rcParams[\"grid.linestyle\"] = '--'\nplt.rcParams[\"font.family\"] = \"monospace\"\n\nplt.rcParams['axes.edgecolor'] = 'black'\nplt.rcParams['figure.frameon'] = False\nplt.rcParams['axes.spines.left'] = True\nplt.rcParams['axes.spines.bottom'] = True\nplt.rcParams['axes.spines.top'] = False\nplt.rcParams['axes.spines.right'] = False\nplt.rcParams['axes.linewidth'] = 1.0\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-06-01T17:43:29.245779Z","iopub.execute_input":"2022-06-01T17:43:29.246214Z","iopub.status.idle":"2022-06-01T17:43:30.892811Z","shell.execute_reply.started":"2022-06-01T17:43:29.246135Z","shell.execute_reply":"2022-06-01T17:43:30.892217Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Reading Data","metadata":{}},{"cell_type":"code","source":"train = pd.read_feather('../input/parquet-files-amexdefault-prediction/train_data.ftr')\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T17:43:30.894262Z","iopub.execute_input":"2022-06-01T17:43:30.895081Z","iopub.status.idle":"2022-06-01T17:43:48.521479Z","shell.execute_reply.started":"2022-06-01T17:43:30.895049Z","shell.execute_reply":"2022-06-01T17:43:48.520692Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test = pd.read_feather('../input/parquet-files-amexdefault-prediction/test_data.ftr')\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T17:43:48.522388Z","iopub.execute_input":"2022-06-01T17:43:48.522649Z","iopub.status.idle":"2022-06-01T17:44:18.108530Z","shell.execute_reply.started":"2022-06-01T17:43:48.522626Z","shell.execute_reply":"2022-06-01T17:44:18.107585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we check if there's any overlap.\n","metadata":{}},{"cell_type":"code","source":"print(f\"Number of overlapping ID's between Train and Test set: {train['customer_ID'].isin(test['customer_ID']).sum()}\")\n","metadata":{"execution":{"iopub.status.busy":"2022-06-01T17:44:18.110087Z","iopub.execute_input":"2022-06-01T17:44:18.110349Z","iopub.status.idle":"2022-06-01T17:44:19.036087Z","shell.execute_reply.started":"2022-06-01T17:44:18.110326Z","shell.execute_reply":"2022-06-01T17:44:19.035096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We're going to separate the general datatypes here, basically for categoricals, continious, date and targets. We also found some categorical-like columns so we add them into categoricals.","metadata":{}},{"cell_type":"code","source":"all_cols = train.columns.to_list()\ncat_cols = ['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']\n\nstr_cols = [\"customer_ID\"]\ndate_cols = [\"S_2\"]\ntarg_col = [\"target\"]\n\ncont_cols = [col for col in all_cols if col not in cat_cols + str_cols + date_cols + targ_col]\n\n# finding other categorical cols\nsec_cats = [col for col in cont_cols if train[col].nunique()<=10 and col not in cat_cols]\ncat_cols = cat_cols + sec_cats\n\nfor col in sec_cats:\n    cont_cols.remove(col)\n\n# setting categorical cols\ntrain[cat_cols] = train[cat_cols].astype('str').astype('category')\ntest[cat_cols] = test[cat_cols].astype('str').astype('category')","metadata":{"execution":{"iopub.status.busy":"2022-06-01T17:44:19.037132Z","iopub.execute_input":"2022-06-01T17:44:19.037399Z","iopub.status.idle":"2022-06-01T17:45:32.835675Z","shell.execute_reply.started":"2022-06-01T17:44:19.037377Z","shell.execute_reply":"2022-06-01T17:45:32.834408Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_test_nonmatch = []\nfor col in cat_cols:\n    if train[col].nunique(dropna=False) != test[col].nunique(dropna=False):\n        train_test_nonmatch.append(col)\n    else:\n        continue","metadata":{"execution":{"iopub.status.busy":"2022-06-01T17:45:32.837006Z","iopub.execute_input":"2022-06-01T17:45:32.837234Z","iopub.status.idle":"2022-06-01T17:45:33.753407Z","shell.execute_reply.started":"2022-06-01T17:45:32.837212Z","shell.execute_reply":"2022-06-01T17:45:33.752450Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def show_nomatch(train, test, cols):\n    fig, axes = plt.subplots(len(cols),2, sharey=True)\n    axes = axes.flatten()\n    \n    for j in zip(cols, axes[1::2], axes[::2]):\n        order1 = train[j[0]].value_counts().index\n        order2 = test[j[0]].value_counts().index\n\n        sns.countplot(train[j[0]],ax=j[2], label='train', palette=orange_black, order=order1)\n        sns.countplot(test[j[0]],ax=j[1], label='test', palette=orange_black, order=order2)\n        j[1].set_yticklabels([])\n        j[1].set_ylabel('')\n        j[2].set_ylabel('')\n        \n    col_tit = ['Train', 'Test']\n        \n    for i, ax in enumerate(axes[:2]):\n        ax.set_title(\"{}\".format(col_tit[i]), fontweight='bold', fontsize=12)\n    plt.suptitle('Unique Categorical Values by Train/Test Splits', fontweight='bold', fontsize=16)\n    plt.tight_layout()\n        \nshow_nomatch(train, test, train_test_nonmatch)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-06-01T17:45:33.755170Z","iopub.execute_input":"2022-06-01T17:45:33.755980Z","iopub.status.idle":"2022-06-01T17:45:37.393556Z","shell.execute_reply.started":"2022-06-01T17:45:33.755938Z","shell.execute_reply":"2022-06-01T17:45:37.392849Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that there are some values in train set that doesn't represented in test set. We can drop them.","metadata":{}},{"cell_type":"code","source":"shape_old = train.shape[0]\nfor col in train_test_nonmatch:\n    drop = set(train[col].value_counts().index) - set(test[col].value_counts().index)\n    for i in list(drop):\n        train = train[train[col]!=i]\n        train[col] = train[col].cat.remove_unused_categories()\n        \ntrain.reset_index(drop=True, inplace=True)\nshape_new = train.shape[0]\nprint(f\"Train Rows dropped: {shape_old - shape_new}\")\ndel shape_old, shape_new\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T17:45:37.395131Z","iopub.execute_input":"2022-06-01T17:45:37.395667Z","iopub.status.idle":"2022-06-01T17:46:19.617597Z","shell.execute_reply.started":"2022-06-01T17:45:37.395629Z","shell.execute_reply":"2022-06-01T17:46:19.616498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Missing Values","metadata":{}},{"cell_type":"code","source":"miss = train.isna().sum().sort_values(ascending=False).reset_index(drop=False).rename({0:'missing_count'}, axis=1)\nmiss['miss_ratio'] = (miss['missing_count'] / train.shape[0]) * 100\n\nsns.barplot(data=miss[:25], x= 'miss_ratio', y='index', palette='YlOrBr_r', linewidth=0.7, edgecolor=\".2\")\nplt.title('Missing Value Ratios Per Column')\nplt.ylabel('Column')\nplt.xlabel('Percent of Missing Values')\ndel miss\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2022-06-01T17:46:19.620248Z","iopub.execute_input":"2022-06-01T17:46:19.620536Z","iopub.status.idle":"2022-06-01T17:46:24.357157Z","shell.execute_reply.started":"2022-06-01T17:46:19.620515Z","shell.execute_reply":"2022-06-01T17:46:24.356194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"miss = train.isna().sum(axis=1).reset_index(drop=False).rename({0:'missing_count'}, axis=1)\nmiss['miss_ratio'] = (miss['missing_count'] / (train.shape[1]-1)) * 100\n\nsns.histplot(miss.miss_ratio, stat='percent', bins=50, linewidth=0.7, edgecolor=\".2\", color=orange_black[0])\nplt.title(\"Distribution of Missing Percentage by Row\")\nplt.xlabel('Missing Value Ratio per Row')\nplt.ylabel(\"Frequency\")\ndel miss\nplt.show()\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T17:46:24.358245Z","iopub.execute_input":"2022-06-01T17:46:24.358525Z","iopub.status.idle":"2022-06-01T17:46:40.237785Z","shell.execute_reply.started":"2022-06-01T17:46:24.358497Z","shell.execute_reply":"2022-06-01T17:46:40.237035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Target Column","metadata":{}},{"cell_type":"code","source":"train_labels = pd.read_csv('../input/amex-default-prediction/train_labels.csv')\ntrain_labels['target'] = train_labels['target'].astype('uint8')\ntrain = train.merge(train_labels, how='left', on='customer_ID')\n\ndel train_labels\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T17:46:40.238897Z","iopub.execute_input":"2022-06-01T17:46:40.239138Z","iopub.status.idle":"2022-06-01T17:49:20.707976Z","shell.execute_reply.started":"2022-06-01T17:46:40.239115Z","shell.execute_reply":"2022-06-01T17:49:20.706918Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"g=sns.countplot(train.target, palette='autumn',  linewidth=0.7, edgecolor=\".2\")\n\n# Adding percentages\n\ntotal = float(len(train.target))\n\nfor p in g.patches:\n    height = p.get_height()\n    g.text(p.get_x() + p.get_width() / 2.,\n            height + height/100,\n            '{:1.2f}%'.format((height / total) * 100),\n            ha='center',\n           fontsize=8,\n          bbox=dict(boxstyle='round',facecolor='black', alpha=0.5))\n\nplt.ylabel('Count')    \nplt.title('Target Distribution', weight='bold')\nplt.tight_layout()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T17:49:20.709349Z","iopub.execute_input":"2022-06-01T17:49:20.709932Z","iopub.status.idle":"2022-06-01T17:49:22.587426Z","shell.execute_reply.started":"2022-06-01T17:49:20.709891Z","shell.execute_reply":"2022-06-01T17:49:22.586922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"correlations = train.corrwith(train['target']).iloc[:-1].to_frame()\ncorrelations['Abs Corr'] = correlations[0].abs()\nsorted_correlations = correlations.sort_values('Abs Corr', ascending=False)['Abs Corr']\nfig, ax = plt.subplots(figsize=(6,8))\nsns.heatmap(sorted_correlations.iloc[1:].to_frame()[sorted_correlations>=.25], cmap='inferno', annot=True, vmin=-1, vmax=1, ax=ax, cbar=False)\n\nplt.ylabel('Feature')\nplt.title('Feature Correlations With Target')\ndel correlations, sorted_correlations\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T17:49:22.588494Z","iopub.execute_input":"2022-06-01T17:49:22.588767Z","iopub.status.idle":"2022-06-01T17:49:39.576338Z","shell.execute_reply.started":"2022-06-01T17:49:22.588739Z","shell.execute_reply":"2022-06-01T17:49:39.575406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Parsing Date","metadata":{}},{"cell_type":"code","source":"train['S_2']= pd.to_datetime(train['S_2'], format='%Y-%m-%d')\ntest['S_2']= pd.to_datetime(test['S_2'], format='%Y-%m-%d')\ntrain['day_name'] = train['S_2'].dt.day_name().astype('category')","metadata":{"execution":{"iopub.status.busy":"2022-06-01T17:49:39.577553Z","iopub.execute_input":"2022-06-01T17:49:39.577841Z","iopub.status.idle":"2022-06-01T17:49:46.105629Z","shell.execute_reply.started":"2022-06-01T17:49:39.577816Z","shell.execute_reply":"2022-06-01T17:49:46.104920Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.countplot(data=train, x='day_name', hue='target', palette=orange_black[1::5])\nplt.title(\"Weekday of the Dataset Entries by Target\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T17:49:46.106938Z","iopub.execute_input":"2022-06-01T17:49:46.107401Z","iopub.status.idle":"2022-06-01T17:49:47.544096Z","shell.execute_reply.started":"2022-06-01T17:49:46.107363Z","shell.execute_reply":"2022-06-01T17:49:47.543200Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Categorical Data","metadata":{}},{"cell_type":"code","source":"def categorical_dist(df, t_df, cols, rows, columns):\n    fig, axes = plt.subplots(rows, columns, figsize=(30, 25), constrained_layout=True)\n    axes = axes.flatten()\n    for j in zip(cols, axes[1::2], axes[::2]):\n        order = df[j[0]].value_counts().index\n        sns.countplot(data=df, x=str(j[0]), ax=j[2], order=order, hue='target', palette=orange_black[2::3])\n        sns.countplot(data=t_df, x=str(j[0]),ax=j[1], order=order, label='Test', color=orange_black[0], alpha=0.8)\n        sns.countplot(data=df, x=str(j[0]),ax=j[1], order=order, label='Train', color=orange_black[-1], alpha=0.8)\n        j[1].set_yticklabels([])\n        j[1].set_ylabel('')\n        j[2].set_ylabel('')\n        j[1].legend(title='Dataset')\n    col_tit = ['Train by Target', 'Train vs Test']    \n    for i, ax in enumerate(axes[:2]):\n        ax.set_title(\"{}\".format(col_tit[i]), fontweight='bold', fontsize=18)\n    plt.suptitle('Unique Categorical Values by Train/Test Splits', fontweight='bold', fontsize=16)\n\n    plt.tight_layout()\n    \n    \ncategorical_dist(train, test, cat_cols, 6, 2)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-06-01T17:49:47.545480Z","iopub.execute_input":"2022-06-01T17:49:47.545811Z","iopub.status.idle":"2022-06-01T17:49:59.016790Z","shell.execute_reply.started":"2022-06-01T17:49:47.545770Z","shell.execute_reply":"2022-06-01T17:49:59.015832Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"corr = train[cont_cols].fillna(0).sample(frac=0.1, random_state=42).corr()\nsns.clustermap(corr, metric=\"correlation\", figsize=(20, 20), dendrogram_ratio=(.1, .2), cmap=\"coolwarm\")\nplt.suptitle('Correlations Between Features', fontsize=24, weight='bold')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T17:49:59.017795Z","iopub.execute_input":"2022-06-01T17:49:59.018094Z","iopub.status.idle":"2022-06-01T17:50:44.993126Z","shell.execute_reply.started":"2022-06-01T17:49:59.018068Z","shell.execute_reply":"2022-06-01T17:50:44.992042Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"corr = corr.abs()\n\ncorrs = corr.unstack()\npair = corrs.sort_values(ascending=False)\npair = pair.reset_index(name='correlation').rename(columns={'level_0': 'feature_a', 'level_1': 'feature_b', 0: 'correlation'})\npair = pair[pair['feature_a'] != pair['feature_b']].iloc[::2,:]\npair['Features'] = pair.feature_a +' / '+ pair.feature_b","metadata":{"execution":{"iopub.status.busy":"2022-06-01T17:50:44.994588Z","iopub.execute_input":"2022-06-01T17:50:44.995109Z","iopub.status.idle":"2022-06-01T17:50:45.025887Z","shell.execute_reply.started":"2022-06-01T17:50:44.995077Z","shell.execute_reply":"2022-06-01T17:50:45.025252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axes = plt.subplots(figsize=(25,20))\nsns.barplot(data=pair[:50], x= 'correlation', y='Features', palette='YlOrBr_r', linewidth=0.7, edgecolor=\".2\")\nplt.title('Highest Correlation Pairs Between Features')\nplt.ylabel('Pairs')\nplt.xlabel('Correlation')\n\nplt.show()\n\ndel corr, pair\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T17:50:45.027005Z","iopub.execute_input":"2022-06-01T17:50:45.027935Z","iopub.status.idle":"2022-06-01T17:50:47.777405Z","shell.execute_reply.started":"2022-06-01T17:50:45.027904Z","shell.execute_reply":"2022-06-01T17:50:47.776486Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's get to the continious features next, but there are too many of them. It'd be better if we group them as hosts did, if you recall:\n\n- D_* = Delinquency variables\n- S_* = Spend variables\n- P_* = Payment variables\n- B_* = Balance variables\n- R_* = Risk variables\n\n","metadata":{}},{"cell_type":"code","source":"cont_d = [col for col in train.columns.tolist() if col.startswith('D') and col not in cat_cols]\ncont_s = [col for col in train.columns.tolist() if col.startswith('S') and col not in cat_cols]\ncont_p = [col for col in train.columns.tolist() if col.startswith('P') and col not in cat_cols]\ncont_b = [col for col in train.columns.tolist() if col.startswith('B') and col not in cat_cols]\ncont_r = [col for col in train.columns.tolist() if col.startswith('R') and col not in cat_cols]\n\n\ncat_d = [col for col in train.columns.tolist() if col.startswith('D') and col in cat_cols]\ncat_s = [col for col in train.columns.tolist() if col.startswith('S') and col in cat_cols]\ncat_p = [col for col in train.columns.tolist() if col.startswith('P') and col in cat_cols]\ncat_b = [col for col in train.columns.tolist() if col.startswith('B') and col in cat_cols]\ncat_r = [col for col in train.columns.tolist() if col.startswith('R') and col in cat_cols]\n\nspec_cont = [cont_d,cont_s,cont_p,cont_b,cont_r]\nspec_cat = [cat_d,cat_s,cat_p,cat_b,cat_r]\nspec_cat = [col for col in spec_cat if col != []]","metadata":{"execution":{"iopub.status.busy":"2022-06-01T17:50:47.778251Z","iopub.execute_input":"2022-06-01T17:50:47.778464Z","iopub.status.idle":"2022-06-01T17:50:47.790114Z","shell.execute_reply.started":"2022-06-01T17:50:47.778443Z","shell.execute_reply":"2022-06-01T17:50:47.789058Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Delinquency Variables","metadata":{}},{"cell_type":"code","source":"class ContiniousDist():\n    def __init__(self, train, test_df):\n        self.train = train\n        self.test = test_df\n        \n        self.df_0 = self.train[self.train['target']==0]\n        self.df_1 = self.train[self.train['target']==1]\n        \n    def histplot(self, cols, title:str='', figsize:tuple=(45, 90), row_factor:int=5, n_cols:int=5):\n        fig, axes = plt.subplots(len(cols)//row_factor, n_cols, figsize=figsize, constrained_layout=True)\n        axes = axes.flatten()\n        for i, col in tqdm(enumerate(cols)):\n            axes[i].hist(self.df_0[col], bins=100, alpha=0.5, color=orange_black[1], density=True,  linewidth=0.2, edgecolor=\".2\", label='Target: 0', histtype='stepfilled')\n            axes[i].hist(self.df_1[col], bins=100, alpha=0.6, color=orange_black[4], density=True,  linewidth=0.2, edgecolor=\".2\", label='Target: 1', histtype='stepfilled')\n            axes[i].hist(self.test[col], bins=100, alpha=0.75, color=orange_black[-1], density=True,  linewidth=0.2, edgecolor=\".2\", label='Test', histtype='stepfilled')\n            axes[i].set_title(col)\n            axes[i].legend()\n        plt.suptitle(title, fontsize=25)\n        plt.show()\n        \n    def ecdf(self, cols, title:str='', figsize:tuple=(30, 60), row_factor:int=5, n_cols:int=5, test=False):\n        fig, axes = plt.subplots(len(cols)//row_factor, n_cols, figsize=figsize, constrained_layout=True)\n        axes = axes.flatten()\n        for i, col in tqdm(enumerate(cols)):\n            if test==False:\n                sns.ecdfplot(self.df_0[col].dropna(), color=orange_black[1], ax=axes[i], label='Target: 0')\n                sns.ecdfplot(self.df_1[col].dropna(), color=orange_black[-1], ax=axes[i], label='Target: 1')\n            else:\n                sns.ecdfplot(self.train[col].dropna(), color=orange_black[1], ax=axes[i], label='Train')\n                sns.ecdfplot(self.test[col].dropna(), color=orange_black[-1], ax=axes[i], label='Test')\n\n            axes[i].set_title(col)\n            axes[i].legend()\n        plt.suptitle(title)\n        plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T17:50:47.791269Z","iopub.execute_input":"2022-06-01T17:50:47.791688Z","iopub.status.idle":"2022-06-01T17:50:47.806361Z","shell.execute_reply.started":"2022-06-01T17:50:47.791660Z","shell.execute_reply":"2022-06-01T17:50:47.805657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"hist = ContiniousDist(train, test)","metadata":{"execution":{"iopub.status.busy":"2022-06-01T17:50:47.807415Z","iopub.execute_input":"2022-06-01T17:50:47.807671Z","iopub.status.idle":"2022-06-01T17:50:54.649462Z","shell.execute_reply.started":"2022-06-01T17:50:47.807648Z","shell.execute_reply":"2022-06-01T17:50:54.648636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"hist.histplot(cont_d[:80], title=\"Delinquency Variable Distributions\")\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T17:50:54.651169Z","iopub.execute_input":"2022-06-01T17:50:54.651824Z","iopub.status.idle":"2022-06-01T17:57:18.446778Z","shell.execute_reply.started":"2022-06-01T17:50:54.651786Z","shell.execute_reply":"2022-06-01T17:57:18.445646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The distribution between train, targets and tests seems aren't that distinctive but I can see some differences in some features (like d_47,d_52, d_59, d_74 etc.), these features should be helpful for models or feature engineering. ","metadata":{}},{"cell_type":"code","source":"hist.ecdf(['D_47','D_52', 'D_59', 'D_74'], title=\"Empirical Cumulative Distributions for Interesting Deliquency Variables\", figsize=(12, 12), row_factor=2, n_cols=2)","metadata":{"execution":{"iopub.status.busy":"2022-06-01T17:57:18.451214Z","iopub.execute_input":"2022-06-01T17:57:18.451885Z","iopub.status.idle":"2022-06-01T17:58:31.208677Z","shell.execute_reply.started":"2022-06-01T17:57:18.451842Z","shell.execute_reply":"2022-06-01T17:58:31.207412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can clearly see there are distribution differences between targets for these features which is good.","metadata":{}},{"cell_type":"markdown","source":"# Spend Variables","metadata":{}},{"cell_type":"code","source":"hist.histplot(spec_cont[1][:21], title=\"Spend Variable Distributions\", row_factor=3, n_cols=3)\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T17:58:31.210061Z","iopub.execute_input":"2022-06-01T17:58:31.210361Z","iopub.status.idle":"2022-06-01T18:00:35.699660Z","shell.execute_reply.started":"2022-06-01T17:58:31.210331Z","shell.execute_reply":"2022-06-01T18:00:35.698924Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see some odd shapes here too. Easiest one to see is time feature, notice that test set covers future in timeline and wider interval. On S_11 you can notice a shift between train and test splits, it could be related to time, you can also notice in first look that S_25 has different target distribution for 1's than 0's. Some feature distributions are clusteren in tightly knitted bins around discrete value, this might indicate random noise injection by hosts like @raddar pointed out.","metadata":{}},{"cell_type":"code","source":"hist.ecdf(['S_7','S_11'], title=\"Empirical Cumulative Distributions for Interesting Spend Variables\", figsize=(8, 4), row_factor=2, n_cols=2, test=True)\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T18:00:35.701026Z","iopub.execute_input":"2022-06-01T18:00:35.701516Z","iopub.status.idle":"2022-06-01T18:02:17.735006Z","shell.execute_reply.started":"2022-06-01T18:00:35.701480Z","shell.execute_reply":"2022-06-01T18:02:17.734239Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Just got a random variable to show you what similar distribution between two sets, at S_7, meanwhile you can see the difference in S_11 better above...","metadata":{}},{"cell_type":"markdown","source":"# Payment Variables","metadata":{}},{"cell_type":"code","source":"hist.histplot(cont_p, title=\"Payment Variable Distributions\", figsize=(12, 4), row_factor=3, n_cols=3)","metadata":{"execution":{"iopub.status.busy":"2022-06-01T18:02:17.735916Z","iopub.execute_input":"2022-06-01T18:02:17.736158Z","iopub.status.idle":"2022-06-01T18:02:35.163480Z","shell.execute_reply.started":"2022-06-01T18:02:17.736134Z","shell.execute_reply":"2022-06-01T18:02:35.162527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There isn't big difference between train and test for payment variables but we can see apparent distinction in P_2 between targets.","metadata":{}},{"cell_type":"markdown","source":"# Balance Variables","metadata":{}},{"cell_type":"code","source":"hist.histplot(cont_b[:36], title=\"Balance Variable Distributions\", row_factor=4, n_cols=4)\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T18:02:35.165065Z","iopub.execute_input":"2022-06-01T18:02:35.165546Z","iopub.status.idle":"2022-06-01T18:05:45.648807Z","shell.execute_reply.started":"2022-06-01T18:02:35.165468Z","shell.execute_reply":"2022-06-01T18:05:45.647923Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"hist.ecdf(['B_2','B_16'], title=\"Empirical Cumulative Distributions for Interesting Spend Variables\", figsize=(8, 4), row_factor=2, n_cols=2, test=False)\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T18:05:45.653553Z","iopub.execute_input":"2022-06-01T18:05:45.654015Z","iopub.status.idle":"2022-06-01T18:06:28.998637Z","shell.execute_reply.started":"2022-06-01T18:05:45.653981Z","shell.execute_reply":"2022-06-01T18:06:28.997906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"hist.ecdf(['B_2','B_16'], title=\"Empirical Cumulative Distributions for Interesting Balance Variables\", figsize=(8, 4), row_factor=2, n_cols=2, test=True)\ngc.collect()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-06-01T18:06:28.999598Z","iopub.execute_input":"2022-06-01T18:06:28.999907Z","iopub.status.idle":"2022-06-01T18:08:41.385003Z","shell.execute_reply.started":"2022-06-01T18:06:28.999862Z","shell.execute_reply":"2022-06-01T18:08:41.384215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Again we can see decisive differences between target distributions, test set seems similar though...","metadata":{}},{"cell_type":"markdown","source":"# Risk Variables","metadata":{}},{"cell_type":"code","source":"hist.histplot(cont_r, title=\"Risk Variable Distributions\", row_factor=4, n_cols=4, figsize=(45, 60))\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T18:08:41.386326Z","iopub.execute_input":"2022-06-01T18:08:41.386894Z","iopub.status.idle":"2022-06-01T18:10:36.520367Z","shell.execute_reply.started":"2022-06-01T18:08:41.386838Z","shell.execute_reply":"2022-06-01T18:10:36.519793Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"hist.ecdf(['R_4','R_27'], title=\"Empirical Cumulative Distributions for Interesting Risk Variables\", figsize=(8, 4), row_factor=2, n_cols=2, test=False)\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T18:10:36.521225Z","iopub.execute_input":"2022-06-01T18:10:36.521495Z","iopub.status.idle":"2022-06-01T18:11:18.736594Z","shell.execute_reply.started":"2022-06-01T18:10:36.521468Z","shell.execute_reply":"2022-06-01T18:11:18.735713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del hist\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T18:11:18.737690Z","iopub.execute_input":"2022-06-01T18:11:18.738298Z","iopub.status.idle":"2022-06-01T18:11:18.999123Z","shell.execute_reply.started":"2022-06-01T18:11:18.738265Z","shell.execute_reply":"2022-06-01T18:11:18.998049Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Adversarial Validation ","metadata":{}},{"cell_type":"code","source":"class Adversarial:\n    def __init__(self):\n        self.params =  {\n            'objective': 'binary',  \n            'boosting_type': 'gbdt',\n            'n_jobs': -1,\n            'metric' :   'auc',\n            'max_bin' : 128,\n            'verbose': -1\n        }\n    \n    def _plot_roc_feat(self, y_trues, y_preds, labels, cols, clf, idxs:set=(), x_max:float=1.0):\n        fig = plt.figure(constrained_layout=True, figsize=(12, 12))\n        grid = gridspec.GridSpec(ncols=4, nrows=2, figure=fig)\n        ax1 = fig.add_subplot(grid[0, :2])    \n        for i, y_pred in enumerate(y_preds):\n            y_true = y_trues[i]\n            fpr, tpr, thresholds = roc_curve(y_true, y_pred)\n            auc = roc_auc_score(y_true, y_pred)\n            ax1.plot(fpr, tpr, label='%s; AUC=%.3f' % (labels[i], auc), marker='o', markersize=1)\n        ax1.legend()\n        ax1.grid()\n        ax1.plot(np.linspace(0, 1, 20), np.linspace(0, 1, 20), linestyle='--')\n        ax1.set_title('Adversarial ROC curve')\n        ax1.set_xlabel('False Positive Rate')\n        ax1.set_xlim([-0.01, x_max])\n        _ = ax1.set_ylabel('True Positive Rate')\n        \n        ax2 = fig.add_subplot(grid[0, 2:])\n\n        feature_imp = pd.DataFrame(sorted(zip(clf.feature_importance(),cols)), columns=['Value','Feature'])\n        sns.barplot(x=\"Value\", y=\"Feature\", data=feature_imp.sort_values(by=\"Value\", ascending=False).iloc[:25,:], ax=ax2, palette='YlOrBr_r', linewidth=0.7, edgecolor=\".2\")\n        ax2.set_title('Most Distinctive Features Between Train and Test')\n        \n        \n        ax3 = fig.add_subplot(grid[1, :])\n        ax3.set_title('A')\n        ax3.hist([train.loc[idxs[0], 'S_2'], test.loc[idxs[1], 'S_2']], bins=pd.date_range(\"2017-03-01\", \"2019-11-01\", freq=\"MS\"), histtype=\"barstacked\",\n                label=['Training', 'Test'], color=[orange_black[0], orange_black[-1]])\n        ax3.set_xticks(pd.date_range(\"2017-03-01\", \"2019-11-01\", freq=\"QS\"))\n        ax3.legend()\n        ax3.set_title('Train Test Sample Intervals')\n        plt.show()\n\n    def validate(self, train, test, method:str='full'):\n        train['set'] = 0\n        test['set'] = 1\n\n        train['set'] = train['set'].astype('uint8')\n        test['set'] = test['set'].astype('uint8')\n\n        if method=='full':\n            a = train.drop(['customer_ID', 'S_2', 'day_name', 'target'], axis=1).sample(frac=0.1, random_state=42)\n            b = test.drop(['customer_ID', 'S_2'], axis=1).sample(frac=0.1, random_state=42)\n            full = pd.concat((a, b), axis=0)\n            a_idx = a.index\n            b_idx = b.index\n\n        elif method=='far_gap':\n            a = train[train.S_2<train.S_2.quantile(0.05)].drop(['customer_ID', 'S_2', 'day_name', 'target'], axis=1)\n            b = test[test.S_2>test.S_2.quantile(0.95)].drop(['customer_ID', 'S_2'], axis=1)\n            full = pd.concat((a, b), axis=0)\n            a_idx = a.index\n            b_idx = b.index\n\n        elif method=='close_gap':\n            a = train[train.S_2>train.S_2.quantile(0.95)].drop(['customer_ID', 'S_2', 'day_name', 'target'], axis=1)\n            b = test[test.S_2<test.S_2.quantile(0.05)].drop(['customer_ID', 'S_2'], axis=1)\n            full = pd.concat((a, b), axis=0)\n            a_idx = a.index\n            b_idx = b.index\n            \n        del a, b\n\n        X_train, X_test, y_train, y_test = train_test_split(full.iloc[:,:-1], full['set'], stratify=full['set'], shuffle=True, random_state=42)\n\n        del full\n        gc.collect()\n\n        train_data = lgb.Dataset(data=X_train, label=y_train, params={'verbose': -1},\n                             categorical_feature=cat_cols,\n                     free_raw_data=False\n                    )\n        val_data = lgb.Dataset(data=X_test, label=y_test, params={'verbose': -1},\n                             categorical_feature=cat_cols,\n                     free_raw_data=False\n                          )\n        clf = lgb.train(\n            params=self.params,\n            train_set=train_data,\n            valid_sets=[train_data,val_data],\n            verbose_eval=-1,\n            )\n\n        y_pred = clf.predict(X_test)\n        \n        self._plot_roc_feat(\n            y_trues=[y_test],\n            y_preds=[y_pred],\n            labels=[f'{method} data'],\n            cols=X_train.columns.tolist(),\n            clf = clf,\n            idxs = (a_idx, b_idx)\n        )\n\n        del X_train, X_test, y_train, y_test, train_data, val_data\n        gc.collect()\n        \nadv = Adversarial()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T18:11:19.024406Z","iopub.execute_input":"2022-06-01T18:11:19.024625Z","iopub.status.idle":"2022-06-01T18:11:19.050068Z","shell.execute_reply.started":"2022-06-01T18:11:19.024604Z","shell.execute_reply":"2022-06-01T18:11:19.049419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"adv.validate(train, test, method='full')","metadata":{"execution":{"iopub.status.busy":"2022-06-01T18:11:19.051344Z","iopub.execute_input":"2022-06-01T18:11:19.052223Z","iopub.status.idle":"2022-06-01T18:13:29.258014Z","shell.execute_reply.started":"2022-06-01T18:11:19.052192Z","shell.execute_reply":"2022-06-01T18:13:29.257197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Isn't looking good, our model doing perfectly on task where it needs to predict if given sample coming from train or test sets. We can see the features which is important for the model while deciding train test predictions. In top features I can see some familiar names we have spotted by looking distributions. I suspect these features are related to time variable directly or indirectly, but anyways we cannot assume train and test sets are sampled randomly for now. Let's take a closer look...","metadata":{}},{"cell_type":"code","source":"adv.validate(train, test, method='close_gap')","metadata":{"execution":{"iopub.status.busy":"2022-06-01T18:13:29.259057Z","iopub.execute_input":"2022-06-01T18:13:29.259273Z","iopub.status.idle":"2022-06-01T18:14:32.399079Z","shell.execute_reply.started":"2022-06-01T18:13:29.259252Z","shell.execute_reply":"2022-06-01T18:14:32.398357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So when we close down the timewise gap in our adversarial samples we can see a different story: The AUC score decreases, so the model having little bit more hard time to classify train and test sets. But it still has a great score... You can ntoice some features are getting more importance as the gap tightens, so these features are not likely to related to time shifts.","metadata":{}},{"cell_type":"code","source":"adv.validate(train, test, method='far_gap')","metadata":{"execution":{"iopub.status.busy":"2022-06-01T18:14:32.400244Z","iopub.execute_input":"2022-06-01T18:14:32.400571Z","iopub.status.idle":"2022-06-01T18:15:30.622803Z","shell.execute_reply.started":"2022-06-01T18:14:32.400534Z","shell.execute_reply":"2022-06-01T18:15:30.621888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And when we take earlier parts of the training data and the latter parts of the test data, our adversarial model gets a perfect score! While D_59 still keeping the most important feature status we can see some features like S_27, D_121 gets huge boost, so we can assume these features are more time related than others.","metadata":{}},{"cell_type":"code","source":"del adv\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T18:15:30.624210Z","iopub.execute_input":"2022-06-01T18:15:30.624564Z","iopub.status.idle":"2022-06-01T18:15:30.742124Z","shell.execute_reply.started":"2022-06-01T18:15:30.624526Z","shell.execute_reply":"2022-06-01T18:15:30.741187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Work in Progress\n\nThese are my first insights about this vast data. I hope I'll be doing more EDA for rest of the feautres and do some modelling on them. I hope you find it useful, best of luck everyone :)","metadata":{}}]}