{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Customer Time Series EDA\nKaggle's \"American Express - Default Prediction\" Competition provides us with time series data from credit card customers. For each customer we have 188 data points for each of up to 13 points in time. Therefore most customers have `2444 = 188 * 13` data points. For each customer, we can plot the 188 line plots that show how each variable (for one specific customer) changes over time.\n\nFurthermore we can plot time series for customers who default on their credit payment. This is `target = 1` and is shown in blue below. And we can plot customers who do not default. This is `target = 0` and shown in orange below. By looking at different customers and different variables, we can gain intuition about which variables predict default.\n\nIn this notebook we chose 33 variables that look interesting. For each variable, we plot 32 default customers' time series, and 32 non-default customers' time series. To the right of the line plots, we plot a histogram also. Discussion about this EDA is [here][4].\n\nIf you wish to view the other 155 features, then copy and edit this notebook and display whichever variables you want. From the competition description [here][1], we know there are 5 types of variables. And there are 188 variables in total.\n\n* D_* = 96 Delinquency variables\n* S_* = 21 Spend variables\n* P_* = 3 Payment variables\n* B_* = 40 Balance variables\n* R_* = 28 Risk variables\n\nWhen you are ready to build a time series model like an RNN or Transformer that takes advantage of what you see in these plots, check out my TF GRU starter notebook [here][2] and check out my TF Transformer starter notebook [here][3]\n\n[1]: https://www.kaggle.com/competitions/amex-default-prediction/data\n[2]: https://www.kaggle.com/cdeotte/tensorflow-gru-starter-0-787\n[3]: https://www.kaggle.com/code/cdeotte/tensorflow-transformer-0-790\n[4]: https://www.kaggle.com/competitions/amex-default-prediction/discussion/327761","metadata":{}},{"cell_type":"markdown","source":"# Load Train Data","metadata":{}},{"cell_type":"code","source":"# LOAD LIBRARIES\nimport pandas as pd, numpy as np\nimport matplotlib.pyplot as plt\nfrom matplotlib import gridspec\n\n# LOAD TRAIN DATA AND MERGE TARGETS ONTO FEATURES\ndf = pd.read_csv('../input/amex-default-prediction/train_data.csv', nrows=100_000)\ndf.S_2 = pd.to_datetime(df.S_2)\ndf2 = pd.read_csv('../input/amex-default-prediction/train_labels.csv')\ndf = df.merge(df2,on='customer_ID',how='left')","metadata":{"execution":{"iopub.status.busy":"2022-06-01T15:23:40.116663Z","iopub.execute_input":"2022-06-01T15:23:40.117550Z","iopub.status.idle":"2022-06-01T15:23:48.451099Z","shell.execute_reply.started":"2022-06-01T15:23:40.117456Z","shell.execute_reply":"2022-06-01T15:23:48.450035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_time_series(prefix='D', cols=None, display_ct=32):\n    \n    # DETERMINE WHICH COLUMNS TO PLOT\n    if cols is not None and len(cols)==0: cols = None\n    if cols is None:\n        COLS = df.columns[2:-1]\n        COLS = np.sort( [int(x[2:]) for x in COLS if x[0]==prefix] )\n        COLS = [f'{prefix}_{x}' for x in COLS]\n        print('#'*25)\n        print(f'Plotting all {len(COLS)} columns with prefix {prefix}')\n        print('#'*25)\n    else:\n        COLS = [f'{prefix}_{x}' for x in cols]\n        print('#'*25)\n        print(f'Plotting {len(COLS)} columns with prefix {prefix}')\n        print('#'*25)\n\n    # ITERATE COLUMNS\n    for c in COLS:\n\n        # CONVERT DATAFRAME INTO SERIES WITH COLUMN\n        tmp = df[['customer_ID','S_2',c,'target']].copy()\n        tmp2 = tmp.groupby(['customer_ID','target'])[['S_2',c]].agg(list).reset_index()\n        tmp3 = tmp2.loc[tmp2.target==1]\n        tmp4 = tmp2.loc[tmp2.target==0]\n\n        # FORMAT PLOT\n        spec = gridspec.GridSpec(ncols=2, nrows=1,\n                             width_ratios=[3, 1], wspace=0.1,\n                             hspace=0.5, height_ratios=[1])\n        fig = plt.figure(figsize=(20,10))\n        ax0 = fig.add_subplot(spec[0])\n\n        # PLOT 32 DEFAULT CUSTOMERS AND 32 NON-DEFAULT CUSTOMERS\n        t0 = []; t1 = []\n        for k in range(display_ct):\n            try:\n                # PLOT DEFAULTING CUSTOMERS\n                row = tmp3.iloc[k]\n                ax0.plot(row.S_2,row[c],'-o',color='blue')\n                t1 += row[c]\n                # PLOT NON-DEFAULT CUSTOMERS\n                row = tmp4.iloc[k]\n                ax0.plot(row.S_2,row[c],'-o',color='orange')\n                t0 += row[c]\n            except:\n                pass\n        plt.title(f'Feature {c} (Key: BLUE=DEFAULT, orange=no default)',size=18)\n\n        # PLOT HISTOGRAMS\n        ax1 = fig.add_subplot(spec[1])\n        try:\n            # COMPUTE BINS\n            t = t0+t1; mn = np.nanmin(t); mx = np.nanmax(t)\n            if mx==mn:\n                mx += 0.01; mn -= 0.01\n            bins = np.arange(mn,mx+(mx-mn)/20,(mx-mn)/20 )\n            # PLOT HISTOGRAMS\n            if np.sum(np.isnan(t1))!=len(t1):\n                ax1.hist(t1,bins=bins,orientation=\"horizontal\",alpha = 0.8,color='blue')\n            if np.sum(np.isnan(t0))!=len(t0):\n                ax1.hist(t0,bins=bins,orientation=\"horizontal\",alpha = 0.8,color='orange')\n        except:\n            pass\n        plt.show()","metadata":{"scrolled":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-06-01T15:23:48.452845Z","iopub.execute_input":"2022-06-01T15:23:48.453266Z","iopub.status.idle":"2022-06-01T15:23:48.473133Z","shell.execute_reply.started":"2022-06-01T15:23:48.453230Z","shell.execute_reply":"2022-06-01T15:23:48.472148Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plot Delinquency Variables","metadata":{}},{"cell_type":"code","source":"# LEAVE LIST BLANK TO PLOT ALL\nplot_time_series('D',[39,41,47,45,46,48,54,59,61,62,75,96,105,112,124])","metadata":{"execution":{"iopub.status.busy":"2022-06-01T15:23:48.474443Z","iopub.execute_input":"2022-06-01T15:23:48.474884Z","iopub.status.idle":"2022-06-01T15:24:09.889769Z","shell.execute_reply.started":"2022-06-01T15:23:48.474844Z","shell.execute_reply":"2022-06-01T15:24:09.888582Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plot Spend Variables","metadata":{}},{"cell_type":"code","source":"# LEAVE LIST BLANK TO PLOT ALL\nplot_time_series('S',[3,7,19,23,26])","metadata":{"execution":{"iopub.status.busy":"2022-06-01T15:24:09.892167Z","iopub.execute_input":"2022-06-01T15:24:09.892571Z","iopub.status.idle":"2022-06-01T15:24:17.060643Z","shell.execute_reply.started":"2022-06-01T15:24:09.892535Z","shell.execute_reply":"2022-06-01T15:24:17.059540Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plot Payment Variables","metadata":{}},{"cell_type":"code","source":"# LEAVE LIST BLANK TO PLOT ALL\nplot_time_series('P',[2,3])","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-06-01T15:24:17.062125Z","iopub.execute_input":"2022-06-01T15:24:17.062875Z","iopub.status.idle":"2022-06-01T15:24:19.836219Z","shell.execute_reply.started":"2022-06-01T15:24:17.062831Z","shell.execute_reply":"2022-06-01T15:24:19.835175Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plot Balance Variables","metadata":{}},{"cell_type":"code","source":"# LEAVE LIST BLANK TO PLOT ALL\nplot_time_series('B',[2,3,4,5,7,9,20])","metadata":{"execution":{"iopub.status.busy":"2022-06-01T15:24:19.837367Z","iopub.execute_input":"2022-06-01T15:24:19.837686Z","iopub.status.idle":"2022-06-01T15:24:29.747430Z","shell.execute_reply.started":"2022-06-01T15:24:19.837657Z","shell.execute_reply":"2022-06-01T15:24:29.746418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plot Risk Variables","metadata":{}},{"cell_type":"code","source":"# LEAVE LIST BLANK TO PLOT ALL\nplot_time_series('R',[1,3,13,18])","metadata":{"execution":{"iopub.status.busy":"2022-06-01T15:24:29.748744Z","iopub.execute_input":"2022-06-01T15:24:29.749052Z","iopub.status.idle":"2022-06-01T15:24:35.465703Z","shell.execute_reply.started":"2022-06-01T15:24:29.749025Z","shell.execute_reply":"2022-06-01T15:24:35.464693Z"},"trusted":true},"execution_count":null,"outputs":[]}]}