{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Customer Time Series EDA\nKaggle's \"American Express - Default Prediction\" Competition provides us with time series data from credit card customers. For each customer we have 188 data points for each of up to 13 points in time. Therefore most customers have `2444 = 188 * 13` data points. For each customer, we can plot the 188 line plots that show how each variable (for one specific customer) changes over time.\n\nFurthermore we can plot time series for customers who default on their credit payment. This is `target = 1` and is shown in blue below. And we can plot customers who do not default. This is `target = 0` and shown in orange below. By looking at different customers and different variables, we can gain intuition about which variables predict default.\n\nIn this notebook we chose 33 variables that look interesting. For each variable, we plot 32 default customers' time series, and 32 non-default customers' time series. To the right of the line plots, we plot a histogram also. Discussion about this EDA is [here][4].\n\nIf you wish to view the other 155 features, then copy and edit this notebook and display whichever variables you want. From the competition description [here][1], we know there are 5 types of variables. And there are 188 variables in total.\n\n* D_* = 96 Delinquency variables\n* S_* = 21 Spend variables\n* P_* = 3 Payment variables\n* B_* = 40 Balance variables\n* R_* = 28 Risk variables\n\nWhen you are ready to build a time series model like an RNN or Transformer that takes advantage of what you see in these plots, check out my TF GRU starter notebook [here][2] and check out my TF Transformer starter notebook [here][3]\n\n[1]: https://www.kaggle.com/competitions/amex-default-prediction/data\n[2]: https://www.kaggle.com/cdeotte/tensorflow-gru-starter-0-787\n[3]: https://www.kaggle.com/code/cdeotte/tensorflow-transformer-0-790\n[4]: https://www.kaggle.com/competitions/amex-default-prediction/discussion/327761\n\n# 客戶時間序列 EDA\nKaggle 的“美國運通 - 默認預測”競賽為我們提供了來自信用卡客戶的時間序列數據。對於每個客戶，我們有 188 個數據點，每個數據點最多 13 個時間點。因此，大多數客戶都有“2444 = 188 * 13”的數據點。對於每個客戶，我們可以繪製 188 個線圖，顯示每個變量（對於一個特定客戶）如何隨時間變化。\n\n此外，我們可以為拖欠信用付款的客戶繪製時間序列。這是`target = 1`，如下圖藍色所示。我們可以繪製沒有違約的客戶。這是`target = 0`，下面用橙色顯示。通過查看不同的客戶和不同的變量，我們可以直觀地了解哪些變量可以預測違約。\n\n在這個筆記本中，我們選擇了 33 個看起來很有趣的變量。對於每個變量，我們繪製了 32 個默認客戶的時間序列和 32 個非默認客戶的時間序列。在折線圖的右側，我們還繪製了一個直方圖。關於這個 EDA 的討論是 [這裡][4]。\n\n如果您想查看其他 155 個功能，請複制並編輯此筆記本並顯示您想要的任何變量。從比賽描述 [這裡][1]，我們知道有 5 種類型的變量。總共有188個變量。\n\n* D_* = 96 個拖欠變量\n* S_* = 21 個支出變量\n* P_* = 3 個付款變量\n* B_* = 40 平衡變量\n* R_* = 28 個風險變量\n\n當您準備好利用您在這些圖中看到的內容構建時間序列模型（如 RNN 或 Transformer）時，請查看我的 TF GRU 入門筆記本 [此處][2] 並查看我的 TF Transformer 入門筆記本 [此處] [3]\n\n[1]:https://www.kaggle.com/competitions/amex-default-prediction/data\n[2]:https://www.kaggle.com/cdeotte/tensorflow-gru-starter-0-787\n[3]:https://www.kaggle.com/code/cdeotte/tensorflow-transformer-0-790\n[4]:https://www.kaggle.com/competitions/amex-default-prediction/discussion/327761","metadata":{}},{"cell_type":"markdown","source":"# Load Train Data","metadata":{}},{"cell_type":"code","source":"# LOAD LIBRARIES\n# 加載庫\nimport pandas as pd, numpy as np\nimport matplotlib.pyplot as plt\nfrom matplotlib import gridspec\n\n# LOAD TRAIN DATA AND MERGE TARGETS ONTO FEATURES\n# 加載訓練數據並將目標合併到特徵上\ndf = pd.read_csv('../input/amex-default-prediction/train_data.csv', nrows=100_000)\ndf.S_2 = pd.to_datetime(df.S_2)\ndf2 = pd.read_csv('../input/amex-default-prediction/train_labels.csv')\ndf = df.merge(df2,on='customer_ID',how='left')","metadata":{"execution":{"iopub.status.busy":"2022-06-28T07:15:37.862875Z","iopub.execute_input":"2022-06-28T07:15:37.863365Z","iopub.status.idle":"2022-06-28T07:15:42.956417Z","shell.execute_reply.started":"2022-06-28T07:15:37.863329Z","shell.execute_reply":"2022-06-28T07:15:42.955517Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_time_series(prefix='D', cols=None, display_ct=32):\n    \n    # DETERMINE WHICH COLUMNS TO PLOT\n    # 確定要繪製的列\n    if cols is not None and len(cols)==0: cols = None\n    if cols is None:\n        COLS = df.columns[2:-1]\n        COLS = np.sort( [int(x[2:]) for x in COLS if x[0]==prefix] )\n        COLS = [f'{prefix}_{x}' for x in COLS]\n        print('#'*25)\n        print(f'Plotting all {len(COLS)} columns with prefix {prefix}')\n        print('#'*25)\n    else:\n        COLS = [f'{prefix}_{x}' for x in cols]\n        print('#'*25)\n        print(f'Plotting {len(COLS)} columns with prefix {prefix}')\n        print('#'*25)\n\n    # ITERATE COLUMNS\n    # 迭代列\n    for c in COLS:\n\n        # CONVERT DATAFRAME INTO SERIES WITH COLUMN\n        # 將數據幀轉換為帶有列的序列\n        tmp = df[['customer_ID','S_2',c,'target']].copy()\n        tmp2 = tmp.groupby(['customer_ID','target'])[['S_2',c]].agg(list).reset_index()\n        tmp3 = tmp2.loc[tmp2.target==1]\n        tmp4 = tmp2.loc[tmp2.target==0]\n\n        # FORMAT PLOT\n        # 格式化繪圖\n        spec = gridspec.GridSpec(ncols=2, nrows=1,\n                             width_ratios=[3, 1], wspace=0.1,\n                             hspace=0.5, height_ratios=[1])\n        fig = plt.figure(figsize=(20,10))\n        ax0 = fig.add_subplot(spec[0])\n\n        # PLOT 32 DEFAULT CUSTOMERS AND 32 NON-DEFAULT CUSTOMERS\n        # 繪製 32 位默認客戶和 32 位非默認客戶\n        t0 = []; t1 = []\n        for k in range(display_ct):\n            try:\n                # PLOT DEFAULTING CUSTOMERS\n                # 繪製默認客戶\n                row = tmp3.iloc[k]\n                ax0.plot(row.S_2,row[c],'-o',color='blue')\n                t1 += row[c]\n                # PLOT NON-DEFAULT CUSTOMERS\n                # 繪製非默認客戶\n                row = tmp4.iloc[k]\n                ax0.plot(row.S_2,row[c],'-o',color='orange')\n                t0 += row[c]\n            except:\n                pass\n        plt.title(f'Feature {c} (Key: BLUE=DEFAULT, orange=no default)',size=18)\n\n        # PLOT HISTOGRAMS\n        # 繪製直方圖\n        ax1 = fig.add_subplot(spec[1])\n        try:\n            # COMPUTE BINS\n            # 計算箱\n            t = t0+t1; mn = np.nanmin(t); mx = np.nanmax(t)\n            if mx==mn:\n                mx += 0.01; mn -= 0.01\n            bins = np.arange(mn,mx+(mx-mn)/20,(mx-mn)/20 )\n            # PLOT HISTOGRAMS\n            # 繪製直方圖\n            if np.sum(np.isnan(t1))!=len(t1):\n                ax1.hist(t1,bins=bins,orientation=\"horizontal\",alpha = 0.8,color='blue')\n            if np.sum(np.isnan(t0))!=len(t0):\n                ax1.hist(t0,bins=bins,orientation=\"horizontal\",alpha = 0.8,color='orange')\n        except:\n            pass\n        plt.show()","metadata":{"scrolled":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-06-28T07:15:42.958899Z","iopub.execute_input":"2022-06-28T07:15:42.959364Z","iopub.status.idle":"2022-06-28T07:15:42.981741Z","shell.execute_reply.started":"2022-06-28T07:15:42.959318Z","shell.execute_reply":"2022-06-28T07:15:42.980533Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plot Delinquency Variables\n\n# 繪製拖欠變量","metadata":{}},{"cell_type":"code","source":"# LEAVE LIST BLANK TO PLOT ALL\n# 將列表留空以繪製所有\nplot_time_series('D',[39,41,47,45,46,48,54,59,61,62,75,96,105,112,124])","metadata":{"execution":{"iopub.status.busy":"2022-06-28T07:15:42.983019Z","iopub.execute_input":"2022-06-28T07:15:42.983361Z","iopub.status.idle":"2022-06-28T07:16:06.415410Z","shell.execute_reply.started":"2022-06-28T07:15:42.983333Z","shell.execute_reply":"2022-06-28T07:16:06.414257Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plot Spend Variables\n\n# 繪製支出變量","metadata":{}},{"cell_type":"code","source":"# LEAVE LIST BLANK TO PLOT ALL\n# 將列表留空以繪製所有\nplot_time_series('S',[3,7,19,23,26])","metadata":{"execution":{"iopub.status.busy":"2022-06-28T07:16:06.417718Z","iopub.execute_input":"2022-06-28T07:16:06.418120Z","iopub.status.idle":"2022-06-28T07:16:13.803243Z","shell.execute_reply.started":"2022-06-28T07:16:06.418085Z","shell.execute_reply":"2022-06-28T07:16:13.802525Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plot Payment Variables\n\n# 繪製支付變量","metadata":{}},{"cell_type":"code","source":"# LEAVE LIST BLANK TO PLOT ALL\n# 將列表留空以繪製所有\nplot_time_series('P',[2,3])","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-06-28T07:16:13.804340Z","iopub.execute_input":"2022-06-28T07:16:13.804977Z","iopub.status.idle":"2022-06-28T07:16:16.713597Z","shell.execute_reply.started":"2022-06-28T07:16:13.804942Z","shell.execute_reply":"2022-06-28T07:16:16.712393Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plot Balance Variables\n\n# 繪製平衡變量","metadata":{}},{"cell_type":"code","source":"# LEAVE LIST BLANK TO PLOT ALL\n# 將列表留空以繪製所有\nplot_time_series('B',[2,3,4,5,7,9,20])","metadata":{"execution":{"iopub.status.busy":"2022-06-28T07:16:16.714985Z","iopub.execute_input":"2022-06-28T07:16:16.715426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plot Risk Variables\n\n# 繪製風險變量","metadata":{}},{"cell_type":"code","source":"# LEAVE LIST BLANK TO PLOT ALL\n# 將列表留空以繪製所有\nplot_time_series('R',[1,3,13,18])","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}