{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# AMEX EDA 有意義 ⭐️⭐️⭐️⭐️⭐️\n\n此 EDA 分析數據並提供一些對設計機器學習管道和選擇模型有用的見解。","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport pickle\nfrom matplotlib import pyplot as plt","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-29T18:55:11.993716Z","iopub.execute_input":"2022-05-29T18:55:11.994467Z","iopub.status.idle":"2022-05-29T18:55:12.023622Z","shell.execute_reply.started":"2022-05-29T18:55:11.994343Z","shell.execute_reply":"2022-05-29T18:55:12.022587Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 標籤\n​​​\n我們首先閱讀訓練數據的標籤。 既沒有缺失值，也沒有重複的 customer_ID。 在 458913 個客戶 ID 中，340000 (74%) 個標籤為 0（好客戶，無默認值），119000 個（26%）標籤為 1（壞客戶，默認值）。\n​​​\n我們知道好的客戶已經被二次抽樣了 20 倍； 這意味著實際上有 680 萬好客戶。 98%的客戶是好的； 2%是壞的。\n​​​\n**洞察力：**\n- 班級不平衡。 建議使用 StratifiedKFold 進行交叉驗證。\n- 因為類是不平衡的，準確率將是評估分類器的一個不好的指標。 [競爭指標](https://www.kaggle.com/competitions/amex-default-prediction/discussion/327464) 是 roc 曲線下面積 (auc) 和召回率的混合。# The labels\n\nWe start by reading the labels for the training data. There are neither missing values nor duplicated customer_IDs. Of the 458913 customer_IDs, 340000 (74 %) have a label of 0 (good customer, no default) and 119000 (26 %) have a label of 1 (bad customer, default).\n\nWe know that the good customers have been subsampled by a factor of 20; this means that in reality there are 6.8 million good customers. 98 % of the customers are good; 2 % are bad.\n\n**Insight:**\n- The classes are imbalanced. A StratifiedKFold for cross-validation is recommended.\n- Because the classes are imbalanced, accuracy would be a bad metric to evaluate a classifier. The [competition metric](https://www.kaggle.com/competitions/amex-default-prediction/discussion/327464) is a mix of area under the roc curve (auc) and recall.","metadata":{}},{"cell_type":"code","source":"train_labels = pd.read_csv('../input/amex-default-prediction/train_labels.csv')\ntrain_labels.head(2)","metadata":{"execution":{"iopub.status.busy":"2022-05-29T18:55:12.025458Z","iopub.execute_input":"2022-05-29T18:55:12.025798Z","iopub.status.idle":"2022-05-29T18:55:13.121555Z","shell.execute_reply.started":"2022-05-29T18:55:12.02577Z","shell.execute_reply":"2022-05-29T18:55:13.120542Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check for missing data and duplicated customer_IDs\ntrain_labels.isna().any().any(), train_labels.customer_ID.duplicated().any()","metadata":{"execution":{"iopub.status.busy":"2022-05-29T18:55:13.12319Z","iopub.execute_input":"2022-05-29T18:55:13.123739Z","iopub.status.idle":"2022-05-29T18:55:13.269399Z","shell.execute_reply.started":"2022-05-29T18:55:13.123631Z","shell.execute_reply":"2022-05-29T18:55:13.268356Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"label_stats = pd.DataFrame({'absolute': train_labels.target.value_counts(),\n              'relative': train_labels.target.value_counts() / len(train_labels)})\nlabel_stats['absolute upsampled'] =  label_stats.absolute * np.array([20, 1])\nlabel_stats['relative upsampled'] = label_stats['absolute upsampled'] / label_stats['absolute upsampled'].sum()\nlabel_stats","metadata":{"execution":{"iopub.status.busy":"2022-05-29T18:55:13.272303Z","iopub.execute_input":"2022-05-29T18:55:13.273122Z","iopub.status.idle":"2022-05-29T18:55:13.301254Z","shell.execute_reply.started":"2022-05-29T18:55:13.273071Z","shell.execute_reply":"2022-05-29T18:55:13.300182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"＃ 數據\n​​​\n本次比賽的數據集規模相當大。 如果您閱讀原始 csv 文件，則數據幾乎無法放入內存。 這就是我們從 @munumbutt 的 [AMEX-Feather-Dataset](https://www.kaggle.com/datasets/munumbutt/amexfeather) 讀取數據的原因。 在這個 [Feather](https://arrow.apache.org/docs/python/feather.html) 文件中，浮點精度已從 64 位降低到 16 位。 並且讀取 Feather 文件比讀取 csv 文件更快，因為 Feather 文件格式是二進制的。\n​​​\n訓練數據有 550 萬行，測試數據有 1100 萬行。# The data\n\nThe dataset of this competition has a considerable size. If you read the original csv files, the data barely fits into memory. That's why we read the data from @munumbutt's [AMEX-Feather-Dataset](https://www.kaggle.com/datasets/munumbutt/amexfeather). In this [Feather](https://arrow.apache.org/docs/python/feather.html) file, the floating point precision has been reduced from 64 bit to 16 bit. And reading a Feather file is faster than reading a csv file because the Feather file format is binary.\n\nThere are 5.5 million rows for training and 11 million rows of test data.","metadata":{}},{"cell_type":"code","source":"%%time\ntrain = pd.read_feather('../input/amexfeather/train_data.ftr')\ntest = pd.read_feather('../input/amexfeather/test_data.ftr')\nwith pd.option_context(\"display.min_rows\", 6):\n    display(train)\n    display(test)","metadata":{"execution":{"iopub.status.busy":"2022-05-29T18:55:13.303137Z","iopub.execute_input":"2022-05-29T18:55:13.304285Z","iopub.status.idle":"2022-05-29T18:56:13.321963Z","shell.execute_reply.started":"2022-05-29T18:55:13.304237Z","shell.execute_reply":"2022-05-29T18:56:13.320926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"訓練數據框的目標列對應於 train_labels.csv 的目標列。 在訓練數據的csv文件中，沒有目標列； 為方便起見，它已被加入到 Feather 文件中。\n\nS_2 是報表日期。 所有火車結單日期都在 2017 年 3 月至 2018 年 3 月（13 個月）之間，並且沒有遺漏任何結單日期。 所有測試聲明日期都在 2018 年 4 月和 2019 年 10 月之間。這意味著 train 和 test 的聲明日期不重疊：","metadata":{}},{"cell_type":"code","source":"print('Train statement dates: ', train.S_2.min(), train.S_2.max(), train.S_2.isna().any())\nprint('Test statement dates: ',  test.S_2.min(), test.S_2.max(), test.S_2.isna().any())\n","metadata":{"execution":{"iopub.status.busy":"2022-05-29T18:56:13.323398Z","iopub.execute_input":"2022-05-29T18:56:13.324396Z","iopub.status.idle":"2022-05-29T18:56:13.463856Z","shell.execute_reply.started":"2022-05-29T18:56:13.324347Z","shell.execute_reply":"2022-05-29T18:56:13.462835Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n**洞察力：**\n- 測試數據來自與訓練數據不同的經濟周期階段。 我們的模型無法了解經濟周期的影響。","metadata":{}},{"cell_type":"code","source":"print(f'Train data memory usage: {train.memory_usage().sum() / 1e9} GBytes')\nprint(f'Test data memory usage:  {test.memory_usage().sum() / 1e9} GBytes')\n","metadata":{"execution":{"iopub.status.busy":"2022-05-29T18:56:13.465034Z","iopub.execute_input":"2022-05-29T18:56:13.465784Z","iopub.status.idle":"2022-05-29T18:56:13.495463Z","shell.execute_reply.started":"2022-05-29T18:56:13.465748Z","shell.execute_reply":"2022-05-29T18:56:13.494306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"訓練數據佔用 2.2 GB RAM。 測試數據是訓練數據的兩倍。\n\n**洞察力：**\n- 有了這麼多數據，我們需要關注內存效率。 避免在內存中保留不必要的數據副本，並避免保留不必要的模型副本！\n- 儘管大多數機器學習算法都希望整個訓練數據都在內存中，但我們不需要一次加載所有測試數據。 測試數據可以批量處理。\n- 您可能希望將訓練和推理代碼分離到兩個筆記本中，這樣您就不會同時在內存中擁有訓練和測試數據。","metadata":{}},{"cell_type":"markdown","source":"info 函數顯示大多數其他特徵都有缺失值：\n","metadata":{}},{"cell_type":"code","source":"train.info(max_cols=200, show_counts=True)","metadata":{"execution":{"iopub.status.busy":"2022-05-29T18:56:13.496929Z","iopub.execute_input":"2022-05-29T18:56:13.497255Z","iopub.status.idle":"2022-05-29T18:56:19.868651Z","shell.execute_reply.started":"2022-05-29T18:56:13.497225Z","shell.execute_reply":"2022-05-29T18:56:19.867503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**洞察力：**\n- 有很多列缺失值：刪除所有缺失值的列不是明智的策略。\n- 有很多行缺失值：刪除所有缺失值的行不是一個明智的策略。\n- 許多基於決策樹的算法可以處理缺失值。如果我們選擇這樣的模型，我們不需要更改缺失值。\n- 神經網絡和其他估計器無法處理缺失值。如果我們選擇這樣的模型，我們需要估算值。請參閱 [本指南](https://www.kaggle.com/code/parulpandey/a-guide-to-handling-missing-values-in-python) 了解許多插補選項的概述。\n- 大多數功能都是 16 位浮點數。原始數據（在 csv 文件中）具有更高的精度。通過將其四捨五入到 16 位精度，會丟失一些信息。為了使這種信息丟失更加明顯：1 到 2 之間的每個 float16 數字都是 1/1024 的倍數。這些數字只有小數點後三位！這個精度足以開始比賽；也許我們將不得不在最後切換到更高的精度。","metadata":{}},{"cell_type":"markdown","source":"# 計算每個客戶的報表","metadata":{}},{"cell_type":"markdown","source":"現在我們可以計算每個客戶有多少行（信用卡對帳單）。 我們看到 80% 的客戶有 13 條語句； 其他 20% 的客戶有 1 到 12 條語句。\n\n**洞察：**我們的模型將不得不處理每個客戶的可變大小輸入（除非我們簡化我們的生活並且只查看@inversion 建議的最新聲明[此處]（https://www.kaggle. com/competitions/amex-default-prediction/discussion/327094）或所有聲明的平均值）。","metadata":{}},{"cell_type":"code","source":"fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))\ntrain_sc = train.customer_ID.value_counts().value_counts().sort_index(ascending=False).rename('Train statements per customer')\nax1.pie(train_sc, labels=train_sc.index)\nax1.set_title(train_sc.name)\ntest_sc = test.customer_ID.value_counts().value_counts().sort_index(ascending=False).rename('Test statements per customer')\nax2.pie(test_sc, labels=test_sc.index)\nax2.set_title(test_sc.name)\nplt.show()\n\n# display(train.customer_ID.value_counts().value_counts().sort_index(ascending=False).rename('Train statements per customer'))\n# display(train.customer_ID.value_counts().value_counts().sort_index(ascending=False).rename('Test statements per customer'))\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-29T18:56:19.872771Z","iopub.execute_input":"2022-05-29T18:56:19.873158Z","iopub.status.idle":"2022-05-29T18:56:23.757455Z","shell.execute_reply.started":"2022-05-29T18:56:19.873125Z","shell.execute_reply":"2022-05-29T18:56:23.756316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"讓我們找出這些客戶何時收到最後一份聲明。 最後一次結單日期的直方圖顯示，每位火車客戶在 2018 年 3 月收到最後一次結單。前四個星期六（3 月 3 日、10 日、17 日、24 日）的結單數量高於平均一天。\n\n測試客戶分為兩部分：其中一半在 2019 年 4 月獲得最後一份聲明，另一半在 2019 年 10 月獲得。正如 [此處討論](https://www.kaggle.com/competitions/amex-default-prediction /discussion/327602)，2019 年 4 月的數據用於公共排行榜，2019 年 10 月的數據用於私人排行榜。","metadata":{}},{"cell_type":"code","source":"temp = train.S_2.groupby(train.customer_ID).max()\nplt.figure(figsize=(16, 4))\nplt.hist(temp, bins=pd.date_range(\"2018-03-01\", \"2018-04-01\", freq=\"d\"),\n         rwidth=0.8, color='#ffd700')\nplt.title('When did the train customers get their last statements?', fontsize=20)\nplt.xlabel('Last statement date per customer')\nplt.ylabel('Count')\nplt.gca().set_facecolor('#0057b8')\nplt.show()\ndel temp\n\ntemp = test.S_2.groupby(test.customer_ID).max()\nplt.figure(figsize=(16, 4))\nplt.hist(temp, bins=pd.date_range(\"2019-04-01\", \"2019-11-01\", freq=\"d\"),\n         rwidth=0.74, color='#ffd700')\nplt.title('When did the test customers get their last statements?', fontsize=20)\nplt.xlabel('Last statement date per customer')\nplt.ylabel('Count')\nplt.gca().set_facecolor('#0057b8')\nplt.show()\ndel temp","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-29T19:30:39.016046Z","iopub.execute_input":"2022-05-29T19:30:39.016461Z","iopub.status.idle":"2022-05-29T19:30:44.668376Z","shell.execute_reply.started":"2022-05-29T19:30:39.016426Z","shell.execute_reply":"2022-05-29T19:30:44.667284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**洞察：** 雖然數據是一種時間序列，但我們無法使用 TimeSeriesSplit 進行交叉驗證，因為所有訓練都發生在同一個月。\n\n對於大多數客戶來說，第一個和最後一個聲明相隔大約一年。 再加上我們通常每個客戶有 13 份對帳單，這表明客戶每個月都會收到一張信用卡對帳單。","metadata":{}},{"cell_type":"code","source":"temp = train.S_2.groupby(train.customer_ID).agg(['max', 'min'])\nplt.figure(figsize=(16, 3))\nplt.hist((temp['max'] - temp['min']).dt.days, bins=400, color='#ffd700')\nplt.xlabel('days')\nplt.ylabel('count')\nplt.title('Number of days between first and last statement of customer (train)', fontsize=20)\nplt.gca().set_facecolor('#0057b8')\nplt.show()\n\ntemp = test.S_2.groupby(test.customer_ID).agg(['max', 'min'])\nplt.figure(figsize=(16, 3))\nplt.hist((temp['max'] - temp['min']).dt.days, bins=400, color='#ffd700')\nplt.xlabel('days')\nplt.ylabel('count')\nplt.title('Number of days between first and last statement of customer (test)', fontsize=20)\nplt.gca().set_facecolor('#0057b8')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-29T18:56:30.903113Z","iopub.execute_input":"2022-05-29T18:56:30.903467Z","iopub.status.idle":"2022-05-29T18:56:37.854649Z","shell.execute_reply.started":"2022-05-29T18:56:30.903435Z","shell.execute_reply":"2022-05-29T18:56:37.85337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"如果我們根據它所屬的數據集（訓練、公共 lb 和私有 lb）為每個語句（即訓練或測試行）著色，我們會看到每個數據集涵蓋 13 個月。 訓練和測試不重疊，但公共和私人 lb 時期重疊。","metadata":{}},{"cell_type":"code","source":"temp = pd.concat([train[['customer_ID', 'S_2']], test[['customer_ID', 'S_2']]], axis=0)\ntemp.set_index('customer_ID', inplace=True)\ntemp['last_month'] = temp.groupby('customer_ID').S_2.max().dt.month\n\nplt.figure(figsize=(16, 4))\nplt.hist([temp.S_2[temp.last_month == 3],   # ending 03/18 -> training\n          temp.S_2[temp.last_month == 4],   # ending 04/19 -> public lb\n          temp.S_2[temp.last_month == 10]], # ending 10/19 -> private lb\n         bins=pd.date_range(\"2017-03-01\", \"2019-11-01\", freq=\"MS\"),\n         label=['Training', 'Public leaderboard', 'Private leaderboard'],\n         stacked=True)\nplt.xticks(pd.date_range(\"2017-03-01\", \"2019-11-01\", freq=\"QS\"))\nplt.xlabel('Statement date')\nplt.ylabel('Count')\nplt.title('The three datasets', fontsize=20)\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-29T19:41:33.514842Z","iopub.execute_input":"2022-05-29T19:41:33.515239Z","iopub.status.idle":"2022-05-29T19:41:45.687309Z","shell.execute_reply.started":"2022-05-29T19:41:33.515208Z","shell.execute_reply":"2022-05-29T19:41:45.686591Z"},"_kg_hide-input":false,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 分類特徵\n​​​\n根據【數據描述】（https://www.kaggle.com/competitions/amex-default-prediction/data），有十一個分類特徵。 我們為 target=0 和 target=1 繪製直方圖。 對於有缺失值的十個特徵，缺失值由直方圖最右邊的條形表示。\n​​​# The categorical features\n\nAccording to the [data description](https://www.kaggle.com/competitions/amex-default-prediction/data), there are eleven categorical features. We plot histograms for target=0 and target=1. For the ten features which have missing values, the missing values are represented by the rightmost bar of the histogram.\n","metadata":{}},{"cell_type":"code","source":"cat_features = ['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']\nplt.figure(figsize=(16, 16))\nfor i, f in enumerate(cat_features):\n    plt.subplot(4, 3, i+1)\n    temp = pd.DataFrame(train[f][train.target == 0].value_counts(dropna=False, normalize=True).sort_index().rename('count'))\n    temp.index.name = 'value'\n    temp.reset_index(inplace=True)\n    plt.bar(temp.index, temp['count'], alpha=0.5, label='target=0')\n    temp = pd.DataFrame(train[f][train.target == 1].value_counts(dropna=False, normalize=True).sort_index().rename('count'))\n    temp.index.name = 'value'\n    temp.reset_index(inplace=True)\n    plt.bar(temp.index, temp['count'], alpha=0.5, label='target=1')\n    plt.xlabel(f)\n    plt.ylabel('frequency')\n    plt.legend()\n    plt.xticks(temp.index, temp.value)\nplt.suptitle('Categorical features', fontsize=20, y=0.93)\nplt.show()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-29T18:56:37.855986Z","iopub.execute_input":"2022-05-29T18:56:37.856329Z","iopub.status.idle":"2022-05-29T18:56:40.530513Z","shell.execute_reply.started":"2022-05-29T18:56:37.856299Z","shell.execute_reply":"2022-05-29T18:56:40.52946Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**洞察力：**\n- 每個特徵最多有八個類別（包括一個nan類別）。 One-hot 編碼是可行的。\n- target=0 和 target=1 的分佈不同。 這意味著每個特徵都提供了一些關於目標的信息。\n​​​\n","metadata":{}},{"cell_type":"markdown","source":"# 二進制特徵\n\n兩個特徵是二進制的：\n- B_31 始終為 0 或 1。\n- D_87 始終為 1 或缺失。","metadata":{}},{"cell_type":"code","source":"bin_features = ['B_31', 'D_87']\nplt.figure(figsize=(16, 4))\nfor i, f in enumerate(bin_features):\n    plt.subplot(1, 2, i+1)\n    temp = pd.DataFrame(train[f][train.target == 0].value_counts(dropna=False, normalize=True).sort_index().rename('count'))\n    temp.index.name = 'value'\n    temp.reset_index(inplace=True)\n    plt.bar(temp.index, temp['count'], alpha=0.5, label='target=0')\n    temp = pd.DataFrame(train[f][train.target == 1].value_counts(dropna=False, normalize=True).sort_index().rename('count'))\n    temp.index.name = 'value'\n    temp.reset_index(inplace=True)\n    plt.bar(temp.index, temp['count'], alpha=0.5, label='target=1')\n    plt.xlabel(f)\n    plt.ylabel('frequency')\n    plt.legend()\n    plt.xticks(temp.index, temp.value)\nplt.suptitle('Binary features', fontsize=20)\nplt.show()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-29T18:56:40.531754Z","iopub.execute_input":"2022-05-29T18:56:40.532113Z","iopub.status.idle":"2022-05-29T18:56:41.103432Z","shell.execute_reply.started":"2022-05-29T18:56:40.532082Z","shell.execute_reply":"2022-05-29T18:56:41.102636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**見解：** 如果您為 D_87 估算缺失值，請不要陷入估算平均值的陷阱 - 該功能將變得無用......","metadata":{}},{"cell_type":"markdown","source":"# 連續特徵\n​​​\n如果我們繪製連續特徵的直方圖，我們會看到它們具有各種分佈：# The continuous features\n\nIf we plot histograms of the continuous features, we see that they have all kinds of distributions:","metadata":{}},{"cell_type":"code","source":"cont_features = sorted([f for f in train.columns if f not in cat_features + ['customer_ID', 'target', 'S_2']])\n# print(cont_features)\nncols = 4\nfor i, f in enumerate(cont_features):\n    if i % ncols == 0: \n        if i > 0: plt.show()\n        plt.figure(figsize=(16, 3))\n        if i == 0: plt.suptitle('Continuous features', fontsize=20, y=1.02)\n    plt.subplot(1, ncols, i % ncols + 1)\n    plt.hist(train[f], bins=200)\n    plt.xlabel(f)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-29T18:56:41.104545Z","iopub.execute_input":"2022-05-29T18:56:41.105578Z","iopub.status.idle":"2022-05-29T18:56:53.482314Z","shell.execute_reply.started":"2022-05-29T18:56:41.105512Z","shell.execute_reply":"2022-05-29T18:56:53.480485Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**洞察：** 左端或右端有空白的直方圖表示數據包含異常值。 我們將不得不處理這些異常值。 但這些數據真的是異常值嗎？ 也許它們是，但它們也可能是罕見事件的合法痕跡。 我們不知道...","metadata":{}},{"cell_type":"markdown","source":"\n未完待續...","metadata":{}}]}