{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# American Express - Default Prediction","metadata":{}},{"cell_type":"markdown","source":"\nIn this analysis, I used the code from notebooks:\n- https://www.kaggle.com/code/ambrosm/amex-eda-which-makes-sense/notebook\n- https://www.kaggle.com/code/ihelon/default-prediction-eda-and-modeling\n\nAlso I use information from this discussion:\n- https://www.kaggle.com/competitions/amex-default-prediction/discussion/327464","metadata":{}},{"cell_type":"markdown","source":"## Let's look at the task description","metadata":{}},{"cell_type":"markdown","source":"In this competition, you’ll apply your machine learning skills to predict credit default. Specifically, you will leverage an industrial scale data set to build a machine learning model that challenges the current model in production. Training, validation, and testing datasets include time-series behavioral data and anonymized customer profile information. You're free to explore any technique to create the most powerful model, from creating features to using the data in a more organic way within a model.","metadata":{}},{"cell_type":"markdown","source":"# Competition metric\n","metadata":{}},{"cell_type":"markdown","source":"### 0.5 * (G + D) ","metadata":{}},{"cell_type":"markdown","source":"The competition metric has two components: the normalized Gini coefficient and the default rate captured at 4 %:\n\n* (G) The normalized Gini coefficient is simply a stretched AUC: AUC is the light red area under the curve, which has a value between 0 and 1. The normalized Gini coefficient is equal to 2*AUC-1 and is between -1 and 1. The larger the red area, the better is the score.\n* (D) The default rate captured at 4 % is the true positive rate (recall) for a threshold set at 4 % of the total (weighted) sample count. It corresponds to the y coordinate of the intersection between the green line and the red roc curve (marked with a green dot) and is always between 0 and 1. The higher the intersection point, the better is the score.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport seaborn as sns\nimport missingno as msno\nimport matplotlib.pyplot as plt\nimport missingno as msno\nimport numpy as np\nfrom pathlib import Path\nimport path\nimport os\nimport warnings\nwarnings.filterwarnings(\"ignore\")","metadata":{"execution":{"iopub.status.busy":"2022-06-01T08:58:48.967512Z","iopub.execute_input":"2022-06-01T08:58:48.967985Z","iopub.status.idle":"2022-06-01T08:58:50.246286Z","shell.execute_reply.started":"2022-06-01T08:58:48.967946Z","shell.execute_reply":"2022-06-01T08:58:50.245091Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_path = Path(\"../input/amex-default-prediction\")\nos.listdir(data_path)","metadata":{"execution":{"iopub.status.busy":"2022-06-01T08:14:58.901724Z","iopub.execute_input":"2022-06-01T08:14:58.902456Z","iopub.status.idle":"2022-06-01T08:14:58.912181Z","shell.execute_reply.started":"2022-06-01T08:14:58.902424Z","shell.execute_reply":"2022-06-01T08:14:58.911098Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"n_rows = 15000\n\ntrain_df = pd.read_csv(\"../input/amex-default-prediction/train_data.csv\", chunksize=n_rows)\ntrain_labels_df = pd.read_csv(\"../input/amex-default-prediction/train_labels.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-06-01T08:14:58.913492Z","iopub.execute_input":"2022-06-01T08:14:58.914161Z","iopub.status.idle":"2022-06-01T08:15:00.128202Z","shell.execute_reply.started":"2022-06-01T08:14:58.914116Z","shell.execute_reply":"2022-06-01T08:15:00.127125Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df_example = train_df.__next__()\n","metadata":{"execution":{"iopub.status.busy":"2022-06-01T08:15:00.130526Z","iopub.execute_input":"2022-06-01T08:15:00.131001Z","iopub.status.idle":"2022-06-01T08:15:01.267050Z","shell.execute_reply.started":"2022-06-01T08:15:00.130958Z","shell.execute_reply":"2022-06-01T08:15:01.265981Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df_example.info()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T08:15:01.268575Z","iopub.execute_input":"2022-06-01T08:15:01.269097Z","iopub.status.idle":"2022-06-01T08:15:01.307947Z","shell.execute_reply.started":"2022-06-01T08:15:01.269031Z","shell.execute_reply":"2022-06-01T08:15:01.306518Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df_example.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T08:15:01.309653Z","iopub.execute_input":"2022-06-01T08:15:01.310165Z","iopub.status.idle":"2022-06-01T08:15:01.343358Z","shell.execute_reply.started":"2022-06-01T08:15:01.310105Z","shell.execute_reply":"2022-06-01T08:15:01.342310Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df_example.tail()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T08:15:01.344661Z","iopub.execute_input":"2022-06-01T08:15:01.345724Z","iopub.status.idle":"2022-06-01T08:15:01.373185Z","shell.execute_reply.started":"2022-06-01T08:15:01.345677Z","shell.execute_reply":"2022-06-01T08:15:01.371989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T08:15:01.374565Z","iopub.execute_input":"2022-06-01T08:15:01.374914Z","iopub.status.idle":"2022-06-01T08:15:01.391938Z","shell.execute_reply.started":"2022-06-01T08:15:01.374885Z","shell.execute_reply":"2022-06-01T08:15:01.390583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels_df.shape","metadata":{"execution":{"iopub.status.busy":"2022-06-01T08:15:01.393835Z","iopub.execute_input":"2022-06-01T08:15:01.394352Z","iopub.status.idle":"2022-06-01T08:15:01.402690Z","shell.execute_reply.started":"2022-06-01T08:15:01.394305Z","shell.execute_reply":"2022-06-01T08:15:01.401900Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels_df.customer_ID.duplicated().sum()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T08:15:01.407515Z","iopub.execute_input":"2022-06-01T08:15:01.407926Z","iopub.status.idle":"2022-06-01T08:15:01.495494Z","shell.execute_reply.started":"2022-06-01T08:15:01.407889Z","shell.execute_reply":"2022-06-01T08:15:01.494338Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df_example[train_df_example[\"customer_ID\"] == \"009469964a6c21c6f1f50bb9a9881dce39dcaa47801b4f09d6d65d6610a1e0e9\"]","metadata":{"execution":{"iopub.status.busy":"2022-06-01T08:15:01.496885Z","iopub.execute_input":"2022-06-01T08:15:01.497416Z","iopub.status.idle":"2022-06-01T08:15:01.539227Z","shell.execute_reply.started":"2022-06-01T08:15:01.497383Z","shell.execute_reply":"2022-06-01T08:15:01.538094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cols = list(train_df_example.columns)\nprint(cols)","metadata":{"execution":{"iopub.status.busy":"2022-06-01T08:15:01.540643Z","iopub.execute_input":"2022-06-01T08:15:01.541029Z","iopub.status.idle":"2022-06-01T08:15:01.554415Z","shell.execute_reply.started":"2022-06-01T08:15:01.540985Z","shell.execute_reply":"2022-06-01T08:15:01.553593Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Features are anonymized and normalized, and fall into the following general categories:\n\n* D_* = Delinquency variables\n* S_* = Spend variables\n* P_* = Payment variables\n* B_* = Balance variables\n* R_* = Risk variables","metadata":{}},{"cell_type":"code","source":"cat_cols = [\n    'B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', \n    'D_126', 'D_63', 'D_64', 'D_66', 'D_68',\n]","metadata":{"execution":{"iopub.status.busy":"2022-06-01T08:15:01.555644Z","iopub.execute_input":"2022-06-01T08:15:01.556621Z","iopub.status.idle":"2022-06-01T08:15:01.573129Z","shell.execute_reply.started":"2022-06-01T08:15:01.556582Z","shell.execute_reply":"2022-06-01T08:15:01.571606Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Let's try to work with all the data and not a part","metadata":{}},{"cell_type":"markdown","source":"The dataset of this contest is of considerable size. If you're reading raw CSV files, the data barely fits in memory.That's why we read the data from @munumbutt's AMEX-Feather-Dataset. In this Feather file, floating point precision has been reduced from 64 bits to 16 bits and reading a Feather file is faster than reading a csv file because the Feather file format is binary.","metadata":{}},{"cell_type":"code","source":"train = pd.read_feather('../input/amexfeather/train_data.ftr')\ntest = pd.read_feather('../input/amexfeather/test_data.ftr')","metadata":{"execution":{"iopub.status.busy":"2022-06-01T08:58:56.486504Z","iopub.execute_input":"2022-06-01T08:58:56.486944Z","iopub.status.idle":"2022-06-01T09:00:00.594518Z","shell.execute_reply.started":"2022-06-01T08:58:56.486912Z","shell.execute_reply":"2022-06-01T09:00:00.593364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.shape","metadata":{"execution":{"iopub.status.busy":"2022-06-01T08:16:03.548933Z","iopub.execute_input":"2022-06-01T08:16:03.550265Z","iopub.status.idle":"2022-06-01T08:16:03.558702Z","shell.execute_reply.started":"2022-06-01T08:16:03.550211Z","shell.execute_reply":"2022-06-01T08:16:03.557797Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test.shape","metadata":{"execution":{"iopub.status.busy":"2022-06-01T08:16:03.560222Z","iopub.execute_input":"2022-06-01T08:16:03.560700Z","iopub.status.idle":"2022-06-01T08:16:03.574414Z","shell.execute_reply.started":"2022-06-01T08:16:03.560655Z","shell.execute_reply":"2022-06-01T08:16:03.573621Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Our observations\n* There are almost twice as many test data as training data\n* AMEX-Feather-Dataset is placed in our RAM","metadata":{}},{"cell_type":"code","source":"msno.matrix(train[0:2000], figsize = (10,5))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T08:16:03.575583Z","iopub.execute_input":"2022-06-01T08:16:03.579799Z","iopub.status.idle":"2022-06-01T08:16:04.225238Z","shell.execute_reply.started":"2022-06-01T08:16:03.579742Z","shell.execute_reply":"2022-06-01T08:16:04.224317Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.info(max_cols=200, show_counts=True)\n","metadata":{"execution":{"iopub.status.busy":"2022-06-01T08:16:04.229798Z","iopub.execute_input":"2022-06-01T08:16:04.232248Z","iopub.status.idle":"2022-06-01T08:16:11.378335Z","shell.execute_reply.started":"2022-06-01T08:16:04.232196Z","shell.execute_reply":"2022-06-01T08:16:11.377193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Our observations\n* There are many missing values in the data\n* There are columns with almost no data","metadata":{}},{"cell_type":"code","source":"temp = pd.concat([train[['customer_ID', 'S_2']], test[['customer_ID', 'S_2']]], axis=0)\ntemp.set_index('customer_ID', inplace=True)\ntemp['last_month'] = temp.groupby('customer_ID').S_2.max().dt.month\n\nplt.figure(figsize=(16, 4))\nplt.hist([temp.S_2[temp.last_month == 3],   # ending 03/18 -> training\n          temp.S_2[temp.last_month == 4],   # ending 04/19 -> public lb\n          temp.S_2[temp.last_month == 10]], # ending 10/19 -> private lb\n         bins=pd.date_range(\"2017-03-01\", \"2019-11-01\", freq=\"MS\"),\n         label=['Training', 'Public leaderboard', 'Private leaderboard'],\n         stacked=True)\nplt.xticks(pd.date_range(\"2017-03-01\", \"2019-11-01\", freq=\"QS\"))\nplt.xlabel('Statement date')\nplt.ylabel('Count')\nplt.title('The three datasets', fontsize=20)\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T08:16:11.379677Z","iopub.execute_input":"2022-06-01T08:16:11.380011Z","iopub.status.idle":"2022-06-01T08:16:23.968755Z","shell.execute_reply.started":"2022-06-01T08:16:11.379981Z","shell.execute_reply":"2022-06-01T08:16:23.967559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Our observations\n* There is no date intersection between test data and training data in the data\n* There are intersections between public and private dataset","metadata":{}},{"cell_type":"code","source":"fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))\ntrain_sc = train.customer_ID.value_counts().value_counts().sort_index(ascending=False).rename('Train statements per customer')\nax1.pie(train_sc, labels=train_sc.index)\nax1.set_title(train_sc.name)\ntest_sc = test.customer_ID.value_counts().value_counts().sort_index(ascending=False).rename('Test statements per customer')\nax2.pie(test_sc, labels=test_sc.index)\nax2.set_title(test_sc.name)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T08:16:23.970215Z","iopub.execute_input":"2022-06-01T08:16:23.970624Z","iopub.status.idle":"2022-06-01T08:16:27.352345Z","shell.execute_reply.started":"2022-06-01T08:16:23.970592Z","shell.execute_reply":"2022-06-01T08:16:27.351158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Our observations\n* Most often, we have 13 observations for a client","metadata":{}},{"cell_type":"markdown","source":"# Categorial data","metadata":{}},{"cell_type":"code","source":"cat_features = ['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']\nind = 0\nfor col in cat_features:\n    if ind % 4 == 0:\n        plt.figure(figsize=(16, 3))\n    plt.subplot(1, 4, ind % 4 + 1)\n    \n    sns.countplot(data=train, x=col, hue=\"target\")\n    plt.ylabel(\"\")\n    \n    if ind % 4 == 3:\n        plt.show()\n    \n    ind += 1","metadata":{"execution":{"iopub.status.busy":"2022-06-01T08:16:27.354168Z","iopub.execute_input":"2022-06-01T08:16:27.354859Z","iopub.status.idle":"2022-06-01T08:16:34.502919Z","shell.execute_reply.started":"2022-06-01T08:16:27.354811Z","shell.execute_reply":"2022-06-01T08:16:34.502074Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Our observations\n* In features B30 D116 D63 D64 D66 D68 there are categories that are relatively rare","metadata":{}},{"cell_type":"markdown","source":"# Continuous features","metadata":{}},{"cell_type":"code","source":"for col in list(train.columns):\n    if col in [\"S_2\", \"customer_ID\", \"target\"] + cat_features:\n        continue\n    \n    if ind % 4 == 0:\n        plt.figure(figsize=(16, 4))\n    plt.subplot(1, 4, ind % 4 + 1)\n    \n    sns.histplot(data=train, x=col, hue=\"target\", bins=20)\n    plt.ylabel(\"\")\n    \n    if ind % 4 == 3:\n        plt.show()\n    \n    ind += 1","metadata":{"execution":{"iopub.status.busy":"2022-06-01T08:16:34.504472Z","iopub.execute_input":"2022-06-01T08:16:34.505103Z","iopub.status.idle":"2022-06-01T08:24:14.711822Z","shell.execute_reply.started":"2022-06-01T08:16:34.505059Z","shell.execute_reply":"2022-06-01T08:24:14.710885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Let's look at the data in a different view ","metadata":{}},{"cell_type":"code","source":"cont_features = sorted([f for f in train.columns if f not in cat_features + ['customer_ID', 'target', 'S_2']])\n# print(cont_features)\nncols = 4\nfor i, f in enumerate(cont_features):\n    if i % ncols == 0: \n        if i > 0: plt.show()\n        plt.figure(figsize=(16, 3))\n        if i == 0: plt.suptitle('Continuous features', fontsize=20, y=1.02)\n    plt.subplot(1, ncols, i % ncols + 1)\n    plt.hist(train[f], bins=200)\n    plt.xlabel(f)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T08:24:14.713073Z","iopub.execute_input":"2022-06-01T08:24:14.713394Z","iopub.status.idle":"2022-06-01T08:27:32.440262Z","shell.execute_reply.started":"2022-06-01T08:24:14.713365Z","shell.execute_reply":"2022-06-01T08:27:32.439069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Our observations\n* In features that have large empty spaces, there are outliers. There are quite a lot of features with outliers\n* S8 B18 B16 can be categorical\n* There are strange gaps in the distribution of traits S15 S18 P2 D47","metadata":{}},{"cell_type":"markdown","source":"#  Features correlation","metadata":{}},{"cell_type":"code","source":"correlations = train.corr()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T09:00:00.596526Z","iopub.execute_input":"2022-06-01T09:00:00.596949Z","iopub.status.idle":"2022-06-01T09:06:51.225244Z","shell.execute_reply.started":"2022-06-01T09:00:00.596910Z","shell.execute_reply":"2022-06-01T09:06:51.223433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(14,14))\nsns.heatmap(correlations,ax = ax)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T09:13:36.665315Z","iopub.execute_input":"2022-06-01T09:13:36.666198Z","iopub.status.idle":"2022-06-01T09:13:38.272481Z","shell.execute_reply.started":"2022-06-01T09:13:36.666152Z","shell.execute_reply":"2022-06-01T09:13:38.271205Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"correlations = correlations.unstack()\ncorrelations.sort_values(ascending=False, kind=\"quicksort\").drop_duplicates().head(30)","metadata":{"execution":{"iopub.status.busy":"2022-06-01T09:13:50.067484Z","iopub.execute_input":"2022-06-01T09:13:50.067916Z","iopub.status.idle":"2022-06-01T09:13:50.102561Z","shell.execute_reply.started":"2022-06-01T09:13:50.067883Z","shell.execute_reply":"2022-06-01T09:13:50.101391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"correlations.sort_values(ascending=False, kind=\"quicksort\").drop_duplicates().tail(30)","metadata":{"execution":{"iopub.status.busy":"2022-06-01T09:13:53.784527Z","iopub.execute_input":"2022-06-01T09:13:53.784977Z","iopub.status.idle":"2022-06-01T09:13:53.802381Z","shell.execute_reply.started":"2022-06-01T09:13:53.784939Z","shell.execute_reply":"2022-06-01T09:13:53.800593Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Our observations\n* It can be seen that there are quite a lot of features strongly dependent on each other in the data.","metadata":{}},{"cell_type":"code","source":"correlations.to_csv(\"Corr.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-06-01T09:15:47.780286Z","iopub.execute_input":"2022-06-01T09:15:47.780743Z","iopub.status.idle":"2022-06-01T09:15:47.885565Z","shell.execute_reply.started":"2022-06-01T09:15:47.780705Z","shell.execute_reply":"2022-06-01T09:15:47.884459Z"},"trusted":true},"execution_count":null,"outputs":[]}]}