{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# American Express - Default Prediction\n\n![Amex-intro-image](https://storage.googleapis.com/kaggle-organizations/3804/thumbnail.png)\n\nIn this competition, aim is to predict credit [default](https://www.investopedia.com/terms/d/default2.asp) using an industrial scale data provided by [American Express](https://www.americanexpress.com/en-in/)","metadata":{"_kg_hide-input":false}},{"cell_type":"code","source":"import os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        file_size = round(os.path.getsize(os.path.join(dirname, filename)) / (1e9), 2)\n        print(f\"Filename : {filename} \\t File Size : {file_size} GB\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-06-13T14:38:59.796613Z","iopub.execute_input":"2022-06-13T14:38:59.797482Z","iopub.status.idle":"2022-06-13T14:38:59.839211Z","shell.execute_reply.started":"2022-06-13T14:38:59.797349Z","shell.execute_reply":"2022-06-13T14:38:59.838248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport random\nimport warnings\nfrom tqdm import tqdm\n\nrandom.seed(42)\nplt.style.use('fivethirtyeight')\nwarnings.filterwarnings('ignore')\nsns.color_palette(\"flare\", as_cmap=True)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-06-13T14:38:59.840780Z","iopub.execute_input":"2022-06-13T14:38:59.841313Z","iopub.status.idle":"2022-06-13T14:39:01.239674Z","shell.execute_reply.started":"2022-06-13T14:38:59.841278Z","shell.execute_reply":"2022-06-13T14:39:01.238514Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Loading ...","metadata":{}},{"cell_type":"markdown","source":"## Loading Dataset\nSince the dataset it just tooooooo big, we can't directly read it as it will take up all the space present inside our kaggle kernel\n\n## How to read it then?\n1. Reading in Chunks \n2. Reading in other storage format such as parquet, feather, etc.\n3. Assigning smaller dtype to each columns \n\n## Reading in other format\nUsing this since, \n* Reading in Chunks won't give us a overall look at the distribution and which `CHUNK_SIZE` is better its a question in itself\n* I tried assigning smaller dtypes such as `float16` as shown [here](https://www.kaggle.com/code/sudalairajkumar/simple-explroration-notebook-amex-default?scriptVersionId=96684923&cellId=7) but still it was taking tooo long to load \n\nThanks [raddar](https://www.kaggle.com/raddar) for providing dataset in Parquet Format : https://www.kaggle.com/datasets/raddar/amex-data-integer-dtypes-parquet-format","metadata":{"_kg_hide-input":true}},{"cell_type":"code","source":"%time\ndf_train = pd.read_parquet(\"../input/amex-data-integer-dtypes-parquet-format/train.parquet\")","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-06-13T14:39:01.241724Z","iopub.execute_input":"2022-06-13T14:39:01.242164Z","iopub.status.idle":"2022-06-13T14:39:22.876430Z","shell.execute_reply.started":"2022-06-13T14:39:01.242126Z","shell.execute_reply":"2022-06-13T14:39:22.875480Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Zuppppp ⚡","metadata":{}},{"cell_type":"code","source":"%time\nlabels = pd.read_csv('../input/amex-default-prediction/train_labels.csv')\ndf_train = df_train.merge(labels, left_on='customer_ID', right_on='customer_ID')","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-06-13T14:39:22.877802Z","iopub.execute_input":"2022-06-13T14:39:22.878319Z","iopub.status.idle":"2022-06-13T14:41:12.549080Z","shell.execute_reply.started":"2022-06-13T14:39:22.878287Z","shell.execute_reply":"2022-06-13T14:41:12.547017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This one takes around minute 🤷‍","metadata":{}},{"cell_type":"code","source":"print(\"Shape of dataset :\", df_train.shape)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-06-13T14:41:12.552521Z","iopub.execute_input":"2022-06-13T14:41:12.553460Z","iopub.status.idle":"2022-06-13T14:41:12.561367Z","shell.execute_reply.started":"2022-06-13T14:41:12.553393Z","shell.execute_reply":"2022-06-13T14:41:12.559890Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-13T14:41:12.562915Z","iopub.execute_input":"2022-06-13T14:41:12.563394Z","iopub.status.idle":"2022-06-13T14:41:12.624740Z","shell.execute_reply.started":"2022-06-13T14:41:12.563347Z","shell.execute_reply":"2022-06-13T14:41:12.623826Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Column info\n\nThe dataset contains aggregated profile features for each customer at each statement date. Features are anonymized and normalized, and fall into the following general categories:\n\n* D_* = Delinquency variables (bad or criminal behaviour, especially among young people)\n* S_* = Spend variables\n* P_* = Payment variables\n* B_* = Balance variables\n* R_* = Risk variables\n\nwith the following features being categorical:  \n`['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']`\n\n","metadata":{}},{"cell_type":"markdown","source":"# Lets Explore ... 🚀","metadata":{}},{"cell_type":"markdown","source":"# Missing values","metadata":{}},{"cell_type":"code","source":"null_vals = df_train.isna().sum().sort_values(ascending=False)\nnull_vals[null_vals > 0 ]","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-06-13T14:41:12.626064Z","iopub.execute_input":"2022-06-13T14:41:12.626586Z","iopub.status.idle":"2022-06-13T14:41:15.876179Z","shell.execute_reply.started":"2022-06-13T14:41:12.626552Z","shell.execute_reply":"2022-06-13T14:41:15.874926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.title(\"Distribution of null values\")\nnull_vals[null_vals > 0 ].plot(kind = 'hist');","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-06-13T14:41:15.877875Z","iopub.execute_input":"2022-06-13T14:41:15.878273Z","iopub.status.idle":"2022-06-13T14:41:16.156222Z","shell.execute_reply.started":"2022-06-13T14:41:15.878238Z","shell.execute_reply":"2022-06-13T14:41:16.155070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So there are some columns for which number of missing values is close to **million** which is approximately same as the number of rows in our dataset thus removing those columns would be better...","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(40,10))\nplt.title(\"Null value count\")\nplt.xlabel(\"Columns\")\nplt.ylabel(\"Count\")\nnull_vals[null_vals > 0 ].plot(kind=\"bar\");","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-06-13T14:41:16.157628Z","iopub.execute_input":"2022-06-13T14:41:16.158010Z","iopub.status.idle":"2022-06-13T14:41:17.070359Z","shell.execute_reply.started":"2022-06-13T14:41:16.157977Z","shell.execute_reply":"2022-06-13T14:41:17.069505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Note** :  \nScale on y-axis is in millions which shows that there's just lot of columns which needs to be removed/preprocessed","metadata":{}},{"cell_type":"markdown","source":"# Is the target imbalance?","metadata":{}},{"cell_type":"code","source":"sns.countplot(\n    df_train[\"target\"].values,\n).set_xlabel(\"Target\");","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-06-13T14:41:17.071983Z","iopub.execute_input":"2022-06-13T14:41:17.072585Z","iopub.status.idle":"2022-06-13T14:41:17.789807Z","shell.execute_reply.started":"2022-06-13T14:41:17.072544Z","shell.execute_reply":"2022-06-13T14:41:17.788537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Yes it is!**","metadata":{}},{"cell_type":"markdown","source":"## How is target calculated ?\n\nThe target binary variable is calculated by observing 18 months performance window after the latest credit card statement, and if the customer does not pay due amount in 120 days after their latest statement date it is considered a default event.","metadata":{}},{"cell_type":"markdown","source":"# Customers","metadata":{}},{"cell_type":"code","source":"print(\"Number of unique customer :\",len(df_train[\"customer_ID\"].unique()))","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-06-13T14:41:17.794048Z","iopub.execute_input":"2022-06-13T14:41:17.795073Z","iopub.status.idle":"2022-06-13T14:41:18.752764Z","shell.execute_reply.started":"2022-06-13T14:41:17.795031Z","shell.execute_reply":"2022-06-13T14:41:18.751487Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cust_id = np.random.choice(df_train[\"customer_ID\"])\ndf_train[df_train[\"customer_ID\"] == cust_id]","metadata":{"execution":{"iopub.status.busy":"2022-06-13T14:41:18.754047Z","iopub.execute_input":"2022-06-13T14:41:18.754799Z","iopub.status.idle":"2022-06-13T14:41:22.796037Z","shell.execute_reply.started":"2022-06-13T14:41:18.754754Z","shell.execute_reply":"2022-06-13T14:41:22.794822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train[df_train[\"target\"] == 1].iloc[100].customer_ID","metadata":{"execution":{"iopub.status.busy":"2022-06-13T14:41:22.797958Z","iopub.execute_input":"2022-06-13T14:41:22.798421Z","iopub.status.idle":"2022-06-13T14:41:23.735437Z","shell.execute_reply.started":"2022-06-13T14:41:22.798366Z","shell.execute_reply":"2022-06-13T14:41:23.734262Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cust_id = '000473eb907b57c8c23f652bba40f87fe7261273dda47034d46fc46821017e50'\ndf_train[df_train[\"customer_ID\"] == cust_id]","metadata":{"execution":{"iopub.status.busy":"2022-06-13T14:41:23.737165Z","iopub.execute_input":"2022-06-13T14:41:23.737591Z","iopub.status.idle":"2022-06-13T14:41:24.654869Z","shell.execute_reply.started":"2022-06-13T14:41:23.737554Z","shell.execute_reply":"2022-06-13T14:41:24.653655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train[df_train[\"customer_ID\"] == cust_id][\"S_2\"]","metadata":{"execution":{"iopub.status.busy":"2022-06-13T14:41:24.657088Z","iopub.execute_input":"2022-06-13T14:41:24.657726Z","iopub.status.idle":"2022-06-13T14:41:25.576691Z","shell.execute_reply.started":"2022-06-13T14:41:24.657678Z","shell.execute_reply":"2022-06-13T14:41:25.575114Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So for each customer, we have a data for multiple days  \nLet's check if it is consistent for all the customers...","metadata":{}},{"cell_type":"code","source":"rand_customers = np.unique(df_train[\"customer_ID\"])[:100] # for 100 customers\nid_counts = df_train[df_train[\"customer_ID\"].isin(rand_customers)].groupby(\"customer_ID\").agg(\"count\")","metadata":{"execution":{"iopub.status.busy":"2022-06-13T14:41:25.578762Z","iopub.execute_input":"2022-06-13T14:41:25.579330Z","iopub.status.idle":"2022-06-13T14:41:33.270910Z","shell.execute_reply.started":"2022-06-13T14:41:25.579289Z","shell.execute_reply":"2022-06-13T14:41:33.269773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(20,10))\nid_counts[\"S_2\"].plot(kind='bar');","metadata":{"execution":{"iopub.status.busy":"2022-06-13T14:41:33.272653Z","iopub.execute_input":"2022-06-13T14:41:33.273076Z","iopub.status.idle":"2022-06-13T14:41:36.861829Z","shell.execute_reply.started":"2022-06-13T14:41:33.273040Z","shell.execute_reply":"2022-06-13T14:41:36.859851Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looks like for all customers number of days when default was announced is not consistent","metadata":{}},{"cell_type":"markdown","source":"# How each variables correlate with the target?","metadata":{}},{"cell_type":"markdown","source":"## Count of each type of variables","metadata":{}},{"cell_type":"code","source":"var_count = {}\nfor col in df_train.columns :\n    if col.startswith(\"S_\"):\n        var_count[\"Spend variables\"] = var_count.get(\"Spend variables\", 0) + 1 \n    if col.startswith(\"D_\"):\n        var_count[\"Deliquency variables\"] = var_count.get(\"Deliquency variables\", 0) + 1\n    if col.startswith(\"B_\"):\n        var_count[\"Balance variables\"] = var_count.get(\"Balance variables\", 0) + 1\n    if col.startswith(\"R_\"):\n        var_count[\"Risk variables\"] = var_count.get(\"Risk variables\", 0) + 1\n    if col.startswith(\"P_\"):\n        var_count[\"Payment variables\"] = var_count.get(\"Payment variables\", 0) + 1\nplt.figure(figsize=(15,5))\nsns.barplot(x=list(var_count.keys()), y=list(var_count.values()));","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-06-13T14:41:36.863464Z","iopub.execute_input":"2022-06-13T14:41:36.864546Z","iopub.status.idle":"2022-06-13T14:41:37.090575Z","shell.execute_reply.started":"2022-06-13T14:41:36.864490Z","shell.execute_reply":"2022-06-13T14:41:37.089458Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Payment variables (P_*) vs Target\n\nLet's see how payment variables affects default ?","metadata":{}},{"cell_type":"code","source":"payment_vars = [col for col in df_train.columns if col.startswith(\"P_\")]\ncorr = df_train[payment_vars+[\"target\"]].corr()\nsns.heatmap(corr, annot=True, cmap=\"Purples\");","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-06-13T14:41:37.092545Z","iopub.execute_input":"2022-06-13T14:41:37.094827Z","iopub.status.idle":"2022-06-13T14:41:37.853724Z","shell.execute_reply.started":"2022-06-13T14:41:37.094769Z","shell.execute_reply":"2022-06-13T14:41:37.852669Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axes = plt.subplots(1,3, figsize=(20,5))\naxes = axes.ravel()\n\nfor i, col in enumerate(payment_vars)  :\n    sns.histplot(data = df_train, x = col, hue='target', ax=axes[i])\n\nfig.suptitle(\"Distribution of Payment Variables w.r.t target\")\nfig.tight_layout()","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-06-13T14:41:37.855032Z","iopub.execute_input":"2022-06-13T14:41:37.855526Z","iopub.status.idle":"2022-06-13T14:42:01.165757Z","shell.execute_reply.started":"2022-06-13T14:41:37.855494Z","shell.execute_reply":"2022-06-13T14:42:01.164447Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train[\"P_4\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-06-13T14:42:01.167778Z","iopub.execute_input":"2022-06-13T14:42:01.168426Z","iopub.status.idle":"2022-06-13T14:42:01.340597Z","shell.execute_reply.started":"2022-06-13T14:42:01.168358Z","shell.execute_reply":"2022-06-13T14:42:01.339360Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looks like another case of artifical noise added to that categorical columns `P_4`","metadata":{"execution":{"iopub.status.busy":"2022-06-11T18:25:59.213986Z","iopub.execute_input":"2022-06-11T18:25:59.214561Z","iopub.status.idle":"2022-06-11T18:25:59.220869Z","shell.execute_reply.started":"2022-06-11T18:25:59.214512Z","shell.execute_reply":"2022-06-11T18:25:59.219628Z"}}},{"cell_type":"code","source":"df_train[\"P_4\"] = df_train[\"P_4\"].apply(lambda x : 0 if x == 0 else 1)\nplt.title(\"P_4 w.r.t target\")\nsns.countplot(data = df_train, x = \"P_4\", hue = \"target\");","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-06-13T14:42:01.342882Z","iopub.execute_input":"2022-06-13T14:42:01.343285Z","iopub.status.idle":"2022-06-13T14:42:08.223733Z","shell.execute_reply.started":"2022-06-13T14:42:01.343253Z","shell.execute_reply":"2022-06-13T14:42:08.222489Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* higher the `P_2` lower the chances of default\n* `target = 1` (i.e. default) is following a normal distribution in both `P_2` and `P_3`\n*  when `P_4` is 1, there's 50% of chance of being **default** but when it goes 0 lot of cases seem to be having less default","metadata":{}},{"cell_type":"markdown","source":"More to be added soon...","metadata":{}}]}