{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### 1. Introduction","metadata":{}},{"cell_type":"markdown","source":"Credit default prediction is central to managing risk in a consumer lending business. Credit default prediction allows lenders to optimize lending decisions, which leads to a better customer experience and sound business economics. Current models exist to help manage risk. But it's possible to create better models that can outperform those currently in use.\n\nIn this notebook, I will use machine learning to predict credit default. I will leverage an industrial scale data set to build a machine learning model that challenges the current model in production. Training, validation, and testing datasets include time-series behavioral data and anonymized customer profile information.\n\nThe objective is to predict the probability that a customer does not pay back their credit card balance amount in the future based on their monthly customer profile. The target binary variable is calculated by observing 18 months performance window after the latest credit card statement, and if the customer does not pay due amount in 120 days after their latest statement date it is considered a default event.\n\nThe dataset contains aggregated profile features for each customer at each statement date. Features are anonymized and normalized, and fall into the following general categories:\n\nD_* = Delinquency variables\nS_* = Spend variables\nP_* = Payment variables\nB_* = Balance variables\nR_* = Risk variables\nwith the following features being categorical:\n\n['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']\n\nThe goal is to predict, for each customer_ID, the probability of a future payment default (target = 1).\n\nNote. The negative class has been subsampled for this dataset at 5%, and thus receives a 20x weighting in the scoring metric.","metadata":{}},{"cell_type":"markdown","source":"### 2. Key Findings","metadata":{}},{"cell_type":"markdown","source":"- 26% of customers in the training data have defaulted. We know that the negative class (good customers) have been subsampled by a factor of 20. Because of the class imbalance, I will use StratifiedKFold for cross validation.\n\n- There is a significant number of missing data. Many decision-tree based algorithms can deal with missing values. With decision-tree models, there is no need to change the missing values. Neural networks can not deal with missing values. So I need to impute values for NNs.\n\n- 80%, of customers have 13 statements. The remaining 20% of the customers have 1 to 12 statements. I will model only the last statement using decision-tree models. For time-series modelling, I will only include customers that have 13 statements.\n\n- According to the competition data description, there are eleven categorical features. I plot histograms for target=0 and target=1. We will see that the distributions are different for target=0 and target=1. This shows that there is information in the categorical variables about the target.","metadata":{}},{"cell_type":"markdown","source":"### 3. Import libraries","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport math\npd.set_option('display.max_columns', None)","metadata":{"execution":{"iopub.status.busy":"2023-10-27T13:36:17.167471Z","iopub.execute_input":"2023-10-27T13:36:17.167978Z","iopub.status.idle":"2023-10-27T13:36:17.174667Z","shell.execute_reply.started":"2023-10-27T13:36:17.167938Z","shell.execute_reply":"2023-10-27T13:36:17.173295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4. Load Data","metadata":{}},{"cell_type":"code","source":"train_labels = pd.read_csv('../input/amex-default-prediction/train_labels.csv')\ntrain_labels.head()","metadata":{"execution":{"iopub.status.busy":"2023-10-25T18:14:46.045985Z","iopub.execute_input":"2023-10-25T18:14:46.046887Z","iopub.status.idle":"2023-10-25T18:14:47.129582Z","shell.execute_reply.started":"2023-10-25T18:14:46.046819Z","shell.execute_reply":"2023-10-25T18:14:47.128241Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The dataset of this competition has a considerable size. If I read the original csv files, the data barely fits into memory. That's why I read the data from @munumbutt's AMEX-Feather-Dataset. In this Feather file, the floating point precision has been reduced from 64 bit to 16 bit. And reading a Feather file is faster than reading a csv file because the Feather file format is binary.","metadata":{}},{"cell_type":"code","source":"train_data = pd.read_feather('../input/amexfeather/train_data.ftr')\ntest_data = pd.read_feather('../input/amexfeather/test_data.ftr')\ntrain_data.head()","metadata":{"execution":{"iopub.status.busy":"2023-10-27T13:36:20.610005Z","iopub.execute_input":"2023-10-27T13:36:20.610454Z","iopub.status.idle":"2023-10-27T13:36:33.812846Z","shell.execute_reply.started":"2023-10-27T13:36:20.610417Z","shell.execute_reply":"2023-10-27T13:36:33.811972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 5. Exploratory Data Analysis","metadata":{}},{"cell_type":"markdown","source":"These are the categorical features based on the compeition description.","metadata":{}},{"cell_type":"code","source":"cat_features = ['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']\n\nprint(f\"There are {train_data.shape[1]-1} features in the data. {len(cat_features)} of them are categorical features.\")","metadata":{"execution":{"iopub.status.busy":"2023-10-27T13:36:33.814383Z","iopub.execute_input":"2023-10-27T13:36:33.815011Z","iopub.status.idle":"2023-10-27T13:36:33.824269Z","shell.execute_reply.started":"2023-10-27T13:36:33.814971Z","shell.execute_reply":"2023-10-27T13:36:33.822372Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"There are {train_data.shape[0]/1000000} million rows in the training data and {test_data.shape[0]/1000000} million rows in the test data.\")\nprint(f\"There are {train_data.customer_ID.nunique()} unique customers in the training data.\")","metadata":{"execution":{"iopub.status.busy":"2023-10-26T20:52:33.781654Z","iopub.execute_input":"2023-10-26T20:52:33.782469Z","iopub.status.idle":"2023-10-26T20:52:34.984238Z","shell.execute_reply.started":"2023-10-26T20:52:33.782413Z","shell.execute_reply":"2023-10-26T20:52:34.982902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"There are {train_labels.shape[0]} customer_IDs in the train_labels.\")\nprint(f\"There are {train_labels.isna().sum().sum()} missing data and {train_labels.customer_ID.duplicated().sum()} duplicated customer_IDs in the train_labels.\")","metadata":{"execution":{"iopub.status.busy":"2023-10-25T18:58:59.900721Z","iopub.execute_input":"2023-10-25T18:58:59.901980Z","iopub.status.idle":"2023-10-25T18:59:00.134052Z","shell.execute_reply.started":"2023-10-25T18:58:59.901932Z","shell.execute_reply":"2023-10-25T18:59:00.132907Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 5.1. Target Variable\n\nThe goal of the competition is to use monthly customer historical data to estimate the probability of default, i.e. failing to pay the credit card bills within the 18 months after the latest statement.\n\nLet's look at the distribution of target variable.\n","metadata":{}},{"cell_type":"code","source":"print(f\"Out of the {train_labels.shape[0]} customers, {train_labels['target'].value_counts().iloc[1]} customers have defaulted.\")","metadata":{"execution":{"iopub.status.busy":"2023-10-25T19:41:01.532976Z","iopub.execute_input":"2023-10-25T19:41:01.533381Z","iopub.status.idle":"2023-10-25T19:41:01.545525Z","shell.execute_reply.started":"2023-10-25T19:41:01.533353Z","shell.execute_reply":"2023-10-25T19:41:01.544175Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.countplot(x=train_labels.target)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-03T12:59:28.909214Z","iopub.execute_input":"2023-10-03T12:59:28.910200Z","iopub.status.idle":"2023-10-03T12:59:29.277098Z","shell.execute_reply.started":"2023-10-03T12:59:28.910161Z","shell.execute_reply":"2023-10-03T12:59:29.275803Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels['target'].value_counts().plot(kind = 'pie',autopct='%1.1f%%', startangle=90)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-03T13:01:30.888062Z","iopub.execute_input":"2023-10-03T13:01:30.888423Z","iopub.status.idle":"2023-10-03T13:01:31.012488Z","shell.execute_reply.started":"2023-10-03T13:01:30.888397Z","shell.execute_reply":"2023-10-03T13:01:31.010864Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Key Insights:\n- 25.9% of the training data has target value of 1 (bad customers). The classes are imbalanced. Balanced accuracy is particularly useful here, since the classes are not represented equally. Regular accuracy may not be an accurate metric to evaluate the performance of a classification model.\n\n- Stratified K-Fold cross-validation is useful for this dataset, since the classes are imbalanced. Stratified K-Fold cross-validation ensures that each fold retains the same class distribution as the original dataset, thus providing a more representative estimate of the model's performance.","metadata":{}},{"cell_type":"markdown","source":"### 5.2. Missing Data Analysis","metadata":{}},{"cell_type":"markdown","source":"There is a significant number of missing data. It is not reasonable to drop all columns or rows that have a missing value.\n\nMany decision-tree based algorithms can deal with missing values. With decision-tree models, I don't need to change the missing values. Neural networks can not deal with missing values. So I need to impute values for NNs.\n\nLet's look at the top 30 columns with the most null values.","metadata":{}},{"cell_type":"code","source":"null= pd.DataFrame(train_data.isnull().sum(),columns=['number_of_nulls'])\nnull['percentage_of_null'] = round(((null['number_of_nulls']/len(train_data))*100) , 2)\nnull = null[null['number_of_nulls']>0]\nnull= null.sort_values(by='percentage_of_null',ascending=False)\nnull.head(30)","metadata":{"execution":{"iopub.status.busy":"2023-10-26T04:49:33.295087Z","iopub.execute_input":"2023-10-26T04:49:33.297871Z","iopub.status.idle":"2023-10-26T04:49:38.923267Z","shell.execute_reply.started":"2023-10-26T04:49:33.297757Z","shell.execute_reply":"2023-10-26T04:49:38.921931Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Most of the top null columns are delinquency features.","metadata":{}},{"cell_type":"code","source":"fig = plt.figure(figsize=(15,30))\nsns.barplot(data=null,y=null.index,x=null['percentage_of_null'])\ndel null","metadata":{"execution":{"iopub.status.busy":"2023-10-03T13:11:46.839301Z","iopub.execute_input":"2023-10-03T13:11:46.839770Z","iopub.status.idle":"2023-10-03T13:11:48.306528Z","shell.execute_reply.started":"2023-10-03T13:11:46.839735Z","shell.execute_reply":"2023-10-03T13:11:48.305387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Key Insights\n\n- There are many missing values, especially in the delinquency variables, with D_87 having the highest number of nulls (99.93%).\n\n- Decision-tree models are attractive to use for this dataset, since they handle null values.","metadata":{}},{"cell_type":"markdown","source":"### 5.3. Customer Profiles","metadata":{}},{"cell_type":"markdown","source":"Let's check the number of credit card statements over time. We see that statement dates range from March 2017 to March 2018 in the train data and from April 2018 to October 2019 in the test data.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(15,3))\ntrain_data.groupby(\"S_2\")['customer_ID'].count().plot()\nplt.title(\"Number of the Issued Customer Statements (Train Data)\")\nplt.xlabel('Date')\nplt.show()\n\nfig2, ax2 = plt.subplots(figsize=(15,3))\ntest_data.groupby(\"S_2\")['customer_ID'].count().plot()\nplt.xlabel('Date')\nplt.title(\"Number of the Issued Customer Statements (Test Data)\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-26T04:56:47.749289Z","iopub.execute_input":"2023-10-26T04:56:47.749718Z","iopub.status.idle":"2023-10-26T04:56:50.835774Z","shell.execute_reply.started":"2023-10-26T04:56:47.749684Z","shell.execute_reply":"2023-10-26T04:56:50.834302Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets count how many credit card statements there are per customer. Most customers have 13 statements and others have 1 to 12 statements.\nThis means that the model has to handle a variable number of statements per customer.","metadata":{}},{"cell_type":"code","source":"fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 7))\ntrain_data_value_counts = train_data.customer_ID.value_counts().value_counts().sort_index(ascending=False)\ntest_data_value_counts = test_data.customer_ID.value_counts().value_counts().sort_index(ascending=False)\n\nax1.pie(train_data_value_counts, labels=train_data_value_counts.index)\nax1.set_title('Num of Statements per Customer (Train Data)')\n\nax2.pie(test_data_value_counts, labels=test_data_value_counts.index)\nax2.set_title('Num of Statements per Customer (Test Data)')","metadata":{"execution":{"iopub.status.busy":"2023-10-03T13:25:31.205567Z","iopub.execute_input":"2023-10-03T13:25:31.206085Z","iopub.status.idle":"2023-10-03T13:25:33.517027Z","shell.execute_reply.started":"2023-10-03T13:25:31.206052Z","shell.execute_reply":"2023-10-03T13:25:33.515804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's see when customers received their last statements. The histogram of the last statement dates shows that every train customer got his last statement in March of 2018. Half of the test customer got their last statement in April 2019 and half in October 2019.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(14, 3))\nplt.hist(train_data.S_2.groupby(train_data.customer_ID).max(), bins=pd.date_range(\"2018-03-01\", \"2018-04-01\", freq=\"d\"))\nplt.title('Last Statement Date (Train Data)', fontsize=12)\nplt.xlabel('Date')\nplt.show()\n\nplt.figure(figsize=(14, 3))\nplt.hist(test_data.S_2.groupby(test_data.customer_ID).max(), bins=pd.date_range(\"2019-04-01\", \"2019-11-01\", freq=\"d\"))\nplt.title('Last Statement Date (Test Data)', fontsize=12)\nplt.xlabel('Date')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-26T05:19:19.411596Z","iopub.execute_input":"2023-10-26T05:19:19.412059Z","iopub.status.idle":"2023-10-26T05:19:26.128866Z","shell.execute_reply.started":"2023-10-26T05:19:19.412013Z","shell.execute_reply":"2023-10-26T05:19:26.127492Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There is no overlap between the dates in the train and the test data.","metadata":{}},{"cell_type":"code","source":"train_data['S_2'] = pd.to_datetime(train_data['S_2'])\ntest_data['S_2'] = pd.to_datetime(test_data['S_2'])\n\nprint(f\"Train Data Min Date: {train_data['S_2'].min()}, Max Date: {train_data['S_2'].max()}\")\nprint(f\"Train Data Min Date: {test_data['S_2'].min()}, Max Date: {test_data['S_2'].max()}\")","metadata":{"execution":{"iopub.status.busy":"2023-10-03T13:33:34.909393Z","iopub.execute_input":"2023-10-03T13:33:34.909835Z","iopub.status.idle":"2023-10-03T13:33:35.446525Z","shell.execute_reply.started":"2023-10-03T13:33:34.909801Z","shell.execute_reply":"2023-10-03T13:33:35.445260Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The following histogram plot shows that the first and last statement for most customers is about a year apart. Most customers have 13 statements. This indicates that the customers usually get one statement per month.\n","metadata":{}},{"cell_type":"code","source":"temp = train_data.S_2.groupby(train_data.customer_ID).agg(['max', 'min'])\nplt.figure(figsize=(14, 3))\nplt.hist((temp['max'] - temp['min']).dt.days, bins=400)\nplt.xlabel('Number of Days')\nplt.title('Number of Days Between the First and the Last Credit Card Statement Received (Train Data)')\nplt.show()\n\ntemp = test_data.S_2.groupby(test_data.customer_ID).agg(['max', 'min'])\nplt.figure(figsize=(14, 3))\nplt.hist((temp['max'] - temp['min']).dt.days, bins=400)\nplt.xlabel('Number of Days')\nplt.title('Number of Days Between the First and the Last Credit Card Statement Received (Test Data)')\nplt.show()\ndel temp","metadata":{"execution":{"iopub.status.busy":"2023-10-03T13:38:38.200312Z","iopub.execute_input":"2023-10-03T13:38:38.200684Z","iopub.status.idle":"2023-10-03T13:38:45.242396Z","shell.execute_reply.started":"2023-10-03T13:38:38.200657Z","shell.execute_reply":"2023-10-03T13:38:45.241120Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Key Insights:\n\n- The model will have to handle a variable-sized number of statements per customer, unless we only model the most recent statement or take an average.\n\n- The data is a time series, however all training happens in the same month. This makes it harder to cross-validate.","metadata":{}},{"cell_type":"markdown","source":"### 5.4 Visualization of Categorical Features","metadata":{}},{"cell_type":"markdown","source":"There are eleven categorical features. I will plot histograms for target=0 and target=1.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(16, 16))\nfor i, f in enumerate(cat_features):\n    plt.subplot(4, 3, i+1)\n    temp = pd.DataFrame(train_data[f][train_data.target == 0].value_counts(dropna=False, normalize=True).sort_index().rename('count'))\n    temp.index.name = 'value'\n    temp.reset_index(inplace=True)\n    plt.bar(temp.index, temp['count'], alpha=0.5, label='target=0')\n    temp = pd.DataFrame(train_data[f][train_data.target == 1].value_counts(dropna=False, normalize=True).sort_index().rename('count'))\n    temp.index.name = 'value'\n    temp.reset_index(inplace=True)\n    plt.bar(temp.index, temp['count'], alpha=0.5, label='target=1')\n    plt.xlabel(f)\n    plt.legend()\n    plt.xticks(temp.index, temp.value)\nplt.suptitle('Categorical Features', fontsize=20, y=0.93)\nplt.show()\ndel temp\n","metadata":{"execution":{"iopub.status.busy":"2023-10-03T13:41:08.249776Z","iopub.execute_input":"2023-10-03T13:41:08.250301Z","iopub.status.idle":"2023-10-03T13:41:11.252191Z","shell.execute_reply.started":"2023-10-03T13:41:08.250266Z","shell.execute_reply":"2023-10-03T13:41:11.251113Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Null values in the categorical features\n\ntrain_data[cat_features].isnull().sum().sort_values(ascending=False)","metadata":{"execution":{"iopub.status.busy":"2023-09-28T23:00:10.864574Z","iopub.execute_input":"2023-09-28T23:00:10.864967Z","iopub.status.idle":"2023-09-28T23:00:10.922159Z","shell.execute_reply.started":"2023-09-28T23:00:10.864937Z","shell.execute_reply":"2023-09-28T23:00:10.920661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Key Insights:\n\n- Every categorical feature has at most eight categories. It is possible to use One-hot encoding. I will explore one-hot encoding and ordical enocding in my modelling notebook.\n\n- The distributions for target=0 and target=1 are different. This indicates that the categorical features gives some information about the target. It is worth exploring modelling the categorical variables.\n\n- The following features are binary (they only take values of 0, 1 or missing): D_114, D_116, D_120, D_66.\n","metadata":{}},{"cell_type":"markdown","source":"### 5.5. Visualization of Numerical Features","metadata":{}},{"cell_type":"markdown","source":"### 5.5.1. Delinquency Variables","metadata":{}},{"cell_type":"code","source":"print(f\"Number of delinquency features: {len([f for f in train_data.columns if (f not in  ['customer_ID', 'target', 'S_2']) and (f.startswith(('D')))])}\")\n\ndelin_features = sorted([f for f in train_data.columns if (f not in cat_features + ['customer_ID', 'target', 'S_2']) and (f.startswith(('D')))])\nprint(f\"Number of numerical delinquency features: {len(delin_features)}\")","metadata":{"execution":{"iopub.status.busy":"2023-10-03T22:05:41.158948Z","iopub.execute_input":"2023-10-03T22:05:41.159328Z","iopub.status.idle":"2023-10-03T22:05:41.167204Z","shell.execute_reply.started":"2023-10-03T22:05:41.159298Z","shell.execute_reply":"2023-10-03T22:05:41.166017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_hist(features, title):\n    fig, axs = plt.subplots(int(len(features)/4)+(1 if len(features)%4!=0 else 0),4, figsize=(15,3*(int(len(features)/4)+(1 if len(features)%4!=0 else 0))))\n\n    plt.suptitle(f'\\n{title}', fontsize=20, y=1)\n\n    for i, f in enumerate(features):\n            ax = axs[int(i/4), i%4] if ((int(len(features)/4)+(1 if len(features)%4!=0 else 0))>1) else axs[i%4]\n            ax.hist(train_data[f], bins=200)\n            ax.set_title(f)\n\n            # Create a smaller inset subplot within the larger subplot\n            inset_ax = ax.inset_axes([0.6, 0.6, 0.35, 0.35])  # [left, bottom, width, height]\n            inset_ax.hist(train_data[f], bins=200)\n            inset_ax.set_ylim([0,0.001])\n            inset_ax.set_yticks([])\n            ax.set_yticklabels([])\n\n    plt.tight_layout()\n    plt.show()                ","metadata":{"execution":{"iopub.status.busy":"2023-10-27T13:30:45.890534Z","iopub.execute_input":"2023-10-27T13:30:45.890877Z","iopub.status.idle":"2023-10-27T13:30:45.903371Z","shell.execute_reply.started":"2023-10-27T13:30:45.890848Z","shell.execute_reply":"2023-10-27T13:30:45.902296Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_hist(delin_features, 'Delinquency Features')","metadata":{"execution":{"iopub.status.busy":"2023-10-03T22:05:48.163069Z","iopub.execute_input":"2023-10-03T22:05:48.163701Z","iopub.status.idle":"2023-10-03T22:08:29.683666Z","shell.execute_reply.started":"2023-10-03T22:05:48.163661Z","shell.execute_reply":"2023-10-03T22:08:29.682805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Which features are discrete?\n\nfor f in delin_features:\n    if(train_data[f].nunique()<15):\n        print(f, train_data[f].nunique())","metadata":{"execution":{"iopub.status.busy":"2023-10-03T16:36:06.749933Z","iopub.execute_input":"2023-10-03T16:36:06.750918Z","iopub.status.idle":"2023-10-03T16:36:16.011531Z","shell.execute_reply.started":"2023-10-03T16:36:06.750869Z","shell.execute_reply":"2023-10-03T16:36:16.010256Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data['D_103'].nunique()","metadata":{"execution":{"iopub.status.busy":"2023-09-28T23:08:10.493039Z","iopub.execute_input":"2023-09-28T23:08:10.493455Z","iopub.status.idle":"2023-09-28T23:08:10.593498Z","shell.execute_reply.started":"2023-09-28T23:08:10.493418Z","shell.execute_reply":"2023-09-28T23:08:10.592452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Based on the histogram plots, some of features such as D_103 appear to be discrete at first. But when I zoom in on the plots, there seems to be an added noise on top of the values. For example, D-103 takes the values of [0,0.01] and [1,1.01]. This could be due to adding noise to the data to anonymize customer data by AMEX.","metadata":{}},{"cell_type":"code","source":"fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14,6))\nax1.hist(train_data['D_103'], bins=200)\nax1.set_xlim([-0.01,0.02])\nax1.set_ylim([0,1e-3])\nax1.set_xlabel('D_103')\n\nax2.hist(train_data['D_103'], bins=200)\nax2.set_xlim([0.99,1.02])\nax2.set_ylim([0,1e-3])\nax2.set_xlabel('D_103')","metadata":{"execution":{"iopub.status.busy":"2023-10-03T16:47:57.668110Z","iopub.execute_input":"2023-10-03T16:47:57.668498Z","iopub.status.idle":"2023-10-03T16:47:59.863307Z","shell.execute_reply.started":"2023-10-03T16:47:57.668470Z","shell.execute_reply":"2023-10-03T16:47:59.862165Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Key Insights:\n\n- Some of the plots have white space at the left or right end. This may show that there aer some outliers or rare events in the data.\n\n- It is possible to remove this added noise. This will likely result in higher model accuracy. We can use a transformatiom such as: \ntrain_data['D_103'] = train_data['D_103'].apply(lambda t: np.floor(t))","metadata":{}},{"cell_type":"markdown","source":"### Distribution of Delinquency Variables (Last Credit Card Statement)\n\nLet's explore the kernel density estimate (KDE) plot for the delinquency variables. KDE plot visuales the distribution of data similar to a histogram, but uses a continuous probability distribution. KDE plots apply a kernel smoothing function to create a smooth probability density.\n\nDue to the large amount of data, I was running out of memory trying to create KDE plots. So I selected the latest statement for each customer. ","metadata":{}},{"cell_type":"code","source":"last_statement_df = train_data.groupby('customer_ID').tail(1).set_index('customer_ID')","metadata":{"execution":{"iopub.status.busy":"2023-10-27T13:37:17.034915Z","iopub.execute_input":"2023-10-27T13:37:17.036616Z","iopub.status.idle":"2023-10-27T13:37:24.243653Z","shell.execute_reply.started":"2023-10-27T13:37:17.036551Z","shell.execute_reply":"2023-10-27T13:37:24.240133Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_kde_last_statement(features, title):\n    fig, axs = plt.subplots(int(len(features)/5)+(1 if len(features)%5!=0 else 0),5, figsize=(16,3*(int(len(features)/5)+(1 if len(features)%5!=0 else 0))))\n    fig.suptitle(f'{title}\\n\\n',fontsize=16)\n    \n    for i, f in enumerate(cols[:-1]):\n        ax = axs[int(i/5), i%5] if ((int(len(features)/5)+(1 if len(features)%5!=0 else 0))>1) else axs[i%5]\n        sns.kdeplot(x=f, hue='target', label=['Default','Paid'], data=last_statement_df[features], fill=True, linewidth=2, legend=False, ax=ax)\n        ax.set_ylabel('Density' if i%5==0 else '')\n\n    handles, _ = axs[0,0].get_legend_handles_labels() if ((int(len(features)/5)+(1 if len(features)%5!=0 else 0))>1) else axs[0].get_legend_handles_labels()\n    fig.legend(labels=['Default','Paid'], handles=reversed(handles), ncol=2, bbox_to_anchor=(0.18, 0.983))\n    sns.despine(bottom=True, trim=True)\n\n    plt.tight_layout(rect=[0, 0.2, 1, 0.99])\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-27T13:43:00.725986Z","iopub.execute_input":"2023-10-27T13:43:00.727347Z","iopub.status.idle":"2023-10-27T13:43:00.740991Z","shell.execute_reply.started":"2023-10-27T13:43:00.727295Z","shell.execute_reply":"2023-10-27T13:43:00.739853Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip uninstall seaborn -y\n!pip install seaborn==0.11.2","metadata":{"execution":{"iopub.status.busy":"2023-10-27T13:34:58.138696Z","iopub.execute_input":"2023-10-27T13:34:58.139447Z","iopub.status.idle":"2023-10-27T13:35:17.055425Z","shell.execute_reply.started":"2023-10-27T13:34:58.139397Z","shell.execute_reply":"2023-10-27T13:35:17.053741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import seaborn as sns\nsns.__version__","metadata":{"execution":{"iopub.status.busy":"2023-10-27T13:36:03.875630Z","iopub.execute_input":"2023-10-27T13:36:03.876832Z","iopub.status.idle":"2023-10-27T13:36:04.539278Z","shell.execute_reply.started":"2023-10-27T13:36:03.876782Z","shell.execute_reply":"2023-10-27T13:36:04.537902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cols=[col for col in last_statement_df.columns if (col.startswith(('D','t'))) & (col not in cat_features)]\nplot_kde_last_statement(cols, 'Distribution of Delinquency Features (Last Statement)')","metadata":{"execution":{"iopub.status.busy":"2023-10-26T18:19:59.001092Z","iopub.execute_input":"2023-10-26T18:19:59.001543Z","iopub.status.idle":"2023-10-26T18:22:40.966597Z","shell.execute_reply.started":"2023-10-26T18:19:59.001509Z","shell.execute_reply":"2023-10-26T18:22:40.965247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Key Insights:\n\n- If the two distributions (default vs. paid) are very separate for a  feature, it shows that the feature could have useful information about the target and should be used in predictive modelling. \n\n- For some of the features such as D_134, D_121, etc., the distribution for bad customers is similar in shape to the distribution of good customers, but the distribution is shifted up or down in the y axis. This may show similar overall behavioral pattern for good and bad customers. The shift in the scale of the distribution could show that the scale of the behavior (such as amout spent) is different for the two groups.","metadata":{}},{"cell_type":"markdown","source":"### Correlation Between Delinquency Features","metadata":{}},{"cell_type":"markdown","source":"Let's look at the correlation plot for the delinquency features. Correlation heatmaps visualize the relationship between numerical variables. They help understand which features are related to each other and the strength of this relationship.","metadata":{}},{"cell_type":"code","source":"def plot_corr_heatmap(features, title):\n   \n    corr=last_statement_df[features].iloc[:,:-1].corr()\n\n    mask=np.triu(np.ones_like(corr, dtype=bool))[1:,:-1]\n    corr=corr.iloc[1:,:-1].copy()\n\n    fig, ax = plt.subplots(figsize=(48,48))   \n    sns.heatmap(corr, mask=mask, vmin=-1, vmax=1, center=0, annot=True, fmt='.2f', cmap='inferno', annot_kws={'fontsize':10,'fontweight':'bold'}, cbar=False)\n\n    ax.tick_params(left=False,bottom=False)\n    ax.set_xticklabels(ax.get_xticklabels(), rotation=45, horizontalalignment='right',fontsize=12)\n    ax.set_yticklabels(ax.get_yticklabels(), fontsize=12)\n\n    plt.title(f'{title}\\n', fontsize=26)\n    fig.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-10-27T13:37:08.640109Z","iopub.execute_input":"2023-10-27T13:37:08.640512Z","iopub.status.idle":"2023-10-27T13:37:08.649806Z","shell.execute_reply.started":"2023-10-27T13:37:08.640479Z","shell.execute_reply":"2023-10-27T13:37:08.648434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_corr_heatmap(cols, 'Correlation Between Delinquency Variables')","metadata":{"execution":{"iopub.status.busy":"2023-10-03T20:17:00.506797Z","iopub.execute_input":"2023-10-03T20:17:00.507312Z","iopub.status.idle":"2023-10-03T20:17:15.979097Z","shell.execute_reply.started":"2023-10-03T20:17:00.507275Z","shell.execute_reply":"2023-10-03T20:17:15.975493Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Some of the delinquency variables are highly correlated:","metadata":{}},{"cell_type":"code","source":"cols=[col for col in last_statement_df.columns if (col.startswith(('D','t'))) & (col not in cat_features)]\ncorr=last_statement_df[cols].iloc[:,:-1].corr()\nnp.fill_diagonal(corr.values, 0.0)\nmask = np.triu(np.ones_like(corr, dtype=bool))\ncorr = corr.mask(mask)\nhighly_correlated_features = corr[np.abs(corr) > 0.9].stack().reset_index()\nhighly_correlated_features=highly_correlated_features.iloc[abs(highly_correlated_features[0]).argsort()[::-1]].reset_index()\nhighly_correlated_features","metadata":{"execution":{"iopub.status.busy":"2023-10-26T21:55:38.966592Z","iopub.execute_input":"2023-10-26T21:55:38.967377Z","iopub.status.idle":"2023-10-26T21:55:48.472785Z","shell.execute_reply.started":"2023-10-26T21:55:38.967336Z","shell.execute_reply":"2023-10-26T21:55:48.471643Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_highly_correlated_featuers(features, title, highly_correlated_features):\n    \n    fig, axs = plt.subplots(int(len(highly_correlated_features)/4)+(1 if len(highly_correlated_features)%4!=0 else 0),4, figsize=(16,5*(int(len(highly_correlated_features)/4)+(1 if len(highly_correlated_features)%4!=0 else 0))))\n    i=0\n    for f,g in zip(highly_correlated_features.level_0, highly_correlated_features.level_1):\n\n        \n        ax = axs[int(i/4), i%4] if ((int(len(highly_correlated_features)/4)+(1 if len(highly_correlated_features)%4!=0 else 0))>1) else axs[i%4]\n                \n        if(i==7):\n            ax.plot(last_statement_df[features][f], last_statement_df[features][g], '.')\n            ax.set(xlabel=f,ylabel=g)\n            ax.text(0.05, 0.9, 'Correlation: {:.4f}'.format(highly_correlated_features[0][i]), transform=ax.transAxes, bbox=dict(boxstyle=\"round,pad=0.3\",fc=\"white\"))\n            ax.tick_params(left=False,bottom=False)\n\n            i = i + 1\n            continue\n        \n\n\n        ax.hexbin(x=f, y=g, data=last_statement_df[features], bins='log', gridsize=40, cmap='inferno')\n        ax.set(xlabel=f,ylabel=g)\n\n        ax.text(0.05, 0.9, 'Correlation: {:.4f}'.format(highly_correlated_features[0][i]), transform=ax.transAxes, bbox=dict(boxstyle=\"round,pad=0.3\",fc=\"white\"))\n        ax.tick_params(left=False,bottom=False)\n        i = i + 1\n\n    sns.despine()\n    fig.suptitle(f'\\n{title})',fontsize=14)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-26T22:18:56.927861Z","iopub.execute_input":"2023-10-26T22:18:56.928683Z","iopub.status.idle":"2023-10-26T22:18:56.943097Z","shell.execute_reply.started":"2023-10-26T22:18:56.928644Z","shell.execute_reply":"2023-10-26T22:18:56.941661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_highly_correlated_featuers(features, title, highly_correlated_features):\n    \n    fig, axs = plt.subplots(int(len(highly_correlated_features)/4)+(1 if len(highly_correlated_features)%4!=0 else 0),4, figsize=(16,5*(int(len(highly_correlated_features)/4)+(1 if len(highly_correlated_features)%4!=0 else 0))))\n    i=0\n    for f,g in zip(highly_correlated_features.level_0, highly_correlated_features.level_1):\n\n        \n        ax = axs[int(i/4), i%4] if ((int(len(highly_correlated_features)/4)+(1 if len(highly_correlated_features)%4!=0 else 0))>1) else axs[i%4]\n        \n\n\n        ax.hexbin(x=f, y=g, data=last_statement_df[features], bins='log', gridsize=40, cmap='inferno')\n        ax.set(xlabel=f,ylabel=g)\n\n        ax.text(0.05, 0.9, 'Correlation: {:.4f}'.format(highly_correlated_features[0][i]), transform=ax.transAxes, bbox=dict(boxstyle=\"round,pad=0.3\",fc=\"white\"))\n        ax.tick_params(left=False,bottom=False)\n        i = i + 1\n\n    sns.despine()\n    fig.suptitle(f'\\n{title})',fontsize=14)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-27T13:36:59.607722Z","iopub.execute_input":"2023-10-27T13:36:59.608132Z","iopub.status.idle":"2023-10-27T13:36:59.621767Z","shell.execute_reply.started":"2023-10-27T13:36:59.608101Z","shell.execute_reply":"2023-10-27T13:36:59.619528Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_highly_correlated_featuers(cols, 'Most Highly-Correlated Delinquency Variables (Log Transformed Relationship)', highly_correlated_features)","metadata":{"execution":{"iopub.status.busy":"2023-10-26T22:18:57.283334Z","iopub.execute_input":"2023-10-26T22:18:57.283727Z","iopub.status.idle":"2023-10-26T22:19:03.313914Z","shell.execute_reply.started":"2023-10-26T22:18:57.283697Z","shell.execute_reply":"2023-10-26T22:19:03.312218Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Key Insights: \n\n- Τhere are several highly correlated Delinquency variables, with the highest correlation being 0.999824, between D_77 and D_62.\n\n- There is some missing correlations in the correlation heatmap. This is due to null values in the data.","metadata":{}},{"cell_type":"markdown","source":"### 5.5.2. Spend Variables","metadata":{}},{"cell_type":"markdown","source":"There are 21 Spend variables, all of which are numerical features. Let's look at histograms, KDE plots and correlation heatmaps for the spend features.","metadata":{}},{"cell_type":"code","source":"spend_features = sorted([f for f in train_data.columns if (f not in ['customer_ID', 'target', 'S_2']) and (f.startswith(('S')))])","metadata":{"execution":{"iopub.status.busy":"2023-10-27T13:48:51.330078Z","iopub.execute_input":"2023-10-27T13:48:51.330483Z","iopub.status.idle":"2023-10-27T13:48:51.336431Z","shell.execute_reply.started":"2023-10-27T13:48:51.330455Z","shell.execute_reply":"2023-10-27T13:48:51.335217Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"Number of spend features: {len(spend_features)}\")\nprint(f\"Number of numerical spend features: {len([f for f in spend_features if f not in cat_features])}\")","metadata":{"execution":{"iopub.status.busy":"2023-10-27T13:49:33.618286Z","iopub.execute_input":"2023-10-27T13:49:33.618687Z","iopub.status.idle":"2023-10-27T13:49:33.624948Z","shell.execute_reply.started":"2023-10-27T13:49:33.618657Z","shell.execute_reply":"2023-10-27T13:49:33.623489Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are no categorical spend features.","metadata":{}},{"cell_type":"code","source":"plot_hist(spend_features, 'Spend Features')","metadata":{"execution":{"iopub.status.busy":"2023-10-27T13:30:45.978840Z","iopub.execute_input":"2023-10-27T13:30:45.979364Z","iopub.status.idle":"2023-10-27T13:31:33.411881Z","shell.execute_reply.started":"2023-10-27T13:30:45.979316Z","shell.execute_reply":"2023-10-27T13:31:33.410507Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cols=[col for col in last_statement_df.columns if (col.startswith(('S','t'))) & (col not in cat_features+['S_2'])]\nplot_kde_last_statement(cols, 'Distribution of Spend Features (Last Statement)')","metadata":{"execution":{"iopub.status.busy":"2023-10-27T13:37:33.665453Z","iopub.execute_input":"2023-10-27T13:37:33.665871Z","iopub.status.idle":"2023-10-27T13:38:26.818448Z","shell.execute_reply.started":"2023-10-27T13:37:33.665841Z","shell.execute_reply":"2023-10-27T13:38:26.817365Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_corr_heatmap(cols, 'Correlations Between Spend Variables')","metadata":{"execution":{"iopub.status.busy":"2023-10-03T20:32:27.819814Z","iopub.execute_input":"2023-10-03T20:32:27.820143Z","iopub.status.idle":"2023-10-03T20:32:29.670253Z","shell.execute_reply.started":"2023-10-03T20:32:27.820117Z","shell.execute_reply":"2023-10-03T20:32:29.669036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"corr=last_statement_df[cols].iloc[:,:-1].corr()\nnp.fill_diagonal(corr.values, 0.0)\nmask = np.triu(np.ones_like(corr, dtype=bool))\ncorr = corr.mask(mask)\nhighly_correlated_features = corr[np.abs(corr) > 0.9].stack().reset_index()\nhighly_correlated_features=highly_correlated_features.iloc[abs(highly_correlated_features[0]).argsort()[::-1]].reset_index()\nhighly_correlated_features","metadata":{"execution":{"iopub.status.busy":"2023-10-27T13:44:17.735961Z","iopub.execute_input":"2023-10-27T13:44:17.736977Z","iopub.status.idle":"2023-10-27T13:44:18.520656Z","shell.execute_reply.started":"2023-10-27T13:44:17.736911Z","shell.execute_reply":"2023-10-27T13:44:18.519465Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_highly_correlated_featuers(cols, 'Most Highly-Correlated Spend Variables (Log Transformed Relationship)', highly_correlated_features)","metadata":{"execution":{"iopub.status.busy":"2023-10-27T13:44:47.024386Z","iopub.execute_input":"2023-10-27T13:44:47.024820Z","iopub.status.idle":"2023-10-27T13:44:47.910625Z","shell.execute_reply.started":"2023-10-27T13:44:47.024786Z","shell.execute_reply":"2023-10-27T13:44:47.909509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Key Insights:\n\n- From the KDE plots, the distributions for target=0 and target=1 differ. The spend variables seem informative about the target and should be included in modelling.\n\n- Some of the spend features are highly correlated to eachother, with correlation between S_22 and S_24 being the highest at 0.965. Keep in mind that this is the Pearson's correlation coefficien. It only measures  linear relationships. Nonlinear relationships might not be well-represented by this coefficient.\n","metadata":{}},{"cell_type":"markdown","source":"### 5.5.3. Payment Variables","metadata":{}},{"cell_type":"code","source":"payment_features = sorted([f for f in train_data.columns if (f not in ['customer_ID', 'target', 'S_2']) and (f.startswith(('P')))])\nprint(f\"Number of payment features: {len(payment_features)}\")\nprint(f\"Number of numerical payment features: {len([f for f in payment_features if f not in cat_features])}\")","metadata":{"execution":{"iopub.status.busy":"2023-10-27T13:48:33.127343Z","iopub.execute_input":"2023-10-27T13:48:33.128228Z","iopub.status.idle":"2023-10-27T13:48:33.136489Z","shell.execute_reply.started":"2023-10-27T13:48:33.128184Z","shell.execute_reply":"2023-10-27T13:48:33.135188Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_hist(payment_features, 'Payment Features')","metadata":{"execution":{"iopub.status.busy":"2023-10-03T21:03:25.269241Z","iopub.execute_input":"2023-10-03T21:03:25.269870Z","iopub.status.idle":"2023-10-03T21:03:30.260868Z","shell.execute_reply.started":"2023-10-03T21:03:25.269835Z","shell.execute_reply":"2023-10-03T21:03:30.259828Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cols=[col for col in last_statement_df.columns if (col.startswith(('P','t'))) & (col not in cat_features)]\nplot_kde_last_statement(cols, 'Distribution of Payment Features (Last Statement)')","metadata":{"execution":{"iopub.status.busy":"2023-10-27T13:43:11.173418Z","iopub.execute_input":"2023-10-27T13:43:11.173807Z","iopub.status.idle":"2023-10-27T13:43:18.803199Z","shell.execute_reply.started":"2023-10-27T13:43:11.173778Z","shell.execute_reply":"2023-10-27T13:43:18.801987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_corr_heatmap(cols, 'Correlations Between Payment Variables')","metadata":{"execution":{"iopub.status.busy":"2023-10-03T21:03:36.750534Z","iopub.execute_input":"2023-10-03T21:03:36.750882Z","iopub.status.idle":"2023-10-03T21:03:37.561324Z","shell.execute_reply.started":"2023-10-03T21:03:36.750845Z","shell.execute_reply":"2023-10-03T21:03:37.560157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"corr=last_statement_df[cols].iloc[:,:-1].corr()\ncorr.values","metadata":{"execution":{"iopub.status.busy":"2023-10-03T20:47:16.779887Z","iopub.execute_input":"2023-10-03T20:47:16.780622Z","iopub.status.idle":"2023-10-03T20:47:16.807001Z","shell.execute_reply.started":"2023-10-03T20:47:16.780598Z","shell.execute_reply":"2023-10-03T20:47:16.806378Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Spend Variables are not as highly correlated to each other.","metadata":{}},{"cell_type":"markdown","source":"### Key Insights:\n\n- There are only three payment variables. They are not highly correlated to each other.\n\n- Fro mthe KDE plot, P_2 seems to be the most informative about the target.","metadata":{}},{"cell_type":"markdown","source":"### 5.5.4. Balance Variables","metadata":{}},{"cell_type":"code","source":"balance_features = sorted([f for f in train_data.columns if (f not in  ['customer_ID', 'target', 'S_2']) and (f.startswith(('B')))])\nprint(f\"Number of balance features: {len(balance_features)}\")\nprint(f\"Number of numerical balance features: {len([f for f in balance_features if f not in cat_features])}\")","metadata":{"execution":{"iopub.status.busy":"2023-10-27T13:52:36.202122Z","iopub.execute_input":"2023-10-27T13:52:36.203333Z","iopub.status.idle":"2023-10-27T13:52:36.211134Z","shell.execute_reply.started":"2023-10-27T13:52:36.203291Z","shell.execute_reply":"2023-10-27T13:52:36.209734Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_hist(balance_features, 'Balance Features')","metadata":{"execution":{"iopub.status.busy":"2023-10-03T20:49:32.938673Z","iopub.execute_input":"2023-10-03T20:49:32.939100Z","iopub.status.idle":"2023-10-03T20:50:33.492746Z","shell.execute_reply.started":"2023-10-03T20:49:32.939067Z","shell.execute_reply":"2023-10-03T20:50:33.490548Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cols=[col for col in last_statement_df.columns if (col.startswith(('B','t'))) & (col not in cat_features)]\nplot_kde_last_statement(cols, 'Distribution of Balance Features (Last Statement)')","metadata":{"execution":{"iopub.status.busy":"2023-10-27T13:52:41.083876Z","iopub.execute_input":"2023-10-27T13:52:41.084305Z","iopub.status.idle":"2023-10-27T13:54:09.468632Z","shell.execute_reply.started":"2023-10-27T13:52:41.084275Z","shell.execute_reply":"2023-10-27T13:54:09.467661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_corr_heatmap(cols, 'Correlations Between Balance Variables')","metadata":{"execution":{"iopub.status.busy":"2023-10-03T20:52:04.644847Z","iopub.execute_input":"2023-10-03T20:52:04.645260Z","iopub.status.idle":"2023-10-03T20:52:08.461712Z","shell.execute_reply.started":"2023-10-03T20:52:04.645231Z","shell.execute_reply":"2023-10-03T20:52:08.460536Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"corr=last_statement_df[cols].iloc[:,:-1].corr()\nnp.fill_diagonal(corr.values, 0.0)\nmask = np.triu(np.ones_like(corr, dtype=bool))\ncorr = corr.mask(mask)\nhighly_correlated_features = corr[np.abs(corr) > 0.9].stack().reset_index()\nhighly_correlated_features=highly_correlated_features.iloc[abs(highly_correlated_features[0]).argsort()[::-1]].reset_index()\nhighly_correlated_features","metadata":{"execution":{"iopub.status.busy":"2023-10-27T13:54:30.385103Z","iopub.execute_input":"2023-10-27T13:54:30.385544Z","iopub.status.idle":"2023-10-27T13:54:32.447122Z","shell.execute_reply.started":"2023-10-27T13:54:30.385507Z","shell.execute_reply":"2023-10-27T13:54:32.445962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_highly_correlated_featuers(cols, 'Most Highly-Correlated Balance Variables (Log Transformed Relationship)', highly_correlated_features)","metadata":{"execution":{"iopub.status.busy":"2023-10-27T13:54:55.105487Z","iopub.execute_input":"2023-10-27T13:54:55.105866Z","iopub.status.idle":"2023-10-27T13:54:58.128307Z","shell.execute_reply.started":"2023-10-27T13:54:55.105837Z","shell.execute_reply":"2023-10-27T13:54:58.126983Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Key Insights:\n\n\n- There is 38 numerical balance features and 2 categorical balance features, some of which are highly correlated to eachother.\n\n- They seem to have good amount of information about the target and will be included in the modelling.\n\n- Based on the density plots, some of the variables seem to have outliers.","metadata":{}},{"cell_type":"markdown","source":"### 5.5.5. Risk Variables","metadata":{}},{"cell_type":"code","source":"risk_features = sorted([f for f in train_data.columns if (f not in ['customer_ID', 'target', 'S_2']) and (f.startswith(('R')))])\nprint(f\"Number of risk features: {len(risk_features)}\")\nprint(f\"Number of numerical risk features: {len([f for f in risk_features if f not in cat_features])}\")","metadata":{"execution":{"iopub.status.busy":"2023-10-27T13:59:15.915506Z","iopub.execute_input":"2023-10-27T13:59:15.915960Z","iopub.status.idle":"2023-10-27T13:59:15.923412Z","shell.execute_reply.started":"2023-10-27T13:59:15.915914Z","shell.execute_reply":"2023-10-27T13:59:15.921863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_hist(risk_features, 'Risk Features')","metadata":{"execution":{"iopub.status.busy":"2023-10-03T20:55:05.611198Z","iopub.execute_input":"2023-10-03T20:55:05.611582Z","iopub.status.idle":"2023-10-03T20:55:51.981723Z","shell.execute_reply.started":"2023-10-03T20:55:05.611552Z","shell.execute_reply":"2023-10-03T20:55:51.980037Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cols=[col for col in last_statement_df.columns if (col.startswith(('R','t'))) & (col not in cat_features)]\nplot_kde_last_statement(cols, 'Distribution of Risk Features (Last Statement)')","metadata":{"execution":{"iopub.status.busy":"2023-10-27T14:02:06.069073Z","iopub.execute_input":"2023-10-27T14:02:06.070172Z","iopub.status.idle":"2023-10-27T14:03:13.146895Z","shell.execute_reply.started":"2023-10-27T14:02:06.070129Z","shell.execute_reply":"2023-10-27T14:03:13.145754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_corr_heatmap(cols, 'Correlations Between Risk Variables')","metadata":{"execution":{"iopub.status.busy":"2023-10-03T21:01:31.183642Z","iopub.execute_input":"2023-10-03T21:01:31.184127Z","iopub.status.idle":"2023-10-03T21:01:35.640502Z","shell.execute_reply.started":"2023-10-03T21:01:31.184084Z","shell.execute_reply":"2023-10-03T21:01:35.638266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"corr=last_statement_df[cols].iloc[:,:-1].corr()\nnp.fill_diagonal(corr.values, 0.0)\nmask = np.triu(np.ones_like(corr, dtype=bool))\ncorr = corr.mask(mask)\nhighly_correlated_features = corr[np.abs(corr) > 0.8].stack().reset_index()\nhighly_correlated_features=highly_correlated_features.iloc[abs(highly_correlated_features[0]).argsort()[::-1]].reset_index()\nhighly_correlated_features","metadata":{"execution":{"iopub.status.busy":"2023-10-27T14:05:14.183298Z","iopub.execute_input":"2023-10-27T14:05:14.183708Z","iopub.status.idle":"2023-10-27T14:05:15.287503Z","shell.execute_reply.started":"2023-10-27T14:05:14.183679Z","shell.execute_reply":"2023-10-27T14:05:15.286339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_highly_correlated_featuers(cols, \"Most Highly-Correlated Risk Variables (Log Transformed Relationship)\", highly_correlated_features)","metadata":{"execution":{"iopub.status.busy":"2023-10-27T14:05:27.251868Z","iopub.execute_input":"2023-10-27T14:05:27.252326Z","iopub.status.idle":"2023-10-27T14:05:28.080029Z","shell.execute_reply.started":"2023-10-27T14:05:27.252294Z","shell.execute_reply":"2023-10-27T14:05:28.078815Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Key Insights:\n\n\n- There is 28 numerical risk features and no categorical risk features.\n\n- Most risk features are not highly correlated to eachother. The highest correlation is between R_5 and R_8.\n\n- They seem to have good amount of information about the target and will be included in the modelling.","metadata":{}},{"cell_type":"markdown","source":"### 4.6. Correlation of Variables with the Target","metadata":{}},{"cell_type":"code","source":"#train_data=train_data.drop(cat_features)\ntarg = train_data.iloc[:, :-1].corrwith(train_data['target'], axis=0)\nval = [str(round(v ,1) *100) + '%' for v in targ.values]\n\nfig, ax = plt.subplots(figsize=(7.5, 60))\nbars = ax.barh(targ.index, targ.values, color=next(palette))\n\nfor bar, label in zip(bars, val):\n    ax.text(bar.get_width(), bar.get_y() + bar.get_height() / 2, label,ha='left', va='center', fontsize=12)\n\nax.set_title(\"Correlation of Numerical Features with the Target\", fontsize=16)\nax.set_facecolor('none')\nax.tick_params(axis='y', which='both', left=False)\n\nax.set_xticks([])\nax.set_xticklabels([])\n\nax.spines[['top', 'bottom', 'left', 'right']].set_visible(False)\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-10-03T22:34:28.188572Z","iopub.execute_input":"2023-10-03T22:34:28.188942Z","iopub.status.idle":"2023-10-03T22:35:01.980694Z","shell.execute_reply.started":"2023-10-03T22:34:28.188914Z","shell.execute_reply":"2023-10-03T22:35:01.979709Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sorted_targ = abs(targ).sort_values(ascending=False)","metadata":{"execution":{"iopub.status.busy":"2023-10-03T22:30:07.399184Z","iopub.execute_input":"2023-10-03T22:30:07.400268Z","iopub.status.idle":"2023-10-03T22:30:07.406217Z","shell.execute_reply.started":"2023-10-03T22:30:07.400228Z","shell.execute_reply":"2023-10-03T22:30:07.405038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sorted_targ","metadata":{"execution":{"iopub.status.busy":"2023-10-03T22:30:13.720936Z","iopub.execute_input":"2023-10-03T22:30:13.721285Z","iopub.status.idle":"2023-10-03T22:30:13.730126Z","shell.execute_reply.started":"2023-10-03T22:30:13.721260Z","shell.execute_reply":"2023-10-03T22:30:13.728906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Key Insights: \n\n- Looking at the correlation of features with the target, we see that the correlations range from -61% to 50%. \n\n- P_2 has the highest correlation with the target at -61%.\n\n- It will be interesting to use the top highest correlated features for modelling and see if that leads to good results. I will explore that in my other notebook.","metadata":{}}]}