{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## A. Info about the datasets I'm using:\n\n### A.1 Denoised data:\n\nAs mentioned in [AMEX01](https://www.kaggle.com/code/afonsoalveskggl/amex01-xplore-data), [raddar](https://www.kaggle.com/code/raddar/the-data-has-random-uniform-noise-added) found that some features have added noise (for some reason...). Luckily for us, he created notebooks (for [train data](https://www.kaggle.com/code/raddar/amex-data-int-types-train) and [test data](https://www.kaggle.com/code/raddar/amex-data-int-types-test)) that transform these features into **integer noiseless features**. Such transformation removes random informationless signal from the data, potentially improving model preformance.\n\nYou can find this data in this notebook's `/kaggle/input/amex-data-integer-dtypes-parquet-format`.\n\n**Note about memory:** the original data available when entering the competion is massive and I couldn't load it into memory here on kaggle. Luckily, the transformation into integer, along with the conversion of the data into parquet format massively reduces the memory size of the data (from 16GB to less than 4GB!).\n\n### A.2 Introduction of lag features:\n\nWe saw in [AMEX01](https://www.kaggle.com/code/afonsoalveskggl/amex01-xplore-data) that we normally have 13 lines (statements) per customer, each one containing potentially usefull information in predicting default (which is a single flag value for the customer). Unless we use complex modelling techniques that can take as input multiple lines (such as neural networks), we must only have one line per customer. And here come lag features (if the feature is numeric):\n* **`X_first`**: the value of the first customer statement;\n* **`X_last`**: the value of the last customer statement;\n* **`X_mean`**: the mean value of the customer statements;\n* **`X_std`**: the standard deviation of the customer statements;\n* **`X_min`**: the mininum value of the customer statements;\n* **`X_max`**: the maximum value of the customer statements;\n* **`X_last_lag_sub`**: last value - first value;\n* **`X_last_lag_div`**: last value / first value;\n\nIf the feature is categorical (`B_30`, `B_38`, `D_114`, `D_116`, `D_117`, `D_120`, `D_126`, `D_63`, `D_64`, `D_66`, `D_68`):\n* **`X_first`**: the value of the first customer statement;\n* **`X_last`**: the value of the last customer statement;\n* **`X_nunique`**: number of unique values through out the customer statements;\n* **`X_count`**: number of values through out the customer statements (either equal to the number of customer statements or lower if missing values exist for some customer statements);\n\n**Note about number of customer statements variable (`ntot_statement`)**: this dataset doesn't have a number of customer statements variable or variables created from `S_2` (this would be a potential improvement; for instance, see `S_2` is in the weekend or not, or get the month `S_2` to try to extract sazonality). However, the `X_count` variable of a categorical feature without missing values will correspond to the `ntot_statement` (luckily we have 1 feature like this: `D_63_count`!). I will rename the feature to `ntot_statement`.\n\nThanks to [The Devastator](https://www.kaggle.com/thedevastator) that made available the [lag features data](https://www.kaggle.com/datasets/thedevastator/amex-fe) (also in parquet format) created in his [notebook](https://www.kaggle.com/code/thedevastator/lag-features-are-all-you-need) (that used as a starting point the noiseless integer features).\n\nYou can find this data in this notebook's `/kaggle/input/amex-fe` (the most recent train and test versions that have more features are the `train_fe_plus_plus.parquet` and `test_fe_plus_plus.parquet`).","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-10-28T14:13:03.962071Z","iopub.execute_input":"2023-10-28T14:13:03.962504Z","iopub.status.idle":"2023-10-28T14:13:05.263981Z","shell.execute_reply.started":"2023-10-28T14:13:03.962470Z","shell.execute_reply":"2023-10-28T14:13:05.262771Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Read the single statement data including the mentioned lag features: \ndf_train_lag_pq = pd.read_parquet(\"/kaggle/input/amex-fe/train_fe_plus_plus.parquet\").rename(\n    {'D_63_count':'ntot_statement'}, axis = 1\n)\nprint(df_train_lag_pq.info())\ndf_train_lag_pq.head(3)","metadata":{"execution":{"iopub.status.busy":"2023-10-28T14:13:05.266504Z","iopub.execute_input":"2023-10-28T14:13:05.267168Z","iopub.status.idle":"2023-10-28T14:13:22.394795Z","shell.execute_reply.started":"2023-10-28T14:13:05.267124Z","shell.execute_reply":"2023-10-28T14:13:22.393745Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## B. Handling missing features:\n\nWe saw in [AMEX01](https://www.kaggle.com/code/afonsoalveskggl/amex01-xplore-data?scriptVersionId=147603199) that the data has a lot of missing values. We will predominantly use `lightgbm` models (which can handle missing values internally). However, I suggested in [AMEX01](https://www.kaggle.com/code/afonsoalveskggl/amex01-xplore-data?scriptVersionId=147603199), we can bundle features with high missing percentage to create stronger features. Right now we have many more features (the additional lag features). Let's see how the missing values distribution changed:","metadata":{"execution":{"iopub.status.busy":"2023-10-22T18:17:04.561920Z","iopub.execute_input":"2023-10-22T18:17:04.562737Z","iopub.status.idle":"2023-10-22T18:17:04.567225Z","shell.execute_reply.started":"2023-10-22T18:17:04.562673Z","shell.execute_reply":"2023-10-22T18:17:04.566017Z"}}},{"cell_type":"code","source":"def report_missing_values(df, exclude_columns = ['customer_ID']):\n    feat_with_nan = [col for col in df if df[col].isna().sum() > 0 and col not in exclude_columns]\n    for ix, col in enumerate(feat_with_nan):\n        # Auxiliary dataframes:\n        is_nan = df[df[col].isna()][[col, 'target']]\n        not_nan = df[~df[col].isna()][[col, 'target']]\n        # Compute interesting missing metrics:\n        n_is_nan = len(is_nan)\n        n_not_nan = len(not_nan)\n        mean_target_is_nan = is_nan.target.mean()\n        mean_target_not_nan = not_nan.target.mean()\n        min_not_nan = not_nan[col].min()\n        temp = pd.DataFrame({\n            'feature':[col], \"n_is_nan\":[n_is_nan], \"n_not_nan\":[n_not_nan],\n            \"miss%\":[str(round(n_is_nan/(n_is_nan+n_not_nan)*100,2))+\"%\"],\n            \"mean_target_is_nan\":[mean_target_is_nan], \"mean_target_not_nan\":[mean_target_not_nan],\n            \"min_not_nan\":[min_not_nan]\n        })\n\n        if ix == 0:\n            missing_report = temp.copy()\n        else:\n            missing_report = pd.concat([missing_report, temp], axis = 0)\n        del is_nan, not_nan, temp\n    missing_report = missing_report.sort_values(['n_is_nan'], axis = 0, ascending = False)\n    print(\"Number of features with NaNs: {} ({}%)\".format(len(feat_with_nan),\n                                                              round(len(feat_with_nan)/len(df.columns)*100,1)\n                                                         ))\n    print(\"Number of features with more than 10% missing values: {}\".format(\n        len(missing_report[missing_report.n_is_nan >= len(df)*0.1]))\n    )\n    print(\"Top 20 missing features:\")\n    display(missing_report.head(20))","metadata":{"execution":{"iopub.status.busy":"2023-10-28T09:06:27.591970Z","iopub.execute_input":"2023-10-28T09:06:27.592525Z","iopub.status.idle":"2023-10-28T09:06:27.604649Z","shell.execute_reply.started":"2023-10-28T09:06:27.592495Z","shell.execute_reply":"2023-10-28T09:06:27.603727Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"report_missing_values(df = df_train_lag_pq.sample(n=50000), exclude_columns = ['customer_ID'])","metadata":{"execution":{"iopub.status.busy":"2023-10-28T09:06:48.812907Z","iopub.execute_input":"2023-10-28T09:06:48.813714Z","iopub.status.idle":"2023-10-28T09:07:52.025918Z","shell.execute_reply.started":"2023-10-28T09:06:48.813657Z","shell.execute_reply":"2023-10-28T09:07:52.024922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## C. Correlations in data:\n\nGiven the enormous number of features, correlations likely occur (specially between intra group variables, e.g. between two balance features). One possible preprocessing step for modelling is to remove features above a given threshold. \n\nDue to the high number of features (and to avoid long run times), will only evaluate correlation between **`X_last`** features. ","metadata":{}},{"cell_type":"code","source":"def lower_corr_matrix(n_sample, variables, figsize, fontsize):\n\n    # Compute the correlation matrix (to avoid long computation / CPU timeout I only compute the corr with\n    # n_sample values):\n    corr_matrix_pay_vars = df_train_lag_pq.sample(n = n_sample, random_state = 42)[variables].corr()\n\n    # Create a heatmap to visualize the lower triangle of the correlation matrix\n    mask = np.triu(np.ones_like(corr_matrix_pay_vars, dtype=bool)) # mask to only show lower triangle \n    plt.figure(figsize=figsize)\n    sns.heatmap(corr_matrix_pay_vars, annot=True, cmap=\"coolwarm\", fmt=\".2f\", linewidths=0.5, mask=mask, annot_kws={\"fontsize\": fontsize})\n    display(HTML(\"<style>.output {max-height: 300px;}</style>\"))\n    plt.show()\n    \n    # Make the upper triangle values of the correlation dataframe as nan:\n    corr_matrix_pay_vars = corr_matrix_pay_vars.where(~mask)\n    return corr_matrix_pay_vars\n    \npayment_variables = [col for col in df_train_lag_pq if col.split(\"_\")[0] == \"P\" and 'last' in col and 'lag_sub' not in col and 'lag_div' not in col]\nprint(\">> Correlation Matrix (Lower Triangle) - PAYMENT VARIABLES\")\ncorr_matrix_pay_vars = lower_corr_matrix(n_sample = 10000, variables = payment_variables, figsize = (6, 4), fontsize = 14)","metadata":{"execution":{"iopub.status.busy":"2023-10-28T15:29:43.160941Z","iopub.execute_input":"2023-10-28T15:29:43.161386Z","iopub.status.idle":"2023-10-28T15:29:43.619264Z","shell.execute_reply.started":"2023-10-28T15:29:43.161352Z","shell.execute_reply":"2023-10-28T15:29:43.618396Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"spend_variables = [col for col in df_train_lag_pq if col.split(\"_\")[0] == \"S\" and 'last' in col and 'lag_sub' not in col and 'lag_div' not in col]\nprint(\">> Correlation Matrix (Lower Triangle) - SPEND VARIABLES\")\ncorr_matrix_spend_vars = lower_corr_matrix(\n    n_sample = 10000, variables = spend_variables, figsize = (12, 9), fontsize = 6, \n)","metadata":{"execution":{"iopub.status.busy":"2023-10-28T15:23:31.556251Z","iopub.execute_input":"2023-10-28T15:23:31.556613Z","iopub.status.idle":"2023-10-28T15:23:32.691747Z","shell.execute_reply.started":"2023-10-28T15:23:31.556582Z","shell.execute_reply":"2023-10-28T15:23:32.690753Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"deliquency_variables = [col for col in df_train_lag_pq if col.split(\"_\")[0] == \"D\" and 'last' in col and 'lag_sub' not in col and 'lag_div' not in col]\nprint(\">> Correlation Matrix (Lower Triangle) - DELIQUENCY VARIABLES\")\ncorr_matrix_spend_vars = lower_corr_matrix(\n    n_sample = 10000, variables = deliquency_variables, figsize = (35, 30), fontsize = 4, \n)","metadata":{"execution":{"iopub.status.busy":"2023-10-28T15:32:02.789663Z","iopub.execute_input":"2023-10-28T15:32:02.790099Z","iopub.status.idle":"2023-10-28T15:32:13.515907Z","shell.execute_reply.started":"2023-10-28T15:32:02.790063Z","shell.execute_reply":"2023-10-28T15:32:13.514702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"balance_variables = [col for col in df_train_lag_pq if col.split(\"_\")[0] == \"B\" and 'last' in col and 'lag_sub' not in col and 'lag_div' not in col]\nprint(\">> Correlation Matrix (Lower Triangle) - BALANCE VARIABLES\")\ncorr_matrix_spend_vars = lower_corr_matrix(\n    n_sample = 30000, variables = balance_variables, figsize = (21, 15), fontsize = 7, \n)","metadata":{"execution":{"iopub.status.busy":"2023-10-28T15:32:30.798735Z","iopub.execute_input":"2023-10-28T15:32:30.799110Z","iopub.status.idle":"2023-10-28T15:32:33.738664Z","shell.execute_reply.started":"2023-10-28T15:32:30.799081Z","shell.execute_reply":"2023-10-28T15:32:33.737255Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"risk_variables = [col for col in df_train_lag_pq if col.split(\"_\")[0] == \"R\" and 'last' in col and 'lag_sub' not in col and 'lag_div' not in col]\nprint(\">> Correlation Matrix (Lower Triangle) - RISK VARIABLES\")\ncorr_matrix_spend_vars = lower_corr_matrix(\n    n_sample = 30000, variables = risk_variables, figsize = (12, 10), fontsize = 6, \n)","metadata":{"execution":{"iopub.status.busy":"2023-10-28T15:32:38.281246Z","iopub.execute_input":"2023-10-28T15:32:38.281750Z","iopub.status.idle":"2023-10-28T15:32:40.155257Z","shell.execute_reply.started":"2023-10-28T15:32:38.281625Z","shell.execute_reply":"2023-10-28T15:32:40.154038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if missing_var1 < 95 and missing_var2 < 95 and missing :\n    if corr > 70:\n        -> linear noise feature\n    else:\n        keep feature pair\n        \npca","metadata":{"scrolled":true},"execution_count":null,"outputs":[]}]}