{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# American Express - Default Prediction\n## Predict if a customer will default in the future\n\n### Competition Overview\n\nWhether out at a restaurant or buying tickets to a concert, modern life counts on the convenience of a credit card to make daily purchases. It saves us from carrying large amounts of cash and also can advance a full purchase that can be paid over time. How do card issuers know we’ll pay back what we charge? That’s a complex problem with many existing solutions—and even more potential improvements, to be explored in this competition.\n\nCredit default prediction is central to managing risk in a consumer lending business. Credit default prediction allows lenders to optimize lending decisions, which leads to a better customer experience and sound business economics. Current models exist to help manage risk. But it's possible to create better models that can outperform those currently in use.\n\nAmerican Express is a globally integrated payments company. The largest payment card issuer in the world, they provide customers with access to products, insights, and experiences that enrich lives and build business success.\n\nIn this competition, you’ll apply your machine learning skills to predict credit default. Specifically, you will leverage an industrial scale data set to build a machine learning model that challenges the current model in production. Training, validation, and testing datasets include time-series behavioral data and anonymized customer profile information. You're free to explore any technique to create the most powerful model, from creating features to using the data in a more organic way within a model.\n\nIf successful, you'll help create a better customer experience for cardholders by making it easier to be approved for a credit card. Top solutions could challenge the credit default prediction model used by the world's largest payment card issuer—earning you cash prizes, the opportunity to interview with American Express, and potentially a rewarding new career.\n\n### Data Overview\n\nThe dataset contains aggregated profile features for each customer at each statement date. Features are anonymized and normalized, and fall into the following general categories:\n\nD_* = Delinquency variables\nS_* = Spend variables\nP_* = Payment variables\nB_* = Balance variables\nR_* = Risk variables\nwith the following features being categorical:\n\n['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']\n\nYour task is to predict, for each customer_ID, the probability of a future payment default (target = 1).\n\n### Objective\n\nThe objective of this competition is to predict the probability that a customer does not pay back their credit card balance amount in the future based on their monthly customer profile. The target binary variable is calculated by observing 18 months performance window after the latest credit card statement, and if the customer does not pay due amount in 120 days after their latest statement date it is considered a default event.\n\n### Evaluation\n\nThe evaluation metric, M , for this competition is the mean of two measures of rank ordering: Normalized Gini Coefficient,G, and default rate captured at 4%,D.\n\nM = 0.5(G + D)\n\nThe default rate captured at 4% is the percentage of the positive labels (defaults) captured within the highest-ranked 4% of the predictions, and represents a Sensitivity/Recall statistic.\n\nFor both of the sub-metrics and , the negative labels are given a weight of 20 to adjust for downsampling.\n\nThis metric has a maximum value of 1.0.","metadata":{}},{"cell_type":"markdown","source":"# Table of contents\n\n* <a href=\"#C1\">Part 1: Data import</a>\n* <a href=\"#C2\">Part 2: Data exploration</a>\n    * <a href=\"#C3\">Missing values</a>\n    * <a href=\"#C4\">Attention to customers</a>\n    * <a href=\"#C5\">Variables distribution</a>\n* <a href=\"#C6\">Part 3: Prediction models</a>\n    * <a href=\"#C7\">Data processing</a>\n    * <a href=\"#C8\">Model testing</a>\n* <a href=\"#C9\">Part 4: Final model</a>","metadata":{}},{"cell_type":"markdown","source":"# <a name=\"C1\">Part 1: Data import</a>\nWe start by importing the useful libraries and datasets.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport plotly.graph_objects as go\nimport timeit\nimport pickle\n\nfrom itertools import cycle\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn.metrics import accuracy_score\n\nfrom lightgbm import LGBMClassifier \nfrom catboost import CatBoostClassifier\nfrom sklearn.ensemble import GradientBoostingClassifier\n\npd.set_option('display.max_columns', 100)","metadata":{"execution":{"iopub.status.busy":"2022-07-09T07:34:42.907302Z","iopub.execute_input":"2022-07-09T07:34:42.908226Z","iopub.status.idle":"2022-07-09T07:34:46.871629Z","shell.execute_reply.started":"2022-07-09T07:34:42.908102Z","shell.execute_reply":"2022-07-09T07:34:46.870610Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntrain = pd.read_feather('../input/amex-default-prediction-feather/train.feather')\ntest = pd.read_feather('../input/amex-default-prediction-feather/test.feather')\ntrain_labels = pd.read_csv(\"../input/amex-default-prediction/train_labels.csv\")\nsample_submission = pd.read_csv('../input/amex-default-prediction/sample_submission.csv')\ncategorical_var = ['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68', 'S_2']","metadata":{"execution":{"iopub.status.busy":"2022-07-09T07:34:46.873484Z","iopub.execute_input":"2022-07-09T07:34:46.873872Z","iopub.status.idle":"2022-07-09T07:35:07.357933Z","shell.execute_reply.started":"2022-07-09T07:34:46.873835Z","shell.execute_reply":"2022-07-09T07:35:07.356718Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-07T09:52:36.066844Z","iopub.execute_input":"2022-07-07T09:52:36.067501Z","iopub.status.idle":"2022-07-07T09:52:36.147977Z","shell.execute_reply.started":"2022-07-07T09:52:36.067462Z","shell.execute_reply":"2022-07-07T09:52:36.146866Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"train data shape:\", train.shape)\nprint(\"test data shape:\", test.shape)\nprint(\"train lablels shape:\", train_labels.shape)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T09:55:01.227175Z","iopub.execute_input":"2022-07-07T09:55:01.228313Z","iopub.status.idle":"2022-07-07T09:55:01.234769Z","shell.execute_reply.started":"2022-07-07T09:55:01.228269Z","shell.execute_reply":"2022-07-07T09:55:01.233527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# <a name=\"C2\">Part 2: Data exploration</a>\n\n## <a name=\"C3\">2.1: Missing values</a>\nLet's first have a look at missing values.","metadata":{}},{"cell_type":"code","source":"def quantity_missing_values(data):\n    \"\"\"function to obtain the number and percentage of missing values for each variable of a dataframe, \n    in descending order\"\"\"\n    \n    values = data.isnull().sum()\n    percentage = 100 * values / len(data)\n    table = pd.concat([values, percentage.round(2)], axis=1)\n    table.columns = ['Number of missing values', '% of missing values']\n    \n    return table[table['Number of missing values'] != 0].sort_values('% of missing values', ascending = False).style.background_gradient('OrRd')","metadata":{"execution":{"iopub.status.busy":"2022-07-07T09:55:17.441285Z","iopub.execute_input":"2022-07-07T09:55:17.441720Z","iopub.status.idle":"2022-07-07T09:55:17.449111Z","shell.execute_reply.started":"2022-07-07T09:55:17.441681Z","shell.execute_reply":"2022-07-07T09:55:17.448050Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"quantity_missing_values(train)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T09:55:20.142554Z","iopub.execute_input":"2022-07-07T09:55:20.143568Z","iopub.status.idle":"2022-07-07T09:55:25.694658Z","shell.execute_reply.started":"2022-07-07T09:55:20.143524Z","shell.execute_reply":"2022-07-07T09:55:25.693635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It appears that for some variables, a lot of data is missing. But it should be kept in mind that cases of fraud or non-payment are quite rare, hence a high number of missing values. We will consider that the dataset does not contain any errors and keep the missing values for the moment.\n\n## <a name=\"C4\">2.2: Attention to customers</a>\nLet's now have a look at the number of customers and how many times they appear in the dataset.","metadata":{}},{"cell_type":"code","source":"nb_customers = len(list(train['customer_ID'].unique()))\nprint(\"Number of customers:\", nb_customers)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T09:55:35.302002Z","iopub.execute_input":"2022-07-07T09:55:35.302384Z","iopub.status.idle":"2022-07-07T09:55:36.122859Z","shell.execute_reply.started":"2022-07-07T09:55:35.302352Z","shell.execute_reply":"2022-07-07T09:55:36.121751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y = train.groupby(\"customer_ID\")['customer_ID'].count().values\n\nplt.figure(figsize=(10, 6))\nsns.histplot(data=y, color='red')\nplt.xlabel('Number of months')\nplt.title('Quantity of customers by number of bank statements available')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-07T09:55:36.314207Z","iopub.execute_input":"2022-07-07T09:55:36.317046Z","iopub.status.idle":"2022-07-07T09:55:38.719594Z","shell.execute_reply.started":"2022-07-07T09:55:36.317002Z","shell.execute_reply":"2022-07-07T09:55:38.718642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For most customers, we have the bank statements of the last 13 months.</br>\nLet's now have a look at the quantity of customers with default payment.","metadata":{}},{"cell_type":"code","source":"# connection between the number of bank statements and the target output\ndf = train.groupby(\"customer_ID\")['customer_ID'].count()\ndf = pd.DataFrame({\"customer_ID\":df.index, \"count\": df.values})\n# merge the data with the label data frame\ndf = df.merge(train_labels, on='customer_ID', how='left')\n\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-07T09:55:51.259476Z","iopub.execute_input":"2022-07-07T09:55:51.259885Z","iopub.status.idle":"2022-07-07T09:55:53.700421Z","shell.execute_reply.started":"2022-07-07T09:55:51.259851Z","shell.execute_reply":"2022-07-07T09:55:53.699266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"labels = ['No default payment', 'Default payment']\n\nplt.figure(figsize=(10, 6))\nplt.pie(df['target'].value_counts(), labels=labels, autopct='%1.1f%%')\nplt.axis('equal')\nplt.title('Proportion of customer with default payment')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-07T09:55:59.013310Z","iopub.execute_input":"2022-07-07T09:55:59.013724Z","iopub.status.idle":"2022-07-07T09:55:59.138117Z","shell.execute_reply.started":"2022-07-07T09:55:59.013690Z","shell.execute_reply":"2022-07-07T09:55:59.136720Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A quarter of customers did not pay back their credit card balance amount.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(12, 8))\nsns.countplot(data=df, x='count',hue='target', palette='Paired')\nplt.title('Customers by number of bank statements')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-07T09:56:06.209101Z","iopub.execute_input":"2022-07-07T09:56:06.209486Z","iopub.status.idle":"2022-07-07T09:56:06.566591Z","shell.execute_reply.started":"2022-07-07T09:56:06.209454Z","shell.execute_reply":"2022-07-07T09:56:06.565593Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <a name=\"C5\">2.3: Variables distribution</a>","metadata":{}},{"cell_type":"code","source":"#Number of variables per category\nD_features = [c for c in train.columns if c.startswith('D_')]\nS_features = [c for c in train.columns if c.startswith('S_')]\nP_features = [c for c in train.columns if c.startswith('P_')]\nB_features = [c for c in train.columns if c.startswith('B_')]\nR_features = [c for c in train.columns if c.startswith('R_')]\n\nnb_features = [len(D_features) ,len(S_features), len(P_features), len(B_features),\n              len(R_features)]\nlabels = ['Delinquency', 'Spend', 'Payment', 'Balance', 'Risk']\n\nplt.figure(figsize=(10, 6))\nplt.pie(nb_features, labels=labels, autopct='%1.1f%%')\nplt.axis('equal')\nplt.title('Number of variables per category')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-09T07:36:50.230604Z","iopub.execute_input":"2022-07-09T07:36:50.231039Z","iopub.status.idle":"2022-07-09T07:36:50.414882Z","shell.execute_reply.started":"2022-07-09T07:36:50.231006Z","shell.execute_reply":"2022-07-09T07:36:50.413848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Principal variables are related to delinquency, which are also the ones containing the most missing values. But as mentioned before, delinquency does not concern erverybody (fortunately !).","metadata":{}},{"cell_type":"code","source":"data = train.merge(train_labels, on='customer_ID', how='left')\ndata.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-09T07:30:00.881681Z","iopub.execute_input":"2022-07-09T07:30:00.882502Z","iopub.status.idle":"2022-07-09T07:31:28.134470Z","shell.execute_reply.started":"2022-07-09T07:30:00.882457Z","shell.execute_reply":"2022-07-09T07:31:28.132726Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_histogram(data, columns, nrow, ncol, figsize, title, hue=None):\n    \"\"\"Display function of input variables\"\"\"\n    \n    fig, ax = plt.subplots(nrow, ncol, figsize=figsize)\n    col, row = ncol, nrow\n    cpt = 0\n    for r in range(row):\n        for c in range(col):\n            if cpt < len(columns):\n                if nrow > 1:\n                    sns.kdeplot(data=data, x=columns[cpt], hue=hue, ax=ax[r, c], palette=[\"#FFA384\",\"#74BDCB\"], fill=True, \n                                hue_order=[1,0], legend=True)\n                    ax[r,c].set(xlabel = columns[cpt], ylabel=(\"Density\"))\n                else:\n                    sns.kdeplot(data=data, x=columns[cpt], hue=hue, ax=ax[c], palette=[\"#FFA384\",\"#74BDCB\"], fill=True, \n                                hue_order=[1,0], legend=True)\n                    ax[c].set(xlabel = columns[cpt], ylabel=(\"Density\"))\n            cpt +=1\n    fig.suptitle(title)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-07T12:15:44.768170Z","iopub.execute_input":"2022-07-07T12:15:44.768657Z","iopub.status.idle":"2022-07-07T12:15:44.779518Z","shell.execute_reply.started":"2022-07-07T12:15:44.768617Z","shell.execute_reply":"2022-07-07T12:15:44.778330Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_correlation_matrix(data, figsize, title):\n    \"\"\"Display function of the correlation matrix of input variables\"\"\"\n    plt.figure(figsize=figsize)\n    corr = data.corr()\n    mask = np.triu(np.ones_like(corr, dtype = bool))\n    sns.heatmap(corr, annot=True, cmap='YlGnBu', mask=mask, fmt='.1f', linewidths=.5, square=True, vmax=1)\n    plt.title(title)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-09T07:31:28.137585Z","iopub.execute_input":"2022-07-09T07:31:28.138143Z","iopub.status.idle":"2022-07-09T07:31:28.145257Z","shell.execute_reply.started":"2022-07-09T07:31:28.138100Z","shell.execute_reply":"2022-07-09T07:31:28.144135Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can have a quick look at the distribution of numerical variables, but we must not forget that data is already normalized and we won't have the true distributions. It'll be probably more interesting to have a look at correlations between varibales.","metadata":{}},{"cell_type":"code","source":"D_cat_features = list(set(D_features)-set(categorical_var))\nnrow = 29\nncol = 3\nfigsize = (40, 150)\nhue = 'target'\ntitle = 'Spend variables distributions'\n\nplot_histogram(data, D_cat_features, nrow, ncol, figsize, hue)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T11:22:37.906192Z","iopub.execute_input":"2022-07-07T11:22:37.907543Z","iopub.status.idle":"2022-07-07T11:44:09.391687Z","shell.execute_reply.started":"2022-07-07T11:22:37.907488Z","shell.execute_reply":"2022-07-07T11:44:09.390719Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_correlation_matrix(data[D_cat_features], (11, 11), 'Correlation of Delinquency variables')","metadata":{"execution":{"iopub.status.busy":"2022-07-09T07:31:28.158982Z","iopub.execute_input":"2022-07-09T07:31:28.159500Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"S_cat_features = list(set(S_features)-set(categorical_var))\nnrow = 7\nncol = 3\nfigsize = (16, 18)\nhue = 'target'\ntitle = 'Spend variables distributions'\n\nplot_histogram(data, S_cat_features, nrow, ncol, figsize, title, hue)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T10:02:29.191224Z","iopub.execute_input":"2022-07-07T10:02:29.191767Z","iopub.status.idle":"2022-07-07T10:08:55.792894Z","shell.execute_reply.started":"2022-07-07T10:02:29.191721Z","shell.execute_reply":"2022-07-07T10:08:55.791776Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_correlation_matrix(data[S_cat_features], (11, 11), 'Correlation of Spend variables')","metadata":{"execution":{"iopub.status.busy":"2022-07-07T10:09:48.807199Z","iopub.execute_input":"2022-07-07T10:09:48.807585Z","iopub.status.idle":"2022-07-07T10:10:00.244231Z","shell.execute_reply.started":"2022-07-07T10:09:48.807553Z","shell.execute_reply":"2022-07-07T10:10:00.243199Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"P_cat_features = list(set(P_features)-set(categorical_var))\nnrow = 1\nncol = 3\nfigsize = (12, 4)\nhue = 'target'\ntitle = 'Payment variables distributions'\n\nplot_histogram(data, P_cat_features, nrow, ncol, figsize, title, hue)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T12:11:35.203118Z","iopub.execute_input":"2022-07-07T12:11:35.203948Z","iopub.status.idle":"2022-07-07T12:12:32.732250Z","shell.execute_reply.started":"2022-07-07T12:11:35.203897Z","shell.execute_reply":"2022-07-07T12:12:32.731266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_correlation_matrix(data[P_cat_features], (6, 6), 'Correlation of Payment variables')","metadata":{"execution":{"iopub.status.busy":"2022-07-07T12:12:42.146564Z","iopub.execute_input":"2022-07-07T12:12:42.146927Z","iopub.status.idle":"2022-07-07T12:12:42.621958Z","shell.execute_reply.started":"2022-07-07T12:12:42.146896Z","shell.execute_reply":"2022-07-07T12:12:42.620976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"B_cat_features = list(set(B_features)-set(categorical_var))\nnrow = 13\nncol = 3\nfigsize = (15, 24)\nhue = 'target'\ntitle = 'Balance variables distributions'\n\nplot_histogram(data, B_cat_features, nrow, ncol, figsize, title, hue)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T11:58:48.172910Z","iopub.execute_input":"2022-07-07T11:58:48.173307Z","iopub.status.idle":"2022-07-07T12:10:05.602706Z","shell.execute_reply.started":"2022-07-07T11:58:48.173275Z","shell.execute_reply":"2022-07-07T12:10:05.601575Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_correlation_matrix(data[B_cat_features], (11, 11), 'Correlation of Balance variables')","metadata":{"execution":{"iopub.status.busy":"2022-07-07T12:10:05.605181Z","iopub.execute_input":"2022-07-07T12:10:05.605564Z","iopub.status.idle":"2022-07-07T12:10:29.051262Z","shell.execute_reply.started":"2022-07-07T12:10:05.605529Z","shell.execute_reply":"2022-07-07T12:10:29.050328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"R_cat_features = list(set(R_features)-set(categorical_var))\nnrow = 10\nncol = 3\nfigsize = (18, 23)\nhue = 'target'\ntitle = 'Risk variables distributions'\n\nplot_histogram(data, R_cat_features, nrow, ncol, figsize, title, hue)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T11:46:50.213808Z","iopub.execute_input":"2022-07-07T11:46:50.214190Z","iopub.status.idle":"2022-07-07T11:55:13.661246Z","shell.execute_reply.started":"2022-07-07T11:46:50.214157Z","shell.execute_reply":"2022-07-07T11:55:13.660224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_correlation_matrix(data[R_cat_features], (11, 11), 'Correlation of Risk variables')","metadata":{"execution":{"iopub.status.busy":"2022-07-07T11:55:13.662845Z","iopub.execute_input":"2022-07-07T11:55:13.663216Z","iopub.status.idle":"2022-07-07T11:55:28.182409Z","shell.execute_reply.started":"2022-07-07T11:55:13.663177Z","shell.execute_reply":"2022-07-07T11:55:28.181330Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Generally speaking we do not have big correlation trends.</br>\nWhat about correlation with the traget values ?","metadata":{}},{"cell_type":"code","source":"palette = cycle([\"#ffd670\",\"#70d6ff\",\"#ff4d6d\",\"#8338ec\",\"#90cf8e\"])\ntarg = data.corrwith(data['target'], axis=0)\nval = [str(round(v ,1) *100) + '%' for v in targ.values]\nfig = go.Figure()\nfig.add_trace(go.Bar(y=targ.index, x= targ.values, orientation='h',text = val, marker_color = next(palette)))\nfig.update_layout(title = \"Correlation of variables with Target\",width = 750, height = 3500,\n                  paper_bgcolor='rgb(0,0,0,0)',plot_bgcolor='rgb(0,0,0,0)')","metadata":{"execution":{"iopub.status.busy":"2022-07-07T11:57:31.242744Z","iopub.execute_input":"2022-07-07T11:57:31.243379Z","iopub.status.idle":"2022-07-07T11:57:54.128260Z","shell.execute_reply.started":"2022-07-07T11:57:31.243323Z","shell.execute_reply":"2022-07-07T11:57:54.127143Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Except perhaps fot the varibale `P_2`, we do not make out any strong correlation with the target. Thus, it would be worth to keep as many varibales as possible.","metadata":{}},{"cell_type":"markdown","source":"# <a name=\"C6\">Part 3: Prediction models</a>\n\nWe can now try and compare different prediction models.</br>\nBut first we need a metric to comapre those models. In addition to the computation time and the accuracy score, we will use the metric given by the Amex competition, which can be found [here](https://www.kaggle.com/code/inversion/amex-competition-metric-python).","metadata":{}},{"cell_type":"code","source":"#Evaluation metric \ndef amex_metric(y_true: pd.DataFrame, y_pred: pd.DataFrame) -> float:\n\n    def top_four_percent_captured(y_true: pd.DataFrame, y_pred: pd.DataFrame) -> float:\n        df = (pd.concat([y_true, y_pred], axis='columns')\n              .sort_values('prediction', ascending=False))\n        df['weight'] = df['target'].apply(lambda x: 20 if x==0 else 1)\n        four_pct_cutoff = int(0.04 * df['weight'].sum())\n        df['weight_cumsum'] = df['weight'].cumsum()\n        df_cutoff = df.loc[df['weight_cumsum'] <= four_pct_cutoff]\n        return (df_cutoff['target'] == 1).sum() / (df['target'] == 1).sum()\n        \n    def weighted_gini(y_true: pd.DataFrame, y_pred: pd.DataFrame) -> float:\n        df = (pd.concat([y_true, y_pred], axis='columns')\n              .sort_values('prediction', ascending=False))\n        df['weight'] = df['target'].apply(lambda x: 20 if x==0 else 1)\n        df['random'] = (df['weight'] / df['weight'].sum()).cumsum()\n        total_pos = (df['target'] * df['weight']).sum()\n        df['cum_pos_found'] = (df['target'] * df['weight']).cumsum()\n        df['lorentz'] = df['cum_pos_found'] / total_pos\n        df['gini'] = (df['lorentz'] - df['random']) * df['weight']\n        return df['gini'].sum()\n\n    def normalized_weighted_gini(y_true: pd.DataFrame, y_pred: pd.DataFrame) -> float:\n        y_true_pred = y_true.rename(columns={'target': 'prediction'})\n        return weighted_gini(y_true, y_pred) / weighted_gini(y_true, y_true_pred)\n\n    g = normalized_weighted_gini(y_true, y_pred)\n    d = top_four_percent_captured(y_true, y_pred)\n\n    return 0.5 * (g + d)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T08:35:23.822915Z","iopub.execute_input":"2022-07-07T08:35:23.823307Z","iopub.status.idle":"2022-07-07T08:35:23.838891Z","shell.execute_reply.started":"2022-07-07T08:35:23.823274Z","shell.execute_reply":"2022-07-07T08:35:23.837558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <a name=\"C7\">3.1: Data processing</a>\nFirst, the dataset contains some categorical variables we need to encode. Second, the dataset being very large and our memory space and computation capacity limited, we will consider only 10% of it while testing different models in order to speed up the process.","metadata":{}},{"cell_type":"code","source":"#Get 10% of the training dataset\ndata_reduced = data.sample(frac=0.1)\ndata_reduced.drop(['customer_ID'], axis=1, inplace=True)\ndata_reduced.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-07T12:21:51.246662Z","iopub.execute_input":"2022-07-07T12:21:51.247449Z","iopub.status.idle":"2022-07-07T12:21:55.459384Z","shell.execute_reply.started":"2022-07-07T12:21:51.247399Z","shell.execute_reply":"2022-07-07T12:21:55.458382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Another key point we want to address is the processing of missing values. We will try 3 different approaches:\n- 1. We keep the data as it is, we let the model take care of missing values\n- 2. We delete all variables containing missing values\n- 3. We delete the variables with more than 30% of missing values and fill the others with the mean (for numerical variabe) or most frequent (for categorical variable) value","metadata":{}},{"cell_type":"code","source":"data[categorical_var].isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2022-07-07T12:21:55.461149Z","iopub.execute_input":"2022-07-07T12:21:55.462668Z","iopub.status.idle":"2022-07-07T12:21:56.134627Z","shell.execute_reply.started":"2022-07-07T12:21:55.462624Z","shell.execute_reply":"2022-07-07T12:21:56.133442Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We do not have any missing value for categorical variables, we will just need to complete numerical variables with the mean value, which is always 0 because the data is normalized.","metadata":{}},{"cell_type":"code","source":"#Get rid of variables with NaN values\ndata_reduced_cleaned = data_reduced.dropna(axis=1)\nprint(\"Shape after removing all columns containing any NaN value:\", data_reduced_cleaned.shape)\n\n#Get rid of variables with more than 30% of NaN values\nthreshold = data_reduced.shape[0]*0.7\ndata_reduced_processed = data_reduced.dropna(axis=1, thresh=threshold)\n\nfor var in data_reduced_processed.columns:\n    if data_reduced_processed[var].isnull().sum()>0:\n        data_reduced_processed[var].fillna(0, inplace=True) #data is normalized, mean=0\n        \nprint(\"Shape after processing part of NaN values:\", data_reduced_processed.shape)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T12:22:00.627074Z","iopub.execute_input":"2022-07-07T12:22:00.627821Z","iopub.status.idle":"2022-07-07T12:22:02.721260Z","shell.execute_reply.started":"2022-07-07T12:22:00.627779Z","shell.execute_reply":"2022-07-07T12:22:02.720249Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Encode categorical variables\ndata_reduced = pd.get_dummies(data=data_reduced, columns=categorical_var, dummy_na=True)\ndata_reduced_cleaned = pd.get_dummies(data=data_reduced_cleaned, columns=categorical_var)\ndata_reduced_processed = pd.get_dummies(data=data_reduced_processed, columns=categorical_var)\n\nprint(\"Data shape with NaN values:\", data_reduced.shape)\nprint(\"Data shape without NaN values:\", data_reduced_cleaned.shape)\nprint(\"Data shape with NaN values processed:\", data_reduced_processed.shape)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T12:22:16.639173Z","iopub.execute_input":"2022-07-07T12:22:16.639723Z","iopub.status.idle":"2022-07-07T12:22:22.209747Z","shell.execute_reply.started":"2022-07-07T12:22:16.639675Z","shell.execute_reply":"2022-07-07T12:22:22.208638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Train/test split\nX = data_reduced.drop(['target'], axis=1)\ny = data_reduced['target']\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\n# Checking split \nprint('X_train shape:', X_train.shape)\nprint('y_train shape:', y_train.shape)\nprint('X_test shape:', X_test.shape)\nprint('y_test shape:', y_test.shape)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T12:22:37.568928Z","iopub.execute_input":"2022-07-07T12:22:37.569298Z","iopub.status.idle":"2022-07-07T12:22:41.598873Z","shell.execute_reply.started":"2022-07-07T12:22:37.569267Z","shell.execute_reply":"2022-07-07T12:22:41.597645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will save training data to be sure we compare models trained on the same data.","metadata":{}},{"cell_type":"code","source":"X_train.to_csv('X_train')\nX_test.to_csv('X_test')\ny_train.to_csv('y_train')\ny_test.to_csv('y_test')","metadata":{"execution":{"iopub.status.busy":"2022-07-06T15:29:18.232836Z","iopub.execute_input":"2022-07-06T15:29:18.233565Z","iopub.status.idle":"2022-07-06T15:31:29.65234Z","shell.execute_reply.started":"2022-07-06T15:29:18.233526Z","shell.execute_reply":"2022-07-06T15:31:29.651268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Train/test split\nX = data_reduced_cleaned.drop(['target'], axis=1)\ny = data_reduced_cleaned['target']\n\nX_train_cleaned, X_test_cleaned, y_train_cleaned, y_test_cleaned = train_test_split(X, y, test_size=0.2, random_state=42)\n\n# Checking split \nprint('X_train shape:', X_train_cleaned.shape)\nprint('y_train shape:', y_train_cleaned.shape)\nprint('X_test shape:', X_test_cleaned.shape)\nprint('y_test shape:', y_test_cleaned.shape)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T12:22:51.451098Z","iopub.execute_input":"2022-07-07T12:22:51.451496Z","iopub.status.idle":"2022-07-07T12:22:54.273303Z","shell.execute_reply.started":"2022-07-07T12:22:51.451462Z","shell.execute_reply":"2022-07-07T12:22:54.272195Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train_cleaned.to_csv('X_train_cleaned')\nX_test_cleaned.to_csv('X_test_cleaned')\ny_train_cleaned.to_csv('y_train_cleaned')\ny_test_cleaned.to_csv('y_test_cleaned')","metadata":{"execution":{"iopub.status.busy":"2022-07-06T15:32:30.48792Z","iopub.execute_input":"2022-07-06T15:32:30.488642Z","iopub.status.idle":"2022-07-06T15:33:46.339409Z","shell.execute_reply.started":"2022-07-06T15:32:30.488606Z","shell.execute_reply":"2022-07-06T15:33:46.338297Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Train/test split\nX = data_reduced_processed.drop(['target'], axis=1)\ny = data_reduced_processed['target']\n\nX_train_processed, X_test_processed, y_train_processed, y_test_processed = train_test_split(X, y, test_size=0.2, \n                                                                                            random_state=42)\n\n# Checking split \nprint('X_train shape:', X_train_processed.shape)\nprint('y_train shape:', y_train_processed.shape)\nprint('X_test shape:', X_test_processed.shape)\nprint('y_test shape:', y_test_processed.shape)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T12:22:58.361646Z","iopub.execute_input":"2022-07-07T12:22:58.362414Z","iopub.status.idle":"2022-07-07T12:23:02.215287Z","shell.execute_reply.started":"2022-07-07T12:22:58.362354Z","shell.execute_reply":"2022-07-07T12:23:02.214188Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train_processed.to_csv('X_train_processed')\nX_test_processed.to_csv('X_test_processed')\ny_train_processed.to_csv('y_train_processed')\ny_test_processed.to_csv('y_test_processed')","metadata":{"execution":{"iopub.status.busy":"2022-07-06T15:34:10.485799Z","iopub.execute_input":"2022-07-06T15:34:10.486754Z","iopub.status.idle":"2022-07-06T15:36:04.591365Z","shell.execute_reply.started":"2022-07-06T15:34:10.486708Z","shell.execute_reply":"2022-07-06T15:36:04.59006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <a name=\"C8\">3.2: Model testing</a>\n\n","metadata":{}},{"cell_type":"markdown","source":"We will use the LGBM Classifier and the CatBoost Classifier. We have chosen those models primarily because they can handle missing values by themselves.","metadata":{}},{"cell_type":"code","source":"#Comparison table\nres_tab = pd.DataFrame(index=['computation time', 'metric score', 'accuracy score'])\n\n#Comparison table\nres_tab_cleaned = pd.DataFrame(index=['computation time', 'metric score', 'accuracy score'])\n\n#Comparison table\nres_tab_processed = pd.DataFrame(index=['computation time', 'metric score', 'accuracy score'])","metadata":{"execution":{"iopub.status.busy":"2022-07-07T09:05:26.497199Z","iopub.execute_input":"2022-07-07T09:05:26.497576Z","iopub.status.idle":"2022-07-07T09:05:26.505833Z","shell.execute_reply.started":"2022-07-07T09:05:26.497545Z","shell.execute_reply":"2022-07-07T09:05:26.503816Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### A. With original data","metadata":{}},{"cell_type":"code","source":"X_train = pd.read_csv('../input/data-entrainement-amx/X_train')\nX_train.drop('Unnamed: 0', axis=1, inplace=True)\n\nX_test = pd.read_csv('../input/data-entrainement-amx/X_test')\nX_test.drop('Unnamed: 0', axis=1, inplace=True)\n\ny_train = pd.read_csv('../input/data-entrainement-amx/y_train')\ny_train.drop('Unnamed: 0', axis=1, inplace=True)\n\ny_test = pd.read_csv('../input/data-entrainement-amx/y_test')\ny_test.drop('Unnamed: 0', axis=1, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T08:35:38.680732Z","iopub.execute_input":"2022-07-07T08:35:38.681880Z","iopub.status.idle":"2022-07-07T08:36:24.481235Z","shell.execute_reply.started":"2022-07-07T08:35:38.681830Z","shell.execute_reply":"2022-07-07T08:36:24.480172Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 1. LGBM ","metadata":{}},{"cell_type":"code","source":"search_params = {'n_estimators': [100, 300],\n                'learning_rate': [0.01, 0.1]}\nlgbm = LGBMClassifier()\n\n#GridSearch\nlgbmCV = GridSearchCV(estimator=lgbm, param_grid=search_params, verbose=0)\nlgbmCV.fit(X_train, y_train.values.ravel())\n\nbest_lgbm = lgbmCV.best_estimator_","metadata":{"execution":{"iopub.status.busy":"2022-07-07T08:36:41.895905Z","iopub.execute_input":"2022-07-07T08:36:41.896578Z","iopub.status.idle":"2022-07-07T09:02:10.844198Z","shell.execute_reply.started":"2022-07-07T08:36:41.896529Z","shell.execute_reply":"2022-07-07T09:02:10.843187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lgbmCV.best_params_","metadata":{"execution":{"iopub.status.busy":"2022-07-07T09:04:47.957067Z","iopub.execute_input":"2022-07-07T09:04:47.957911Z","iopub.status.idle":"2022-07-07T09:04:47.964614Z","shell.execute_reply.started":"2022-07-07T09:04:47.957872Z","shell.execute_reply":"2022-07-07T09:04:47.963342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lgbm_time = lgbmCV.cv_results_['mean_fit_time'][3]\nprint(lgbm_time)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T09:04:57.577557Z","iopub.execute_input":"2022-07-07T09:04:57.578442Z","iopub.status.idle":"2022-07-07T09:04:57.584053Z","shell.execute_reply.started":"2022-07-07T09:04:57.578402Z","shell.execute_reply":"2022-07-07T09:04:57.583025Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_preds_lgbm = best_lgbm.predict(X_test)\n\ny_pred_lgbm = pd.DataFrame(y_preds_lgbm, index=y_test.index, columns=['prediction'])\ny_true = pd.DataFrame(y_test, index=y_test.index, columns=['target'])\nM_lgbm = amex_metric(y_true, y_pred_lgbm)\nprint(M_lgbm)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T09:05:06.440204Z","iopub.execute_input":"2022-07-07T09:05:06.440588Z","iopub.status.idle":"2022-07-07T09:05:09.498674Z","shell.execute_reply.started":"2022-07-07T09:05:06.440555Z","shell.execute_reply":"2022-07-07T09:05:09.497578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"accuracy_lgbm = accuracy_score(y_true, y_preds_lgbm)\nprint(accuracy_lgbm)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T09:05:13.334216Z","iopub.execute_input":"2022-07-07T09:05:13.334599Z","iopub.status.idle":"2022-07-07T09:05:13.351399Z","shell.execute_reply.started":"2022-07-07T09:05:13.334566Z","shell.execute_reply":"2022-07-07T09:05:13.350338Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"res_tab.loc['computation time', 'LGBM'] = lgbm_time\nres_tab.loc['metric score', 'LGBM'] = M_lgbm\nres_tab.loc['accuracy score', 'LGBM'] = accuracy_lgbm\nres_tab","metadata":{"execution":{"iopub.status.busy":"2022-07-07T09:05:32.221247Z","iopub.execute_input":"2022-07-07T09:05:32.221639Z","iopub.status.idle":"2022-07-07T09:05:32.239815Z","shell.execute_reply.started":"2022-07-07T09:05:32.221604Z","shell.execute_reply":"2022-07-07T09:05:32.238798Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 2. CatBoost","metadata":{}},{"cell_type":"code","source":"search_params = {'iterations': [50, 100, 500],\n                 'learning_rate': [0.01, 0.1]}\ncat = CatBoostClassifier()\n\n#GridSearch\ncatCV = GridSearchCV(estimator=cat, param_grid=search_params, verbose=0)\ncatCV.fit(X_train, y_train.values.ravel())\n\nbest_cat = catCV.best_estimator_","metadata":{"execution":{"iopub.status.busy":"2022-07-07T09:05:38.913394Z","iopub.execute_input":"2022-07-07T09:05:38.913922Z","iopub.status.idle":"2022-07-07T09:42:29.029507Z","shell.execute_reply.started":"2022-07-07T09:05:38.913881Z","shell.execute_reply":"2022-07-07T09:42:29.028384Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"catCV.best_params_","metadata":{"execution":{"iopub.status.busy":"2022-07-07T09:51:07.447104Z","iopub.execute_input":"2022-07-07T09:51:07.448086Z","iopub.status.idle":"2022-07-07T09:51:07.455692Z","shell.execute_reply.started":"2022-07-07T09:51:07.448045Z","shell.execute_reply":"2022-07-07T09:51:07.454308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cat_time = catCV.cv_results_['mean_fit_time'][5]\nprint(cat_time)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T09:51:17.789412Z","iopub.execute_input":"2022-07-07T09:51:17.790372Z","iopub.status.idle":"2022-07-07T09:51:17.796720Z","shell.execute_reply.started":"2022-07-07T09:51:17.790330Z","shell.execute_reply":"2022-07-07T09:51:17.795672Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_preds_cat = best_cat.predict(X_test)\n\ny_pred_cat = pd.DataFrame(y_preds_cat, index=y_test.index, columns=['prediction'])\ny_true = pd.DataFrame(y_test, index=y_test.index, columns=['target'])\nM_cat = amex_metric(y_true, y_pred_cat)\nprint(M_cat)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T09:51:21.552413Z","iopub.execute_input":"2022-07-07T09:51:21.553140Z","iopub.status.idle":"2022-07-07T09:51:22.091388Z","shell.execute_reply.started":"2022-07-07T09:51:21.553099Z","shell.execute_reply":"2022-07-07T09:51:22.090438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"accuracy_cat = accuracy_score(y_true, y_preds_cat)\nprint(accuracy_cat)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T09:51:24.079065Z","iopub.execute_input":"2022-07-07T09:51:24.079753Z","iopub.status.idle":"2022-07-07T09:51:24.096230Z","shell.execute_reply.started":"2022-07-07T09:51:24.079715Z","shell.execute_reply":"2022-07-07T09:51:24.095168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"res_tab.loc['computation time', 'CatBoost'] = cat_time\nres_tab.loc['metric score', 'CatBoost'] = M_cat\nres_tab.loc['accuracy score', 'CatBoost'] = accuracy_cat\nres_tab","metadata":{"execution":{"iopub.status.busy":"2022-07-07T09:51:27.814693Z","iopub.execute_input":"2022-07-07T09:51:27.815715Z","iopub.status.idle":"2022-07-07T09:51:27.831023Z","shell.execute_reply.started":"2022-07-07T09:51:27.815636Z","shell.execute_reply":"2022-07-07T09:51:27.830032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### B. With cleaned data ","metadata":{}},{"cell_type":"code","source":"X_train_cleaned = pd.read_csv('../input/data-entrainement-amx/X_train_cleaned')\nX_train_cleaned.drop('Unnamed: 0', axis=1, inplace=True)\n\nX_test_cleaned = pd.read_csv('../input/data-entrainement-amx/X_test_cleaned')\nX_test_cleaned.drop('Unnamed: 0', axis=1, inplace=True)\n\ny_train_cleaned = pd.read_csv('../input/data-entrainement-amx/y_train_cleaned')\ny_train_cleaned.drop('Unnamed: 0', axis=1, inplace=True)\n\ny_test_cleaned = pd.read_csv('../input/data-entrainement-amx/y_test_cleaned')\ny_test_cleaned.drop('Unnamed: 0', axis=1, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T06:53:27.666365Z","iopub.execute_input":"2022-07-07T06:53:27.667345Z","iopub.status.idle":"2022-07-07T06:53:57.721865Z","shell.execute_reply.started":"2022-07-07T06:53:27.667298Z","shell.execute_reply":"2022-07-07T06:53:57.720903Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 1. LGBM","metadata":{}},{"cell_type":"code","source":"search_params = {'n_estimators': [100, 300],\n                'learning_rate': [0.01, 0.1]}\nlgbm = LGBMClassifier()\n\n#GridSearch\nlgbmCV = GridSearchCV(estimator=lgbm, param_grid=search_params, verbose=0)\nlgbmCV.fit(X_train_cleaned, y_train_cleaned.values.ravel())\n\nbest_lgbm = lgbmCV.best_estimator_","metadata":{"execution":{"iopub.status.busy":"2022-07-07T06:54:37.844300Z","iopub.execute_input":"2022-07-07T06:54:37.844662Z","iopub.status.idle":"2022-07-07T07:06:49.541777Z","shell.execute_reply.started":"2022-07-07T06:54:37.844619Z","shell.execute_reply":"2022-07-07T07:06:49.540915Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lgbmCV.best_params_","metadata":{"execution":{"iopub.status.busy":"2022-07-07T07:07:53.010328Z","iopub.execute_input":"2022-07-07T07:07:53.010744Z","iopub.status.idle":"2022-07-07T07:07:53.017351Z","shell.execute_reply.started":"2022-07-07T07:07:53.010713Z","shell.execute_reply":"2022-07-07T07:07:53.016270Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lgbm_time = lgbmCV.cv_results_['mean_fit_time'][3]\nprint(lgbm_time)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T07:08:50.339894Z","iopub.execute_input":"2022-07-07T07:08:50.340448Z","iopub.status.idle":"2022-07-07T07:08:50.345872Z","shell.execute_reply.started":"2022-07-07T07:08:50.340412Z","shell.execute_reply":"2022-07-07T07:08:50.344692Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_preds_lgbm = best_lgbm.predict(X_test_cleaned)\n\ny_pred_lgbm = pd.DataFrame(y_preds_lgbm, index=y_test_cleaned.index, columns=['prediction'])\ny_true = pd.DataFrame(y_test_cleaned, index=y_test_cleaned.index, columns=['target'])\nM_lgbm = amex_metric(y_true, y_pred_lgbm)\nprint(M_lgbm)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T07:09:08.716023Z","iopub.execute_input":"2022-07-07T07:09:08.716363Z","iopub.status.idle":"2022-07-07T07:09:10.655397Z","shell.execute_reply.started":"2022-07-07T07:09:08.716335Z","shell.execute_reply":"2022-07-07T07:09:10.654391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"accuracy_lgbm = accuracy_score(y_true, y_preds_lgbm)\nprint(accuracy_lgbm)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T07:09:14.467473Z","iopub.execute_input":"2022-07-07T07:09:14.467838Z","iopub.status.idle":"2022-07-07T07:09:14.485221Z","shell.execute_reply.started":"2022-07-07T07:09:14.467808Z","shell.execute_reply":"2022-07-07T07:09:14.484032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"res_tab_cleaned.loc['computation time', 'LGBM'] = lgbm_time\nres_tab_cleaned.loc['metric score', 'LGBM'] = M_lgbm\nres_tab_cleaned.loc['accuracy score', 'LGBM'] = accuracy_lgbm\nres_tab_cleaned","metadata":{"execution":{"iopub.status.busy":"2022-07-07T07:09:54.482696Z","iopub.execute_input":"2022-07-07T07:09:54.483037Z","iopub.status.idle":"2022-07-07T07:09:54.498773Z","shell.execute_reply.started":"2022-07-07T07:09:54.483010Z","shell.execute_reply":"2022-07-07T07:09:54.497800Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 2. CatBoost","metadata":{}},{"cell_type":"code","source":"search_params = {'iterations': [50, 100, 500],\n                 'learning_rate': [0.01, 0.1]}\ncat = CatBoostClassifier()\n\n#GridSearch\ncatCV = GridSearchCV(estimator=cat, param_grid=search_params, verbose=0)\ncatCV.fit(X_train_cleaned, y_train_cleaned.values.ravel())\n\nbest_cat = catCV.best_estimator_","metadata":{"execution":{"iopub.status.busy":"2022-07-07T07:10:21.447698Z","iopub.execute_input":"2022-07-07T07:10:21.448038Z","iopub.status.idle":"2022-07-07T07:30:22.697783Z","shell.execute_reply.started":"2022-07-07T07:10:21.448011Z","shell.execute_reply":"2022-07-07T07:30:22.696721Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"catCV.best_params_","metadata":{"execution":{"iopub.status.busy":"2022-07-07T07:31:55.200258Z","iopub.execute_input":"2022-07-07T07:31:55.200605Z","iopub.status.idle":"2022-07-07T07:31:55.207339Z","shell.execute_reply.started":"2022-07-07T07:31:55.200575Z","shell.execute_reply":"2022-07-07T07:31:55.206450Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cat_time = catCV.cv_results_['mean_fit_time'][5]\nprint(cat_time)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T07:32:23.191213Z","iopub.execute_input":"2022-07-07T07:32:23.191554Z","iopub.status.idle":"2022-07-07T07:32:23.197101Z","shell.execute_reply.started":"2022-07-07T07:32:23.191524Z","shell.execute_reply":"2022-07-07T07:32:23.195716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_preds_cat = best_cat.predict(X_test_cleaned)\n\ny_pred_cat = pd.DataFrame(y_preds_cat, index=y_test_cleaned.index, columns=['prediction'])\ny_true = pd.DataFrame(y_test_cleaned, index=y_test_cleaned.index, columns=['target'])\nM_cat = amex_metric(y_true, y_pred_cat)\nprint(M_cat)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T07:32:27.862518Z","iopub.execute_input":"2022-07-07T07:32:27.863167Z","iopub.status.idle":"2022-07-07T07:32:28.344894Z","shell.execute_reply.started":"2022-07-07T07:32:27.863126Z","shell.execute_reply":"2022-07-07T07:32:28.343158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"accuracy_cat = accuracy_score(y_true, y_preds_cat)\nprint(accuracy_cat)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T07:32:31.354454Z","iopub.execute_input":"2022-07-07T07:32:31.355415Z","iopub.status.idle":"2022-07-07T07:32:31.371861Z","shell.execute_reply.started":"2022-07-07T07:32:31.355367Z","shell.execute_reply":"2022-07-07T07:32:31.370854Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"res_tab_cleaned.loc['computation time', 'CatBoost'] = cat_time\nres_tab_cleaned.loc['metric score', 'CatBoost'] = M_cat\nres_tab_cleaned.loc['accuracy score', 'CatBoost'] = accuracy_cat\nres_tab_cleaned","metadata":{"execution":{"iopub.status.busy":"2022-07-07T07:32:34.755982Z","iopub.execute_input":"2022-07-07T07:32:34.756326Z","iopub.status.idle":"2022-07-07T07:32:34.770530Z","shell.execute_reply.started":"2022-07-07T07:32:34.756297Z","shell.execute_reply":"2022-07-07T07:32:34.769591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### C. With processed data","metadata":{}},{"cell_type":"code","source":"X_train_processed = pd.read_csv('../input/data-entrainement-amx/X_train_processed')\nX_train_processed.drop('Unnamed: 0', axis=1, inplace=True)\n\nX_test_processed = pd.read_csv('../input/data-entrainement-amx/X_test_processed')\nX_test_processed.drop('Unnamed: 0', axis=1, inplace=True)\n\ny_train_processed = pd.read_csv('../input/data-entrainement-amx/y_train_processed')\ny_train_processed.drop('Unnamed: 0', axis=1, inplace=True)\n\ny_test_processed = pd.read_csv('../input/data-entrainement-amx/y_test_processed')\ny_test_processed.drop('Unnamed: 0', axis=1, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T07:32:41.128067Z","iopub.execute_input":"2022-07-07T07:32:41.128405Z","iopub.status.idle":"2022-07-07T07:33:22.941324Z","shell.execute_reply.started":"2022-07-07T07:32:41.128378Z","shell.execute_reply":"2022-07-07T07:33:22.940359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 1. LGBM","metadata":{}},{"cell_type":"code","source":"search_params = {'n_estimators': [100, 300],\n                'learning_rate': [0.01, 0.1]}\nlgbm = LGBMClassifier()\n\n#GridSearch\nlgbmCV = GridSearchCV(estimator=lgbm, param_grid=search_params, verbose=0)\nlgbmCV.fit(X_train_processed, y_train_processed.values.ravel())\n\nbest_lgbm = lgbmCV.best_estimator_","metadata":{"execution":{"iopub.status.busy":"2022-07-07T07:34:31.133523Z","iopub.execute_input":"2022-07-07T07:34:31.134230Z","iopub.status.idle":"2022-07-07T07:56:05.851023Z","shell.execute_reply.started":"2022-07-07T07:34:31.134191Z","shell.execute_reply":"2022-07-07T07:56:05.850066Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lgbmCV.best_params_","metadata":{"execution":{"iopub.status.busy":"2022-07-07T07:57:39.683434Z","iopub.execute_input":"2022-07-07T07:57:39.684160Z","iopub.status.idle":"2022-07-07T07:57:39.696217Z","shell.execute_reply.started":"2022-07-07T07:57:39.684111Z","shell.execute_reply":"2022-07-07T07:57:39.694632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lgbm_time = lgbmCV.cv_results_['mean_fit_time'][3]\nprint(lgbm_time)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T07:57:56.092957Z","iopub.execute_input":"2022-07-07T07:57:56.093303Z","iopub.status.idle":"2022-07-07T07:57:56.099167Z","shell.execute_reply.started":"2022-07-07T07:57:56.093275Z","shell.execute_reply":"2022-07-07T07:57:56.098191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_preds_lgbm = best_lgbm.predict(X_test_processed)\n\ny_pred_lgbm = pd.DataFrame(y_preds_lgbm, index=y_test_processed.index, columns=['prediction'])\ny_true = pd.DataFrame(y_test_processed, index=y_test_processed.index, columns=['target'])\nM_lgbm = amex_metric(y_true, y_pred_lgbm)\nprint(M_lgbm)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T07:58:04.060360Z","iopub.execute_input":"2022-07-07T07:58:04.060754Z","iopub.status.idle":"2022-07-07T07:58:06.191095Z","shell.execute_reply.started":"2022-07-07T07:58:04.060721Z","shell.execute_reply":"2022-07-07T07:58:06.189230Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"accuracy_lgbm = accuracy_score(y_true, y_preds_lgbm)\nprint(accuracy_lgbm)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T07:58:11.605112Z","iopub.execute_input":"2022-07-07T07:58:11.605505Z","iopub.status.idle":"2022-07-07T07:58:11.622106Z","shell.execute_reply.started":"2022-07-07T07:58:11.605472Z","shell.execute_reply":"2022-07-07T07:58:11.621045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"res_tab_processed.loc['computation time', 'LGBM'] = lgbm_time\nres_tab_processed.loc['metric score', 'LGBM'] = M_lgbm\nres_tab_processed.loc['accuracy score', 'LGBM'] = accuracy_lgbm\nres_tab_processed","metadata":{"execution":{"iopub.status.busy":"2022-07-07T07:58:20.571782Z","iopub.execute_input":"2022-07-07T07:58:20.572351Z","iopub.status.idle":"2022-07-07T07:58:20.584166Z","shell.execute_reply.started":"2022-07-07T07:58:20.572307Z","shell.execute_reply":"2022-07-07T07:58:20.582148Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 2. CatBoost","metadata":{}},{"cell_type":"code","source":"search_params = {'iterations': [50, 100, 500],\n                 'learning_rate': [0.01, 0.1]}\ncat = CatBoostClassifier()\n\n#GridSearch\ncatCV = GridSearchCV(estimator=cat, param_grid=search_params, verbose=0)\ncatCV.fit(X_train_processed, y_train_processed.values.ravel())\n\nbest_cat = catCV.best_estimator_","metadata":{"execution":{"iopub.status.busy":"2022-07-07T07:58:29.612210Z","iopub.execute_input":"2022-07-07T07:58:29.612547Z","iopub.status.idle":"2022-07-07T08:31:00.229831Z","shell.execute_reply.started":"2022-07-07T07:58:29.612519Z","shell.execute_reply":"2022-07-07T08:31:00.228791Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"catCV.best_params_","metadata":{"execution":{"iopub.status.busy":"2022-07-07T08:32:54.931731Z","iopub.execute_input":"2022-07-07T08:32:54.932824Z","iopub.status.idle":"2022-07-07T08:32:54.939182Z","shell.execute_reply.started":"2022-07-07T08:32:54.932774Z","shell.execute_reply":"2022-07-07T08:32:54.938173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cat_time = catCV.cv_results_['mean_fit_time'][5]\nprint(cat_time)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T08:33:09.118833Z","iopub.execute_input":"2022-07-07T08:33:09.119201Z","iopub.status.idle":"2022-07-07T08:33:09.125232Z","shell.execute_reply.started":"2022-07-07T08:33:09.119172Z","shell.execute_reply":"2022-07-07T08:33:09.124184Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_preds_cat = best_cat.predict(X_test_processed)\n\ny_pred_cat = pd.DataFrame(y_preds_cat, index=y_test_processed.index, columns=['prediction'])\ny_true = pd.DataFrame(y_test_processed, index=y_test_processed.index, columns=['target'])\nM_cat = amex_metric(y_true, y_pred_cat)\nprint(M_cat)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T08:33:41.640484Z","iopub.execute_input":"2022-07-07T08:33:41.641199Z","iopub.status.idle":"2022-07-07T08:33:42.165804Z","shell.execute_reply.started":"2022-07-07T08:33:41.641159Z","shell.execute_reply":"2022-07-07T08:33:42.164621Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"accuracy_cat = accuracy_score(y_true, y_preds_cat)\nprint(accuracy_cat)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T08:33:46.312127Z","iopub.execute_input":"2022-07-07T08:33:46.312494Z","iopub.status.idle":"2022-07-07T08:33:46.328952Z","shell.execute_reply.started":"2022-07-07T08:33:46.312465Z","shell.execute_reply":"2022-07-07T08:33:46.327879Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"res_tab_processed.loc['computation time', 'CatBoost'] = cat_time\nres_tab_processed.loc['metric score', 'CatBoost'] = M_cat\nres_tab_processed.loc['accuracy score', 'CatBoost'] = accuracy_cat\nres_tab_processed","metadata":{"execution":{"iopub.status.busy":"2022-07-07T08:33:51.339936Z","iopub.execute_input":"2022-07-07T08:33:51.340340Z","iopub.status.idle":"2022-07-07T08:33:51.353640Z","shell.execute_reply.started":"2022-07-07T08:33:51.340293Z","shell.execute_reply":"2022-07-07T08:33:51.352220Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In each case both models give similar results, but the LGBM is almost two times faster.</br>\nAbout the data, deleting all the variables with missing values does not seem to be the best idea since it reduces the quality of prediction. Certainly the difference is small but we used only 10% of available data. For the two other methodswe obtain close results, a little bit longer unprocessed data.</br>\nWe remain convinced that the delinquency variables, which contain many missing values, can reveal small details that are very important in pointing to payment default. Even if this does not improve the general quality of the model, taking into account all the variables would perhaps make it possible to avoid certain non-repayments or fraud and thus avoid dramatic consequences given the central role of banks in our societies.","metadata":{}},{"cell_type":"markdown","source":"# <a name=\"C9\">Part 4: Final model</a>\n\nGiven the previous results and our understanding of the problem, our model choice is the LGBM Classifier trained on the original data without any processing of missing values.","metadata":{}},{"cell_type":"code","source":"train.drop(['customer_ID'], axis=1, inplace=True)\ntrain = pd.get_dummies(data=train, columns=categorical_var, dummy_na=True)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lgbm = LGBMClassifier(learning_rate=0.1, n_estimators=300)\nlgbm.fit(train, train_labels['target'])\n\n#Save final model\npickle.dump(lgbm, open('final_model_lgbm.pkl','wb'))","metadata":{"execution":{"iopub.status.busy":"2022-07-08T08:20:15.677038Z","iopub.execute_input":"2022-07-08T08:20:15.677423Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Load model\nlgbm = pd.read_pickle('../input/data-entrainement-amx/final_model_lgbm.pkl')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preds = lgbm.predict(test)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission[\"prediction\"] = preds\nsample_submission.to_csv('final_sample_submission.csv', index=False)","metadata":{},"execution_count":null,"outputs":[]}]}