{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"#importing required libraries\n\nimport numpy as np \nimport pandas as pd \nimport os\n\n#for dirname, _, filenames in os.walk('/kaggle/input'):\n    #for filename in filenames:\n        #print(os.path.join(dirname, filename))\ntrain = pd.read_parquet('../input/amex-parquet/train_data.parquet') #importing train dataset\ntrain.shape","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-07-26T04:52:37.261641Z","iopub.execute_input":"2022-07-26T04:52:37.262115Z","iopub.status.idle":"2022-07-26T04:53:00.535119Z","shell.execute_reply.started":"2022-07-26T04:52:37.262080Z","shell.execute_reply":"2022-07-26T04:53:00.534176Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* I have used parquet dataset as it loads very fast the entire dataset, as we see here, Train dataset has *190 independent* columns(X) and 1 Target column(y) with *5531451 Rows.* \n* This size of dataset would have taken comparitively more time to run if we had used csv format. ","metadata":{}},{"cell_type":"markdown","source":"**Grouping the customer_ID and extracting the latest entry of each customer**<br>\nThe final submission file should have 924621 unique values, which is unique number of customers. So I have grouped the customers with their customer_Id and taken the last transaction done by them.  ","metadata":{}},{"cell_type":"code","source":"train=train.groupby('customer_ID').tail(1)# tail gives the last entry\ntrain.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:00.537740Z","iopub.execute_input":"2022-07-26T04:53:00.538289Z","iopub.status.idle":"2022-07-26T04:53:02.812887Z","shell.execute_reply.started":"2022-07-26T04:53:00.538255Z","shell.execute_reply":"2022-07-26T04:53:02.811802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Hence, we see observe that we have 458913 unique cstomer_IDs. This approach also helps us by reducing the number of rows we are handling and making other executions faster. ","metadata":{}},{"cell_type":"markdown","source":"\n**Droppping columns **\nFurther lets think about how to reduce the no.of columns:\n1. There are many columns with missing values greater than 85%. So I have decided to drop them as they may not cotibute significantly to the prediction of defaulters\n2. ","metadata":{}},{"cell_type":"code","source":"train.dropna(axis=1, thresh=int(0.85*len(train)),inplace=True) #Droppping columns with missing values greater than 85% \ntrain.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:02.815234Z","iopub.execute_input":"2022-07-26T04:53:02.815626Z","iopub.status.idle":"2022-07-26T04:53:03.199650Z","shell.execute_reply.started":"2022-07-26T04:53:02.815595Z","shell.execute_reply":"2022-07-26T04:53:03.198513Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation**<br>\n* We are left with 155 columns after dropping columns with 85% missing values","metadata":{}},{"cell_type":"markdown","source":"Lets check the distribution of D_63 and D_64 with respect to target column","metadata":{}},{"cell_type":"code","source":"train[train.target==0].D_63.value_counts().plot.bar()\n","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:03.202761Z","iopub.execute_input":"2022-07-26T04:53:03.203687Z","iopub.status.idle":"2022-07-26T04:53:03.716578Z","shell.execute_reply.started":"2022-07-26T04:53:03.203651Z","shell.execute_reply":"2022-07-26T04:53:03.714896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[train.target==1].D_63.value_counts().plot.bar()","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:03.718392Z","iopub.execute_input":"2022-07-26T04:53:03.718736Z","iopub.status.idle":"2022-07-26T04:53:04.132887Z","shell.execute_reply.started":"2022-07-26T04:53:03.718697Z","shell.execute_reply":"2022-07-26T04:53:04.131988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation**<br>\n* There is not much difference in the distribution so I am dropping it","metadata":{}},{"cell_type":"code","source":"train[train.target==1].D_64.value_counts().plot.bar()","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:04.134668Z","iopub.execute_input":"2022-07-26T04:53:04.135268Z","iopub.status.idle":"2022-07-26T04:53:04.360887Z","shell.execute_reply.started":"2022-07-26T04:53:04.135229Z","shell.execute_reply":"2022-07-26T04:53:04.359934Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[train.target==0].D_64.value_counts().plot.bar()","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:04.362602Z","iopub.execute_input":"2022-07-26T04:53:04.363217Z","iopub.status.idle":"2022-07-26T04:53:04.759720Z","shell.execute_reply.started":"2022-07-26T04:53:04.363178Z","shell.execute_reply":"2022-07-26T04:53:04.758490Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation**\n* There is difference in distribution of categories so we will keep it and do one hot encoding ","metadata":{}},{"cell_type":"markdown","source":"**Dropping the categorical columns**<br>\n1. customer_ID: All the values are unique\n2. S_2 is date column and we have taken only the last transaction for each customer so S_2 has become insignificant\n3. D_63: There is not major change in the distribution between the target=0 and target=1","metadata":{}},{"cell_type":"code","source":"train.drop(['customer_ID','S_2','D_63'],axis=1,inplace=True)# dropping the categorical columns\ntrain.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:04.761660Z","iopub.execute_input":"2022-07-26T04:53:04.762219Z","iopub.status.idle":"2022-07-26T04:53:04.843060Z","shell.execute_reply.started":"2022-07-26T04:53:04.762184Z","shell.execute_reply":"2022-07-26T04:53:04.841599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# one hot encoding for categorical column D_64\ndef OHE(original_dataframe, col):    \n    dummies = pd.get_dummies(original_dataframe[[col]])\n    res = pd.concat([original_dataframe, dummies], axis=1)\n    res = res.drop([col], axis=1)\n    return(res) \ntrain = OHE(train,'D_64')","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:04.845228Z","iopub.execute_input":"2022-07-26T04:53:04.845583Z","iopub.status.idle":"2022-07-26T04:53:05.136189Z","shell.execute_reply.started":"2022-07-26T04:53:04.845551Z","shell.execute_reply":"2022-07-26T04:53:05.135077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:05.139393Z","iopub.execute_input":"2022-07-26T04:53:05.139908Z","iopub.status.idle":"2022-07-26T04:53:05.147392Z","shell.execute_reply.started":"2022-07-26T04:53:05.139880Z","shell.execute_reply":"2022-07-26T04:53:05.145577Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Dropping columns who have correlation less than 0.3 with target**<br> \n* Columns which have less correlation with the target column, *may* not contribute significantly to the prediction of the target column\ncorrwith() function finds the correlation of each column with the target columns and those whose value is less than 0.3 are being dropped.","metadata":{}},{"cell_type":"code","source":"train.drop(train.columns[train.corrwith(train['target']).abs()<0.3],axis=1,inplace=True)#Droppping columns who have correlation less than 0.3 with target \ntrain.shape\n","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:05.149341Z","iopub.execute_input":"2022-07-26T04:53:05.149721Z","iopub.status.idle":"2022-07-26T04:53:06.118523Z","shell.execute_reply.started":"2022-07-26T04:53:05.149690Z","shell.execute_reply":"2022-07-26T04:53:06.117004Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation**<br>\n* After dropping columns with low correlation with target we are left with 37 columns which includes 'target'.\n","metadata":{}},{"cell_type":"code","source":"train.columns","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:06.120096Z","iopub.execute_input":"2022-07-26T04:53:06.120471Z","iopub.status.idle":"2022-07-26T04:53:06.129059Z","shell.execute_reply.started":"2022-07-26T04:53:06.120415Z","shell.execute_reply":"2022-07-26T04:53:06.128024Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To make the fetching of the test data faster, we will fetch only the 36 columns which are left in the train dataset. \n* We will remove target from the column list \n* Add customer_ID so we can groupby customer_ID and get unique customer list","metadata":{}},{"cell_type":"code","source":"c=list(train.columns)\nc.remove('target')# as target is not present in test data\nc.append('customer_ID')# it will be used for groupby\nc","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:06.130937Z","iopub.execute_input":"2022-07-26T04:53:06.131612Z","iopub.status.idle":"2022-07-26T04:53:06.146917Z","shell.execute_reply.started":"2022-07-26T04:53:06.131572Z","shell.execute_reply":"2022-07-26T04:53:06.145824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#'columns' is used to specify the column list to retrieve the test data \ntest = pd.read_parquet('../input/amex-parquet/test_data.parquet',columns=c) # importing test data with the latest column list from train\ntest.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:06.148369Z","iopub.execute_input":"2022-07-26T04:53:06.148714Z","iopub.status.idle":"2022-07-26T04:53:16.905372Z","shell.execute_reply.started":"2022-07-26T04:53:06.148683Z","shell.execute_reply":"2022-07-26T04:53:16.904181Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation**\n* test data has 11363762 rows","metadata":{}},{"cell_type":"code","source":"test=test.groupby('customer_ID').tail(1)#groupby to get only unique customer ids\n","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:16.908023Z","iopub.execute_input":"2022-07-26T04:53:16.909559Z","iopub.status.idle":"2022-07-26T04:53:18.944877Z","shell.execute_reply.started":"2022-07-26T04:53:16.909504Z","shell.execute_reply":"2022-07-26T04:53:18.941295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:18.948103Z","iopub.execute_input":"2022-07-26T04:53:18.949036Z","iopub.status.idle":"2022-07-26T04:53:18.965976Z","shell.execute_reply.started":"2022-07-26T04:53:18.948953Z","shell.execute_reply":"2022-07-26T04:53:18.963980Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation**\n* We got the unique no.of customers: 924621 ","metadata":{}},{"cell_type":"code","source":"test.drop(['customer_ID'],axis=1,inplace=True) #droping to match the train dataframe column list","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:18.967740Z","iopub.execute_input":"2022-07-26T04:53:18.968321Z","iopub.status.idle":"2022-07-26T04:53:19.083589Z","shell.execute_reply.started":"2022-07-26T04:53:18.968276Z","shell.execute_reply":"2022-07-26T04:53:19.080985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y = train.pop('target')","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:19.088986Z","iopub.execute_input":"2022-07-26T04:53:19.090586Z","iopub.status.idle":"2022-07-26T04:53:19.108083Z","shell.execute_reply.started":"2022-07-26T04:53:19.090535Z","shell.execute_reply":"2022-07-26T04:53:19.105085Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Splitting data into training and testing sets with using Validation Test Data as 25%\nfrom sklearn.model_selection import train_test_split                # To split the data in training and testing part     \nfrom sklearn.linear_model import LogisticRegression  \n#-------------------------------------------------------------------------------------------------------------------------------\nfrom sklearn.metrics import accuracy_score                          # For calculating the accuracy for the model\nfrom sklearn.metrics import precision_score                         # For calculating the Precision of the model\nfrom sklearn.metrics import recall_score                            # For calculating the recall of the model\nfrom sklearn.metrics import precision_recall_curve                  # For precision and recall metric estimation\nfrom sklearn.metrics import confusion_matrix                        # For verifying model performance using confusion matrix\nfrom sklearn.metrics import f1_score                                # For Checking the F1-Score of our model  \nfrom sklearn.metrics import roc_curve                               # For Roc-Auc metric estimation\nfrom sklearn.metrics import plot_roc_curve\nX_train, X_test, y_train, y_test = train_test_split(train, y, test_size=0.25, random_state=42)\n\n# Display the shape of training and testing data\nprint('X_train shape: ', X_train.shape)\nprint('y_train shape: ', y_train.shape)\nprint('X_test shape: ', X_test.shape)\nprint('y_test shape: ', y_test.shape)\nX_train.info()\n\nX_train.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:19.110662Z","iopub.execute_input":"2022-07-26T04:53:19.113026Z","iopub.status.idle":"2022-07-26T04:53:19.603587Z","shell.execute_reply.started":"2022-07-26T04:53:19.112911Z","shell.execute_reply":"2022-07-26T04:53:19.598978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from xgboost import XGBClassifier\n\nXB=XGBClassifier(max_depth = 5,n_estimators=15)","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:19.607362Z","iopub.execute_input":"2022-07-26T04:53:19.607873Z","iopub.status.idle":"2022-07-26T04:53:19.620677Z","shell.execute_reply.started":"2022-07-26T04:53:19.607844Z","shell.execute_reply":"2022-07-26T04:53:19.616729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dt = XB\ndt.fit(X_train, y_train.values.ravel())\nprint('\\nXGBoost Classifier : {:.3f}'.format(accuracy_score(y_test, dt.predict(X_test))))","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:19.625493Z","iopub.execute_input":"2022-07-26T04:53:19.628651Z","iopub.status.idle":"2022-07-26T04:53:43.578906Z","shell.execute_reply.started":"2022-07-26T04:53:19.628529Z","shell.execute_reply":"2022-07-26T04:53:43.577996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_train_pred_count = dt.predict(X_train)\ny_test_pred_count = dt.predict(X_test)","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:43.580690Z","iopub.execute_input":"2022-07-26T04:53:43.581314Z","iopub.status.idle":"2022-07-26T04:53:43.843236Z","shell.execute_reply.started":"2022-07-26T04:53:43.581279Z","shell.execute_reply":"2022-07-26T04:53:43.842117Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import classification_report                   # To generate classification report\nfrom sklearn.metrics import plot_confusion_matrix   \ntrain_report = classification_report(y_train, y_train_pred_count)\ntest_report = classification_report(y_test, y_test_pred_count)\nprint('                    Training Data Report          ')\nprint(train_report)\nprint('                    Test Validation Data Report           ')\nprint(test_report)","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:43.844941Z","iopub.execute_input":"2022-07-26T04:53:43.845521Z","iopub.status.idle":"2022-07-26T04:53:44.466524Z","shell.execute_reply.started":"2022-07-26T04:53:43.845484Z","shell.execute_reply":"2022-07-26T04:53:44.465371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def amex_metric_mod(y_true, y_pred):\n\n    labels     = np.transpose(np.array([y_true, y_pred]))\n    labels     = labels[labels[:, 1].argsort()[::-1]]\n    weights    = np.where(labels[:,0]==0, 20, 1)\n    cut_vals   = labels[np.cumsum(weights) <= int(0.04 * np.sum(weights))]\n    top_four   = np.sum(cut_vals[:,0]) / np.sum(labels[:,0])\n\n    gini = [0,0]\n    for i in [1,0]:\n        labels         = np.transpose(np.array([y_true, y_pred]))\n        labels         = labels[labels[:, i].argsort()[::-1]]\n        weight         = np.where(labels[:,0]==0, 20, 1)\n        weight_random  = np.cumsum(weight / np.sum(weight))\n        total_pos      = np.sum(labels[:, 0] *  weight)\n        cum_pos_found  = np.cumsum(labels[:, 0] * weight)\n        lorentz        = cum_pos_found / total_pos\n        gini[i]        = np.sum((lorentz - weight_random) * weight)\n\n    return 0.5 * (gini[1]/gini[0] + top_four)\nprint(amex_metric_mod(y_train, y_train_pred_count)) ","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:44.468072Z","iopub.execute_input":"2022-07-26T04:53:44.468427Z","iopub.status.idle":"2022-07-26T04:53:44.574300Z","shell.execute_reply.started":"2022-07-26T04:53:44.468399Z","shell.execute_reply":"2022-07-26T04:53:44.573479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(amex_metric_mod(y_test, y_test_pred_count)) ","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:44.575573Z","iopub.execute_input":"2022-07-26T04:53:44.575829Z","iopub.status.idle":"2022-07-26T04:53:44.602376Z","shell.execute_reply.started":"2022-07-26T04:53:44.575804Z","shell.execute_reply":"2022-07-26T04:53:44.600597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_pred_final=dt.predict(test)","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:44.604602Z","iopub.execute_input":"2022-07-26T04:53:44.605062Z","iopub.status.idle":"2022-07-26T04:53:45.109326Z","shell.execute_reply.started":"2022-07-26T04:53:44.605018Z","shell.execute_reply":"2022-07-26T04:53:45.108140Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"c=['customer_ID']\ntest1 = pd.read_parquet('../input/amex-parquet/test_data.parquet',columns=c)","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:45.110856Z","iopub.execute_input":"2022-07-26T04:53:45.111154Z","iopub.status.idle":"2022-07-26T04:53:47.296044Z","shell.execute_reply.started":"2022-07-26T04:53:45.111124Z","shell.execute_reply":"2022-07-26T04:53:47.294340Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test1=test1.groupby('customer_ID').tail(1)","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:47.304474Z","iopub.execute_input":"2022-07-26T04:53:47.305861Z","iopub.status.idle":"2022-07-26T04:53:48.175397Z","shell.execute_reply.started":"2022-07-26T04:53:47.305797Z","shell.execute_reply":"2022-07-26T04:53:48.173620Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df3 = pd.DataFrame({\"customer_ID\":test1.customer_ID,\"prediction\":y_pred_final})","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:48.177522Z","iopub.execute_input":"2022-07-26T04:53:48.178968Z","iopub.status.idle":"2022-07-26T04:53:48.227249Z","shell.execute_reply.started":"2022-07-26T04:53:48.178883Z","shell.execute_reply":"2022-07-26T04:53:48.225925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df3.prediction.value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:48.228796Z","iopub.execute_input":"2022-07-26T04:53:48.229169Z","iopub.status.idle":"2022-07-26T04:53:48.246049Z","shell.execute_reply.started":"2022-07-26T04:53:48.229112Z","shell.execute_reply":"2022-07-26T04:53:48.245181Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df3.to_csv('cust_default_classification_output.csv',index=False, header=True)","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:48.247182Z","iopub.execute_input":"2022-07-26T04:53:48.247890Z","iopub.status.idle":"2022-07-26T04:53:51.693393Z","shell.execute_reply.started":"2022-07-26T04:53:48.247859Z","shell.execute_reply":"2022-07-26T04:53:51.692304Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}