{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-08-11T09:34:39.654606Z","iopub.execute_input":"2022-08-11T09:34:39.655203Z","iopub.status.idle":"2022-08-11T09:34:39.676767Z","shell.execute_reply.started":"2022-08-11T09:34:39.655134Z","shell.execute_reply":"2022-08-11T09:34:39.675487Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"table-of-content\"></a>\n# Table of Content\n### [1. Setup](#setup)\n- [Import Libraries](#import-libraries)\n- [Dataset](#dataset)\n\n### [2. Handling Missing Values](#missing)\n- [Removes columns with >50% missing values](#50%-missing)\n- [Simple Imputer](#simple-imputer)\n\n### [Go to end](#end)","metadata":{}},{"cell_type":"markdown","source":"# Setup\n<a id=\"setup\"></a>","metadata":{}},{"cell_type":"markdown","source":"# Import Libraries\n<a id=\"import-libraries\"></a>","metadata":{}},{"cell_type":"code","source":"# data preparation\nimport pandas as pd\nimport numpy as np\n\n# visualization\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\n# data preprocessing\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import accuracy_score  \nfrom sklearn.metrics import precision_score                         \nfrom sklearn.metrics import recall_score\nfrom sklearn.impute import SimpleImputer\nfrom sklearn import preprocessing\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.model_selection import StratifiedKFold\nfrom lightgbm import LGBMClassifier, log_evaluation\n","metadata":{"execution":{"iopub.status.busy":"2022-08-11T09:34:39.679368Z","iopub.execute_input":"2022-08-11T09:34:39.680097Z","iopub.status.idle":"2022-08-11T09:34:39.695914Z","shell.execute_reply.started":"2022-08-11T09:34:39.680058Z","shell.execute_reply":"2022-08-11T09:34:39.694327Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Dataset\n<a id=\"dataset\"></a>","metadata":{}},{"cell_type":"markdown","source":"- The original dataset provided in csv is too large (50GB)\n- The data cannot fit into memory\n- [Amex-Feather-Dataset](#https://www.kaggle.com/datasets/munumbutt/amexfeather) provided by [@munum](#https://www.kaggle.com/munumbutt) is a [feather file](#https://arrow.apache.org/docs/python/feather.html) that has smaller size than an equivalent csv file","metadata":{}},{"cell_type":"code","source":"%%time\ntrain_data = pd.read_feather('../input/amexfeather/train_data.ftr')\ntest_data = pd.read_feather('../input/amexfeather/test_data.ftr')\ntrain=train_data.groupby('customer_ID').tail(1)\ntrain=train.set_index(['customer_ID'])\n# There are multiple transactions. Lets take only the latest transaction from each customer.\ntest=test_data.groupby('customer_ID').tail(1)\ntest=test.set_index(['customer_ID'])\ndel train_data\ndel test_data","metadata":{"execution":{"iopub.status.busy":"2022-08-11T09:34:39.698292Z","iopub.execute_input":"2022-08-11T09:34:39.699851Z","iopub.status.idle":"2022-08-11T09:35:47.372700Z","shell.execute_reply.started":"2022-08-11T09:34:39.699777Z","shell.execute_reply":"2022-08-11T09:35:47.370129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.info(max_cols=200, show_counts=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-11T09:35:47.376748Z","iopub.execute_input":"2022-08-11T09:35:47.377359Z","iopub.status.idle":"2022-08-11T09:35:48.585636Z","shell.execute_reply.started":"2022-08-11T09:35:47.377308Z","shell.execute_reply":"2022-08-11T09:35:48.584500Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Insights (Train):\n- There are many columns with missing values: Some with a lot missing values, some not much. So we wouldn't drop all columns with missing values.\n- We wouldn't know whether the column is important because the column names are anonymous. Hence, we need to find statistical proof to justify dropping the columns.","metadata":{}},{"cell_type":"markdown","source":"# Handling Missing Values\n<a id=\"missing\"></a>","metadata":{}},{"cell_type":"code","source":"# Calculate how many % instances are missing in each column\nnull_percent = ((train.isnull().sum())/train.shape[0]).tolist()\n# null_percent","metadata":{"execution":{"iopub.status.busy":"2022-08-11T09:35:48.589613Z","iopub.execute_input":"2022-08-11T09:35:48.590641Z","iopub.status.idle":"2022-08-11T09:35:49.072231Z","shell.execute_reply.started":"2022-08-11T09:35:48.590600Z","shell.execute_reply":"2022-08-11T09:35:49.070839Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Removes columns with >50% missing values\n<a id=\"50%-missing\"></a>","metadata":{}},{"cell_type":"code","source":"# n where n is the nth column in the data frame that has more than 50% instances as null\nnull_list = []\nfor i in range(0,len(null_percent)):\n    if null_percent[i]>=0.5:\n        null_list.append(i)\nnull_list # return a list that contains which column has >50% missing values\ndel null_percent","metadata":{"execution":{"iopub.status.busy":"2022-08-11T09:35:49.073921Z","iopub.execute_input":"2022-08-11T09:35:49.074353Z","iopub.status.idle":"2022-08-11T09:35:49.082049Z","shell.execute_reply.started":"2022-08-11T09:35:49.074316Z","shell.execute_reply":"2022-08-11T09:35:49.080592Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Drop columns that have .50% missing values\ntrain_drop = train.drop(train.columns[null_list],axis=1)\ntest_drop = test.drop(test.columns[null_list],axis=1)\ndel train\ndel test\ndel null_list","metadata":{"execution":{"iopub.status.busy":"2022-08-11T09:35:49.083464Z","iopub.execute_input":"2022-08-11T09:35:49.083878Z","iopub.status.idle":"2022-08-11T09:35:50.311679Z","shell.execute_reply.started":"2022-08-11T09:35:49.083841Z","shell.execute_reply":"2022-08-11T09:35:50.310670Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"When more than half of the columns are missing, probably the column is not going to be useful regardless of what imputing method you use unless it tells something about the column like (missing = no account) or something. Which it is impossible for us to know unless you can speak to the data owner.\n\nHence, we dropped 30 columns (>50% missing values) and now we have only 161 columns left.","metadata":{}},{"cell_type":"code","source":"train_drop.info(max_cols=200, show_counts=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-11T09:35:50.313228Z","iopub.execute_input":"2022-08-11T09:35:50.313878Z","iopub.status.idle":"2022-08-11T09:35:50.735698Z","shell.execute_reply.started":"2022-08-11T09:35:50.313840Z","shell.execute_reply":"2022-08-11T09:35:50.734371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's use simple imputer to impute the other columns with missing values","metadata":{}},{"cell_type":"markdown","source":"# Preprocessing Data","metadata":{}},{"cell_type":"code","source":"df_drop = train_drop.drop(columns=[\"target\",'S_2'], axis=1)\ntest_final = test_drop.drop(columns=['S_2'], axis=1)\n# X_train.info(max_cols=200, show_counts=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-11T09:35:50.737360Z","iopub.execute_input":"2022-08-11T09:35:50.737850Z","iopub.status.idle":"2022-08-11T09:35:51.567700Z","shell.execute_reply.started":"2022-08-11T09:35:50.737786Z","shell.execute_reply":"2022-08-11T09:35:51.566498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y = train_drop[\"target\"]\ndel train_drop\ndel test_drop","metadata":{"execution":{"iopub.status.busy":"2022-08-11T09:35:51.569201Z","iopub.execute_input":"2022-08-11T09:35:51.569577Z","iopub.status.idle":"2022-08-11T09:35:51.610883Z","shell.execute_reply.started":"2022-08-11T09:35:51.569543Z","shell.execute_reply":"2022-08-11T09:35:51.609712Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train, X_test, y_train, y_test = train_test_split(df_drop, y, test_size = 0.1)","metadata":{"execution":{"iopub.status.busy":"2022-08-11T09:35:51.612857Z","iopub.execute_input":"2022-08-11T09:35:51.613285Z","iopub.status.idle":"2022-08-11T09:35:52.810793Z","shell.execute_reply.started":"2022-08-11T09:35:51.613250Z","shell.execute_reply":"2022-08-11T09:35:52.809565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"clf = LGBMClassifier(n_estimators=1200,\n                          learning_rate=0.03, reg_lambda=50,\n                          min_child_samples=2400,\n                          num_leaves=95,\n                          colsample_bytree=0.19,\n                          max_bins=511, random_state=1)","metadata":{"execution":{"iopub.status.busy":"2022-08-11T09:35:52.812456Z","iopub.execute_input":"2022-08-11T09:35:52.813775Z","iopub.status.idle":"2022-08-11T09:35:52.819991Z","shell.execute_reply.started":"2022-08-11T09:35:52.813717Z","shell.execute_reply":"2022-08-11T09:35:52.818874Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"clf.fit(X_train,y_train)","metadata":{"execution":{"iopub.status.busy":"2022-08-11T09:35:52.821348Z","iopub.execute_input":"2022-08-11T09:35:52.821666Z","iopub.status.idle":"2022-08-11T09:38:06.889043Z","shell.execute_reply.started":"2022-08-11T09:35:52.821638Z","shell.execute_reply":"2022-08-11T09:38:06.887855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Test the model\ny_predict=clf.predict(X_test)\nprint('LGBM Classifier Accuracy: {:.3f}'.format(accuracy_score(y_test, y_predict)))\n# Achieved 88.4% accuracy","metadata":{"execution":{"iopub.status.busy":"2022-08-11T09:38:06.892730Z","iopub.execute_input":"2022-08-11T09:38:06.893141Z","iopub.status.idle":"2022-08-11T09:38:10.886014Z","shell.execute_reply.started":"2022-08-11T09:38:06.893106Z","shell.execute_reply":"2022-08-11T09:38:10.884576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Predict probabilities of default\ny_test_predict=clf.predict_proba(test_final)","metadata":{"execution":{"iopub.status.busy":"2022-08-11T09:38:10.887613Z","iopub.execute_input":"2022-08-11T09:38:10.888773Z","iopub.status.idle":"2022-08-11T09:39:32.062537Z","shell.execute_reply.started":"2022-08-11T09:38:10.888723Z","shell.execute_reply":"2022-08-11T09:39:32.060781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"a = pd.DataFrame({\"prediction\":y_predict})","metadata":{"execution":{"iopub.status.busy":"2022-08-11T09:39:32.065025Z","iopub.execute_input":"2022-08-11T09:39:32.066106Z","iopub.status.idle":"2022-08-11T09:39:32.073002Z","shell.execute_reply.started":"2022-08-11T09:39:32.066040Z","shell.execute_reply":"2022-08-11T09:39:32.071908Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"a['prediction'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-08-11T09:39:32.074281Z","iopub.execute_input":"2022-08-11T09:39:32.075350Z","iopub.status.idle":"2022-08-11T09:39:32.090821Z","shell.execute_reply.started":"2022-08-11T09:39:32.075312Z","shell.execute_reply":"2022-08-11T09:39:32.089468Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_test_predict","metadata":{"execution":{"iopub.status.busy":"2022-08-11T09:41:35.250228Z","iopub.execute_input":"2022-08-11T09:41:35.251473Z","iopub.status.idle":"2022-08-11T09:41:35.258498Z","shell.execute_reply.started":"2022-08-11T09:41:35.251427Z","shell.execute_reply":"2022-08-11T09:41:35.257648Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Retrieve the probability of default\ny_predict_final=y_test_predict[:,-1]\n\n# Merge the prediction and customer_ID into submission dataframe\nsubmission = pd.DataFrame({\"customer_ID\":test_final.index,\"prediction\":y_predict_final})\n\nsubmission.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-08-11T09:41:47.405881Z","iopub.execute_input":"2022-08-11T09:41:47.406433Z","iopub.status.idle":"2022-08-11T09:41:51.065735Z","shell.execute_reply.started":"2022-08-11T09:41:47.406383Z","shell.execute_reply":"2022-08-11T09:41:51.064342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Work in Progress\n- Model Optimization\n- Try different aggregation method\n- Stratified KFold validation\n- Feature Engineering","metadata":{}},{"cell_type":"markdown","source":"# References\n<a id=\"references\"></a>\n- [AMEX EDA which makes sense](#https://www.kaggle.com/code/ambrosm/amex-eda-which-makes-sense)\n- [AMEX - Light GBM](#https://www.kaggle.com/code/lixinqi98/amex-lightgbm/notebook)\n- [AMEX Default Prediction - EDA & Prediction](#https://www.kaggle.com/code/aryanml007/amex-default-prediction-eda-prediction)","metadata":{}},{"cell_type":"markdown","source":"# [Back](#table-of-content)\n<a id=\"end\"></a>","metadata":{}}]}