{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport gc\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-07-02T15:43:26.748179Z","iopub.execute_input":"2022-07-02T15:43:26.748543Z","iopub.status.idle":"2022-07-02T15:43:26.763238Z","shell.execute_reply.started":"2022-07-02T15:43:26.748513Z","shell.execute_reply":"2022-07-02T15:43:26.762119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prologue\nThis session is Part Two of https://www.kaggle.com/code/aaron874/amex-competition-eda-and-model-selections. Due to the limitation Kaggle memory and large scale of the compeition data, we have seperated the notebook for our work. We will use the best parameters of the best models (XGBoost) as our prediction submission. ","metadata":{}},{"cell_type":"code","source":"!pip install XGBosst\n!pip install imblearn","metadata":{"execution":{"iopub.status.busy":"2022-07-02T15:42:58.672742Z","iopub.status.idle":"2022-07-02T15:42:58.674093Z","shell.execute_reply.started":"2022-07-02T15:42:58.673747Z","shell.execute_reply":"2022-07-02T15:42:58.673780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Data Precessing \nfrom sklearn.preprocessing import OneHotEncoder\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.preprocessing import MinMaxScaler\nfrom sklearn.impute import SimpleImputer\n\n# Model\nimport xgboost as xgb \nfrom sklearn.decomposition import PCA","metadata":{"execution":{"iopub.status.busy":"2022-07-02T15:42:58.676028Z","iopub.status.idle":"2022-07-02T15:42:58.677037Z","shell.execute_reply.started":"2022-07-02T15:42:58.676709Z","shell.execute_reply":"2022-07-02T15:42:58.676741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Train_df = pd.read_feather('../input/amexfeather/train_data.ftr')\nTrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T15:42:58.678913Z","iopub.status.idle":"2022-07-02T15:42:58.679920Z","shell.execute_reply.started":"2022-07-02T15:42:58.679597Z","shell.execute_reply":"2022-07-02T15:42:58.679629Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Train_df = Train_df.groupby('customer_ID').tail(1)\nTrain_df.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-02T15:42:58.681670Z","iopub.status.idle":"2022-07-02T15:42:58.682595Z","shell.execute_reply.started":"2022-07-02T15:42:58.682300Z","shell.execute_reply":"2022-07-02T15:42:58.682330Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Null_Check = pd.DataFrame({'Columns':Train_df.columns,\n                           'Null Ratio':Train_df.isna().sum().values / len(Train_df)}).sort_values(by = ['Null Ratio'], ascending = False)\nNull_Check.head(20)","metadata":{"execution":{"iopub.status.busy":"2022-07-02T15:42:58.684428Z","iopub.status.idle":"2022-07-02T15:42:58.685344Z","shell.execute_reply.started":"2022-07-02T15:42:58.685036Z","shell.execute_reply":"2022-07-02T15:42:58.685075Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in np.linspace(0,1, 11).round(1):\n    print(i, len(Null_Check[Null_Check['Null Ratio'] > i]))\n    \nDrop_Columns = Null_Check[Null_Check['Null Ratio'] > 0.7]['Columns']\nDrop_Columns","metadata":{"execution":{"iopub.status.busy":"2022-07-02T15:42:58.687115Z","iopub.status.idle":"2022-07-02T15:42:58.688076Z","shell.execute_reply.started":"2022-07-02T15:42:58.687749Z","shell.execute_reply":"2022-07-02T15:42:58.687780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Train_df = Train_df.drop(columns = Drop_Columns)\nTrain_df.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-02T15:42:58.689756Z","iopub.status.idle":"2022-07-02T15:42:58.690655Z","shell.execute_reply.started":"2022-07-02T15:42:58.690365Z","shell.execute_reply":"2022-07-02T15:42:58.690394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Prepare for a PCA \nMaster_df = Train_df[['customer_ID','target']].reset_index(drop = True)\n\n# Categorial\nPCA_Cat = Train_df.select_dtypes(include='category').reset_index(drop = True)\n\nfor i in PCA_Cat.columns:\n    PCA_Cat[i].fillna(PCA_Cat[i].quantile(.5), inplace = True)\n    \nPCA_Cat = pd.get_dummies(PCA_Cat, drop_first= True)\n\n# Numeric and Normalize\nPCA_Numeric = Train_df.select_dtypes(include=['float16']).reset_index(drop = True)\n\nfor i in PCA_Numeric.columns:\n    PCA_Numeric[i] = PCA_Numeric[i].astype('float64')\n    PCA_Numeric[i] = PCA_Numeric[i].fillna(PCA_Numeric[i].mean())\n\nPCA_Numeric = pd.DataFrame(StandardScaler().fit_transform(PCA_Numeric), columns = PCA_Numeric.columns)\n    \n# Concat\nPCA_df = pd.concat([PCA_Cat, PCA_Numeric], axis = 1)\n\n# PCA\nPCA_Model = PCA(n_components=6, random_state=0)\nTemp = pd.DataFrame(PCA_Model.fit_transform(PCA_df))\nMaster_df = pd.concat([Master_df.iloc[:, :2], Temp], axis = 1)\nMaster_df","metadata":{"execution":{"iopub.status.busy":"2022-07-02T15:42:58.810606Z","iopub.execute_input":"2022-07-02T15:42:58.811015Z","iopub.status.idle":"2022-07-02T15:42:58.890675Z","shell.execute_reply.started":"2022-07-02T15:42:58.810980Z","shell.execute_reply":"2022-07-02T15:42:58.888371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from imblearn.over_sampling import SMOTE \nsm = SMOTE(random_state=42)\nX, y = sm.fit_resample(Master_df.iloc[:,2:], Master_df['target'])","metadata":{"execution":{"iopub.status.busy":"2022-07-02T15:42:58.891697Z","iopub.status.idle":"2022-07-02T15:42:58.892405Z","shell.execute_reply.started":"2022-07-02T15:42:58.892189Z","shell.execute_reply":"2022-07-02T15:42:58.892212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y.value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T15:42:58.894139Z","iopub.status.idle":"2022-07-02T15:42:58.894518Z","shell.execute_reply.started":"2022-07-02T15:42:58.894340Z","shell.execute_reply":"2022-07-02T15:42:58.894357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def amex_metric(y_true: pd.DataFrame, y_pred: pd.DataFrame) -> float:\n\n    def top_four_percent_captured(y_true: pd.DataFrame, y_pred: pd.DataFrame) -> float:\n        df = (pd.concat([y_true, y_pred], axis='columns')\n              .sort_values('prediction', ascending=False))\n        df['weight'] = df['target'].apply(lambda x: 20 if x==0 else 1)\n        four_pct_cutoff = int(0.04 * df['weight'].sum())\n        df['weight_cumsum'] = df['weight'].cumsum()\n        df_cutoff = df.loc[df['weight_cumsum'] <= four_pct_cutoff]\n        return (df_cutoff['target'] == 1).sum() / (df['target'] == 1).sum()\n        \n    def weighted_gini(y_true: pd.DataFrame, y_pred: pd.DataFrame) -> float:\n        df = (pd.concat([y_true, y_pred], axis='columns')\n              .sort_values('prediction', ascending=False))\n        df['weight'] = df['target'].apply(lambda x: 20 if x==0 else 1)\n        df['random'] = (df['weight'] / df['weight'].sum()).cumsum()\n        total_pos = (df['target'] * df['weight']).sum()\n        df['cum_pos_found'] = (df['target'] * df['weight']).cumsum()\n        df['lorentz'] = df['cum_pos_found'] / total_pos\n        df['gini'] = (df['lorentz'] - df['random']) * df['weight']\n        return df['gini'].sum()\n\n    def normalized_weighted_gini(y_true: pd.DataFrame, y_pred: pd.DataFrame) -> float:\n        y_true_pred = y_true.rename(columns={'target': 'prediction'})\n        return weighted_gini(y_true, y_pred) / weighted_gini(y_true, y_true_pred)\n\n    g = normalized_weighted_gini(y_true, y_pred)\n    d = top_four_percent_captured(y_true, y_pred)\n\n    return 0.5 * (g + d)","metadata":{"execution":{"iopub.status.busy":"2022-07-02T15:42:58.895811Z","iopub.status.idle":"2022-07-02T15:42:58.896505Z","shell.execute_reply.started":"2022-07-02T15:42:58.896293Z","shell.execute_reply":"2022-07-02T15:42:58.896315Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"best_parameters = {'max_depth': 12, 'min_child_weight': 7, 'eta': 0.1, 'objective': 'binary:logistic', 'tree_method': 'gpu_hist', 'eval_metric': 'rmsle'}\n\nXGB_Model = xgb.XGBClassifier(**best_parameters,\n                              verbosity = 1,\n                              n_jobs = -1).fit(X, y)\n        \n","metadata":{"execution":{"iopub.status.busy":"2022-07-02T15:42:58.897664Z","iopub.status.idle":"2022-07-02T15:42:58.898071Z","shell.execute_reply.started":"2022-07-02T15:42:58.897873Z","shell.execute_reply":"2022-07-02T15:42:58.897891Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del Train_df, PCA_Numeric, PCA_Cat, PCA_df, Master_df, Temp\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T15:42:58.899725Z","iopub.status.idle":"2022-07-02T15:42:58.900270Z","shell.execute_reply.started":"2022-07-02T15:42:58.900062Z","shell.execute_reply":"2022-07-02T15:42:58.900083Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Test","metadata":{}},{"cell_type":"code","source":"Test_df = pd.read_feather('../input/amexfeather/test_data.ftr')\nTest_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T15:42:58.901634Z","iopub.status.idle":"2022-07-02T15:42:58.902016Z","shell.execute_reply.started":"2022-07-02T15:42:58.901815Z","shell.execute_reply":"2022-07-02T15:42:58.901831Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Test_df = Test_df.groupby('customer_ID').tail(1)\nTest_df.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-02T15:42:58.903350Z","iopub.status.idle":"2022-07-02T15:42:58.903718Z","shell.execute_reply.started":"2022-07-02T15:42:58.903530Z","shell.execute_reply":"2022-07-02T15:42:58.903547Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Test_df = Test_df.drop(columns = Drop_Columns)\nTest_df.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-02T15:42:58.904795Z","iopub.status.idle":"2022-07-02T15:42:58.905166Z","shell.execute_reply.started":"2022-07-02T15:42:58.904985Z","shell.execute_reply":"2022-07-02T15:42:58.905008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Prepare for a PCA \nMaster_df = Test_df[['customer_ID']].reset_index(drop = True)\n\n# Categorial\nPCA_Cat = Test_df.select_dtypes(include='category').reset_index(drop = True)\n\nfor i in PCA_Cat.columns:\n    PCA_Cat[i].fillna(PCA_Cat[i].quantile(.5), inplace = True)\n    \nPCA_Cat = pd.get_dummies(PCA_Cat, drop_first= True)\n\n# Numeric and Normalize\nPCA_Numeric = Test_df.select_dtypes(include=['float16']).reset_index(drop = True)\n\nfor i in PCA_Numeric.columns:\n    PCA_Numeric[i] = PCA_Numeric[i].astype('float64')\n    PCA_Numeric[i] = PCA_Numeric[i].fillna(PCA_Numeric[i].mean())\n\nPCA_Numeric = pd.DataFrame(StandardScaler().fit_transform(PCA_Numeric), columns = PCA_Numeric.columns)\n    \n# Concat\nPCA_df = pd.concat([PCA_Cat, PCA_Numeric], axis = 1)\n\n# PCA\nPCA_Model = PCA(n_components=6, random_state=0)\nTemp = pd.DataFrame(PCA_Model.fit_transform(PCA_df))\nMaster_df = pd.concat([Master_df.iloc[:, :2], Temp], axis = 1)\nMaster_df","metadata":{"execution":{"iopub.status.busy":"2022-07-02T15:42:58.906666Z","iopub.status.idle":"2022-07-02T15:42:58.907036Z","shell.execute_reply.started":"2022-07-02T15:42:58.906843Z","shell.execute_reply":"2022-07-02T15:42:58.906874Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"prediction = pd.DataFrame({'customer_ID':Master_df['customer_ID'],\n                           'prediction':XGB_Model.predict_proba(Master_df.iloc[:,1:])[:, 1]})\n\ndel PCA_Numeric, PCA_Cat, PCA_df, Master_df, Temp\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T15:42:58.908036Z","iopub.status.idle":"2022-07-02T15:42:58.908382Z","shell.execute_reply.started":"2022-07-02T15:42:58.908209Z","shell.execute_reply":"2022-07-02T15:42:58.908225Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"prediction.to_csv('submission.csv', index=False)\nprediction.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T15:42:58.909621Z","iopub.status.idle":"2022-07-02T15:42:58.909984Z","shell.execute_reply.started":"2022-07-02T15:42:58.909796Z","shell.execute_reply":"2022-07-02T15:42:58.909812Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Note: This submission performance is 0.739 eventually. Although the score is not that high among the leaderboard. However, the model performance unexpectedly works better than our expection. From the previous findings, the amex scores of different models range at 0.68 to 0.69 only, but the out-sample predict have reached nearly a 5% increment to ~0.74 now, which is a significant improvement. This is already an achievement! \n\nOf course, there are still a lot rooms for enhancements. As discussed in my previous notebook, given resources limitations and enormous dataset, we cannot make too many investigation in a single notebook and the Kaggle environment. If we have more resources, we will definitely explore on the following directions:\n\n    1) Reconsider dropping null ratio > 0.7 as we do no have information on schema, whose may be important features!\n\n    2) Rethink PCA as it will make some loss some losses on features during compression \n\n    3) Investigate the time-serial features for each variables and find the most profound ones (e.g. rank the date descending and replace by time step)\n\n    4) Work on featuer engineering (take previous t-n mean/ median/ model/ quantile/ max/ min)\n\n    5) Generative model: clustering and create corrsponding models for each largely-differentative customers segments (e.g. always default, good conduct, forgotten etc.)\n\n    6) Default defection as a problem of anomality detection? (isolated forest may be a more suitable and scalable solution)\n\nLet us see any chance we can work on theme in the future! Cheers!","metadata":{}}]}