{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<div style=\"padding:10px;color:BLACK;margin:0;font-size:200%;text-align:center;display:fill;border-radius:10px;background-color:#00BFFF;font-weight:800\">AMEX Default Prediction</div>","metadata":{}},{"cell_type":"markdown","source":"### 0 - overview","metadata":{}},{"cell_type":"markdown","source":"The objective of this competition is to predict the probability that a customer does not pay back their credit card balance amount in the future based on their monthly customer profile. The target binary variable is calculated by observing 18 months performance window after the latest credit card statement, and if the customer does not pay due amount in 120 days after their latest statement date it is considered a default event.\n\nThe dataset contains aggregated profile features for each customer at each statement date. Features are anonymized and normalized, and fall into the following general categories:\n\nD_* = Delinquency variables\nS_* = Spend variables\nP_* = Payment variables\nB_* = Balance variables\nR_* = Risk variables\nwith the following features being categorical:\n\n['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']\n\nYour task is to predict, for each customer_ID, the probability of a future payment default (target = 1).\n\nNote that the negative class has been subsampled for this dataset at 5%, and thus receives a 20x weighting in the scoring metric.","metadata":{}},{"cell_type":"markdown","source":"### 1 -  load enviroments\n","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt","metadata":{"execution":{"iopub.status.busy":"2022-08-11T11:00:19.289751Z","iopub.execute_input":"2022-08-11T11:00:19.290227Z","iopub.status.idle":"2022-08-11T11:00:19.321396Z","shell.execute_reply.started":"2022-08-11T11:00:19.290138Z","shell.execute_reply":"2022-08-11T11:00:19.320504Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2 - Import Data","metadata":{}},{"cell_type":"code","source":"%%time\n\ndef read_data(data_path, label_path = None):\n    \n    # read train data\n    df = pd.read_parquet(train_path)\n    \n    # read labels\n    labels = pd.read_csv(label_path)\n    \n    # merge data\n    df = df.merge(labels, left_on='customer_ID', right_on='customer_ID')\n    \n    return df\n\n\n\ndef data_view(df):\n    print(\"Take a look first five roll of the data\")\n    df.head(5)\n    \n    print(\"The info\")\n    df.info()\n    \n    \n    D_columns = [col for col in df.columns if 'D_' in col]\n    print (\"Total number of columns about Delinquency variables is\", len(D_columns))\n    S_columns = [col for col in df.columns if 'S_' in col]\n    print (\"Total number of columns about Spend variables is\", len(S_columns))\n    P_columns = [col for col in df.columns if 'P_' in col]\n    print (\"Total number of columns about Payment variables is\", len(P_columns))\n    B_columns = [col for col in df.columns if 'B_' in col]\n    print (\"Total number of columns about Balance variables is\", len(B_columns))\n    R_columns = [col for col in df.columns if 'R_' in col]\n    print (\"Total number of columns about Risk variables is\", len(R_columns))\n    \n    D_missing = [col for col in D_columns if df[col].isnull().sum() > 0 ]\n    print (\"Total number of columns in Delinquency variables have missing value is\", len(D_missing)) \n    S_missing = [col for col in S_columns if df[col].isnull().sum() > 0 ]\n    print (\"Total number of columns in Spend variables have missing value is\", len(S_missing))\n    P_missing = [col for col in P_columns if df[col].isnull().sum() > 0 ]\n    print (\"Total number of columns in Payment variables have missing value is\", len(P_missing))\n    B_missing = [col for col in B_columns if df[col].isnull().sum() > 0 ]\n    print (\"Total number of columns in Balance variables have missing value is\", len(B_missing))\n    R_missing = [col for col in R_columns if df[col].isnull().sum() > 0 ]\n    print (\"Total number of columns in Risk variables have missing value is\", len(R_missing))\n     \n        \n        \ndef data_cleaning(df):\n\n    # transfer S_2  to timestamp dtype\n    df['S_2'] = pd.to_datetime(df['S_2'])\n    \n    \n    \n    all_cols = [c for c in list(df.columns) if c not in ['customer_ID','S_2']]\n    cat_features = [\"B_30\",\"B_38\",\"D_114\",\"D_116\",\"D_117\",\"D_120\",\"D_126\",\"D_63\",\"D_64\",\"D_66\",\"D_68\"]\n    num_features = [col for col in all_cols if col not in cat_features]\n    \n    \n    # change features from cat_features to category dtype\n    df[cat_features] = df[cat_features].astype(\"category\")\\\n    \n    # fill out all miss values\n    # if is cate, fill with Na\n    # if is num, fill with avg\n    \n    for column in list(df.columns[df.isnull().sum() > 0]):\n        if column in num_features:\n            mean_val = df[column].mean()\n            df[column].fillna(mean_val, inplace=True)\n        else:\n            df[column].fillna(\"Na\", inplace=True)\n            \n            \n    for column in df.columns.values.tolist():\n        miss = df[column].isnull().sum()\n        print(f\"column {column}  has number of missing value is\", miss)\n    return df\n    \n    \n\ndef main_function(df):\n    \n    # take look\n    data_view(df)\n    \n    # data clean\n    df_cleaned = data_cleaning(df)\n    \n    return df_cleaned\n#df_raw.customer_ID.duplicated().any()","metadata":{"execution":{"iopub.status.busy":"2022-08-11T11:00:19.982624Z","iopub.execute_input":"2022-08-11T11:00:19.983051Z","iopub.status.idle":"2022-08-11T11:00:19.999359Z","shell.execute_reply.started":"2022-08-11T11:00:19.983014Z","shell.execute_reply":"2022-08-11T11:00:19.997982Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntrain_path = \"../input/amex-data-integer-dtypes-parquet-format/train.parquet\"\nlabel_path = '../input/amex-default-prediction/train_labels.csv'\ndf_raw = read_data(train_path, label_path)","metadata":{"execution":{"iopub.status.busy":"2022-08-11T11:00:22.555948Z","iopub.execute_input":"2022-08-11T11:00:22.556676Z","iopub.status.idle":"2022-08-11T11:03:27.627643Z","shell.execute_reply.started":"2022-08-11T11:00:22.556638Z","shell.execute_reply":"2022-08-11T11:03:27.625773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#df_11 = df_raw[100000:100500]","metadata":{"execution":{"iopub.status.busy":"2022-08-11T01:30:52.526084Z","iopub.execute_input":"2022-08-11T01:30:52.526533Z","iopub.status.idle":"2022-08-11T01:30:52.532578Z","shell.execute_reply.started":"2022-08-11T01:30:52.526495Z","shell.execute_reply":"2022-08-11T01:30:52.531153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_cleaned  = main_function(df_raw)","metadata":{"execution":{"iopub.status.busy":"2022-08-11T11:21:58.103377Z","iopub.execute_input":"2022-08-11T11:21:58.104409Z","iopub.status.idle":"2022-08-11T11:22:12.584092Z","shell.execute_reply.started":"2022-08-11T11:21:58.104334Z","shell.execute_reply":"2022-08-11T11:22:12.582967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_cleaned.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-11T01:33:05.401494Z","iopub.execute_input":"2022-08-11T01:33:05.401930Z","iopub.status.idle":"2022-08-11T01:33:05.439282Z","shell.execute_reply.started":"2022-08-11T01:33:05.401893Z","shell.execute_reply":"2022-08-11T01:33:05.438083Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_cleaned.info()","metadata":{"execution":{"iopub.status.busy":"2022-08-11T01:33:15.190395Z","iopub.execute_input":"2022-08-11T01:33:15.191393Z","iopub.status.idle":"2022-08-11T01:33:15.214393Z","shell.execute_reply.started":"2022-08-11T01:33:15.191345Z","shell.execute_reply":"2022-08-11T01:33:15.213245Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_cleaned[\"B_30\"].isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2022-08-11T11:24:49.127918Z","iopub.execute_input":"2022-08-11T11:24:49.128849Z","iopub.status.idle":"2022-08-11T11:24:49.153130Z","shell.execute_reply.started":"2022-08-11T11:24:49.128783Z","shell.execute_reply":"2022-08-11T11:24:49.152079Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for column in df_cleaned.columns.values.tolist():\n    miss = df_cleaned[column].isnull().sum()\n    print(f\"  column {column}  has number of missing value is\", miss)","metadata":{"execution":{"iopub.status.busy":"2022-08-11T11:32:32.532578Z","iopub.execute_input":"2022-08-11T11:32:32.533779Z","iopub.status.idle":"2022-08-11T11:32:34.375974Z","shell.execute_reply.started":"2022-08-11T11:32:32.533736Z","shell.execute_reply":"2022-08-11T11:32:34.374612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}