{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import pandas as pd\n\ndef read_data(cols):\n    \n    print('Reading data...')\n    \n    df = pd.read_parquet('../input/amex-data-integer-dtypes-parquet-format/train.parquet', columns=cols)\n    \n    # simplify cus_id\n    unique_cus_ids = df.customer_ID.unique()\n    assignment     = dict(zip(unique_cus_ids, list(range(len(unique_cus_ids)))))\n    df.customer_ID = df.customer_ID.apply(lambda x: assignment[x]).astype('int32')\n    \n    print('shape of data:', df.shape)\n    \n    return df","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-08-03T19:16:53.679411Z","iopub.execute_input":"2022-08-03T19:16:53.680455Z","iopub.status.idle":"2022-08-03T19:16:53.685581Z","shell.execute_reply.started":"2022-08-03T19:16:53.680407Z","shell.execute_reply":"2022-08-03T19:16:53.684898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# How to make sense out of 'Risk Variables' ?","metadata":{}},{"cell_type":"markdown","source":"A general problem I see in most of the approaches is the exclusive reliance on features like 'last', 'first', 'max' etc.. I find this features to ignore the history of the customer. Those features assume that imporant information is to be found at ONLY ONE SPECIFIC point in time. But a customer's record is more than just one point in time.<br>\n<br>\nI have seen other approaches using 'mean', 'median' or 'std', which indeed contain some aggregated information on the history of some customer. But this seems a little bornig. I have not seen further approaches. <br> \n<br> \nIn this notebook I will show you how to compute from low information binary features a high information feature, which is based on the history of a customer.<br>\n<br>\nMy (non-public) EDA has found some features to be binary. Below you see the 'risk' features, which I found to be binary.","metadata":{}},{"cell_type":"code","source":"r_bin = ['R_2', 'R_4', 'R_15', 'R_19', 'R_21', 'R_22', 'R_23', 'R_24', 'R_25', 'R_28', 'R_7', 'R_12', 'R_14']","metadata":{"execution":{"iopub.status.busy":"2022-08-03T19:31:43.485061Z","iopub.execute_input":"2022-08-03T19:31:43.485350Z","iopub.status.idle":"2022-08-03T19:31:43.488842Z","shell.execute_reply.started":"2022-08-03T19:31:43.485328Z","shell.execute_reply":"2022-08-03T19:31:43.488241Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Read features","metadata":{}},{"cell_type":"code","source":"# read only 'risk' binary features\ntrain  = read_data(['customer_ID'] + r_bin)\n# read targets\ntarget = pd.read_csv('../input/amex-default-prediction/train_labels.csv', usecols=['target'])","metadata":{"execution":{"iopub.status.busy":"2022-08-03T20:25:00.272427Z","iopub.execute_input":"2022-08-03T20:25:00.272795Z","iopub.status.idle":"2022-08-03T20:25:04.503543Z","shell.execute_reply.started":"2022-08-03T20:25:00.272770Z","shell.execute_reply":"2022-08-03T20:25:04.503020Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preprocessing","metadata":{}},{"cell_type":"markdown","source":"First of all, I lied:) We have two numerical features. 'R_7' and 'R_14'. You will see now why I classified them as binary.","metadata":{}},{"cell_type":"code","source":"train.R_7.value_counts(dropna=False)","metadata":{"execution":{"iopub.status.busy":"2022-08-03T19:42:39.360294Z","iopub.execute_input":"2022-08-03T19:42:39.360580Z","iopub.status.idle":"2022-08-03T19:42:39.454155Z","shell.execute_reply.started":"2022-08-03T19:42:39.360558Z","shell.execute_reply":"2022-08-03T19:42:39.453261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.R_14.value_counts(dropna=False)","metadata":{"execution":{"iopub.status.busy":"2022-08-03T19:42:39.455199Z","iopub.execute_input":"2022-08-03T19:42:39.455512Z","iopub.status.idle":"2022-08-03T19:42:39.528911Z","shell.execute_reply.started":"2022-08-03T19:42:39.455482Z","shell.execute_reply":"2022-08-03T19:42:39.528296Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Note both features are mostly 0. There is so little information in this ~1% of values, which are not equal to 1. Therefore, set them equal to 1 and consider the features as binary. ","metadata":{}},{"cell_type":"code","source":"train.loc[train.R_7 != 0, 'R_7']   = 1\ntrain.loc[train.R_14 != 0, 'R_14'] = 1","metadata":{"execution":{"iopub.status.busy":"2022-08-03T20:25:04.504517Z","iopub.execute_input":"2022-08-03T20:25:04.505843Z","iopub.status.idle":"2022-08-03T20:25:04.549014Z","shell.execute_reply.started":"2022-08-03T20:25:04.505794Z","shell.execute_reply":"2022-08-03T20:25:04.548142Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now see that we have also 'R_12' having some missing values.","metadata":{}},{"cell_type":"code","source":"train.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-08-03T19:42:43.331152Z","iopub.execute_input":"2022-08-03T19:42:43.333189Z","iopub.status.idle":"2022-08-03T19:42:43.411112Z","shell.execute_reply.started":"2022-08-03T19:42:43.333157Z","shell.execute_reply":"2022-08-03T19:42:43.410362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Moreover, like the two other features above, 'R_12' has some continious values.","metadata":{}},{"cell_type":"code","source":"train.R_12.value_counts(dropna=False)","metadata":{"execution":{"iopub.status.busy":"2022-08-03T19:42:45.723306Z","iopub.execute_input":"2022-08-03T19:42:45.723727Z","iopub.status.idle":"2022-08-03T19:42:45.842933Z","shell.execute_reply.started":"2022-08-03T19:42:45.723694Z","shell.execute_reply":"2022-08-03T19:42:45.842197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# but missing in majority class\ntrain.loc[train.R_12.isna(), 'R_12'] = 1\n# set continious values equal 0\ntrain.loc[train.R_12 != 1, 'R_12']   = 0","metadata":{"execution":{"iopub.status.busy":"2022-08-03T20:25:06.586629Z","iopub.execute_input":"2022-08-03T20:25:06.587015Z","iopub.status.idle":"2022-08-03T20:25:06.623131Z","shell.execute_reply.started":"2022-08-03T20:25:06.586989Z","shell.execute_reply":"2022-08-03T20:25:06.622210Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we are ready! We have only ones and zeros in the data.","metadata":{}},{"cell_type":"code","source":"print(train[r_bin].max().max())\nprint(train[r_bin].min().min())","metadata":{"execution":{"iopub.status.busy":"2022-08-03T20:25:08.974493Z","iopub.execute_input":"2022-08-03T20:25:08.974803Z","iopub.status.idle":"2022-08-03T20:25:09.261805Z","shell.execute_reply.started":"2022-08-03T20:25:08.974780Z","shell.execute_reply":"2022-08-03T20:25:09.261047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's check the default rate in binary variables. ","metadata":{}},{"cell_type":"code","source":"for feature in r_bin:\n    \n    print(feature)\n    zero_cus = train.loc[train[feature] == 0, 'customer_ID'].to_list()\n    rate     = target.loc[zero_cus].mean().values[0]\n    print(f'default rate for \"0\": {rate}, number of observations: {len(zero_cus)}')\n    \n    one_cus = train.loc[train[feature] == 1, 'customer_ID'].to_list()\n    rate    = target.loc[one_cus].mean().values[0]\n    print(f'default rate for \"1\": {rate} , number of observations: {len(one_cus)}')\n    print('\\n')","metadata":{"execution":{"iopub.status.busy":"2022-08-03T20:25:31.692393Z","iopub.execute_input":"2022-08-03T20:25:31.693614Z","iopub.status.idle":"2022-08-03T20:25:42.107557Z","shell.execute_reply.started":"2022-08-03T20:25:31.693584Z","shell.execute_reply":"2022-08-03T20:25:42.106286Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Cool! So we see that we have a very large difference in the default rate in the binary variable. BUT! But unfortunately the high default class has only very little members -> Very little information IF you look at the features seperatly :( <br>\n<br>\nBut what if we look at them all at once? how? Well... <br>\n<br>\nFor all features, let's change the value (0 or 1) to the corresponding default rate of all customers displaying the given value. Example: Let's say for some feature R_ customers without zero values have default rate 20%. Then all zeros in this feature will become 0.20. And if customers with one have default rate 60% then all ones become 0.6.","metadata":{}},{"cell_type":"code","source":"for feature in r_bin:\n\n    rate = float(target.loc[train.loc[train[feature] == 0, 'customer_ID'].unique()].mean())\n    train.loc[train[feature] == 0, feature] = rate\n    \n    rate = float(target.loc[train.loc[train[feature] == 1, 'customer_ID'].unique()].mean())\n    train.loc[train[feature] == 1, feature] = rate","metadata":{"execution":{"iopub.status.busy":"2022-08-03T20:26:16.877700Z","iopub.execute_input":"2022-08-03T20:26:16.878896Z","iopub.status.idle":"2022-08-03T20:26:19.761038Z","shell.execute_reply.started":"2022-08-03T20:26:16.878812Z","shell.execute_reply":"2022-08-03T20:26:19.759939Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now if we take the mean of this features we get a feature, which takes into account how many risky moment a customer had over time AND which can be conmpare among all customers.","metadata":{}},{"cell_type":"code","source":"agg = train.groupby('customer_ID').agg('mean')","metadata":{"execution":{"iopub.status.busy":"2022-08-03T20:26:22.611751Z","iopub.execute_input":"2022-08-03T20:26:22.612613Z","iopub.status.idle":"2022-08-03T20:26:23.208262Z","shell.execute_reply.started":"2022-08-03T20:26:22.612557Z","shell.execute_reply":"2022-08-03T20:26:23.207357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we used rates in computing while computing this features, we can compare all those features with each other. Therefore let's sum them together for all customers.","metadata":{}},{"cell_type":"code","source":"agg['risk_binary'] = agg.sum(axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-08-03T20:26:31.431307Z","iopub.execute_input":"2022-08-03T20:26:31.431634Z","iopub.status.idle":"2022-08-03T20:26:31.459357Z","shell.execute_reply.started":"2022-08-03T20:26:31.431611Z","shell.execute_reply":"2022-08-03T20:26:31.458809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"See that we have 325444 customers, which never displayed one risky moment. But the rest had at least one risky moment at some point in time.","metadata":{}},{"cell_type":"code","source":"agg.risk_binary.value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-08-03T20:26:32.964495Z","iopub.execute_input":"2022-08-03T20:26:32.964813Z","iopub.status.idle":"2022-08-03T20:26:32.981176Z","shell.execute_reply.started":"2022-08-03T20:26:32.964788Z","shell.execute_reply":"2022-08-03T20:26:32.980335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"agg.risk_binary.plot(kind='hist',bins=200)","metadata":{"execution":{"iopub.status.busy":"2022-08-03T20:26:49.451910Z","iopub.execute_input":"2022-08-03T20:26:49.452686Z","iopub.status.idle":"2022-08-03T20:26:50.323832Z","shell.execute_reply.started":"2022-08-03T20:26:49.452658Z","shell.execute_reply":"2022-08-03T20:26:50.322648Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have basically seperated customers, which never had risky moment from customers which had at least one risky moment. At the same time, we have accounted for the porpotinal risk of different features! -Note that a '1' in featue 'R_24' is asociated with more risk than in a '1' in 'R_23'.\n\nIn my permutation importance this feature ranks always in top 20%.","metadata":{}},{"cell_type":"markdown","source":"You can do the same for other groups of binary features... or maybe for all binary features together?:) Maybe you find other mhigh-information features.","metadata":{}},{"cell_type":"markdown","source":"Note, when you use this method in your model, you will have to apply the same default rates to the test set. So do all at once! Otherwise you get lost later. ","metadata":{}}]}