{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import pandas as pd \nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nx = pd.read_csv('../input/amex-default-prediction/train_data.csv', usecols=['B_2'])","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:13:52.974837Z","iopub.execute_input":"2022-05-28T11:13:52.975288Z","iopub.status.idle":"2022-05-28T11:16:12.708643Z","shell.execute_reply.started":"2022-05-28T11:13:52.975245Z","shell.execute_reply":"2022-05-28T11:16:12.707823Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Context \n\nMoney has this unique property that values although highly cardinal it is still discrete with an increment of $0.01.\n\nIt can be expected that some debit cards would have a balance of $0.00 or credit cards maxed out the credit limit. Therefore spikes should appear in the money related feature distribution plots. It should be observed one way or another, even though the data has been anonymized & normalized (as per competition description).","metadata":{}},{"cell_type":"markdown","source":"### Distribution of B\\_2\n\nLet's pick a random balance feature B\\_2 and look at its distribution:","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(12, 8))\nsns.distplot(x['B_2'], bins=500, kde=False)","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:18:06.881521Z","iopub.execute_input":"2022-05-28T11:18:06.882245Z","iopub.status.idle":"2022-05-28T11:18:08.360145Z","shell.execute_reply.started":"2022-05-28T11:18:06.882195Z","shell.execute_reply":"2022-05-28T11:18:08.35875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's zoom in into these 2 spikes","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(12, 8))\nsns.distplot(x.loc[x.B_2>0.99,'B_2'],bins=500,kde=False)","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:18:22.449006Z","iopub.execute_input":"2022-05-28T11:18:22.450106Z","iopub.status.idle":"2022-05-28T11:18:23.480035Z","shell.execute_reply.started":"2022-05-28T11:18:22.450048Z","shell.execute_reply":"2022-05-28T11:18:23.479034Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that the spike is in range in (1, 1.01) and not a single value. I guess the max value for this feature in raw form is 1 and random uniform value from [0, 0.01] was added for each observation.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(12, 8))\nsns.distplot(x.loc[(x.B_2>0.8) & (x.B_2<0.83),'B_2'],bins=500,kde=False)","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:18:29.104371Z","iopub.execute_input":"2022-05-28T11:18:29.10541Z","iopub.status.idle":"2022-05-28T11:18:30.064313Z","shell.execute_reply.started":"2022-05-28T11:18:29.105364Z","shell.execute_reply":"2022-05-28T11:18:30.063463Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The second spike confirms the idea of [0, 0.01] random uniform noise.","metadata":{}},{"cell_type":"markdown","source":"### The other way to look at it...","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:21:02.688366Z","iopub.execute_input":"2022-05-28T11:21:02.688885Z","iopub.status.idle":"2022-05-28T11:21:02.693046Z","shell.execute_reply.started":"2022-05-28T11:21:02.688851Z","shell.execute_reply":"2022-05-28T11:21:02.692056Z"}}},{"cell_type":"code","source":"len(x['B_2'].unique())/len(x['B_2'])","metadata":{"execution":{"iopub.status.busy":"2022-05-28T11:24:39.867776Z","iopub.execute_input":"2022-05-28T11:24:39.868195Z","iopub.status.idle":"2022-05-28T11:24:40.853725Z","shell.execute_reply.started":"2022-05-28T11:24:39.868164Z","shell.execute_reply":"2022-05-28T11:24:40.85262Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"99.9% values are unique. This goes against the nature of how maney is distributed in bank accounts. This confirms the hypothesis of random noise injection.","metadata":{}},{"cell_type":"markdown","source":"# BONUS","metadata":{}},{"cell_type":"code","source":"sns.distplot(x.loc[x.B_2<0.1,'B_2'],bins=500,kde=False)","metadata":{},"execution_count":null,"outputs":[]}]}