{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"#### Setup data paths:\nRunning on kaggle...","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd \nimport datetime\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport warnings\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as\n# output when you create a version using \"Save & Run All\".\n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-31T19:25:51.040214Z","iopub.execute_input":"2023-10-31T19:25:51.041117Z","iopub.status.idle":"2023-10-31T19:25:52.456647Z","shell.execute_reply.started":"2023-10-31T19:25:51.040964Z","shell.execute_reply":"2023-10-31T19:25:52.455030Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# A. Explore train data\n\nWe can see that we have **190 columns** (185 float, 1 int, 4 object). One of the object columns is `customer_ID`, which can be used to link the multiple statements a client has in the data.\n\nAverage train target is **0.26345**. Keep in mind that the negative class was subsampled at 5%. This means that the actual train target mean is:\n```python\nlen(target[target.target == 1])/(len(target[target.target == 0])*20 + len(target[target.target == 1])) = 0.01757\n```","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(\"/kaggle/input/amex-default-prediction/train_data.csv\", nrows=20000)\ntarget = pd.read_csv(\"/kaggle/input/amex-default-prediction/train_labels.csv\", nrows=20000)\nprint(df.info())\npd.set_option(\"display.max_columns\", 300)\ndisplay(df.head(5))\n\nprint(\"\\nTrain target average:\", target.target.mean())","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-31T19:25:52.459735Z","iopub.execute_input":"2023-10-31T19:25:52.460301Z","iopub.status.idle":"2023-10-31T19:25:54.688114Z","shell.execute_reply.started":"2023-10-31T19:25:52.460247Z","shell.execute_reply":"2023-10-31T19:25:54.686731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## A.1 `customer_ID` index: credit card statement analysis\nThe data contains multiple credit card statements per customer. _\"The target binary variable is calculated by observing 18 months performance window after the latest credit card statement, and if the customer does not pay due amount in 120 days after their latest statement date it is considered a default event.\"_\n","metadata":{}},{"cell_type":"code","source":"unique_customer_ids_in_train = len(df.customer_ID.unique())\nprint(\"#unique customers in train data: {} out of {} datalines.\".format(\n    unique_customer_ids_in_train, len(df)\n))\n\nunique_customer_ids_in_target = len(target.customer_ID.unique())\nprint(\"#unique customers in train target: {} out of {} datalines.\".format(\n    unique_customer_ids_in_target,len(target)\n))\n\ndf.merge(\n    target, how = 'left', left_on = ['customer_ID'], right_on = ['customer_ID']\n).groupby(['customer_ID']).agg(\n    {'customer_ID':'count', 'target':'mean'}\n    #'target':'max' would give the same results, since the target is same for all customer stateme.\n).sort_index().rename({'customer_ID':'ntot_statements'}, axis = 1).groupby(['ntot_statements']).agg(\n    {'ntot_statements':'count', 'target':'mean'}\n).sort_index().rename({'ntot_statements':'n_customers'}, axis = 1)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-31T19:25:54.689690Z","iopub.execute_input":"2023-10-31T19:25:54.690966Z","iopub.status.idle":"2023-10-31T19:25:54.839009Z","shell.execute_reply.started":"2023-10-31T19:25:54.690898Z","shell.execute_reply":"2023-10-31T19:25:54.837630Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that we have mostly **13 statements per customer** (and not 18). We can also see that the **'target' is a single value per 'customer_ID' (and not per statement)**, which links to the fact that these statements might be the 'performance window' in which we measure default. In terms of default, we can see that the **average is much higher for customers with `n_statements<13`**.\n\nThis raises questions regarding how the default is computed for the data - if a customer entered default and has 13 statements, can we assume it was in the 13th month they entered the default state? Or could it have been anytime in those 13 months? This information would make the **last customer statements _potentially_ more \"valuable\" in predicting default**. Such can be explored by comparing the performance of models built with the first statements Vs last statements.","metadata":{}},{"cell_type":"markdown","source":"### A.1.1 Time distribution","metadata":{}},{"cell_type":"code","source":"def week_from_S_2(x):\n    dt = x.split(\"-\")\n    day, month, year = int(dt[2]), int(dt[1]), int(dt[0])\n    \n    week = datetime.date(year, month, day).isocalendar()[1]\n    \n    # year of date could be different from the week's year (ex: 1 Jan 2023 belongs to week 52, 2022)\n    if month == 1 and week > 50: # date belongs to previous year's last week\n        week_label = str(year-1)\n    else:\n        week_label = str(year)\n    week_label = week_label + \"_w\" + (\"0\"+str(week) if week < 10 else str(week))\n    return week_label\n\n# Get week label:\ndf[\"week\"] = df.S_2.apply(lambda x: week_from_S_2(x))\n\n# Get n_statements:\ndf = df.sort_values(['customer_ID', 'S_2'], axis = 0, ascending = [True, True])\nn_statement_aux = 1\nfor row in range(len(df)):\n    if row == 0:\n        df.loc[row, 'n_statement'] = n_statement_aux\n    else:\n        c_ID = df.loc[row,'customer_ID']\n        c_ID_prev = df.loc[row-1,'customer_ID']\n        \n        if c_ID == c_ID_prev: # if we are still in the same customer\n            n_statement_aux += 1\n            df.loc[row, 'n_statement'] = n_statement_aux\n        else:\n            n_statement_aux = 1\n            df.loc[row, 'n_statement'] = n_statement_aux\n            \n# Get ntot_statements:\ntry: # code in case I run this cell multiple times\n    df = df.drop(['ntot_statement'], axis = 1)\nexcept:\n    pass\ndf = df.merge(\n    df.groupby(['customer_ID']).agg({'n_statement':'max'}).rename({'n_statement':'ntot_statement'}, axis=1),\n    how = 'left', left_on = ['customer_ID'], right_on = ['customer_ID']\n)\n#df[['week', 'n_statement', 'ntot_statement']].head()\n\n# All statements plot:\ndf.groupby(['week']).agg({'week':'count'}).rename(\n    {'week':'statement_count'}, axis = 1\n).sort_index().reset_index().plot.bar(x='week', y='statement_count', figsize=(24,5),)\nprint(\">> All statements:\")\nplt.show()\n\n# Only first statements plot:\ncolor_map_statement = {\n    1:'#A1FBCA', 2:'#A1FBDD', 3:'#A1FBEA', 4:'#A1F9FB', 5:'#A1D9FB', 6:'#A1BAFB', 7:'#A1A1FB',\n    8:'#C4A1FB', 9:'#E4A1FB', 10:'#FBA1F3', 11:'#FBA1D3', 12:'#FBA1A2', 13:'#F55759'\n}\n_, ax = plt.subplots(figsize=(24, 5))\nbottom = np.zeros(len(df.week.unique()))\nfor tot_value in df.ntot_statement.sort_values(ascending=True).unique():\n    # Prepare data for plot:\n    plot1 = df[(df.n_statement == 1) & (df.ntot_statement == tot_value)].groupby(\n        ['week', 'n_statement']).agg(\n            {'week':'count'}\n    ).rename({'week':'statement_count'}, axis = 1).sort_index().reset_index()\n    \n    # Merge additional week values to obtain the same xrange:\n    plot1 = pd.concat([plot1,\n                   pd.DataFrame({\n                       'week':df[~df.week.isin(plot1.week.tolist())].week.unique().tolist(),\n                       'statement_count':0})\n                  ], axis = 0).sort_values(['week'], axis = 0)\n    \n    # Plot:\n    plot1.plot.bar(x='week', y='statement_count', color = color_map_statement[int(tot_value)],\n                   ax = ax, label = \"ntot_statement = \"+str(tot_value), bottom=bottom)\n    bottom += np.array(plot1.statement_count.tolist()) # to stack plots, keep raising the bottom\nprint(\">> Only first customer statements: (n_statement = 1)\")\nplt.show()\n\n\n# Only latest statements plot:\ncolor_map_statement = {\n    1:'#A1FBCA', 2:'#A1FBDD', 3:'#A1FBEA', 4:'#A1F9FB', 5:'#A1D9FB', 6:'#A1BAFB', 7:'#A1A1FB',\n    8:'#C4A1FB', 9:'#E4A1FB', 10:'#FBA1F3', 11:'#FBA1D3', 12:'#FBA1A2', 13:'#F55759'\n}\n_, ax = plt.subplots(figsize=(24, 5))\nbottom = np.zeros(len(df.week.unique()))\nfor tot_value in df.ntot_statement.sort_values(ascending=True).unique():\n    # Prepare data for plot:\n    plot1 = df[(df.n_statement == df.ntot_statement) & (df.ntot_statement == tot_value)].groupby(\n        ['week', 'n_statement']).agg(\n            {'week':'count'}\n    ).rename({'week':'statement_count'}, axis = 1).sort_index().reset_index()\n    \n    # Merge additional week values to obtain the same xrange:\n    plot1 = pd.concat([plot1,\n                   pd.DataFrame({\n                       'week':df[~df.week.isin(plot1.week.tolist())].week.unique().tolist(),\n                       'statement_count':0})\n                  ], axis = 0).sort_values(['week'], axis = 0)\n    \n    # Plot:\n    plot1.plot.bar(x='week', y='statement_count', color = color_map_statement[int(tot_value)],\n                   ax = ax, label = \"ntot_statement = \"+str(tot_value), bottom=bottom)\n    bottom += np.array(plot1.statement_count.tolist()) # to stack plots, keep raising the bottom\nprint(\">> Only last customer statements: (n_statement = ntot_statement)\")\nplt.show()\ndel plot1","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-31T19:25:54.842373Z","iopub.execute_input":"2023-10-31T19:25:54.842897Z","iopub.status.idle":"2023-10-31T19:26:10.737303Z","shell.execute_reply.started":"2023-10-31T19:25:54.842852Z","shell.execute_reply":"2023-10-31T19:26:10.735737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that most customers start in **March 2017** (with a small number of customers joining through the rest of the year). Almost all customers have their final statement in **March 2018**, meaning most customers have **1 year of statements in the train data**.  \n\nOnly 1 customer had their final statement outside the weeks `['2018_w09', '2018_w10', '2018_w11', '2018_w12', '2018_w13']`:","metadata":{"execution":{"iopub.status.busy":"2023-08-04T13:52:06.453335Z","iopub.execute_input":"2023-08-04T13:52:06.453858Z","iopub.status.idle":"2023-08-04T13:52:06.478335Z","shell.execute_reply.started":"2023-08-04T13:52:06.453812Z","shell.execute_reply":"2023-08-04T13:52:06.476718Z"}}},{"cell_type":"code","source":"outlier_customer = df[(df.n_statement == df.ntot_statement) & \n   (~df.week.isin(['2018_w09', '2018_w10', '2018_w11', '2018_w12', '2018_w13']))\n]\ndisplay(outlier_customer)\nprint(\"Customer_ID:\", outlier_customer.customer_ID.tolist()[0])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-31T19:26:10.738660Z","iopub.execute_input":"2023-10-31T19:26:10.739046Z","iopub.status.idle":"2023-10-31T19:26:10.888399Z","shell.execute_reply.started":"2023-10-31T19:26:10.739014Z","shell.execute_reply":"2023-10-31T19:26:10.886612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## B. Categorical variables:\n\nLets now see the distribution and target behaviour of the categorical features.  \nI will assume that if a feature has `n_unique_values < 16` it's categorical. Using this definition we see bellow the categorical features:\n```python\ncategorical_features = [\n    'B_31', 'D_87', 'D_114', 'D_116', 'D_120', 'D_66', 'D_126', 'B_30', 'D_64', 'D_63', 'B_38', 'D_117', 'D_68'\n]\n```","metadata":{}},{"cell_type":"code","source":"#Ln_uniq_vals, Lcols = [], []\n#for col in df.columns:\n#    if col not in ['week', 'n_statement', 'ntot_statement', 'customer_ID', 'target']:\n#        n_uniq_vals = len(df[col].value_counts(dropna = False).values.tolist())\n#        Ln_uniq_vals.append(n_uniq_vals)\n#        Lcols.append(col)\n#display(pd.DataFrame(\n#    {'variable':Lcols, 'n_unique_values':Ln_uniq_vals}\n#).sort_values(['n_unique_values'], axis = 0).head(14))\n\ncategorical_features = [\n    'B_31', 'D_87', 'D_114', 'D_116', 'D_120', 'D_66', 'D_126', 'B_30', 'D_64', 'D_63', 'B_38', 'D_117', 'D_68'\n]\n\n# Join target to df:\ntry:\n    df = df.drop(['target'], axis = 1)\nexcept:\n    pass\ndf = df.merge(target,how='left', left_on = ['customer_ID'], right_on = ['customer_ID'])\n\n# Create a new figure and the array of axes with the specified figsize\nfig, axs = plt.subplots(5, 3, figsize=(24, 24))\naxs = axs.flatten()\n\nmean_target = df['target'].mean() # mean target\n\n# Loop through each DataFrame in 'df_list' and create subplots with secondary y-axes\nfor i, cat_feat in enumerate(categorical_features):\n    plot2 = df.groupby([cat_feat], dropna = False).agg(\n        {cat_feat:lambda x: len(x), 'target':'mean'}).rename({cat_feat:'freq'}, \n                                                      axis = 1).sort_index().reset_index()\n    plot2[cat_feat] = plot2[cat_feat].apply(lambda x: str(x))\n    \n    ax1 = axs[i]\n\n    # Plot on the primary y-axis (left side)\n    ax1.bar(plot2[cat_feat], plot2['freq'], color='#A4CDFF', label='freq. ')\n    ax1.set_xlabel(cat_feat, fontsize = 16)\n    ax1.set_xticks(plot2[cat_feat])\n    ax1.set_xticklabels(ax1.get_xticks(), fontsize=18)\n    ax1.set_ylabel('freq')\n    ax1.tick_params(axis='y')\n\n    # Create a second y-axis that shares the same x-axis\n    ax2 = ax1.twinx()\n\n    # Plot on the secondary y-axis (right side)\n    ax2.plot(plot2[cat_feat], plot2['target'], color='#FB0C70', label='Target',\n            marker = 'v', markersize=12)\n    ax2.set_ylabel('target',)\n    ax2.tick_params(axis='y',)\n    # Add horizontal mean line:\n    ax2.axhline(y=mean_target, color='#FB0C70', linestyle='dashed', label='Target avg')\n\n    # Show the legend for both plots\n    lines, labels = ax1.get_legend_handles_labels()\n    lines2, labels2 = ax2.get_legend_handles_labels()\n    ax2.legend(lines + lines2, labels + labels2, loc='upper left', fontsize=15)\n    ax2.grid()\n# Adjust the layout to avoid overlapping labels\nplt.tight_layout()\nprint(\">> Categorical variables (statistics by statements):\")\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-31T19:26:10.891856Z","iopub.execute_input":"2023-10-31T19:26:10.892450Z","iopub.status.idle":"2023-10-31T19:26:15.979992Z","shell.execute_reply.started":"2023-10-31T19:26:10.892389Z","shell.execute_reply":"2023-10-31T19:26:15.978900Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Generally we see predictive potential in the above features. I highlight the `B_30` and `B_37` features has apparently having high predictive power.","metadata":{"execution":{"iopub.status.busy":"2023-08-16T10:28:49.036322Z","iopub.execute_input":"2023-08-16T10:28:49.036982Z","iopub.status.idle":"2023-08-16T10:28:49.043919Z","shell.execute_reply.started":"2023-08-16T10:28:49.036944Z","shell.execute_reply":"2023-08-16T10:28:49.042239Z"}}},{"cell_type":"markdown","source":"## C. Numerical Features\n\nNow that we have analysed the categorical features, lets see the distribution + target behaviour of the numerical features.","metadata":{}},{"cell_type":"code","source":"cont_features = sorted(\n    [f for f in df.columns if f not in categorical_features + ['customer_ID', 'target', 'S_2',\n                                                               'n_statement', 'ntot_statement', 'week']])\n\nncols = 4\n\nfig, axs = plt.subplots(35, 5, figsize=(24, 24*5))\naxs = axs.flatten()\nmean_target = round(df.target.mean(),2)\nfor i, f in enumerate(cont_features):\n    # Create bins via np.histogram:\n    hist, bin_edges = np.histogram(df[f].dropna(), bins=8,) # using 10 bins\n    # Need to use .dropna() for np.histogram to work\n    # This will create bins based only on non missing values (of course)\n\n    # Create bin labels from bin_edges:\n    bin_labels = [f\"{i+1:02}_<={bin_edges[i+1]:.2f}\" for i in range(len(bin_edges)-1)]\n    def get_bins(x, bin_edges, bin_labels):\n        for idx, bin_ in enumerate(bin_edges[1:]):\n            if x <= bin_:\n                return bin_labels[idx]\n    # Create new binned variable in df:\n    df[f+'_bin'] = df[f].apply(lambda x: \"00_nan\" if pd.isna(x) else get_bins(x, bin_edges, bin_labels))\n    \n    # Aggregate according to the new variable: compute mean and frequency\n    plot4_g = df.groupby(f+'_bin').agg({'target':'mean', f+\"_bin\":'count'}).rename({f+'_bin':'freq'}, axis = 1).sort_index().reset_index()\n    df = df.drop([f+'_bin'], axis = 1) # drop bin feature\n    \n    ax1 = axs[i]\n\n    # Plot on the primary y-axis (left side):\n    ax1.bar(plot4_g[f+'_bin'], plot4_g['freq'], color='#A4CDFF', label='freq. ')\n    ax1.set_xlabel(f, fontsize = 18)\n    ax1.set_xticks(plot4_g[f+'_bin'])\n    ax1.set_xticklabels(ax1.get_xticks(), fontsize=14)\n    ax1.set_ylabel('freq', fontsize = 11)\n    ax1.tick_params(axis='y')\n\n    # Create a second y-axis that shares the same x-axis:\n    ax2 = ax1.twinx()\n\n    # Plot on the secondary y-axis (right side):\n    ax2.plot(plot4_g[f+'_bin'], plot4_g['target'], color='#FB0C70', label='Target',\n            marker = 'v', markersize=10)\n    ax2.set_ylabel('target', fontsize = 11)\n    ax2.tick_params(axis='y',)\n    \n    # Add horizontal mean line:\n    ax2.axhline(y=mean_target, color='#FB0C70', linestyle='dashed', label='Target avg')\n\n    # Show the legend for both plots\n    lines, labels = ax1.get_legend_handles_labels()\n    lines2, labels2 = ax2.get_legend_handles_labels()\n    ax2.legend(lines + lines2, labels + labels2, loc='upper left', fontsize=11)\n    ax2.grid()\n    \n# Adjust the layout to avoid overlapping labels\nplt.tight_layout()\nprint(\">> Continuous variables (statistics by statements):\")\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-31T19:26:15.981443Z","iopub.execute_input":"2023-10-31T19:26:15.982593Z","iopub.status.idle":"2023-10-31T19:27:29.164374Z","shell.execute_reply.started":"2023-10-31T19:26:15.982547Z","shell.execute_reply":"2023-10-31T19:27:29.162700Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## D. Missing values\n\nLots of features have missing values in this dataset. Lets make a quick analysis of the problem.","metadata":{}},{"cell_type":"code","source":"df[cont_features].isna().sum().sort_values(ascending=False).head(50)\n\ncont_feat_with_nan = [col for col in cont_features if df[col].isna().sum() > 0]\nfor ix, col in enumerate(cont_feat_with_nan):\n    # Auxiliary dataframes:\n    is_nan = df[df[col].isna()][[col, 'target']]\n    not_nan = df[~df[col].isna()][[col, 'target']]\n    # Compute interesting missing metrics:\n    n_is_nan = len(is_nan)\n    n_not_nan = len(not_nan)\n    mean_target_is_nan = is_nan.target.mean()\n    mean_target_not_nan = not_nan.target.mean()\n    min_not_nan = not_nan[col].min()\n    temp = pd.DataFrame({\n        'feature':[col], \"n_is_nan\":[n_is_nan], \"n_not_nan\":[n_not_nan],\n        \"miss%\":[str(round(n_is_nan/(n_is_nan+n_not_nan)*100,2))+\"%\"],\n        \"mean_target_is_nan\":[mean_target_is_nan], \"mean_target_not_nan\":[mean_target_not_nan],\n        \"min_not_nan\":[min_not_nan]\n    })\n    \n    if ix == 0:\n        missing_report = temp.copy()\n    else:\n        missing_report = pd.concat([missing_report, temp], axis = 0)\n    del is_nan, not_nan, temp\nmissing_report = missing_report.sort_values(['n_is_nan'], axis = 0, ascending = False)\nprint(\"Number of features with NaNs: {} ({}%)\".format(len(cont_feat_with_nan),\n                                                          round(len(cont_feat_with_nan)/len(df.columns)*100,1)\n                                                     ))\nprint(\"Number of features with more than 10% missing values: {}\".format(\n    len(missing_report[missing_report.n_is_nan >= len(df)*0.1]))\n)\nprint(\"Top 20 missing features:\")\ndisplay(missing_report.head(20))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-31T19:27:29.166498Z","iopub.execute_input":"2023-10-31T19:27:29.167314Z","iopub.status.idle":"2023-10-31T19:27:31.513977Z","shell.execute_reply.started":"2023-10-31T19:27:29.167269Z","shell.execute_reply":"2023-10-31T19:27:31.513001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that half (54.1%) of features have missing values, with 37 features having more than 10% of it's values missing.  \nWe can also observe that in many situations having a missing value does increase the `target` (see `mean_target_is_nan` and `mean_target_not_nan`).  \n\nThe top 4 features ([`D_88`, `D_111`, `D_110`, `B_39`]) have `n_not_nan < 100`. These aren't enough observations for the variables to be considered for modelling (and here we are seeing all the count considering all statements!).  \n\nHowever, given that the NaN values of these features have valuable information (see their `mean_target_not_nan`), we can for instance build a aggregation variable:\n```python\ndf['D88_D111_D110_B39_bin'] = df.apply(\n    lambda row: 1 if ~pd.isna(row['D_88']) or ~pd.isna(row['D_111']) or ~pd.isna(row['D_110']) or ~pd.isna(row['B_39']) else 0, axis = 1\n) \n```","metadata":{"execution":{"iopub.status.busy":"2023-09-28T19:25:35.857689Z","iopub.execute_input":"2023-09-28T19:25:35.858190Z","iopub.status.idle":"2023-09-28T19:25:35.867784Z","shell.execute_reply.started":"2023-09-28T19:25:35.858152Z","shell.execute_reply":"2023-09-28T19:25:35.866160Z"}}},{"cell_type":"markdown","source":"## E. Random noise added to some features\n\n(Property found by [@raddar](https://www.kaggle.com/raddar) in is [notebook](https://www.kaggle.com/code/raddar/the-data-has-random-uniform-noise-added))  \n\nLet's explore the distribution of `B_2`, one the \"balance\" features.","metadata":{}},{"cell_type":"code","source":"warnings.filterwarnings(\"ignore\", category=FutureWarning) # ignore FutureWarning warning:\n\nplt.figure(figsize=(9, 4))\nsns.distplot(df['B_2'], bins=500, kde=False)\nplt.show()\nprint(\"(zoom in on skike near 1.0)\")\nplt.figure(figsize=(9, 4))\nsns.distplot(df.loc[df.B_2>0.99,'B_2'],bins=500,kde=False)\nplt.show()\nprint(\"(zoom in on skike near 0.81)\")\nplt.figure(figsize=(9, 4))\nsns.distplot(df.loc[(df.B_2>0.8) & (df.B_2<0.83),'B_2'],bins=500,kde=False)\nplt.show()\n\nwarnings.filterwarnings(\"default\", category=FutureWarning) # change it back to default","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-31T19:27:31.515661Z","iopub.execute_input":"2023-10-31T19:27:31.516245Z","iopub.status.idle":"2023-10-31T19:27:35.133841Z","shell.execute_reply.started":"2023-10-31T19:27:31.516209Z","shell.execute_reply":"2023-10-31T19:27:35.132385Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\"We can see that the spike is in range in (1, 1.01) and not a single value ((same for the (0.81, 0.82) spike)). I guess the max value for this feature in raw form is 1 and random uniform value from [0, 0.01] was added for each observation.\" ([@raddar](https://www.kaggle.com/code/raddar/the-data-has-random-uniform-noise-added?scriptVersionId=96828067&cellId=7))\n\nWe can see that most the values of `B_2` are in fact unique:","metadata":{}},{"cell_type":"code","source":"len(df.B_2.unique())/len(df.B_2)","metadata":{"execution":{"iopub.status.busy":"2023-10-31T19:27:35.138284Z","iopub.execute_input":"2023-10-31T19:27:35.138901Z","iopub.status.idle":"2023-10-31T19:27:35.152598Z","shell.execute_reply.started":"2023-10-31T19:27:35.138839Z","shell.execute_reply":"2023-10-31T19:27:35.150268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Has a balance feature, we are expecting a distribution with moderate cardinality. We can clearly see that these features have noise added to them.\n\nThere are more features with added noise. Luckily, [raddar](https://www.kaggle.com/raddar) has already found all these features! \n\nIn the [notebook_train](https://www.kaggle.com/code/raddar/amex-data-int-types-train) and [notebook_test](https://www.kaggle.com/code/raddar/amex-data-int-types-test) he presents the code to change to integer the noised features. I will use his code to build denoise features in my next notebooks.","metadata":{}},{"cell_type":"markdown","source":"## F. Distribution of test set\n\nWe should also check if the statement distribution of the test set is the same as the train data. Differences in the number of statements per `customer_ID`, or time of statements (which means potential data drift) are important information. Namely when building the validation set (to eval the ML models). ","metadata":{}},{"cell_type":"code","source":"df_test = pd.read_csv(\"/kaggle/input/amex-default-prediction/test_data.csv\", nrows=20000)","metadata":{"execution":{"iopub.status.busy":"2023-10-31T20:41:59.991608Z","iopub.execute_input":"2023-10-31T20:41:59.992105Z","iopub.status.idle":"2023-10-31T20:42:01.098276Z","shell.execute_reply.started":"2023-10-31T20:41:59.992066Z","shell.execute_reply":"2023-10-31T20:42:01.097021Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"unique_customer_ids_in_train = len(df_test.customer_ID.unique())\nprint(\"#unique customers in train data: {} out of {} datalines.\".format(\n    unique_customer_ids_in_train, len(df_test)\n))\n\n\ndf_test.groupby(['customer_ID']).agg(\n    {'customer_ID':'count'}\n).sort_index().rename({'customer_ID':'ntot_statements'}, axis = 1).groupby(['ntot_statements']).agg(\n    {'ntot_statements':'count'}\n).sort_index().rename({'ntot_statements':'n_customers'}, axis = 1)","metadata":{"execution":{"iopub.status.busy":"2023-10-31T19:34:51.344678Z","iopub.execute_input":"2023-10-31T19:34:51.345149Z","iopub.status.idle":"2023-10-31T19:34:51.380709Z","shell.execute_reply.started":"2023-10-31T19:34:51.345112Z","shell.execute_reply":"2023-10-31T19:34:51.379374Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def week_from_S_2(x):\n    dt = x.split(\"-\")\n    day, month, year = int(dt[2]), int(dt[1]), int(dt[0])\n    \n    week = datetime.date(year, month, day).isocalendar()[1]\n    \n    # year of date could be different from the week's year (ex: 1 Jan 2023 belongs to week 52, 2022)\n    if month == 1 and week > 50: # date belongs to previous year's last week\n        week_label = str(year-1)\n    else:\n        week_label = str(year)\n    week_label = week_label + \"_w\" + (\"0\"+str(week) if week < 10 else str(week))\n    return week_label\n\n# Get week label:\ndf_test[\"week\"] = df_test.S_2.apply(lambda x: week_from_S_2(x))\n\n# Get n_statements:\ndf_test = df_test.sort_values(['customer_ID', 'S_2'], axis = 0, ascending = [True, True])\nn_statement_aux = 1\nfor row in range(len(df_test)):\n    if row == 0:\n        df_test.loc[row, 'n_statement'] = n_statement_aux\n    else:\n        c_ID = df_test.loc[row,'customer_ID']\n        c_ID_prev = df_test.loc[row-1,'customer_ID']\n        \n        if c_ID == c_ID_prev: # if we are still in the same customer\n            n_statement_aux += 1\n            df_test.loc[row, 'n_statement'] = n_statement_aux\n        else:\n            n_statement_aux = 1\n            df_test.loc[row, 'n_statement'] = n_statement_aux\n            \n# Get ntot_statements:\ntry: # code in case I run this cell multiple times\n    df_test = df_test.drop(['ntot_statement'], axis = 1)\nexcept:\n    pass\ndf_test = df_test.merge(\n    df_test.groupby(['customer_ID']).agg({'n_statement':'max'}).rename({'n_statement':'ntot_statement'}, axis=1),\n    how = 'left', left_on = ['customer_ID'], right_on = ['customer_ID']\n)\n#df_test[['week', 'n_statement', 'ntot_statement']].head()\n\n# All statements plot:\ndf_test.groupby(['week']).agg({'week':'count'}).rename(\n    {'week':'statement_count'}, axis = 1\n).sort_index().reset_index().plot.bar(x='week', y='statement_count', figsize=(24,5),)\nprint(\">> All statements:\")\nplt.show()\n\n# Only first statements plot:\ncolor_map_statement = {\n    1:'#A1FBCA', 2:'#A1FBDD', 3:'#A1FBEA', 4:'#A1F9FB', 5:'#A1D9FB', 6:'#A1BAFB', 7:'#A1A1FB',\n    8:'#C4A1FB', 9:'#E4A1FB', 10:'#FBA1F3', 11:'#FBA1D3', 12:'#FBA1A2', 13:'#F55759'\n}\n_, ax = plt.subplots(figsize=(24, 5))\nbottom = np.zeros(len(df_test.week.unique()))\nfor tot_value in df_test.ntot_statement.sort_values(ascending=True).unique():\n    # Prepare data for plot:\n    plot1 = df_test[(df_test.n_statement == 1) & (df_test.ntot_statement == tot_value)].groupby(\n        ['week', 'n_statement']).agg(\n            {'week':'count'}\n    ).rename({'week':'statement_count'}, axis = 1).sort_index().reset_index()\n    \n    # Merge additional week values to obtain the same xrange:\n    plot1 = pd.concat([plot1,\n                   pd.DataFrame({\n                       'week':df_test[~df_test.week.isin(plot1.week.tolist())].week.unique().tolist(),\n                       'statement_count':0})\n                  ], axis = 0).sort_values(['week'], axis = 0)\n    \n    # Plot:\n    plot1.plot.bar(x='week', y='statement_count', color = color_map_statement[int(tot_value)],\n                   ax = ax, label = \"ntot_statement = \"+str(tot_value), bottom=bottom)\n    bottom += np.array(plot1.statement_count.tolist()) # to stack plots, keep raising the bottom\nprint(\">> Only first customer statements: (n_statement = 1)\")\nplt.show()\n\n\n# Only latest statements plot:\ncolor_map_statement = {\n    1:'#A1FBCA', 2:'#A1FBDD', 3:'#A1FBEA', 4:'#A1F9FB', 5:'#A1D9FB', 6:'#A1BAFB', 7:'#A1A1FB',\n    8:'#C4A1FB', 9:'#E4A1FB', 10:'#FBA1F3', 11:'#FBA1D3', 12:'#FBA1A2', 13:'#F55759'\n}\n_, ax = plt.subplots(figsize=(24, 5))\nbottom = np.zeros(len(df_test.week.unique()))\nfor tot_value in df_test.ntot_statement.sort_values(ascending=True).unique():\n    # Prepare data for plot:\n    plot1 = df_test[(df_test.n_statement == df_test.ntot_statement) & (df_test.ntot_statement == tot_value)].groupby(\n        ['week', 'n_statement']).agg(\n            {'week':'count'}\n    ).rename({'week':'statement_count'}, axis = 1).sort_index().reset_index()\n    \n    # Merge additional week values to obtain the same xrange:\n    plot1 = pd.concat([plot1,\n                   pd.DataFrame({\n                       'week':df_test[~df_test.week.isin(plot1.week.tolist())].week.unique().tolist(),\n                       'statement_count':0})\n                  ], axis = 0).sort_values(['week'], axis = 0)\n    \n    # Plot:\n    plot1.plot.bar(x='week', y='statement_count', color = color_map_statement[int(tot_value)],\n                   ax = ax, label = \"ntot_statement = \"+str(tot_value), bottom=bottom)\n    bottom += np.array(plot1.statement_count.tolist()) # to stack plots, keep raising the bottom\nprint(\">> Only last customer statements: (n_statement = ntot_statement)\")\nplt.show()\ndel plot1","metadata":{"execution":{"iopub.status.busy":"2023-10-31T19:35:34.342984Z","iopub.execute_input":"2023-10-31T19:35:34.343841Z","iopub.status.idle":"2023-10-31T19:35:53.995634Z","shell.execute_reply.started":"2023-10-31T19:35:34.343797Z","shell.execute_reply":"2023-10-31T19:35:53.993845Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see that while the **train last statements** are concentrated in **March 2018**, the **test last statements** are located in two periods: **April 2019 and October 2019**, corresponding to the **public and private leaderboard customers**, respectively (see this [discussion post](https://www.kaggle.com/competitions/amex-default-prediction/discussion/327926) by [Ryota](https://www.kaggle.com/ryotak12)!).","metadata":{}}]}