{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Table of Contents\n\n* [Overview](#section-one)\n* [Part1: Interesting attributes of ‘target’](#section-two)\n* [Part2: Time Series analysis deep dive](#section-three)\n    - [Overview & Thoughts](#subsection-overview)\n    - [Delinquency Variables](#subsection-delinquency)\n    - [Spend Variables](#subsection-spend)\n    - [Payment Variables](#subsection-payment)\n    - [Balance Variables](#subsection-balance)\n    - [Risk Variables](#subsection-risk)","metadata":{}},{"cell_type":"markdown","source":"<a id=\"section-one\"></a>\n# Overview\n\nI have been lurking on Kaggle and specifically this compeition for some time and thought to write down some of my thoughts here. I see a lot of the EDAs have been focusing on the whole dataset, whereas most models are built on the aggregated data of each customer (e.g. take min/max of a feature over all the statements, take first/last, etc.) \n\n**This notebook tries to bridge the gap in between these two by using a time series perspective EDA to take a deeper look into how the statements evolve over time for each customer, spot patterns, and propose ideas to engineer features so that when we compress the original train dataset to an customer-level aggregated train dataset, we can create as many variables that would pack information as possible.**\n\nIn this notebook would mainly consist of two parts. The first part looks into some interesting attributes of ‘target’ - like the fact that the defaulting average goes up when there are more observations in train, it goes higher on Sundays, and it is also higher when the customer has less statements. The second part paints plots time series charts on the different categories of variables (like spend, delinquency, etc.), and looks at how the mean, std, distribution changes across the different time stamps. In both sections I would highlight my ideas in detail on what I found interesting from the data, and how it can be used for feature engineering.\n\nAll in all, it was a lot of fun for me to work on this AMEX default project, and hopefully this notebook could be of use to everybody as it had been for me! Please give me a upvote if it were helpful and all recommendations are greatly welcomed. I am working on the feature engineering problem based off of my findings in this notebook and I will publish it as soon as they are done. \n\nMany thanks to:\n1. CHRIS DEOTTE - For the time series EDA in [this notebook](http://https://www.kaggle.com/code/cdeotte/time-series-eda) which gave me inspiration for this notebook\n2. RADDAR - For providing integer dtypes - parquet format in [this](http://https://www.kaggle.com/datasets/raddar/amex-data-integer-dtypes-parquet-format) dataset \n3. Oleh Kondratenko and Burrito Dan for following up in my [previous discussion post](https://www.kaggle.com/competitions/amex-default-prediction/discussion/337329#1856691) about their observations on S_2","metadata":{}},{"cell_type":"markdown","source":"# Setup","metadata":{"_kg_hide-input":true}},{"cell_type":"code","source":"#essentials\nimport numpy as np\nimport pandas as pd\nimport datetime\n\n#plotting libraries\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n#os\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        pass\n        #print(os.path.join(dirname, filename))\n        \n#misc.\nimport random\nimport gc","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-19T16:51:03.270489Z","iopub.execute_input":"2022-07-19T16:51:03.270952Z","iopub.status.idle":"2022-07-19T16:51:03.916025Z","shell.execute_reply.started":"2022-07-19T16:51:03.270857Z","shell.execute_reply":"2022-07-19T16:51:03.915222Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#'global' variables\nplot_width = 22","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-19T16:51:03.917578Z","iopub.execute_input":"2022-07-19T16:51:03.918269Z","iopub.status.idle":"2022-07-19T16:51:03.921936Z","shell.execute_reply.started":"2022-07-19T16:51:03.918238Z","shell.execute_reply":"2022-07-19T16:51:03.921131Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_data(input_dir, test=True):\n    train = pd.read_parquet(input_dir + 'train.parquet')\n    if (test):\n        test = pd.read_parquet(input_dir + 'test.parquet')\n    else:\n        test = None\n    \n    train_labels = pd.read_csv('../input/amex-default-prediction/train_labels.csv')\n    train_labels_dict = pd.Series(train_labels.target.values,index=train_labels.customer_ID).to_dict()\n    train['target'] = train['customer_ID'].map(train_labels_dict)\n\n    return train, test\n\ntrain, test = read_data('../input/amex-data-integer-dtypes-parquet-format/', test=True)\ntrain.S_2 = pd.to_datetime(train.S_2)\nprint('train shape is {}'.format(train.shape))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-19T16:51:03.923202Z","iopub.execute_input":"2022-07-19T16:51:03.923539Z","iopub.status.idle":"2022-07-19T16:52:17.252299Z","shell.execute_reply.started":"2022-07-19T16:51:03.923510Z","shell.execute_reply":"2022-07-19T16:52:17.250677Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-two\"></a>\n# Part 1: Interesting attributes of ‘target’","metadata":{}},{"cell_type":"markdown","source":"**Observation:** this is a chart of the number of data points at each timestamp and the mean of the target at that same time stamp. Although the target mean fluctuates around 0.25 (as the detailed in the metric description about upsampling) it does bounce around, also seems like the less samples gathered the higher the percentage of defaults for that observation. The correlation between these two variables are -36%! We also see in the test data the # of observations bounces around.","metadata":{}},{"cell_type":"code","source":"#pretty consistent and stable throughout the observation period\n#target also fluctuates around 0.25 (as the prompt has shown) but bounces around, also seems like the less samples gathered the higher the defaults for that observation\nfig, ax = plt.subplots(figsize=(plot_width, 4))\nax2 = ax.twinx()\n\ntemp = train.groupby(['S_2']).agg({'target' : np.mean, 'S_2': np.size})\ntemp = temp.rename(columns={'S_2': 'observation count', 'target':'target average'})\n\ntemp['observation count'].plot(ax=ax, color='steelblue', label='observation count')\ntemp['target average'].plot(ax=ax2, color='orange', label='target_average')\nax.legend(loc=2)\nax2.legend(loc=0)\nfig.suptitle('number of observations (left) vs average of target (right) in training')\n\nplt.savefig('train_observations_vs_target_avg.png')\nplt.show()\n\nsns.heatmap(temp.corr().round(3), annot=True)\n#temp.corr().round(3).to_csv('corr.csv')","metadata":{"execution":{"iopub.status.busy":"2022-07-19T16:52:17.255311Z","iopub.execute_input":"2022-07-19T16:52:17.255697Z","iopub.status.idle":"2022-07-19T16:52:19.102828Z","shell.execute_reply.started":"2022-07-19T16:52:17.255659Z","shell.execute_reply":"2022-07-19T16:52:19.102052Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ttemp = test.groupby(['S_2']).agg({'S_2': np.size})\nttemp = ttemp.rename(columns={'S_2': 'observation count', 'target':'target average'})\nfig, ax = plt.subplots(figsize=(plot_width, 4))\nttemp['observation count'].plot(ax=ax, color='steelblue', label='observation count')\nax.legend(loc=2)\nfig.suptitle('number of observations (left) in testing')\nplt.savefig('testing_observation_count.png')\nplt.show()\n\ndel test, ttemp, temp\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-07-19T16:52:19.103848Z","iopub.execute_input":"2022-07-19T16:52:19.104792Z","iopub.status.idle":"2022-07-19T16:52:23.178168Z","shell.execute_reply.started":"2022-07-19T16:52:19.104758Z","shell.execute_reply":"2022-07-19T16:52:23.177016Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#del default_rate_by_weekday, num_obs_by_weekday\n#train.drop(columns=['dayofweek'], inplace=True)\ngc.collect()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-19T16:52:23.179725Z","iopub.execute_input":"2022-07-19T16:52:23.180060Z","iopub.status.idle":"2022-07-19T16:52:23.309013Z","shell.execute_reply.started":"2022-07-19T16:52:23.180030Z","shell.execute_reply":"2022-07-19T16:52:23.307928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:** This is a chart of the average default rate distribution by dayofweek and also the number of observations on each dayofweek. Kind of similar to the observation above, we see that Saturday has the most observations and lowest default rate, whereas Sunday has the lowest observations and highest default rate. \n\n**Idea:** I think there can be a series of features engineered from S_2 itself, both with the day of the week and the number of observations on that day. Not sure which causes which but it does seem like there is a relationship going on here. ","metadata":{}},{"cell_type":"code","source":"train['dayofweek'] = train['S_2'].dt.day_name()\ndefault_rate_by_weekday = train.groupby(['dayofweek'])['target'].mean()\nnum_obs_by_weekday = train.groupby(['dayofweek']).size()\n\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(plot_width, 10))\nweekday_order = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday']\nax1.set_title('default rate by dayofweek')\ndefault_rate_by_weekday.reindex(weekday_order).plot(ax=ax1, color='steelblue')\nax2.set_title('number of observations by dayofweek')\nnum_obs_by_weekday.reindex(weekday_order).plot(ax=ax2)\n","metadata":{"execution":{"iopub.status.busy":"2022-07-19T16:52:23.310319Z","iopub.execute_input":"2022-07-19T16:52:23.310642Z","iopub.status.idle":"2022-07-19T16:52:27.578369Z","shell.execute_reply.started":"2022-07-19T16:52:23.310613Z","shell.execute_reply":"2022-07-19T16:52:27.577276Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_sorted_list_intersection(l1, l2):\n    temp = list(set(l1) & set(l2))\n    temp.sort(key=lambda x: int(x.split('_')[1]))\n    return temp","metadata":{"execution":{"iopub.status.busy":"2022-07-19T16:52:27.579966Z","iopub.execute_input":"2022-07-19T16:52:27.580331Z","iopub.status.idle":"2022-07-19T16:52:27.586607Z","shell.execute_reply.started":"2022-07-19T16:52:27.580300Z","shell.execute_reply":"2022-07-19T16:52:27.585492Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"yes_defaulted_IDs = train[train['target'] == 1].customer_ID.unique().tolist()\nno_defaulted_IDs = train[train['target'] == 0].customer_ID.unique().tolist()\n\ncustomer_appearance_count_series = train.groupby(['customer_ID']).cumcount().add(1)\ntrain['appearance_count'] = customer_appearance_count_series","metadata":{"execution":{"iopub.status.busy":"2022-07-19T16:52:27.588456Z","iopub.execute_input":"2022-07-19T16:52:27.588794Z","iopub.status.idle":"2022-07-19T16:52:36.530013Z","shell.execute_reply.started":"2022-07-19T16:52:27.588766Z","shell.execute_reply":"2022-07-19T16:52:36.528812Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#distribution of total statements by default group\n\n_defaulted = train.loc[train['customer_ID'].isin(yes_defaulted_IDs)].groupby(['customer_ID'])['appearance_count'].max().value_counts().sort_index().rename('defaulted')\n_paid = train.loc[train['customer_ID'].isin(no_defaulted_IDs)].groupby(['customer_ID'])['appearance_count'].max().value_counts().sort_index().rename('paid')","metadata":{"execution":{"iopub.status.busy":"2022-07-19T16:52:36.534159Z","iopub.execute_input":"2022-07-19T16:52:36.534501Z","iopub.status.idle":"2022-07-19T16:52:42.290178Z","shell.execute_reply.started":"2022-07-19T16:52:36.534472Z","shell.execute_reply":"2022-07-19T16:52:42.289151Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:** these two are really interesting charts. First we can see that a. The default rate is significantly lower when the customer has all 13 of observations and b. Most customers have 13 observations (right chart). \n\n**Idea:** I think this can be due to ‘early-stopping’ like if the customer defaults then they stop collecting info on the customer (or put them into another basket). Not sure how robust this is in train vs test but definitely something to experiment on.","metadata":{}},{"cell_type":"code","source":"fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(plot_width, 10))\n\n_df = pd.concat([_defaulted, _paid], axis=1)\n_df['total'] = _df['paid'] + _df['defaulted']\n_df['paid ratio'] = _df['paid'] / _df['total']\n_df['defaulted ratio'] = _df['defaulted'] / _df['total']\n\n\n_df[['defaulted ratio', 'paid ratio']].plot(ax=ax1, kind='line', color=['steelblue', 'orange'])\nax1.set_title('default rate by num of observations per customer')\n_df[['defaulted', 'paid']].plot(ax=ax2, kind='bar', stacked=True, color=['steelblue', 'orange'])\nax2.set_title('count of num of observations per customer')","metadata":{"execution":{"iopub.status.busy":"2022-07-19T16:52:42.291680Z","iopub.execute_input":"2022-07-19T16:52:42.292108Z","iopub.status.idle":"2022-07-19T16:52:42.785877Z","shell.execute_reply.started":"2022-07-19T16:52:42.292063Z","shell.execute_reply":"2022-07-19T16:52:42.784699Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-three\"></a>\n# Part 2: Time Series analysis deep dive","metadata":{}},{"cell_type":"code","source":"'''\nD_* = Delinquency variables\nS_* = Spend variables\nP_* = Payment variables\nB_* = Balance variables\nR_* = Risk variables\n'''\n\nall_variables = list(train)\n\ndtypes_df = train.dtypes\ncategorical_features_list = ['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68'] #from the description\nnon_categorical_features_list = [x for x in all_variables if x not in categorical_features_list]\n\n\ndelinquency_variables = [i for i in all_variables if i.startswith('D_')]\nspend_variables = [i for i in all_variables if (i.startswith('S_') and i != 'S_2')]\npayment_variables = [i for i in all_variables if i.startswith('P_')]\nbalance_variables = [i for i in all_variables if i.startswith('B_')]\nrisk_variables = [i for i in all_variables if i.startswith('R_')]","metadata":{"execution":{"iopub.status.busy":"2022-07-19T16:52:42.787999Z","iopub.execute_input":"2022-07-19T16:52:42.789064Z","iopub.status.idle":"2022-07-19T16:52:42.801053Z","shell.execute_reply.started":"2022-07-19T16:52:42.789016Z","shell.execute_reply":"2022-07-19T16:52:42.799815Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_helper(df1, df2, ax, label1='', label2='', title='', recenter_na=True, std=1, legend=True):\n    df1_mean_series = df1.mean()\n    df2_mean_series = df2.mean()\n    \n    df1_std_series = df1.std()\n    df2_std_series = df2.std()\n    \n    if (recenter_na):\n        df1_diff_mean_series = df1.diff(axis=1).mean()\n        df2_diff_mean_series = df2.diff(axis=1).mean()\n        \n        r1 = (df1_mean_series.iloc[0] + df1_diff_mean_series.cumsum().fillna(0))\n        r2 = (df2_mean_series.iloc[0] + df2_diff_mean_series.cumsum().fillna(0))\n    else:\n        r1 = df1_mean_series\n        r2 = df2_mean_series\n    \n    r1.plot(ax=ax, color='steelblue', label=label1)\n    r2.plot(ax=ax, color='orange', label=label2)\n    \n    if (std):\n        (r1 + (std*df1_std_series)).plot(ax=ax, color='steelblue', style='--', label='{} + {} sd'.format(label1, std))\n        (r1 - (std*df1_std_series)).plot(ax=ax, color='steelblue', style='--', label='{} - {} sd'.format(label1, std))\n\n        (r2 + (std*df2_std_series)).plot(ax=ax, color='orange', style='--', label='{} + {} sd'.format(label2, std))\n        (r2 - (std*df2_std_series)).plot(ax=ax, color='orange', style='--', label='{} - {} sd'.format(label2, std))\n    \n    if (legend):\n        ax.legend()\n    ax.set_title(title)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-19T16:52:42.802840Z","iopub.execute_input":"2022-07-19T16:52:42.803354Z","iopub.status.idle":"2022-07-19T16:52:42.818395Z","shell.execute_reply.started":"2022-07-19T16:52:42.803313Z","shell.execute_reply":"2022-07-19T16:52:42.817572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def distribution_plot_helper(df1, df2, ax, title='', timestamps_to_plot = [1, 6, 13], \n                             legend=True, logscale=True, num_bins=100, clip=True):\n    if (clip):\n        _min = min(df1.quantile(0.001).min(), df2.quantile(0.001).min())\n        _max = max(df1.quantile(0.999).max(), df2.quantile(0.999).max())\n        df1.clip(lower=_min)\n        df2.clip(lower=_min)\n        df1.clip(upper=_max)\n        df2.clip(upper=_max)\n    else:\n        _min = min(df1.min(), df2.min())\n        _max = max(df1.max(), df2.max())\n    bins = np.linspace(_min, _max, num_bins)\n    \n    #default_colors_dic = {1:'lightblue', 6:'steelblue', 13:'navy'}\n    #paid_colors_dic = {1:'peachpuff', 6:'orange', 13:'darkorange'}\n    \n    default_colors_dic = {1:'steelblue', 13:'navy'}\n    paid_colors_dic = {1:'orange', 13:'darkorange'}\n    \n    for col in timestamps_to_plot:\n        df1[col].hist(bins=bins, ax=ax, alpha=0.6, color=default_colors_dic[col], label='default time={}'.format(col))\n        \n    #separating into for loops makes the legend order correct...\n    for col in timestamps_to_plot:\n        df2[col].hist(bins=bins, ax=ax, alpha=0.6, color=paid_colors_dic[col], label='paid time={}'.format(col))\n    \n    if (legend):\n        ax.legend()\n    if (logscale):\n        ax.set_yscale('log')\n\n    ax.set_title(title)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-19T16:52:42.819846Z","iopub.execute_input":"2022-07-19T16:52:42.820717Z","iopub.status.idle":"2022-07-19T16:52:42.834544Z","shell.execute_reply.started":"2022-07-19T16:52:42.820666Z","shell.execute_reply":"2022-07-19T16:52:42.833689Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def kde_plot_helper(df1, df2, ax, title='', timestamps_to_plot = [1, 6, 13], \n                             legend=True, logscale=True, alpha=0.7, clip=True):\n    #default_colors_dic = {1:'lightblue', 6:'steelblue', 13:'navy'}\n    #paid_colors_dic = {1:'peachpuff', 6:'orange', 13:'darkorange'}\n    \n    if (clip):\n        _min = min(df1.quantile(0.001).min(), df2.quantile(0.001).min())\n        _max = max(df1.quantile(0.999).max(), df2.quantile(0.999).max())\n        df1.clip(lower=_min)\n        df2.clip(lower=_min)\n        df1.clip(upper=_max)\n        df2.clip(upper=_max)\n    \n    default_colors_dic = {1:'steelblue', 13:'purple'}\n    paid_colors_dic = {1:'orange', 13:'saddlebrown'}\n    \n    for col in timestamps_to_plot:\n        sns.kdeplot(x=col, data=df1, fill=False, linewidth=2, legend=False, ax=ax, label='default time={}'.format(col), alpha=alpha, color=default_colors_dic[col])\n        \n    #separating into for loops makes the legend order correct...\n    for col in timestamps_to_plot:\n        sns.kdeplot(x=col, data=df2, fill=False, linewidth=2, legend=False, ax=ax, label='paid time={}'.format(col), alpha=alpha, color=paid_colors_dic[col])\n    \n    if (legend):\n        ax.legend()\n    if (logscale):\n        ax.set_yscale('log')\n\n    ax.set_title(title)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-19T16:52:42.835686Z","iopub.execute_input":"2022-07-19T16:52:42.836389Z","iopub.status.idle":"2022-07-19T16:52:42.851137Z","shell.execute_reply.started":"2022-07-19T16:52:42.836355Z","shell.execute_reply":"2022-07-19T16:52:42.849826Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def categorical_distribution_plot_helper(df1, df2, ax, title='', timestamps_to_plot = [1, 6, 13], \n                                         legend=True, logscale=True, num_bins=100):\n    \n    default_colors_dic = {1:'lightblue', 6:'steelblue', 13:'navy'}\n    paid_colors_dic = {1:'peachpuff', 6:'orange', 13:'darkorange'}\n    \n    for col in timestamps_to_plot:\n        df1[col].hist(ax=ax, alpha=0.6, color=default_colors_dic[col], label='default time={}'.format(col))\n        \n    #separating into for loops makes the legend order correct...\n    for col in timestamps_to_plot:\n        df2[col].hist(ax=ax, alpha=0.6, color=paid_colors_dic[col], label='paid time={}'.format(col))\n    \n    if (legend):\n        ax.legend()\n    if (logscale):\n        ax.set_yscale('log')\n\n    ax.set_title(title)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-19T16:52:42.853602Z","iopub.execute_input":"2022-07-19T16:52:42.854499Z","iopub.status.idle":"2022-07-19T16:52:42.868044Z","shell.execute_reply.started":"2022-07-19T16:52:42.854437Z","shell.execute_reply":"2022-07-19T16:52:42.867148Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_all(variables_list, verbose=False):\n    #non categorical variables\n    target_cols = get_sorted_list_intersection(non_categorical_features_list, variables_list)\n    for col in target_cols:\n        temp = train.pivot(index='customer_ID', columns='appearance_count', values=col)\n        defaulted_temp = temp.loc[yes_defaulted_IDs]\n        paid_temp = temp.loc[no_defaulted_IDs]\n        defaulted_temp_diff = temp.loc[yes_defaulted_IDs].diff(axis=1)\n        paid_temp_diff = temp.loc[no_defaulted_IDs].diff(axis=1)\n\n        fig, (ax1, ax2, ax3, ax4, ax5) = plt.subplots(1, 5, figsize=(25, 4))\n        plot_helper(defaulted_temp, paid_temp, ax1, label1='defaulted', label2='paid', title='{} mean'.format(col), recenter_na=True, std=0)\n        plot_helper(defaulted_temp, paid_temp, ax2, label1='defaulted', label2='paid', title='{} mean_w_{}_sd'.format(col, 1), recenter_na=True, std=1)\n        plot_helper(defaulted_temp_diff, paid_temp_diff, ax3, label1='defaulted', label2='paid', title='{} period_diff_w_{}_sd'.format(col, 1), recenter_na=False, std=1)\n        distribution_plot_helper(defaulted_temp, paid_temp, ax4, title='{} distribution hist plot'.format(col), timestamps_to_plot=[1])\n        distribution_plot_helper(defaulted_temp, paid_temp, ax5, title='{} distribution hist plot'.format(col), timestamps_to_plot=[13])\n        #kde_plot_helper(defaulted_temp, paid_temp, ax5, title='{} kde plot'.format(col), timestamps_to_plot=[1, 13], logscale=False)\n\n        del temp, defaulted_temp, paid_temp, defaulted_temp_diff, paid_temp_diff\n        gc.collect()\n        if (verbose):\n            print(col)\n    \n    #plot categorical variables\n    target_cols = get_sorted_list_intersection(categorical_features_list, variables_list)\n    for col in target_cols:\n        fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(15, 4))\n\n        temp = train.pivot(index='customer_ID', columns='appearance_count', values=col)\n        defaulted_temp = temp.loc[yes_defaulted_IDs]\n        paid_temp = temp.loc[no_defaulted_IDs]\n        categorical_distribution_plot_helper(defaulted_temp, paid_temp, ax1, title='{} distribution hist plot'.format(col), timestamps_to_plot=[1])\n        categorical_distribution_plot_helper(defaulted_temp, paid_temp, ax2, title='{} distribution hist plot'.format(col), timestamps_to_plot=[6])\n        categorical_distribution_plot_helper(defaulted_temp, paid_temp, ax3, title='{} distribution hist plot'.format(col), timestamps_to_plot=[13])\n        \n        del temp, defaulted_temp, paid_temp\n        gc.collect()\n        if (verbose):\n            print(col)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-19T16:52:42.869393Z","iopub.execute_input":"2022-07-19T16:52:42.870258Z","iopub.status.idle":"2022-07-19T16:52:42.887751Z","shell.execute_reply.started":"2022-07-19T16:52:42.870215Z","shell.execute_reply":"2022-07-19T16:52:42.886617Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"subsection-overview\"></a>\n## Overview for Part 2","metadata":{}},{"cell_type":"markdown","source":"### Numerical Variables\nBefore I start I would like to give an explanation on how these charts work. D_65 here is an example of a numerical variable’s chart. There are five smaller charts per variable, and from an order of left to right they are:\nChart 1. The mean of the variable split grouped by defaulted and paid across the 13 statements <br>\nChart 2. Same as the first chart but with the standard deviation (at each appearance) also plotted <br>\nChart 3. This is similar to the second chart but instead of plotting the mean of the variable itself it plots the difference between each period (think of it as mean(t1_mean - t0_mean)) <br>\nChart 4. This plots the histogram of the variable at the first statement <br>\nChart 5. Same as chart 4 but for the 13th statement (ignores NaNs) <br>\n","metadata":{}},{"cell_type":"markdown","source":"**Observation:** The distribution definitely changes more drastically on the defaulted variables than the paid ones. In D_65 as an example as can see the defaulted mean just kept on climbing whereas the paid one basically stayed constant (chart 1). We can also see that D_65’s distribution is also more spare for defaulted observations vs paid observations (blue dotted vs orange dotted lines). In the two rightmost charts (chart 4 & chart 5) we can see that the defaulted distribution became a lot more right skewed compared the paid one. ","metadata":{}},{"cell_type":"code","source":"plot_all(['D_65'])","metadata":{"execution":{"iopub.status.busy":"2022-07-19T16:52:42.889106Z","iopub.execute_input":"2022-07-19T16:52:42.890073Z","iopub.status.idle":"2022-07-19T16:52:51.932227Z","shell.execute_reply.started":"2022-07-19T16:52:42.890040Z","shell.execute_reply":"2022-07-19T16:52:51.931133Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Categorical Variables\nFor categorical variables from left to right it represents:\n1. it is the histogram plot of the variable at t=1\n2. same as above but for t=6\n3. same as above but for t=13 \n\nso that we can observe how the categorical variables evolve over time.","metadata":{}},{"cell_type":"code","source":"plot_all(['D_63'])","metadata":{"execution":{"iopub.status.busy":"2022-07-19T16:52:51.933665Z","iopub.execute_input":"2022-07-19T16:52:51.933973Z","iopub.status.idle":"2022-07-19T16:52:57.216509Z","shell.execute_reply.started":"2022-07-19T16:52:51.933944Z","shell.execute_reply":"2022-07-19T16:52:57.215251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### **Ideas**\nI think many features can be engineered by looking at these distribution plots. I think for feature engineering there should be two areas in progressive order.\n1. **first and foremost is the level of the variable**. This can be taken care of by just looking at the first/last, mean, etc for each customer across the different observation periods \n2. **The second and probably the harder part is the evolution for the levels and distribution**. This makes a lot of sense to me since probably the paid and defaulted customers kind of start of similarily (or else they wouldn't be approved of a credit card in the first place), and then the defaulted ones just kind of 'worsen' over time. A rudimentary approach to this would be to take diff (first - last), std(all observations), skew(all observations) but I think more can be done here. ","metadata":{}},{"cell_type":"markdown","source":"<a id=\"subsection-delinquency\"></a>\n## Deliquency Variables","metadata":{}},{"cell_type":"code","source":"%%time\nplot_all(delinquency_variables)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T16:52:57.220101Z","iopub.execute_input":"2022-07-19T16:52:57.220778Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"subsection-spend\"></a>\n\n## Spend Variables","metadata":{}},{"cell_type":"code","source":"%%time\nplot_all(spend_variables)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"subsection-payment\"></a>\n\n## Payment Variables","metadata":{}},{"cell_type":"code","source":"%%time\nplot_all(payment_variables)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"subsection-balance\"></a>\n\n## Balance Variables","metadata":{}},{"cell_type":"code","source":"%%time\nplot_all(balance_variables)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"subsection-risk\"></a>\n\n## Risk Variables","metadata":{}},{"cell_type":"code","source":"%%time\nplot_all(risk_variables)","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}