{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Ch 01. View Each Separately\n\nWhen we don't really understand what our data looks like, it's best to look at in smaller chunks. In this first chapter, we separate the five column types(D,S,P,B,R) and take a look at the two-dimensional distribution.","metadata":{}},{"cell_type":"markdown","source":"> Table of Contents\n```\nStep 1. Load Samle Data\nStep 2. Separate Data\nStep 3. Check Outliers\nStep 4. PCA (Focus on 'D' type)\n```\n","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-06-03T11:51:18.539067Z","iopub.execute_input":"2022-06-03T11:51:18.539470Z","iopub.status.idle":"2022-06-03T11:51:18.546681Z","shell.execute_reply.started":"2022-06-03T11:51:18.539431Z","shell.execute_reply":"2022-06-03T11:51:18.545826Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Step 1. Load Sample Data","metadata":{}},{"cell_type":"markdown","source":"\nBecause the data volume is too large, we will only use 10000 rows in the tutorial.","metadata":{}},{"cell_type":"code","source":"train_data_path = '/kaggle/input/amex-default-prediction/train_data.csv'\ntrain_label_path = '/kaggle/input/amex-default-prediction/train_labels.csv'","metadata":{"execution":{"iopub.status.busy":"2022-06-03T11:10:50.118786Z","iopub.execute_input":"2022-06-03T11:10:50.119186Z","iopub.status.idle":"2022-06-03T11:10:50.123768Z","shell.execute_reply.started":"2022-06-03T11:10:50.119154Z","shell.execute_reply":"2022-06-03T11:10:50.122609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_iter = pd.read_csv(train_data_path, chunksize=10000)\ntrain_iter","metadata":{"execution":{"iopub.status.busy":"2022-06-03T11:10:51.010923Z","iopub.execute_input":"2022-06-03T11:10:51.011398Z","iopub.status.idle":"2022-06-03T11:10:51.022107Z","shell.execute_reply.started":"2022-06-03T11:10:51.011359Z","shell.execute_reply":"2022-06-03T11:10:51.021388Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_sample = train_iter.__next__() # first 10000 rows\ntrain_sample","metadata":{"execution":{"iopub.status.busy":"2022-06-03T11:10:52.142693Z","iopub.execute_input":"2022-06-03T11:10:52.143081Z","iopub.status.idle":"2022-06-03T11:10:52.525408Z","shell.execute_reply.started":"2022-06-03T11:10:52.143050Z","shell.execute_reply":"2022-06-03T11:10:52.524384Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Check how many cases are counted for each customer ID","metadata":{}},{"cell_type":"code","source":"train_sample['customer_ID'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-06-03T10:59:49.412831Z","iopub.execute_input":"2022-06-03T10:59:49.413212Z","iopub.status.idle":"2022-06-03T10:59:49.427207Z","shell.execute_reply.started":"2022-06-03T10:59:49.413180Z","shell.execute_reply":"2022-06-03T10:59:49.426343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"When unfolding as a two-dimensional distribution, the target will be expressed in color. So, Let's take the target information as well and combine it with our sample data.","metadata":{}},{"cell_type":"code","source":"train_labels = pd.read_csv(train_label_path)\ntrain_labels","metadata":{"execution":{"iopub.status.busy":"2022-06-03T11:13:49.166109Z","iopub.execute_input":"2022-06-03T11:13:49.166507Z","iopub.status.idle":"2022-06-03T11:13:50.244559Z","shell.execute_reply.started":"2022-06-03T11:13:49.166475Z","shell.execute_reply":"2022-06-03T11:13:50.243589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_df = pd.merge(left=train_sample, right=train_labels, on=['customer_ID'], how='left')\nsample_df","metadata":{"execution":{"iopub.status.busy":"2022-06-03T11:13:52.015646Z","iopub.execute_input":"2022-06-03T11:13:52.016070Z","iopub.status.idle":"2022-06-03T11:13:52.244478Z","shell.execute_reply.started":"2022-06-03T11:13:52.016027Z","shell.execute_reply":"2022-06-03T11:13:52.243365Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Step 2. Separate data","metadata":{}},{"cell_type":"markdown","source":"We will check the data separately for each column type. So, Let's separate the columns.","metadata":{}},{"cell_type":"code","source":"d_cols = list(filter(lambda x : x.startswith(\"D\"), train_sample.columns.tolist()))\ns_cols = list(filter(lambda x : x.startswith(\"S\"), train_sample.columns.tolist()))\np_cols = list(filter(lambda x : x.startswith(\"P\"), train_sample.columns.tolist()))\nb_cols = list(filter(lambda x : x.startswith(\"B\"), train_sample.columns.tolist()))\nr_cols = list(filter(lambda x : x.startswith(\"R\"), train_sample.columns.tolist()))\n\nprint(f\"number of 'd's : {len(d_cols)}\\n\",\n      f\"number of 's's : {len(s_cols)}\\n\", \n      f\"number of 'p's : {len(p_cols)}\\n\", \n      f\"number of 'b's : {len(b_cols)}\\n\", \n      f\"number of 'r's : {len(r_cols)}\\n\")","metadata":{"execution":{"iopub.status.busy":"2022-06-03T11:14:31.064520Z","iopub.execute_input":"2022-06-03T11:14:31.064932Z","iopub.status.idle":"2022-06-03T11:14:31.072932Z","shell.execute_reply.started":"2022-06-03T11:14:31.064898Z","shell.execute_reply":"2022-06-03T11:14:31.071904Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings('ignore')\n\nuser_info = sample_df[['customer_ID', 'target']]\nts_info = sample_df[['S_2']]\nts_info['S_2'] = ts_info['S_2'].astype('datetime64')\n\nsample_df_d = sample_df.loc[:,d_cols]\nsample_df_d = pd.concat([ts_info, user_info, sample_df_d], axis=1)\nsample_df_d.set_index('S_2', inplace=True)\nsample_df_d.sort_index(inplace=True)\n\nsample_df_s = sample_df.loc[:,s_cols[1:]]\nsample_df_s = pd.concat([ts_info, user_info, sample_df_s], axis=1)\nsample_df_s.set_index('S_2', inplace=True)\nsample_df_s.sort_index(inplace=True)\n\nsample_df_p = sample_df.loc[:,p_cols]\nsample_df_p = pd.concat([ts_info, user_info, sample_df_p], axis=1)\nsample_df_p.set_index('S_2', inplace=True)\nsample_df_p.sort_index(inplace=True)\n\nsample_df_b = sample_df.loc[:,b_cols]\nsample_df_b = pd.concat([ts_info, user_info, sample_df_b], axis=1)\nsample_df_b.set_index('S_2', inplace=True)\nsample_df_b.sort_index(inplace=True)\n\nsample_df_r = sample_df.loc[:,r_cols]\nsample_df_r = pd.concat([ts_info, user_info, sample_df_r], axis=1)\nsample_df_r.set_index('S_2', inplace=True)\nsample_df_r.sort_index(inplace=True)\n\nprint(f\"[sample_df_d] number of rows : {len(sample_df_d)}\\n\",\n      f\"[sample_df_d] number of cols : {len(sample_df_d.columns)}\\n\",\n      \n      f\"[sample_df_s] number of rows : {len(sample_df_s)}\\n\",             \n      f\"[sample_df_s] number of cols : {len(sample_df_s.columns)}\\n\", \n      \n      f\"[sample_df_p] number of rows : {len(sample_df_p)}\\n\", \n      f\"[sample_df_p] number of cols : {len(sample_df_p.columns)}\\n\", \n      \n      f\"[sample_df_b] number of rows : {len(sample_df_b)}\\n\", \n      f\"[sample_df_b] number of cols : {len(sample_df_b.columns)}\\n\", \n      \n      f\"[sample_df_r] number of rows : {len(sample_df_r)}\\n\",\n      f\"[sample_df_r] number of cols : {len(sample_df_r.columns)}\\n\")","metadata":{"execution":{"iopub.status.busy":"2022-06-03T11:15:24.678066Z","iopub.execute_input":"2022-06-03T11:15:24.678541Z","iopub.status.idle":"2022-06-03T11:15:24.737964Z","shell.execute_reply.started":"2022-06-03T11:15:24.678506Z","shell.execute_reply":"2022-06-03T11:15:24.737013Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If the data types are the same, the independence between columns is likely to be weak. And if the independence is weak, it is necessary to eliminate some of the highly correlated variables. To check these points, let's check the correlation distribution. The darker the blue color, the lower the independence and the stronger the correlation.","metadata":{}},{"cell_type":"code","source":"import seaborn as sns\nimport matplotlib.pyplot as plt\n\nplt.figure(figsize=(12,12))\n\nplt.subplot(321)\nsns.heatmap(sample_df_d.loc[:,d_cols].corr(), cmap='Blues')\nplt.title('col_type : D')\n\nplt.subplot(322)\nsns.heatmap(sample_df_s.loc[:,s_cols[1:]].corr(), cmap='Blues')\nplt.title('col_type : S')\n\nplt.subplot(323)\nsns.heatmap(sample_df_p.loc[:,p_cols].corr(), cmap='Blues')\nplt.title('col_type : P')\n\nplt.subplot(324)\nsns.heatmap(sample_df_b.loc[:,b_cols].corr(), cmap='Blues')\nplt.title('col_type : B')\n\nplt.subplot(325)\nsns.heatmap(sample_df_r.loc[:,r_cols].corr(), cmap='Blues')\nplt.title('col_type : R')\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-03T07:47:36.238462Z","iopub.execute_input":"2022-06-03T07:47:36.238847Z","iopub.status.idle":"2022-06-03T07:47:39.149590Z","shell.execute_reply.started":"2022-06-03T07:47:36.238817Z","shell.execute_reply":"2022-06-03T07:47:39.148615Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Considering the amount of variables, it is fortunate that there are not many variables that are related to each other. However, there are several strongly correlated variables. So,Keep in mind that these variables will need to be removed when training the machine learning model later(not this chapter).","metadata":{}},{"cell_type":"markdown","source":"## Step 3. Check Outliers","metadata":{}},{"cell_type":"markdown","source":"Since it is a matter of matching whether the target is 0 or 1, it is necessary to check the proportion of appearance of 1 for each date. In case of excessive or very rare occurrence, the date can be treated as an outlier, and this outlier may be helpful for our analysis.","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\n\nplt.figure(figsize=(12,4))\ndaily_cnt_df = pd.DataFrame(sample_df_d.resample('D')['target'].value_counts())\ndaily_cnt_df.columns = ['count']\ndaily_cnt_df.reset_index(inplace=True)\n\nplt.subplot(211)\ndata_0 = daily_cnt_df[daily_cnt_df['target']==0]\nsns.lineplot(data=data_0, x='S_2', y='count', linewidth=0.8, alpha=0.8, label='target:0')\ndata_1 = daily_cnt_df[daily_cnt_df['target']==1]\nsns.lineplot(data=data_1, x='S_2', y='count', linewidth=0.8, alpha=0.8, label='target:1')\nplt.title('count of targets')\n\nplt.subplot(212)\ndata_ratio_1 = data_1.set_index('S_2')['count'] / (data_1.set_index('S_2')['count'] + data_0.set_index('S_2')['count'])\nsns.lineplot(data=data_ratio_1, linewidth=0.8, alpha=0.8, color='g')\nplt.axhspan(ymin=data_ratio_1.mean()-3*data_ratio_1.std(), ymax=data_ratio_1.mean()+3*data_ratio_1.std(), alpha=0.25, color='pink')\nv_idx = data_ratio_1[(data_ratio_1 > data_ratio_1.mean()+3*data_ratio_1.std()) | (data_ratio_1 < data_ratio_1.mean()-3*data_ratio_1.std())]\nfor i in v_idx.index:\n    plt.axvline(x=i, color='r', linestyle='--', alpha=0.7, linewidth=0.95)\nplt.ylabel('ratio')\nplt.title('ratio of target(1)')\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-03T11:21:44.127890Z","iopub.execute_input":"2022-06-03T11:21:44.128237Z","iopub.status.idle":"2022-06-03T11:21:45.350184Z","shell.execute_reply.started":"2022-06-03T11:21:44.128209Z","shell.execute_reply":"2022-06-03T11:21:45.349243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The red area on the `ratio of target(1)` graph indicates values that fall within 3 standard deviations. + and - standard deviations are 99% or more of the section that contains most of the data, and values protruding outside the corresponding range can be judged as outliers.","metadata":{}},{"cell_type":"code","source":"v_idx # outliers","metadata":{"execution":{"iopub.status.busy":"2022-06-03T11:29:31.587193Z","iopub.execute_input":"2022-06-03T11:29:31.587981Z","iopub.status.idle":"2022-06-03T11:29:31.594714Z","shell.execute_reply.started":"2022-06-03T11:29:31.587943Z","shell.execute_reply":"2022-06-03T11:29:31.593967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Step 4. PCA (Focus on 'D' type)","metadata":{}},{"cell_type":"markdown","source":"In this tutorial, we will analyze 'D' type columns as an example among the five data types divided above.\n\nAnd Among the categorical variables specified in the guide, the variables corresponding to type D are as follows.","metadata":{}},{"cell_type":"code","source":"cat_cols = ['D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']","metadata":{"execution":{"iopub.status.busy":"2022-06-03T11:33:10.266709Z","iopub.execute_input":"2022-06-03T11:33:10.267081Z","iopub.status.idle":"2022-06-03T11:33:10.272146Z","shell.execute_reply.started":"2022-06-03T11:33:10.267052Z","shell.execute_reply":"2022-06-03T11:33:10.271082Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There should be no missing values for PCA. However, since the loss of information is enormous when all missing values are removed, we will fill in the missing values with the median of the average values of each D-type data.","metadata":{}},{"cell_type":"code","source":"from sklearn.decomposition import PCA\n\npca = PCA(n_components=2)\ndecomposed_data = pca.fit_transform(sample_df_d.drop(cat_cols,axis=1).iloc[:,2:].fillna(sample_df_d[d_cols].mean().median()))\ndecomposed_df = pd.DataFrame(data=decomposed_data, columns=['x1','y1'])\ndecomposed_df","metadata":{"execution":{"iopub.status.busy":"2022-06-03T11:35:39.988435Z","iopub.execute_input":"2022-06-03T11:35:39.988788Z","iopub.status.idle":"2022-06-03T11:35:40.408211Z","shell.execute_reply.started":"2022-06-03T11:35:39.988761Z","shell.execute_reply":"2022-06-03T11:35:40.406975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now all D-type data has been reduced to two-dimensional data. Let's add target information here.","metadata":{}},{"cell_type":"code","source":"pd.concat([decomposed_df, sample_df_d['target'].reset_index(drop=True)], axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-06-03T11:36:40.933413Z","iopub.execute_input":"2022-06-03T11:36:40.933817Z","iopub.status.idle":"2022-06-03T11:36:40.949462Z","shell.execute_reply.started":"2022-06-03T11:36:40.933782Z","shell.execute_reply":"2022-06-03T11:36:40.948554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"When this data is spread out on a two-dimensional plane, it looks like the following.","metadata":{}},{"cell_type":"code","source":"decomposed_2d = pd.concat([decomposed_df, sample_df_d['target'].reset_index(drop=True)], axis=1)\nsns.scatterplot(data=decomposed_2d, x='x1', y='y1', hue='target') \nplt.title('decomposed 2d')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-03T09:32:46.000593Z","iopub.execute_input":"2022-06-03T09:32:46.001029Z","iopub.status.idle":"2022-06-03T09:32:46.521286Z","shell.execute_reply.started":"2022-06-03T09:32:46.000993Z","shell.execute_reply":"2022-06-03T09:32:46.520176Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It seems that the x-axis values are reduced to discrete. It may be a problem because the existing data values have different scales for each column. Therefore, let's standardize the data and then proceed with dimensionality reduction again.","metadata":{}},{"cell_type":"code","source":"num_d = sample_df_d.drop(cat_cols,axis=1).iloc[:,2:]\nscaled_d = (num_d - num_d.mean()) / num_d.std()\nscaled_d.describe()","metadata":{"execution":{"iopub.status.busy":"2022-06-03T11:40:35.851484Z","iopub.execute_input":"2022-06-03T11:40:35.851890Z","iopub.status.idle":"2022-06-03T11:40:36.105744Z","shell.execute_reply.started":"2022-06-03T11:40:35.851860Z","shell.execute_reply":"2022-06-03T11:40:36.104708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pca = PCA(n_components=2)\ndecomposed_scaled_data = pca.fit_transform(scaled_d.fillna(scaled_d.mean().median()))\ndecomposed_scaled_df = pd.DataFrame(data=decomposed_scaled_data, columns=['x1','y1'])\ndecomposed_scaled_df","metadata":{"execution":{"iopub.status.busy":"2022-06-03T11:40:38.799941Z","iopub.execute_input":"2022-06-03T11:40:38.800356Z","iopub.status.idle":"2022-06-03T11:40:38.978015Z","shell.execute_reply.started":"2022-06-03T11:40:38.800319Z","shell.execute_reply":"2022-06-03T11:40:38.976871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"decomposed_scaled_2d = pd.concat([decomposed_scaled_df, \n                                  sample_df_d['target'].reset_index(drop=True)], axis=1)\nsns.scatterplot(data=decomposed_scaled_2d, x='x1', y='y1', hue='target') \nplt.title('decomposed_scaled_2d')\n\nplt.grid(alpha=0.4)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-03T11:40:41.502594Z","iopub.execute_input":"2022-06-03T11:40:41.502983Z","iopub.status.idle":"2022-06-03T11:40:41.962146Z","shell.execute_reply.started":"2022-06-03T11:40:41.502951Z","shell.execute_reply":"2022-06-03T11:40:41.961059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now it is well expressed as a continuous variable. It can be seen that target-0 and target-1 can be distinguished to some extend only with D type variables.\n\nIn addition, let's check where the outlier(date) extracted earlier is located.","metadata":{}},{"cell_type":"code","source":"decomposed_scaled_2d = pd.concat([decomposed_scaled_df, \n                                  sample_df_d['target'].reset_index(drop=True), \n                                  sample_df_d.reset_index()['S_2'].isin(v_idx.index)], axis=1)\nsns.scatterplot(data=decomposed_scaled_2d, x='x1', y='y1', color='grey', alpha=0.2)\nsns.scatterplot(data=decomposed_scaled_2d[decomposed_scaled_2d['S_2']==True], x='x1', y='y1', hue='target')\nplt.xlim((decomposed_scaled_2d['x1'].min(), decomposed_scaled_2d['x1'].max()))\nplt.ylim((decomposed_scaled_2d['y1'].min(), decomposed_scaled_2d['y1'].max()))\nplt.title('decomposed_scaled_2d')\n\nplt.grid(alpha=0.4)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-03T09:46:57.109093Z","iopub.execute_input":"2022-06-03T09:46:57.109525Z","iopub.status.idle":"2022-06-03T09:46:57.545212Z","shell.execute_reply.started":"2022-06-03T09:46:57.109491Z","shell.execute_reply":"2022-06-03T09:46:57.544255Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The points where the outliers exist are irregular and are not clustered in a specific location. This means that it is difficult to confirm information about outliers with at leat D-type data.\n\nSo. Chapter 1 of this tutorial is finished. By analyzing S-type, P-type, etc. in this way, we can determine which type of data best distributes the target category, and through it, we can select the variables we want to focus on little by little. In this tutorial, we only checked type D variables(column), but I hope you will actively check other type variabels in this way and share your insights in the comments.\n\nThanks for joining the tutorial!","metadata":{}}]}