{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 1 | Introduction\n\n\nCredit default risk modeling is a very crucial and important in the domain of BFSI. In this AMEX provided dataset we have very large data which poses computing and processing hurdles .\n \n In this notebook we shall try to explore the various features and conduct preliminary EDA to make way for model building! Apart from this we shall be exploring Feature Partitioning in such high dimensional dataset and building ensemble models for each partition!\n \n Let us dive in !\n \n","metadata":{}},{"cell_type":"markdown","source":"
\n
1.1 | Load Libraries
\n
","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport os\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport gc\n\npd.options.display.max_rows = 200","metadata":{"execution":{"iopub.status.busy":"2022-06-04T04:19:28.024453Z","iopub.execute_input":"2022-06-04T04:19:28.024954Z","iopub.status.idle":"2022-06-04T04:19:28.030504Z","shell.execute_reply.started":"2022-06-04T04:19:28.024916Z","shell.execute_reply":"2022-06-04T04:19:28.029831Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2 | Load Large Datasets\n\nWe can use one of the following strategies to load large dataset:\n1. Reduce the default dtype space when reading by downcasting columns\n2. Use Feather dataset\n3. Use Dask Distributed Computing \n4. Read in chunks(but analysis could not be done at once for all)\n\nHere am using the feather datset [here](https://www.kaggle.com/datasets/seefun/amex-default-prediction-feather). This dataset is of only 4GB and excludes target label.\n\n\n","metadata":{}},{"cell_type":"markdown","source":"
\n\n**Size of Train** : (5531451, 190) ~5.5 million rows\n\n**Size of Test** : (5531451, 190) ~5.5 million rows\n\n**Number of Unique Customers** : 458913 \n\nObservations from below:\n* Each customer's credit card statement is present for 13 months in majority of cases\n* All the credit card ","metadata":{}},{"cell_type":"code","source":"train_feather.dtypes","metadata":{"execution":{"iopub.status.busy":"2022-06-04T04:19:33.148033Z","iopub.execute_input":"2022-06-04T04:19:33.148409Z","iopub.status.idle":"2022-06-04T04:19:33.165122Z","shell.execute_reply.started":"2022-06-04T04:19:33.148378Z","shell.execute_reply":"2022-06-04T04:19:33.163915Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n
3.2 | Missing Value Features
\n
\n\nWe can observe many features having more than 50% of missing values. So removing those features which is having more than 50% of NAN and common in both train and test set.","metadata":{}},{"cell_type":"code","source":"gc.collect()\nTRAIN_SHAPE = train_feather.shape\nmissing_df = train_feather.isnull().sum().sort_values(ascending=False)\nmissing_percent = missing_df.apply(lambda x:round(x/TRAIN_SHAPE[0],2))\n#clean up\ndel missing_df\ngc.collect()\nmissing_percent","metadata":{"execution":{"iopub.status.busy":"2022-06-04T04:19:33.167792Z","iopub.execute_input":"2022-06-04T04:19:33.168897Z","iopub.status.idle":"2022-06-04T04:19:39.020440Z","shell.execute_reply.started":"2022-06-04T04:19:33.168849Z","shell.execute_reply":"2022-06-04T04:19:39.019750Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gc.collect()\ntest_feather = pd.read_feather(\"../input/amex-default-prediction-feather/test.feather\")\nTEST_SHAPE = test_feather.shape\n","metadata":{"execution":{"iopub.status.busy":"2022-06-04T04:19:39.021483Z","iopub.execute_input":"2022-06-04T04:19:39.022011Z","iopub.status.idle":"2022-06-04T04:20:11.764476Z","shell.execute_reply.started":"2022-06-04T04:19:39.021977Z","shell.execute_reply":"2022-06-04T04:20:11.763411Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"missing_df_test = test_feather.isnull().sum().sort_values(ascending=False)\nmissing_percent_test = missing_df_test.apply(lambda x:round(x/TEST_SHAPE[0],2))\n#clean up\ndel missing_df_test \ndel test_feather\ngc.collect()\n\nmissing_percent_test","metadata":{"execution":{"iopub.status.busy":"2022-06-04T04:41:46.087947Z","iopub.execute_input":"2022-06-04T04:41:46.088377Z","iopub.status.idle":"2022-06-04T04:41:59.238935Z","shell.execute_reply.started":"2022-06-04T04:41:46.088339Z","shell.execute_reply":"2022-06-04T04:41:59.237960Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_missing = set(missing_percent.index[missing_percent>0.5])\ntest_missing = set(missing_percent_test.index[missing_percent_test>0.5])\n#finding features which is 50% missing in both train & test set\ncommon_miss_feature = train_missing.intersection(test_missing) \nprint(\"The 50% missing feautres in train and test set : \\n\",common_miss_feature)\n\n#clean up\n","metadata":{"execution":{"iopub.status.busy":"2022-06-04T04:40:42.113587Z","iopub.execute_input":"2022-06-04T04:40:42.114052Z","iopub.status.idle":"2022-06-04T04:40:42.120265Z","shell.execute_reply.started":"2022-06-04T04:40:42.114017Z","shell.execute_reply":"2022-06-04T04:40:42.119456Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n
3.3 | Basic EDA
\n
\n\n## Categorical Variables\n\n['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']\n\n## Binary Variables\n\n* B_31 is always 0 or 1.\n* D_87 is missing.\n\n## Count of 5 Feature Type Variables\nHere we count the number of features in each of the 5 types of attributes Delinquency , Spend, Payment, Balance, Risk","metadata":{}},{"cell_type":"code","source":"train_features = train_feather.columns\n\nprint(\"Count Delinquency Features :\",sum(train_features.str.startswith('D')))\nprint(\"Count Spend Features :\",sum(train_features.str.startswith('S')))\nprint(\"Count Payment Features :\",sum(train_features.str.startswith('P')))\nprint(\"Count Balance Features :\",sum(train_features.str.startswith('B')))\nprint(\"Count Risk Features :\",sum(train_features.str.startswith('R')))","metadata":{"execution":{"iopub.status.busy":"2022-06-04T04:40:19.229193Z","iopub.execute_input":"2022-06-04T04:40:19.229633Z","iopub.status.idle":"2022-06-04T04:40:19.241151Z","shell.execute_reply.started":"2022-06-04T04:40:19.229598Z","shell.execute_reply":"2022-06-04T04:40:19.239946Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Customer Statements (S_2 feature)\n\nLet us count the number of credit card statements present for each unique customer ID . The `S_2` feature is the date of receiving statements.\n\n**Observation**\n* We can see from the plot that majority of customers have received 13 credit card statements .\n* And if we inspect the `last` row in each group by customer_ID then last statement for customer were all in month of March 2018.","metadata":{}},{"cell_type":"code","source":"gc.collect()\ncutomer_statements = (train_feather['customer_ID'].value_counts()).value_counts()\ncutomer_statements","metadata":{"execution":{"iopub.status.busy":"2022-06-04T04:20:25.355010Z","iopub.execute_input":"2022-06-04T04:20:25.355609Z","iopub.status.idle":"2022-06-04T04:20:26.041289Z","shell.execute_reply.started":"2022-06-04T04:20:25.355559Z","shell.execute_reply":"2022-06-04T04:20:26.040208Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig,ax = plt.subplots(1,1,figsize=(15,5))\nsns.barplot(x=cutomer_statements.index,y=cutomer_statements,palette='Reds').set_xlabel('No. of credit card statements for each customer')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-04T04:20:26.043612Z","iopub.execute_input":"2022-06-04T04:20:26.043953Z","iopub.status.idle":"2022-06-04T04:20:26.334112Z","shell.execute_reply.started":"2022-06-04T04:20:26.043926Z","shell.execute_reply":"2022-06-04T04:20:26.332974Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#clean up \ndel cutomer_statements\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-04T04:20:26.335525Z","iopub.execute_input":"2022-06-04T04:20:26.335935Z","iopub.status.idle":"2022-06-04T04:20:26.458947Z","shell.execute_reply.started":"2022-06-04T04:20:26.335903Z","shell.execute_reply":"2022-06-04T04:20:26.457951Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4 | Feature Engineering and Transformation\n\n
\n
4.1 | Custom Feature Engg Function
\n
\nThe below is a custom function to transform and engineer features. It does the following:\n1. Drop commong missing columns\n2. Group data by `customer_ID` \n3. Agrregate the groups : \n * For Numerical: Calculate the mean,std,max,min, last(this will be the last statement received by the customers).\n * For Categorical: Calculate the nsmallest, nlargest ,count, nunique last(this will be the last statement received by the customers).\n4. Number of missing features in each row.","metadata":{}},{"cell_type":"code","source":"gc.collect()\n\n#Custom function for feature engg.\n\ndef feature_engg(data, missing_f, num_f, cat_f):\n #drop\n data = data.drop(columns=missing_f,axis=1)\n \n #group num\n data_num_agg = data.groupby('customer_ID')[num_f].agg(['mean','std','min','max','last'])\n #join each column which is a tuple for each feature\n data_num_agg.columns = ['_'.join(c) for c in data_num_agg.columns]\n \n #group cat\n data_cat_agg = data.groupby('customer_ID')[cat_f].agg(['count','nunique','nlargest','nsmallest','last'])\n data_cat_agg.columns = ['_'.join(c) for c in data_cat_agg.columns]\n \n del data\n gc.collect()\n \n #concat\n df_out = pd.concat([data_num_agg,data_cat_agg],axis=1)\n del data_cat_agg, data_num_agg\n gc.collect()\n \n #count of missing features per row\n df_out['na_count'] = df_out.isna().sum(axis=1)\n \n print('Shape of data after feature engineering',df_out.shape)\n \n return df_out\n \n ","metadata":{"execution":{"iopub.status.busy":"2022-06-04T04:39:25.822374Z","iopub.execute_input":"2022-06-04T04:39:25.822897Z","iopub.status.idle":"2022-06-04T04:39:25.962744Z","shell.execute_reply.started":"2022-06-04T04:39:25.822862Z","shell.execute_reply":"2022-06-04T04:39:25.961449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"execution":{"iopub.status.busy":"2022-06-04T05:10:09.726271Z","iopub.execute_input":"2022-06-04T05:10:09.726835Z","iopub.status.idle":"2022-06-04T05:10:10.681546Z","shell.execute_reply.started":"2022-06-04T05:10:09.726793Z","shell.execute_reply":"2022-06-04T05:10:10.680362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n#Define features\nmissing_f = list( {'D_53', 'R_26', 'D_50', 'D_136', 'D_88', 'D_134', 'D_137', 'D_111', 'D_132', 'D_73', 'B_17', 'D_87', 'D_142', 'B_42', 'D_138', 'B_29', 'D_110', 'D_106', 'D_135', 'D_108', 'R_9', 'D_76', 'D_49', 'D_105', 'B_39', 'D_42', 'D_82'})\ncat_f = ['B_30','B_31','B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']\nexclude_f = ['customer_ID']+missing_f+cat_f \nnum_f= [i for i in train_feather.columns if i not in exclude_f]\nprint('Number of missing features:', missing_f.__len__())\nprint('Number of cat features:', cat_f.__len__())\nprint('Number of num features:', num_f.__len__())\n\n#call function\ntrain_trans = feature_engg(train_feather, missing_f, num_f, cat_f)","metadata":{"execution":{"iopub.status.busy":"2022-06-04T05:05:58.953505Z","iopub.execute_input":"2022-06-04T05:05:58.954436Z","iopub.status.idle":"2022-06-04T05:05:58.975846Z","shell.execute_reply.started":"2022-06-04T05:05:58.954395Z","shell.execute_reply":"2022-06-04T05:05:58.974087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_trans","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_trans.to_pickle('./train_train.pkl')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"