{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 1 Introduction\nKaggle describes this competition as [follows](https://www.kaggle.com/competitions/amex-default-prediction/data):\n\nThe objective of this competition is to predict the **probability** that a customer does not pay back their credit card balance amount in the future based on their monthly customer profile. The target binary variable is calculated by observing 18 months performance window after the latest credit card statement, and if the customer does not pay due amount in 120 days after their latest statement date it is considered a default event.","metadata":{}},{"cell_type":"markdown","source":"# 2 Loading and Exploring Data","metadata":{}},{"cell_type":"markdown","source":"# 2.1 Loading the libraries and data","metadata":{}},{"cell_type":"markdown","source":"Loading Python packages","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom category_encoders import TargetEncoder","metadata":{"execution":{"iopub.status.busy":"2022-08-07T11:46:01.597400Z","iopub.execute_input":"2022-08-07T11:46:01.598217Z","iopub.status.idle":"2022-08-07T11:46:02.720672Z","shell.execute_reply.started":"2022-08-07T11:46:01.598089Z","shell.execute_reply":"2022-08-07T11:46:02.718999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Loading train (train_x and train_y) dataset from csv files.","metadata":{}},{"cell_type":"code","source":"train_x= pd.read_parquet(\"../input/amex-data-integer-dtypes-parquet-format/train.parquet\")\ntrain_y= pd.read_csv(\"../input/amex-default-prediction/train_labels.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-08-07T11:46:02.722900Z","iopub.execute_input":"2022-08-07T11:46:02.723360Z","iopub.status.idle":"2022-08-07T11:46:23.236362Z","shell.execute_reply.started":"2022-08-07T11:46:02.723322Z","shell.execute_reply":"2022-08-07T11:46:23.234845Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2.2 Data size and structure","metadata":{}},{"cell_type":"markdown","source":"From the shape of train_x and train_y we could conclude that train_x contain multiple statments for a single customer and train_y is the prediction that whether a particular customer with a unique customer ID is a defaulter or not.","metadata":{}},{"cell_type":"code","source":"train_x.shape","metadata":{"execution":{"iopub.status.busy":"2022-08-07T11:46:23.237997Z","iopub.execute_input":"2022-08-07T11:46:23.238376Z","iopub.status.idle":"2022-08-07T11:46:23.253772Z","shell.execute_reply.started":"2022-08-07T11:46:23.238342Z","shell.execute_reply":"2022-08-07T11:46:23.247691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_y.shape","metadata":{"execution":{"iopub.status.busy":"2022-08-07T11:46:23.259595Z","iopub.execute_input":"2022-08-07T11:46:23.260453Z","iopub.status.idle":"2022-08-07T11:46:23.271718Z","shell.execute_reply.started":"2022-08-07T11:46:23.260393Z","shell.execute_reply":"2022-08-07T11:46:23.270380Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Below is a glimps of the training data and training lables","metadata":{}},{"cell_type":"code","source":"pd.set_option(\"display.max_columns\",1000)\npd.set_option(\"display.max_rows\",1000)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T11:46:23.273435Z","iopub.execute_input":"2022-08-07T11:46:23.273909Z","iopub.status.idle":"2022-08-07T11:46:23.283380Z","shell.execute_reply.started":"2022-08-07T11:46:23.273866Z","shell.execute_reply":"2022-08-07T11:46:23.282085Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_x.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T11:46:23.287240Z","iopub.execute_input":"2022-08-07T11:46:23.287582Z","iopub.status.idle":"2022-08-07T11:46:23.435268Z","shell.execute_reply.started":"2022-08-07T11:46:23.287538Z","shell.execute_reply":"2022-08-07T11:46:23.434054Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_y.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T11:46:23.438953Z","iopub.execute_input":"2022-08-07T11:46:23.439401Z","iopub.status.idle":"2022-08-07T11:46:23.450440Z","shell.execute_reply.started":"2022-08-07T11:46:23.439368Z","shell.execute_reply":"2022-08-07T11:46:23.449259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Thus there are 189 features(without customer_ID) out of which ['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68'] are categorical features and 1 binary target.","metadata":{}},{"cell_type":"markdown","source":"# 3 Exploring some of the most important variables","metadata":{}},{"cell_type":"markdown","source":"# 3.1 Response variable - target","metadata":{}},{"cell_type":"markdown","source":"Note that the negative class has been subsampled for this dataset at 5%, and thus receives a 20x weighting in the scoring metric. Thus from the plot we can conclude that in real life the number of defaulter is significantly less.\n\ndefaulter : not_defaulter =1:3.5 (Approx)\n\nnot_defaulter is subsampled at 5 % so propotional number of not_defaulter =3.5 * 20 = 70.\n\nThus the new ratio should be equal to 1:70.","metadata":{}},{"cell_type":"code","source":"sns.catplot(x=\"target\",kind=\"count\",data=train_y)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T11:46:23.451803Z","iopub.execute_input":"2022-08-07T11:46:23.452635Z","iopub.status.idle":"2022-08-07T11:46:23.908028Z","shell.execute_reply.started":"2022-08-07T11:46:23.452578Z","shell.execute_reply":"2022-08-07T11:46:23.906731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3.2 Most Important Numeric predictor","metadata":{}},{"cell_type":"markdown","source":"Here I try to find which numeric variables have high correlation with target variable. This helps us to find the important variables. There are a total of 177 numerical features.","metadata":{}},{"cell_type":"markdown","source":"# 3.2.1 Correlation with target","metadata":{}},{"cell_type":"code","source":"numericalFeatures=[]\ncategoricalFeatures=['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']\n\ntrain_x.dtypes\nfor x in train_x.columns:\n    if x != \"customer_ID\" and x!= \"S_2\" and x not in categoricalFeatures:\n        numericalFeatures.append(x)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T11:46:23.911944Z","iopub.execute_input":"2022-08-07T11:46:23.912320Z","iopub.status.idle":"2022-08-07T11:46:23.920356Z","shell.execute_reply.started":"2022-08-07T11:46:23.912288Z","shell.execute_reply":"2022-08-07T11:46:23.918710Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(numericalFeatures)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T11:46:23.926628Z","iopub.execute_input":"2022-08-07T11:46:23.927386Z","iopub.status.idle":"2022-08-07T11:46:23.935009Z","shell.execute_reply.started":"2022-08-07T11:46:23.927333Z","shell.execute_reply":"2022-08-07T11:46:23.934025Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We cannot yet find correlation between train lables and train features because they have different number of rows. So we must apply some join function to match the number of rows and join two tables.","metadata":{}},{"cell_type":"code","source":"train=pd.merge(left=train_y,right=train_x,on=\"customer_ID\",how=\"outer\")","metadata":{"execution":{"iopub.status.busy":"2022-08-07T11:46:23.936156Z","iopub.execute_input":"2022-08-07T11:46:23.936527Z","iopub.status.idle":"2022-08-07T11:49:05.791228Z","shell.execute_reply.started":"2022-08-07T11:46:23.936494Z","shell.execute_reply":"2022-08-07T11:49:05.789206Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T11:49:05.793570Z","iopub.execute_input":"2022-08-07T11:49:05.794127Z","iopub.status.idle":"2022-08-07T11:49:05.923595Z","shell.execute_reply.started":"2022-08-07T11:49:05.794072Z","shell.execute_reply":"2022-08-07T11:49:05.922213Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.shape","metadata":{"execution":{"iopub.status.busy":"2022-08-07T11:49:05.925150Z","iopub.execute_input":"2022-08-07T11:49:05.925491Z","iopub.status.idle":"2022-08-07T11:49:05.934311Z","shell.execute_reply.started":"2022-08-07T11:49:05.925459Z","shell.execute_reply":"2022-08-07T11:49:05.932666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del train_x\ndel train_y","metadata":{"execution":{"iopub.status.busy":"2022-08-07T11:49:05.936315Z","iopub.execute_input":"2022-08-07T11:49:05.936730Z","iopub.status.idle":"2022-08-07T11:49:05.945337Z","shell.execute_reply.started":"2022-08-07T11:49:05.936694Z","shell.execute_reply":"2022-08-07T11:49:05.944090Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we have only one table and the number of rows in predictor variable is equal to number of rows in feature variable.","metadata":{}},{"cell_type":"markdown","source":"Here we find the correlation between each columns. But, unfortunately we cannot vectorize it due to memory constrains as vectorizing it will require a lot of memory so we find correlation between two columns one at a time.","metadata":{}},{"cell_type":"code","source":"col=[]\ncol=numericalFeatures.copy()\ncol.append(\"target\")\n\ncoefficient=[]\nfor x in col:\n    temp=[]\n    for y in col:\n        temp.append(abs(train[x].corr(train[y])))\n    coefficient.append(temp)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T11:49:05.947301Z","iopub.execute_input":"2022-08-07T11:49:05.947795Z","iopub.status.idle":"2022-08-07T12:44:53.869929Z","shell.execute_reply.started":"2022-08-07T11:49:05.947748Z","shell.execute_reply":"2022-08-07T12:44:53.868085Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"coefficient=pd.DataFrame(coefficient)\ncoefficient.index=col\ncoefficient.columns=col\ncoefficient","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:44:53.872705Z","iopub.execute_input":"2022-08-07T12:44:53.873646Z","iopub.status.idle":"2022-08-07T12:44:54.924416Z","shell.execute_reply.started":"2022-08-07T12:44:53.873559Z","shell.execute_reply":"2022-08-07T12:44:54.921288Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"P_2 and D_48 have greater effect on target variable than other variables as they have a correlational coefficient greater than 0.5. So, now we will look into this variables closely.","metadata":{}},{"cell_type":"markdown","source":"B_2, B_7, B_9, B_18, B_33, D_44, D_48, D_55, D_74, D_75, have a high correlation with P_2 variable. P_2, B_2, B_3, B_4, B_7, B_9, B_16, B_18, D_44, D_55, D_58, D_61, have a high correlation with D_48 variable. Thus there is a multicollinearity issue.","metadata":{}},{"cell_type":"markdown","source":"# 3.3.2 P_2 variable","metadata":{}},{"cell_type":"markdown","source":"We found that P_2 (a payment variable) is dependent on some balance variables and some delinquency variables which is quite expected as a person having enough balance in his/her bank would tend to pay back his credit and a person who have failed to pay back his/her debt in time often tends to avoid paying back the credit.","metadata":{}},{"cell_type":"code","source":"plt.scatter(train[\"P_2\"],train[\"target\"])\nplt.xlabel(\"P_2\")\nplt.ylabel(\"Target\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:44:54.926229Z","iopub.execute_input":"2022-08-07T12:44:54.927010Z","iopub.status.idle":"2022-08-07T12:45:04.890375Z","shell.execute_reply.started":"2022-08-07T12:44:54.926960Z","shell.execute_reply":"2022-08-07T12:45:04.889408Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3.3.3 D_48 Variable","metadata":{}},{"cell_type":"markdown","source":"We found that delinquency variables is dependent on balance variable which is quite obvious and it also depend on paymennt variable and other deliquency variables.","metadata":{}},{"cell_type":"code","source":"plt.scatter(train[\"D_48\"],train[\"target\"])\nplt.xlabel(\"D_48\")\nplt.ylabel(\"Target\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:04.891994Z","iopub.execute_input":"2022-08-07T12:45:04.892665Z","iopub.status.idle":"2022-08-07T12:45:13.566047Z","shell.execute_reply.started":"2022-08-07T12:45:04.892627Z","shell.execute_reply":"2022-08-07T12:45:13.564634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The points right of 8 might be outliers.","metadata":{}},{"cell_type":"markdown","source":"# 4 Missing data, label encoding, and factorizing variables","metadata":{}},{"cell_type":"markdown","source":"# 4.1 Completeness of Data","metadata":{}},{"cell_type":"markdown","source":"First of all we will look for missing values.","metadata":{}},{"cell_type":"code","source":"train.isnull().sum().sort_values(ascending=False)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:13.568508Z","iopub.execute_input":"2022-08-07T12:45:13.569434Z","iopub.status.idle":"2022-08-07T12:45:16.836919Z","shell.execute_reply.started":"2022-08-07T12:45:13.569383Z","shell.execute_reply":"2022-08-07T12:45:16.835532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Drop D_87, D_88, D_110, D_111, B_39, D_73, B_42, D_134, B_29, D_132, D_76, D_42, D_142, D_53, D_50, B_17, D_105, D_56 as they have null values more than 50%.","metadata":{}},{"cell_type":"code","source":"dropCols=[\"D_87\", \"D_88\", \"D_110\", \"D_111\", \"B_39\", \"D_73\", \"B_42\", \"D_134\", \"B_29\", \"D_132\", \"D_76\", \"D_42\", \"D_142\", \"D_53\", \"D_50\", \"B_17\", \"D_105\", \"D_56\"]\n\nfor col in dropCols:\n    if coefficient[\"target\"][col]>=0.25:\n        dropCols.remove(col)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:16.840235Z","iopub.execute_input":"2022-08-07T12:45:16.840954Z","iopub.status.idle":"2022-08-07T12:45:16.848831Z","shell.execute_reply.started":"2022-08-07T12:45:16.840904Z","shell.execute_reply":"2022-08-07T12:45:16.847813Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"D_42 and B_17 have correlation >=0.25 with target variable. So we will not drop them.","metadata":{"execution":{"iopub.status.busy":"2022-07-25T04:09:17.320735Z","iopub.execute_input":"2022-07-25T04:09:17.321199Z","iopub.status.idle":"2022-07-25T04:09:17.329068Z","shell.execute_reply.started":"2022-07-25T04:09:17.321163Z","shell.execute_reply":"2022-07-25T04:09:17.328020Z"}}},{"cell_type":"code","source":"train.drop(labels=dropCols,axis=1,inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:16.850509Z","iopub.execute_input":"2022-08-07T12:45:16.851195Z","iopub.status.idle":"2022-08-07T12:45:23.056591Z","shell.execute_reply.started":"2022-08-07T12:45:16.851161Z","shell.execute_reply":"2022-08-07T12:45:23.055177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del coefficient\ndel dropCols","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:23.058500Z","iopub.execute_input":"2022-08-07T12:45:23.059252Z","iopub.status.idle":"2022-08-07T12:45:23.066473Z","shell.execute_reply.started":"2022-08-07T12:45:23.059194Z","shell.execute_reply":"2022-08-07T12:45:23.065196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We find that risk and spend variables effect the output least from correlation so we will handel them first.","metadata":{}},{"cell_type":"markdown","source":"* Spend variables to be handled - S_9, S_7, S_3, S_22, S_24, S_25, S_26, S_23\n\n* Risk variables to be handled - R_27, R_12, R_14, R_7\n\n* Balance variables to be handled - B_13, B_8, B_15, B_25, B_2, B_27, B_3, B_26, B_6, B_37, B_40, B_17\n\n* Delinquency variables to be handled - D_77, D_43, D_46, D_62, D_48 (important variable), D_61, D_69, D_55, D_118, D_121, D_115, D_119, D_131, D_130, D_104, D_141, D_128, D_133, D_144, D_102, D_52, D_112, D_45, D_54, D_41, D_42\n\n* Payment variables to be handled - P_3, P_2 (important variable)","metadata":{}},{"cell_type":"markdown","source":"# 4.2 Imputing missing values","metadata":{}},{"cell_type":"markdown","source":"# 4.2.1 Spend Variables","metadata":{}},{"cell_type":"markdown","source":"Here we look for correlation between the spend variables.","metadata":{}},{"cell_type":"code","source":"col=[\"S_9\", \"S_7\", \"S_3\", \"S_22\", \"S_24\", \"S_25\", \"S_26\", \"S_23\"]\n\ncoefficient_S=[]\nfor x in col:\n    temp=[]\n    for y in col:\n        temp.append(abs(train[x].corr(train[y])))\n    coefficient_S.append(temp)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:23.068121Z","iopub.execute_input":"2022-08-07T12:45:23.069466Z","iopub.status.idle":"2022-08-07T12:45:30.437360Z","shell.execute_reply.started":"2022-08-07T12:45:23.069388Z","shell.execute_reply":"2022-08-07T12:45:30.436088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"coefficient_S=pd.DataFrame(coefficient_S)\ncoefficient_S","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:30.438987Z","iopub.execute_input":"2022-08-07T12:45:30.439333Z","iopub.status.idle":"2022-08-07T12:45:30.455837Z","shell.execute_reply.started":"2022-08-07T12:45:30.439303Z","shell.execute_reply":"2022-08-07T12:45:30.454757Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"S_3 and S_7 are highly correlated.\nS_22 and S_24 are highly correlated.\n\nThus one of them can be dropped.","metadata":{}},{"cell_type":"code","source":"del coefficient_S","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:30.457582Z","iopub.execute_input":"2022-08-07T12:45:30.458451Z","iopub.status.idle":"2022-08-07T12:45:30.467199Z","shell.execute_reply.started":"2022-08-07T12:45:30.458400Z","shell.execute_reply":"2022-08-07T12:45:30.466233Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[(train[\"S_3\"].isnull() & ~(train[\"S_7\"].isnull())) | (~(train[\"S_3\"].isnull()) & train[\"S_7\"].isnull())]","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:30.469134Z","iopub.execute_input":"2022-08-07T12:45:30.469942Z","iopub.status.idle":"2022-08-07T12:45:30.540928Z","shell.execute_reply.started":"2022-08-07T12:45:30.469901Z","shell.execute_reply":"2022-08-07T12:45:30.539678Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There is not a single row there S_3 is null but S_7 is not null or S_3 is not null but S_7 is null. So, drop S_3 as S_3 has less correlation with target variable than S_7.","metadata":{}},{"cell_type":"code","source":"train[((~(train[\"S_22\"].isnull()) & train[\"S_24\"].isnull() | train[\"S_22\"].isnull() & ~(train[\"S_24\"].isnull())))][[\"S_22\",\"S_24\"]]","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:30.542392Z","iopub.execute_input":"2022-08-07T12:45:30.542757Z","iopub.status.idle":"2022-08-07T12:45:30.618114Z","shell.execute_reply.started":"2022-08-07T12:45:30.542723Z","shell.execute_reply":"2022-08-07T12:45:30.616989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So we should drop S_22.","metadata":{}},{"cell_type":"code","source":"train.drop(labels=[\"S_3\",\"S_22\"],axis=1,inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:30.626625Z","iopub.execute_input":"2022-08-07T12:45:30.627018Z","iopub.status.idle":"2022-08-07T12:45:32.010492Z","shell.execute_reply.started":"2022-08-07T12:45:30.626985Z","shell.execute_reply":"2022-08-07T12:45:32.009361Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"S_7\"].fillna(value=train[\"S_7\"].mean(),inplace=True)\ntrain[\"S_24\"].fillna(value=train[\"S_24\"].mean(),inplace=True)\ntrain[\"S_9\"].fillna(value=train[\"S_9\"].mean(),inplace=True)\ntrain[\"S_25\"].fillna(value=train[\"S_25\"].mean(),inplace=True)\ntrain[\"S_26\"].fillna(value=train[\"S_26\"].mean(),inplace=True)\ntrain[\"S_23\"].fillna(value=train[\"S_23\"].mean(),inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:32.011665Z","iopub.execute_input":"2022-08-07T12:45:32.012584Z","iopub.status.idle":"2022-08-07T12:45:32.223300Z","shell.execute_reply.started":"2022-08-07T12:45:32.012546Z","shell.execute_reply":"2022-08-07T12:45:32.221974Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Normalize S_8 columns","metadata":{}},{"cell_type":"code","source":"train[\"S_8\"]=train[\"S_8\"]/(train[\"S_8\"].max()-train[\"S_8\"].min())","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:32.224710Z","iopub.execute_input":"2022-08-07T12:45:32.225072Z","iopub.status.idle":"2022-08-07T12:45:32.317097Z","shell.execute_reply.started":"2022-08-07T12:45:32.225040Z","shell.execute_reply":"2022-08-07T12:45:32.315724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4.2.2 Risk Variables","metadata":{}},{"cell_type":"markdown","source":"Find the correlation between risk variables.","metadata":{}},{"cell_type":"code","source":"col=[\"R_27\", \"R_12\", \"R_14\", \"R_7\"]\n\ncoefficient_R=[]\nfor x in col:\n    temp=[]\n    for y in col:\n        temp.append(abs(train[x].corr(train[y])))\n    coefficient_R.append(temp)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:32.319857Z","iopub.execute_input":"2022-08-07T12:45:32.320250Z","iopub.status.idle":"2022-08-07T12:45:34.482170Z","shell.execute_reply.started":"2022-08-07T12:45:32.320215Z","shell.execute_reply":"2022-08-07T12:45:34.480971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"coefficient_R=pd.DataFrame(coefficient_R)\ncoefficient_R","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:34.483502Z","iopub.execute_input":"2022-08-07T12:45:34.483868Z","iopub.status.idle":"2022-08-07T12:45:34.496740Z","shell.execute_reply.started":"2022-08-07T12:45:34.483836Z","shell.execute_reply":"2022-08-07T12:45:34.495626Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del coefficient_R","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:34.497888Z","iopub.execute_input":"2022-08-07T12:45:34.498626Z","iopub.status.idle":"2022-08-07T12:45:34.509391Z","shell.execute_reply.started":"2022-08-07T12:45:34.498570Z","shell.execute_reply":"2022-08-07T12:45:34.508325Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The problem of multicolinearity does not exist in risk variables.","metadata":{}},{"cell_type":"code","source":"train[\"R_7\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:34.511172Z","iopub.execute_input":"2022-08-07T12:45:34.512026Z","iopub.status.idle":"2022-08-07T12:45:34.656176Z","shell.execute_reply.started":"2022-08-07T12:45:34.511964Z","shell.execute_reply":"2022-08-07T12:45:34.654745Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"R_7 has maximum values as 0. So we need to scale the larger values down.","metadata":{}},{"cell_type":"code","source":"train[\"R_7\"]=train[\"R_7\"]/train[\"R_7\"].max()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:34.657908Z","iopub.execute_input":"2022-08-07T12:45:34.658452Z","iopub.status.idle":"2022-08-07T12:45:34.701133Z","shell.execute_reply.started":"2022-08-07T12:45:34.658396Z","shell.execute_reply":"2022-08-07T12:45:34.699721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"R_12\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:34.703714Z","iopub.execute_input":"2022-08-07T12:45:34.704198Z","iopub.status.idle":"2022-08-07T12:45:34.900484Z","shell.execute_reply.started":"2022-08-07T12:45:34.704163Z","shell.execute_reply":"2022-08-07T12:45:34.899232Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"No need to scale or standardize R_12","metadata":{}},{"cell_type":"code","source":"train[\"R_14\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:34.901768Z","iopub.execute_input":"2022-08-07T12:45:34.902108Z","iopub.status.idle":"2022-08-07T12:45:35.024558Z","shell.execute_reply.started":"2022-08-07T12:45:34.902078Z","shell.execute_reply":"2022-08-07T12:45:35.023045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"R_14 has maximum values as 0. So we need to scale the larger values down.","metadata":{}},{"cell_type":"code","source":"train[\"R_14\"]=train[\"R_14\"]/train[\"R_14\"].max()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:35.026294Z","iopub.execute_input":"2022-08-07T12:45:35.026736Z","iopub.status.idle":"2022-08-07T12:45:35.067379Z","shell.execute_reply.started":"2022-08-07T12:45:35.026700Z","shell.execute_reply":"2022-08-07T12:45:35.065347Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we need to fill the null values.","metadata":{"execution":{"iopub.status.busy":"2022-07-26T04:53:01.255213Z","iopub.execute_input":"2022-07-26T04:53:01.255725Z","iopub.status.idle":"2022-07-26T04:53:01.468515Z","shell.execute_reply.started":"2022-07-26T04:53:01.255700Z","shell.execute_reply":"2022-07-26T04:53:01.467302Z"}}},{"cell_type":"code","source":"train[\"R_7\"].fillna(value=train[\"R_7\"].median(),inplace=True)\ntrain[\"R_7\"].fillna(value=train[\"R_12\"].mean(),inplace=True)\ntrain[\"R_7\"].fillna(value=train[\"R_14\"].median(),inplace=True)\ntrain[\"R_7\"].fillna(value=train[\"R_27\"].mean(),inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:35.070804Z","iopub.execute_input":"2022-08-07T12:45:35.071418Z","iopub.status.idle":"2022-08-07T12:45:35.336132Z","shell.execute_reply.started":"2022-08-07T12:45:35.071360Z","shell.execute_reply":"2022-08-07T12:45:35.334521Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4.2.3 Balance Variables","metadata":{}},{"cell_type":"markdown","source":"Find correlation between balance variables.","metadata":{}},{"cell_type":"code","source":"col=[\"B_13\", \"B_8\", \"B_15\", \"B_25\", \"B_2\", \"B_27\", \"B_3\", \"B_26\", \"B_6\", \"B_37\", \"B_40\", \"B_17\"]\n\ncoefficient_B=[]\nfor x in col:\n    temp=[]\n    for y in col:\n        temp.append(abs(train[x].corr(train[y])))\n    coefficient_B.append(temp)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:35.338133Z","iopub.execute_input":"2022-08-07T12:45:35.338783Z","iopub.status.idle":"2022-08-07T12:45:55.118286Z","shell.execute_reply.started":"2022-08-07T12:45:35.338721Z","shell.execute_reply":"2022-08-07T12:45:55.116926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"coefficient_B=pd.DataFrame(coefficient_B)\ncoefficient_B","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:55.120222Z","iopub.execute_input":"2022-08-07T12:45:55.121282Z","iopub.status.idle":"2022-08-07T12:45:55.145971Z","shell.execute_reply.started":"2022-08-07T12:45:55.121240Z","shell.execute_reply":"2022-08-07T12:45:55.144646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* B_13 and B_15 \n* B_25 and B_37\n* B_2 and B_3 and B_37\n* B_2 and B_17\nare correlated.","metadata":{}},{"cell_type":"code","source":"del coefficient_B","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:55.147911Z","iopub.execute_input":"2022-08-07T12:45:55.151165Z","iopub.status.idle":"2022-08-07T12:45:55.156880Z","shell.execute_reply.started":"2022-08-07T12:45:55.151112Z","shell.execute_reply":"2022-08-07T12:45:55.155988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[(train[\"B_25\"].isnull() | train[\"B_37\"].isnull() | train[\"B_2\"].isnull() | train[\"B_3\"].isnull() | train[\"B_17\"].isnull()) & ~(train[\"B_25\"].isnull() & train[\"B_37\"].isnull() & train[\"B_2\"].isnull() & train[\"B_3\"].isnull() & train[\"B_17\"].isnull())][[\"B_25\",\"B_37\",\"B_2\",\"B_3\",\"B_17\"]]","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:55.158132Z","iopub.execute_input":"2022-08-07T12:45:55.158640Z","iopub.status.idle":"2022-08-07T12:45:57.517931Z","shell.execute_reply.started":"2022-08-07T12:45:55.158595Z","shell.execute_reply":"2022-08-07T12:45:57.516542Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"B_37\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:57.519767Z","iopub.execute_input":"2022-08-07T12:45:57.520131Z","iopub.status.idle":"2022-08-07T12:45:57.795399Z","shell.execute_reply.started":"2022-08-07T12:45:57.520099Z","shell.execute_reply":"2022-08-07T12:45:57.793952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"B_25\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:57.797365Z","iopub.execute_input":"2022-08-07T12:45:57.797831Z","iopub.status.idle":"2022-08-07T12:45:58.073982Z","shell.execute_reply.started":"2022-08-07T12:45:57.797788Z","shell.execute_reply":"2022-08-07T12:45:58.072568Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.loc[train[\"B_37\"].isnull() & ~train[\"B_25\"].isnull(),\"B_37\"]=(train[\"B_37\"].corr(train[\"B_25\"])*train[\"B_37\"].std()/train[\"B_25\"].std())*(train.loc[train[\"B_37\"].isnull() & ~train[\"B_25\"].isnull(),\"B_25\"]-train[\"B_25\"].mean())+train[\"B_37\"].mean()\ntrain.loc[~train[\"B_37\"].isnull() & train[\"B_25\"].isnull(),\"B_25\"]=(train[\"B_37\"].corr(train[\"B_25\"])*train[\"B_25\"].std()/train[\"B_37\"].std())*(train.loc[~train[\"B_37\"].isnull() & train[\"B_25\"].isnull(),\"B_37\"]-train[\"B_37\"].mean())+train[\"B_25\"].mean()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:58.075962Z","iopub.execute_input":"2022-08-07T12:45:58.076910Z","iopub.status.idle":"2022-08-07T12:45:58.882733Z","shell.execute_reply.started":"2022-08-07T12:45:58.076856Z","shell.execute_reply":"2022-08-07T12:45:58.880995Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"B_37\"].fillna(train[\"B_37\"].mean(),inplace=True)\ntrain[\"B_25\"].fillna(train[\"B_25\"].mean(),inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:58.884974Z","iopub.execute_input":"2022-08-07T12:45:58.886034Z","iopub.status.idle":"2022-08-07T12:45:58.959014Z","shell.execute_reply.started":"2022-08-07T12:45:58.885966Z","shell.execute_reply":"2022-08-07T12:45:58.957355Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Column B_37 and B_25 is free from any null values.","metadata":{}},{"cell_type":"code","source":"train[(~train[\"B_13\"].isnull() & train[\"B_15\"].isnull()) | (train[\"B_13\"].isnull() & ~train[\"B_15\"].isnull()) ][[\"B_13\",\"B_15\"]]","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:58.961078Z","iopub.execute_input":"2022-08-07T12:45:58.962167Z","iopub.status.idle":"2022-08-07T12:45:59.158924Z","shell.execute_reply.started":"2022-08-07T12:45:58.962096Z","shell.execute_reply":"2022-08-07T12:45:59.157382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"B_13\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:59.161126Z","iopub.execute_input":"2022-08-07T12:45:59.161571Z","iopub.status.idle":"2022-08-07T12:45:59.411473Z","shell.execute_reply.started":"2022-08-07T12:45:59.161529Z","shell.execute_reply":"2022-08-07T12:45:59.409900Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"B_15\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:59.413330Z","iopub.execute_input":"2022-08-07T12:45:59.414705Z","iopub.status.idle":"2022-08-07T12:45:59.714925Z","shell.execute_reply.started":"2022-08-07T12:45:59.414637Z","shell.execute_reply":"2022-08-07T12:45:59.713459Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will use correlational coefficient and the know values to predict the unknown values.","metadata":{}},{"cell_type":"code","source":"train.loc[train[\"B_13\"].isnull() & ~train[\"B_15\"].isnull(),\"B_13\"]=(train[\"B_13\"].corr(train[\"B_15\"])*train[\"B_13\"].std()/train[\"B_15\"].std())*(train.loc[train[\"B_13\"].isnull() & ~train[\"B_15\"].isnull(),\"B_15\"]-train[\"B_15\"].mean())+train[\"B_13\"].mean()\ntrain.loc[~train[\"B_13\"].isnull() & train[\"B_15\"].isnull(),\"B_15\"]=(train[\"B_13\"].corr(train[\"B_15\"])*train[\"B_15\"].std()/train[\"B_13\"].std())*(train.loc[~train[\"B_13\"].isnull() & train[\"B_15\"].isnull(),\"B_13\"]-train[\"B_13\"].mean())+train[\"B_15\"].mean()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:45:59.716570Z","iopub.execute_input":"2022-08-07T12:45:59.717473Z","iopub.status.idle":"2022-08-07T12:46:00.368325Z","shell.execute_reply.started":"2022-08-07T12:45:59.717437Z","shell.execute_reply":"2022-08-07T12:46:00.367017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Fill the remaning null values.","metadata":{"execution":{"iopub.status.busy":"2022-07-27T10:52:23.531180Z","iopub.status.idle":"2022-07-27T10:52:23.531865Z","shell.execute_reply.started":"2022-07-27T10:52:23.531660Z","shell.execute_reply":"2022-07-27T10:52:23.531680Z"}}},{"cell_type":"code","source":"train[\"B_13\"].fillna(train[\"B_13\"].mean(),inplace=True)\ntrain[\"B_15\"].fillna(train[\"B_15\"].mean(),inplace=True)","metadata":{"scrolled":true,"execution":{"iopub.status.busy":"2022-08-07T12:46:00.370291Z","iopub.execute_input":"2022-08-07T12:46:00.370726Z","iopub.status.idle":"2022-08-07T12:46:00.440128Z","shell.execute_reply.started":"2022-08-07T12:46:00.370689Z","shell.execute_reply":"2022-08-07T12:46:00.438874Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"B_13\"]=train[\"B_13\"]/(train[\"B_13\"].max()-train[\"B_13\"].min())","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:00.441183Z","iopub.execute_input":"2022-08-07T12:46:00.441550Z","iopub.status.idle":"2022-08-07T12:46:00.485905Z","shell.execute_reply.started":"2022-08-07T12:46:00.441518Z","shell.execute_reply":"2022-08-07T12:46:00.484680Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Thus B_13 and B_15 is handled.","metadata":{}},{"cell_type":"markdown","source":"Calculating Values of B_2 and B_3 from B_37","metadata":{}},{"cell_type":"code","source":"train.loc[train[\"B_2\"].isnull() & ~train[\"B_37\"].isnull(),\"B_2\"]=(train[\"B_2\"].corr(train[\"B_37\"])*train[\"B_2\"].std()/train[\"B_37\"].std())*(train.loc[train[\"B_2\"].isnull() & ~train[\"B_37\"].isnull(),\"B_37\"]-train[\"B_37\"].mean())+train[\"B_2\"].mean()\ntrain.loc[train[\"B_3\"].isnull() & ~train[\"B_37\"].isnull(),\"B_3\"]=(train[\"B_3\"].corr(train[\"B_37\"])*train[\"B_3\"].std()/train[\"B_37\"].std())*(train.loc[train[\"B_3\"].isnull() & ~train[\"B_37\"].isnull(),\"B_37\"]-train[\"B_37\"].mean())+train[\"B_3\"].mean()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:00.488029Z","iopub.execute_input":"2022-08-07T12:46:00.488401Z","iopub.status.idle":"2022-08-07T12:46:01.097845Z","shell.execute_reply.started":"2022-08-07T12:46:00.488369Z","shell.execute_reply":"2022-08-07T12:46:01.096457Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Calculating B_17 from B_2","metadata":{}},{"cell_type":"code","source":"train.loc[train[\"B_17\"].isnull() & ~train[\"B_2\"].isnull(),\"B_17\"]=(train[\"B_17\"].corr(train[\"B_2\"])*train[\"B_17\"].std()/train[\"B_2\"].std())*(train.loc[train[\"B_17\"].isnull() & ~train[\"B_2\"].isnull(),\"B_2\"]-train[\"B_2\"].mean())+train[\"B_17\"].mean()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:01.100277Z","iopub.execute_input":"2022-08-07T12:46:01.100726Z","iopub.status.idle":"2022-08-07T12:46:01.506479Z","shell.execute_reply.started":"2022-08-07T12:46:01.100684Z","shell.execute_reply":"2022-08-07T12:46:01.504863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4.2.4 Payment Variables","metadata":{}},{"cell_type":"code","source":"train[\"P_2\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:01.508311Z","iopub.execute_input":"2022-08-07T12:46:01.508894Z","iopub.status.idle":"2022-08-07T12:46:01.756340Z","shell.execute_reply.started":"2022-08-07T12:46:01.508838Z","shell.execute_reply":"2022-08-07T12:46:01.754782Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"P_2\"].fillna(value=train[\"P_2\"].mean(),inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:01.758465Z","iopub.execute_input":"2022-08-07T12:46:01.759250Z","iopub.status.idle":"2022-08-07T12:46:01.793816Z","shell.execute_reply.started":"2022-08-07T12:46:01.759208Z","shell.execute_reply":"2022-08-07T12:46:01.792661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"P_2 variable is handled","metadata":{}},{"cell_type":"code","source":"train[\"P_3\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:01.795620Z","iopub.execute_input":"2022-08-07T12:46:01.796092Z","iopub.status.idle":"2022-08-07T12:46:02.077687Z","shell.execute_reply.started":"2022-08-07T12:46:01.796052Z","shell.execute_reply":"2022-08-07T12:46:02.076456Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"P_3\"].fillna(value=train[\"P_3\"].mean(),inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:02.079399Z","iopub.execute_input":"2022-08-07T12:46:02.079926Z","iopub.status.idle":"2022-08-07T12:46:02.117713Z","shell.execute_reply.started":"2022-08-07T12:46:02.079891Z","shell.execute_reply":"2022-08-07T12:46:02.116625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"P_3 column is free of any null values","metadata":{}},{"cell_type":"markdown","source":"# 4.2.5 Delinquency Variables","metadata":{}},{"cell_type":"code","source":"train.isnull().sum().sort_values(ascending=False)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:02.119124Z","iopub.execute_input":"2022-08-07T12:46:02.119876Z","iopub.status.idle":"2022-08-07T12:46:04.050008Z","shell.execute_reply.started":"2022-08-07T12:46:02.119837Z","shell.execute_reply":"2022-08-07T12:46:04.048706Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.drop(labels=[\"D_42\",\"D_77\"],axis=1,inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:04.051966Z","iopub.execute_input":"2022-08-07T12:46:04.052374Z","iopub.status.idle":"2022-08-07T12:46:05.479509Z","shell.execute_reply.started":"2022-08-07T12:46:04.052338Z","shell.execute_reply":"2022-08-07T12:46:05.478124Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.loc[train[\"D_43\"].isnull() & ~train[\"R_3\"].isnull(),\"D_43\"]=(train[\"D_43\"].corr(train[\"R_3\"])*train[\"D_43\"].std()/train[\"R_3\"].std())*(train.loc[train[\"D_43\"].isnull() & ~train[\"R_3\"].isnull(),\"R_3\"]-train[\"R_3\"].mean())+train[\"D_43\"].mean()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:05.481292Z","iopub.execute_input":"2022-08-07T12:46:05.482236Z","iopub.status.idle":"2022-08-07T12:46:05.838631Z","shell.execute_reply.started":"2022-08-07T12:46:05.482197Z","shell.execute_reply":"2022-08-07T12:46:05.837153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_46\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:05.840314Z","iopub.execute_input":"2022-08-07T12:46:05.840772Z","iopub.status.idle":"2022-08-07T12:46:06.108545Z","shell.execute_reply.started":"2022-08-07T12:46:05.840735Z","shell.execute_reply":"2022-08-07T12:46:06.107354Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_46\"].fillna(value=train[\"D_46\"].mean(),inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:06.110196Z","iopub.execute_input":"2022-08-07T12:46:06.110719Z","iopub.status.idle":"2022-08-07T12:46:06.153510Z","shell.execute_reply.started":"2022-08-07T12:46:06.110670Z","shell.execute_reply":"2022-08-07T12:46:06.152480Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_62\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:06.155008Z","iopub.execute_input":"2022-08-07T12:46:06.156270Z","iopub.status.idle":"2022-08-07T12:46:06.408417Z","shell.execute_reply.started":"2022-08-07T12:46:06.156218Z","shell.execute_reply":"2022-08-07T12:46:06.406447Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_62\"].fillna(value=train[\"D_62\"].mean(),inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:06.410912Z","iopub.execute_input":"2022-08-07T12:46:06.411555Z","iopub.status.idle":"2022-08-07T12:46:06.461139Z","shell.execute_reply.started":"2022-08-07T12:46:06.411494Z","shell.execute_reply":"2022-08-07T12:46:06.459453Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_48\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:06.463689Z","iopub.execute_input":"2022-08-07T12:46:06.464186Z","iopub.status.idle":"2022-08-07T12:46:06.730803Z","shell.execute_reply.started":"2022-08-07T12:46:06.464137Z","shell.execute_reply":"2022-08-07T12:46:06.729500Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.loc[train[\"D_48\"].isnull() & ~train[\"P_2\"].isnull(),\"D_48\"]=(train[\"D_48\"].corr(train[\"P_2\"])*train[\"D_48\"].std()/train[\"P_2\"].std())*(train.loc[train[\"D_48\"].isnull() & ~train[\"P_2\"].isnull(),\"P_2\"]-train[\"P_2\"].mean())+train[\"D_48\"].mean()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:06.732500Z","iopub.execute_input":"2022-08-07T12:46:06.733203Z","iopub.status.idle":"2022-08-07T12:46:07.081098Z","shell.execute_reply.started":"2022-08-07T12:46:06.733166Z","shell.execute_reply":"2022-08-07T12:46:07.079838Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_61\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:07.082410Z","iopub.execute_input":"2022-08-07T12:46:07.082796Z","iopub.status.idle":"2022-08-07T12:46:07.337792Z","shell.execute_reply.started":"2022-08-07T12:46:07.082761Z","shell.execute_reply":"2022-08-07T12:46:07.336378Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_61\"]=train[\"D_61\"]/(train[\"D_61\"].max()-train[\"D_61\"].min())","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:07.339419Z","iopub.execute_input":"2022-08-07T12:46:07.342099Z","iopub.status.idle":"2022-08-07T12:46:07.409665Z","shell.execute_reply.started":"2022-08-07T12:46:07.342058Z","shell.execute_reply":"2022-08-07T12:46:07.408070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_61\"].fillna(value=train[\"D_61\"].mean(),inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:07.411316Z","iopub.execute_input":"2022-08-07T12:46:07.411742Z","iopub.status.idle":"2022-08-07T12:46:07.454070Z","shell.execute_reply.started":"2022-08-07T12:46:07.411706Z","shell.execute_reply":"2022-08-07T12:46:07.452942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_69\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:07.455783Z","iopub.execute_input":"2022-08-07T12:46:07.456185Z","iopub.status.idle":"2022-08-07T12:46:07.744110Z","shell.execute_reply.started":"2022-08-07T12:46:07.456141Z","shell.execute_reply":"2022-08-07T12:46:07.742540Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_69\"]=train[\"D_69\"]/(train[\"D_69\"].max()-train[\"D_69\"].min())","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:07.745739Z","iopub.execute_input":"2022-08-07T12:46:07.746114Z","iopub.status.idle":"2022-08-07T12:46:07.810503Z","shell.execute_reply.started":"2022-08-07T12:46:07.746081Z","shell.execute_reply":"2022-08-07T12:46:07.809130Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_69\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:07.812477Z","iopub.execute_input":"2022-08-07T12:46:07.813036Z","iopub.status.idle":"2022-08-07T12:46:08.104140Z","shell.execute_reply.started":"2022-08-07T12:46:07.812987Z","shell.execute_reply":"2022-08-07T12:46:08.102856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_69\"].fillna(value=train[\"D_69\"].mean(),inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:08.105448Z","iopub.execute_input":"2022-08-07T12:46:08.105801Z","iopub.status.idle":"2022-08-07T12:46:08.143552Z","shell.execute_reply.started":"2022-08-07T12:46:08.105769Z","shell.execute_reply":"2022-08-07T12:46:08.142260Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_55\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:08.145123Z","iopub.execute_input":"2022-08-07T12:46:08.145845Z","iopub.status.idle":"2022-08-07T12:46:08.401188Z","shell.execute_reply.started":"2022-08-07T12:46:08.145796Z","shell.execute_reply":"2022-08-07T12:46:08.399875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_55\"].fillna(value=train[\"D_55\"].mean(),inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:08.402648Z","iopub.execute_input":"2022-08-07T12:46:08.402996Z","iopub.status.idle":"2022-08-07T12:46:08.439175Z","shell.execute_reply.started":"2022-08-07T12:46:08.402966Z","shell.execute_reply":"2022-08-07T12:46:08.437586Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_118\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:08.440731Z","iopub.execute_input":"2022-08-07T12:46:08.441106Z","iopub.status.idle":"2022-08-07T12:46:08.676926Z","shell.execute_reply.started":"2022-08-07T12:46:08.441072Z","shell.execute_reply":"2022-08-07T12:46:08.675364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_118\"].fillna(value=train[\"D_118\"].mean(),inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:08.679025Z","iopub.execute_input":"2022-08-07T12:46:08.679811Z","iopub.status.idle":"2022-08-07T12:46:08.718544Z","shell.execute_reply.started":"2022-08-07T12:46:08.679760Z","shell.execute_reply":"2022-08-07T12:46:08.716969Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_121\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:08.720321Z","iopub.execute_input":"2022-08-07T12:46:08.721046Z","iopub.status.idle":"2022-08-07T12:46:08.944891Z","shell.execute_reply.started":"2022-08-07T12:46:08.720950Z","shell.execute_reply":"2022-08-07T12:46:08.943633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_121\"].fillna(value=train[\"D_121\"].mean(),inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:08.947124Z","iopub.execute_input":"2022-08-07T12:46:08.948020Z","iopub.status.idle":"2022-08-07T12:46:08.986086Z","shell.execute_reply.started":"2022-08-07T12:46:08.947971Z","shell.execute_reply":"2022-08-07T12:46:08.984705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_115\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:09.003067Z","iopub.execute_input":"2022-08-07T12:46:09.003490Z","iopub.status.idle":"2022-08-07T12:46:09.222889Z","shell.execute_reply.started":"2022-08-07T12:46:09.003449Z","shell.execute_reply":"2022-08-07T12:46:09.221521Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_115\"].fillna(value=train[\"D_115\"].mean(),inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:09.224938Z","iopub.execute_input":"2022-08-07T12:46:09.225423Z","iopub.status.idle":"2022-08-07T12:46:09.262774Z","shell.execute_reply.started":"2022-08-07T12:46:09.225380Z","shell.execute_reply":"2022-08-07T12:46:09.261578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_119\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:09.265027Z","iopub.execute_input":"2022-08-07T12:46:09.265513Z","iopub.status.idle":"2022-08-07T12:46:09.493429Z","shell.execute_reply.started":"2022-08-07T12:46:09.265468Z","shell.execute_reply":"2022-08-07T12:46:09.491987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_119\"].fillna(value=train[\"D_119\"].mean(),inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:09.495599Z","iopub.execute_input":"2022-08-07T12:46:09.496129Z","iopub.status.idle":"2022-08-07T12:46:09.533854Z","shell.execute_reply.started":"2022-08-07T12:46:09.496085Z","shell.execute_reply":"2022-08-07T12:46:09.532491Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_131\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:09.535935Z","iopub.execute_input":"2022-08-07T12:46:09.536661Z","iopub.status.idle":"2022-08-07T12:46:09.732438Z","shell.execute_reply.started":"2022-08-07T12:46:09.536595Z","shell.execute_reply":"2022-08-07T12:46:09.731139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_131\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:09.733955Z","iopub.execute_input":"2022-08-07T12:46:09.734403Z","iopub.status.idle":"2022-08-07T12:46:09.926272Z","shell.execute_reply.started":"2022-08-07T12:46:09.734370Z","shell.execute_reply":"2022-08-07T12:46:09.924965Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_131\"].fillna(value=0,inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:09.927636Z","iopub.execute_input":"2022-08-07T12:46:09.928478Z","iopub.status.idle":"2022-08-07T12:46:09.942751Z","shell.execute_reply.started":"2022-08-07T12:46:09.928436Z","shell.execute_reply":"2022-08-07T12:46:09.941123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_130\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:09.944591Z","iopub.execute_input":"2022-08-07T12:46:09.945003Z","iopub.status.idle":"2022-08-07T12:46:10.145968Z","shell.execute_reply.started":"2022-08-07T12:46:09.944967Z","shell.execute_reply":"2022-08-07T12:46:10.144344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_130\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:10.147331Z","iopub.execute_input":"2022-08-07T12:46:10.147691Z","iopub.status.idle":"2022-08-07T12:46:10.329745Z","shell.execute_reply.started":"2022-08-07T12:46:10.147659Z","shell.execute_reply":"2022-08-07T12:46:10.328370Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_130\"].fillna(value=0,inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:10.331860Z","iopub.execute_input":"2022-08-07T12:46:10.332625Z","iopub.status.idle":"2022-08-07T12:46:10.347233Z","shell.execute_reply.started":"2022-08-07T12:46:10.332552Z","shell.execute_reply":"2022-08-07T12:46:10.346232Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_104\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:10.348331Z","iopub.execute_input":"2022-08-07T12:46:10.348716Z","iopub.status.idle":"2022-08-07T12:46:10.567704Z","shell.execute_reply.started":"2022-08-07T12:46:10.348681Z","shell.execute_reply":"2022-08-07T12:46:10.566461Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_104\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:10.569483Z","iopub.execute_input":"2022-08-07T12:46:10.570071Z","iopub.status.idle":"2022-08-07T12:46:10.972781Z","shell.execute_reply.started":"2022-08-07T12:46:10.570031Z","shell.execute_reply":"2022-08-07T12:46:10.971415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_104\"].fillna(value=0,inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:10.974748Z","iopub.execute_input":"2022-08-07T12:46:10.975250Z","iopub.status.idle":"2022-08-07T12:46:10.990148Z","shell.execute_reply.started":"2022-08-07T12:46:10.975203Z","shell.execute_reply":"2022-08-07T12:46:10.988656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_141\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:10.991826Z","iopub.execute_input":"2022-08-07T12:46:10.992210Z","iopub.status.idle":"2022-08-07T12:46:11.188660Z","shell.execute_reply.started":"2022-08-07T12:46:10.992177Z","shell.execute_reply":"2022-08-07T12:46:11.186977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_141\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:11.190634Z","iopub.execute_input":"2022-08-07T12:46:11.191685Z","iopub.status.idle":"2022-08-07T12:46:11.455282Z","shell.execute_reply.started":"2022-08-07T12:46:11.191632Z","shell.execute_reply":"2022-08-07T12:46:11.454054Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_141\"].fillna(value=0,inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:11.456396Z","iopub.execute_input":"2022-08-07T12:46:11.457274Z","iopub.status.idle":"2022-08-07T12:46:11.470862Z","shell.execute_reply.started":"2022-08-07T12:46:11.457232Z","shell.execute_reply":"2022-08-07T12:46:11.469691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_128\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:11.473804Z","iopub.execute_input":"2022-08-07T12:46:11.474522Z","iopub.status.idle":"2022-08-07T12:46:11.724009Z","shell.execute_reply.started":"2022-08-07T12:46:11.474479Z","shell.execute_reply":"2022-08-07T12:46:11.722789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_128\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:11.725448Z","iopub.execute_input":"2022-08-07T12:46:11.725828Z","iopub.status.idle":"2022-08-07T12:46:12.017576Z","shell.execute_reply.started":"2022-08-07T12:46:11.725782Z","shell.execute_reply":"2022-08-07T12:46:12.016090Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_128\"].fillna(value=0,inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:12.019284Z","iopub.execute_input":"2022-08-07T12:46:12.020059Z","iopub.status.idle":"2022-08-07T12:46:12.034564Z","shell.execute_reply.started":"2022-08-07T12:46:12.020002Z","shell.execute_reply":"2022-08-07T12:46:12.033141Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_133\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:12.036068Z","iopub.execute_input":"2022-08-07T12:46:12.036691Z","iopub.status.idle":"2022-08-07T12:46:12.326856Z","shell.execute_reply.started":"2022-08-07T12:46:12.036639Z","shell.execute_reply":"2022-08-07T12:46:12.325264Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_133\"].fillna(value=train[\"D_133\"].mean(),inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:12.330887Z","iopub.execute_input":"2022-08-07T12:46:12.331277Z","iopub.status.idle":"2022-08-07T12:46:12.367064Z","shell.execute_reply.started":"2022-08-07T12:46:12.331243Z","shell.execute_reply":"2022-08-07T12:46:12.365565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_144\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:12.369362Z","iopub.execute_input":"2022-08-07T12:46:12.369915Z","iopub.status.idle":"2022-08-07T12:46:12.647294Z","shell.execute_reply.started":"2022-08-07T12:46:12.369862Z","shell.execute_reply":"2022-08-07T12:46:12.645881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_144\"].fillna(value=train[\"D_144\"].mean(),inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:12.649500Z","iopub.execute_input":"2022-08-07T12:46:12.650057Z","iopub.status.idle":"2022-08-07T12:46:12.686383Z","shell.execute_reply.started":"2022-08-07T12:46:12.650008Z","shell.execute_reply":"2022-08-07T12:46:12.685185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_102\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:12.688096Z","iopub.execute_input":"2022-08-07T12:46:12.688782Z","iopub.status.idle":"2022-08-07T12:46:12.937926Z","shell.execute_reply.started":"2022-08-07T12:46:12.688727Z","shell.execute_reply":"2022-08-07T12:46:12.936567Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_102\"].fillna(value=train[\"D_102\"].mean(),inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:12.939397Z","iopub.execute_input":"2022-08-07T12:46:12.939798Z","iopub.status.idle":"2022-08-07T12:46:12.974389Z","shell.execute_reply.started":"2022-08-07T12:46:12.939762Z","shell.execute_reply":"2022-08-07T12:46:12.973190Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_52\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:12.976105Z","iopub.execute_input":"2022-08-07T12:46:12.976470Z","iopub.status.idle":"2022-08-07T12:46:13.185486Z","shell.execute_reply.started":"2022-08-07T12:46:12.976437Z","shell.execute_reply":"2022-08-07T12:46:13.184270Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_52\"].fillna(value=train[\"D_52\"].mean(),inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:13.187110Z","iopub.execute_input":"2022-08-07T12:46:13.187576Z","iopub.status.idle":"2022-08-07T12:46:13.222112Z","shell.execute_reply.started":"2022-08-07T12:46:13.187529Z","shell.execute_reply":"2022-08-07T12:46:13.220951Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_112\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:13.223584Z","iopub.execute_input":"2022-08-07T12:46:13.224137Z","iopub.status.idle":"2022-08-07T12:46:13.415234Z","shell.execute_reply.started":"2022-08-07T12:46:13.224084Z","shell.execute_reply":"2022-08-07T12:46:13.413913Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_112\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:13.417594Z","iopub.execute_input":"2022-08-07T12:46:13.418341Z","iopub.status.idle":"2022-08-07T12:46:13.718285Z","shell.execute_reply.started":"2022-08-07T12:46:13.418297Z","shell.execute_reply":"2022-08-07T12:46:13.716791Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_112\"].fillna(value=1,inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:13.720656Z","iopub.execute_input":"2022-08-07T12:46:13.721040Z","iopub.status.idle":"2022-08-07T12:46:13.733707Z","shell.execute_reply.started":"2022-08-07T12:46:13.721002Z","shell.execute_reply":"2022-08-07T12:46:13.732007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_45\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:13.735446Z","iopub.execute_input":"2022-08-07T12:46:13.735853Z","iopub.status.idle":"2022-08-07T12:46:13.959772Z","shell.execute_reply.started":"2022-08-07T12:46:13.735815Z","shell.execute_reply":"2022-08-07T12:46:13.958862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_45\"].fillna(value=train[\"D_45\"].mean(),inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:13.961036Z","iopub.execute_input":"2022-08-07T12:46:13.961363Z","iopub.status.idle":"2022-08-07T12:46:13.996216Z","shell.execute_reply.started":"2022-08-07T12:46:13.961335Z","shell.execute_reply":"2022-08-07T12:46:13.995069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_54\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:13.997642Z","iopub.execute_input":"2022-08-07T12:46:13.998010Z","iopub.status.idle":"2022-08-07T12:46:14.188380Z","shell.execute_reply.started":"2022-08-07T12:46:13.997978Z","shell.execute_reply":"2022-08-07T12:46:14.187253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_54\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:14.190014Z","iopub.execute_input":"2022-08-07T12:46:14.190356Z","iopub.status.idle":"2022-08-07T12:46:14.348997Z","shell.execute_reply.started":"2022-08-07T12:46:14.190324Z","shell.execute_reply":"2022-08-07T12:46:14.348142Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_54\"].fillna(value=1,inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:14.350245Z","iopub.execute_input":"2022-08-07T12:46:14.350568Z","iopub.status.idle":"2022-08-07T12:46:14.364228Z","shell.execute_reply.started":"2022-08-07T12:46:14.350539Z","shell.execute_reply":"2022-08-07T12:46:14.363050Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_41\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:14.365718Z","iopub.execute_input":"2022-08-07T12:46:14.366094Z","iopub.status.idle":"2022-08-07T12:46:14.558726Z","shell.execute_reply.started":"2022-08-07T12:46:14.366061Z","shell.execute_reply":"2022-08-07T12:46:14.557872Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_41\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:14.560047Z","iopub.execute_input":"2022-08-07T12:46:14.561068Z","iopub.status.idle":"2022-08-07T12:46:14.795371Z","shell.execute_reply.started":"2022-08-07T12:46:14.561030Z","shell.execute_reply":"2022-08-07T12:46:14.793856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"D_41\"].fillna(value=0,inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:14.797238Z","iopub.execute_input":"2022-08-07T12:46:14.797889Z","iopub.status.idle":"2022-08-07T12:46:14.811256Z","shell.execute_reply.started":"2022-08-07T12:46:14.797850Z","shell.execute_reply":"2022-08-07T12:46:14.810131Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4.2.6 Remaining Unhandled Variables","metadata":{}},{"cell_type":"markdown","source":"Check for unhandled columns.","metadata":{}},{"cell_type":"code","source":"train.isnull().sum().sort_values(ascending=False)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:14.812726Z","iopub.execute_input":"2022-08-07T12:46:14.813966Z","iopub.status.idle":"2022-08-07T12:46:16.701883Z","shell.execute_reply.started":"2022-08-07T12:46:14.813926Z","shell.execute_reply":"2022-08-07T12:46:16.700436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"S_27\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:16.703437Z","iopub.execute_input":"2022-08-07T12:46:16.704204Z","iopub.status.idle":"2022-08-07T12:46:16.975784Z","shell.execute_reply.started":"2022-08-07T12:46:16.704163Z","shell.execute_reply":"2022-08-07T12:46:16.974918Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"S_27\"].fillna(value=train[\"S_27\"].mean(),inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:16.976942Z","iopub.execute_input":"2022-08-07T12:46:16.977824Z","iopub.status.idle":"2022-08-07T12:46:17.027299Z","shell.execute_reply.started":"2022-08-07T12:46:16.977789Z","shell.execute_reply":"2022-08-07T12:46:17.025978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"R_27\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:17.030112Z","iopub.execute_input":"2022-08-07T12:46:17.030848Z","iopub.status.idle":"2022-08-07T12:46:17.317857Z","shell.execute_reply.started":"2022-08-07T12:46:17.030799Z","shell.execute_reply":"2022-08-07T12:46:17.316233Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"R_27\"].fillna(value=train[\"R_27\"].mean(),inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:17.319733Z","iopub.execute_input":"2022-08-07T12:46:17.320168Z","iopub.status.idle":"2022-08-07T12:46:17.358494Z","shell.execute_reply.started":"2022-08-07T12:46:17.320122Z","shell.execute_reply":"2022-08-07T12:46:17.357352Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"B_8\"].describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:17.360171Z","iopub.execute_input":"2022-08-07T12:46:17.361128Z","iopub.status.idle":"2022-08-07T12:46:17.614240Z","shell.execute_reply.started":"2022-08-07T12:46:17.361081Z","shell.execute_reply":"2022-08-07T12:46:17.612745Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"B_8\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:17.616338Z","iopub.execute_input":"2022-08-07T12:46:17.616821Z","iopub.status.idle":"2022-08-07T12:46:17.981946Z","shell.execute_reply.started":"2022-08-07T12:46:17.616777Z","shell.execute_reply":"2022-08-07T12:46:17.980775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"R_27\"].fillna(value=0,inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:17.984083Z","iopub.execute_input":"2022-08-07T12:46:17.985698Z","iopub.status.idle":"2022-08-07T12:46:18.003402Z","shell.execute_reply.started":"2022-08-07T12:46:17.985631Z","shell.execute_reply":"2022-08-07T12:46:18.001018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"B_27\"].fillna(value=train[\"B_27\"].mean(),inplace=True)\ntrain[\"B_26\"].fillna(value=train[\"B_26\"].mean(),inplace=True)\ntrain[\"B_6\"].fillna(value=train[\"B_6\"].mean(),inplace=True)\ntrain[\"B_8\"].fillna(value=train[\"B_8\"].mean(),inplace=True)\ntrain[\"B_40\"].fillna(value=train[\"B_40\"].mean(),inplace=True)\ntrain[\"R_12\"].fillna(value=train[\"R_12\"].mean(),inplace=True)\ntrain[\"R_14\"].fillna(value=train[\"R_14\"].mean(),inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:18.007104Z","iopub.execute_input":"2022-08-07T12:46:18.008123Z","iopub.status.idle":"2022-08-07T12:46:18.246709Z","shell.execute_reply.started":"2022-08-07T12:46:18.008072Z","shell.execute_reply":"2022-08-07T12:46:18.245222Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#  4.3 Label encoding/factorizing the remaining character variables","metadata":{}},{"cell_type":"markdown","source":"First of all we need to convert date into factors.","metadata":{}},{"cell_type":"code","source":"train.groupby(by=\"customer_ID\")\ntrain","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:18.248864Z","iopub.execute_input":"2022-08-07T12:46:18.249352Z","iopub.status.idle":"2022-08-07T12:46:18.420811Z","shell.execute_reply.started":"2022-08-07T12:46:18.249310Z","shell.execute_reply":"2022-08-07T12:46:18.419253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Remove the day from the S_2 columns as it is of not much significance.","metadata":{}},{"cell_type":"markdown","source":"I think recent transaction indicates lower probability of defaulting (Contradictory with our finding from target variable). So I factories the date column.","metadata":{}},{"cell_type":"code","source":"train[\"S_2\"]=train[\"S_2\"].str.slice(start=0,stop=7)\nmapping={\"2017-03\":0,\"2017-04\":1,\"2017-05\":2,\"2017-06\":3,\"2017-07\":4,\"2017-08\":5,\"2017-09\":6,\"2017-10\":7,\"2017-11\":8,\"2017-12\":9,\"2018-01\":10,\"2018-02\":11,\"2018-03\":12}\ntrain=train.replace({\"S_2\":mapping})\ntrain[\"S_2\"]","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:18.422631Z","iopub.execute_input":"2022-08-07T12:46:18.423090Z","iopub.status.idle":"2022-08-07T12:46:28.561760Z","shell.execute_reply.started":"2022-08-07T12:46:18.423052Z","shell.execute_reply":"2022-08-07T12:46:28.560239Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"S_2\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:28.563285Z","iopub.execute_input":"2022-08-07T12:46:28.563635Z","iopub.status.idle":"2022-08-07T12:46:28.622220Z","shell.execute_reply.started":"2022-08-07T12:46:28.563588Z","shell.execute_reply":"2022-08-07T12:46:28.620672Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Thus online use of credit card increased over time.","metadata":{}},{"cell_type":"code","source":"train.drop(labels=\"customer_ID\",axis=1,inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:28.624435Z","iopub.execute_input":"2022-08-07T12:46:28.625043Z","iopub.status.idle":"2022-08-07T12:46:34.706413Z","shell.execute_reply.started":"2022-08-07T12:46:28.624993Z","shell.execute_reply":"2022-08-07T12:46:34.705164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The categorical features are the followings ['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']","metadata":{}},{"cell_type":"code","source":"columns=['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68','S_2']\nfor col in columns:\n    print(train[col].value_counts())","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:34.708222Z","iopub.execute_input":"2022-08-07T12:46:34.708722Z","iopub.status.idle":"2022-08-07T12:46:35.214626Z","shell.execute_reply.started":"2022-08-07T12:46:34.708679Z","shell.execute_reply":"2022-08-07T12:46:35.213107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here I have used target encoder for categorical columns.","metadata":{}},{"cell_type":"code","source":"cols=['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68','S_2']\nencoder=TargetEncoder()\nfor c in cols:\n    train[c]=train[c].astype('category')\n    train[c] = encoder.fit_transform(train[c], train[\"target\"])","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:46:35.216311Z","iopub.execute_input":"2022-08-07T12:46:35.216920Z","iopub.status.idle":"2022-08-07T12:48:33.355748Z","shell.execute_reply.started":"2022-08-07T12:46:35.216878Z","shell.execute_reply":"2022-08-07T12:48:33.354107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"columns=['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68','S_2']\nfor col in columns:\n    print(train[col].value_counts())","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:48:33.357716Z","iopub.execute_input":"2022-08-07T12:48:33.358906Z","iopub.status.idle":"2022-08-07T12:48:34.469451Z","shell.execute_reply.started":"2022-08-07T12:48:33.358850Z","shell.execute_reply":"2022-08-07T12:48:34.468058Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.scatterplot(x=np.linspace(start=0,stop=12,num=13),y=train[\"S_2\"].unique())","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:48:34.471269Z","iopub.execute_input":"2022-08-07T12:48:34.472348Z","iopub.status.idle":"2022-08-07T12:48:34.746925Z","shell.execute_reply.started":"2022-08-07T12:48:34.472297Z","shell.execute_reply":"2022-08-07T12:48:34.745673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Thus we can notice that persons who have made recent transactions have a higher probability of defaulting than others.","metadata":{}},{"cell_type":"markdown","source":"We will use the correlation ,standard deviation and mean to derive the values for the lables present in the test set.","metadata":{}},{"cell_type":"code","source":"np.corrcoef(np.linspace(start=0,stop=12,num=13),y=train[\"S_2\"].unique())","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:48:34.748813Z","iopub.execute_input":"2022-08-07T12:48:34.749168Z","iopub.status.idle":"2022-08-07T12:48:34.817894Z","shell.execute_reply.started":"2022-08-07T12:48:34.749137Z","shell.execute_reply":"2022-08-07T12:48:34.816627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.shape","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:48:34.819624Z","iopub.execute_input":"2022-08-07T12:48:34.820011Z","iopub.status.idle":"2022-08-07T12:48:34.827329Z","shell.execute_reply.started":"2022-08-07T12:48:34.819977Z","shell.execute_reply":"2022-08-07T12:48:34.826100Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head(10)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:48:34.829172Z","iopub.execute_input":"2022-08-07T12:48:34.830309Z","iopub.status.idle":"2022-08-07T12:48:34.970454Z","shell.execute_reply.started":"2022-08-07T12:48:34.830238Z","shell.execute_reply":"2022-08-07T12:48:34.969013Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As there was very less space available to me. I use garbage collector to free up some space.","metadata":{}},{"cell_type":"code","source":"import gc\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:48:34.972427Z","iopub.execute_input":"2022-08-07T12:48:34.972846Z","iopub.status.idle":"2022-08-07T12:48:35.153266Z","shell.execute_reply.started":"2022-08-07T12:48:34.972807Z","shell.execute_reply":"2022-08-07T12:48:35.151724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5 Preparing Data for modelling","metadata":{}},{"cell_type":"markdown","source":"# 5.1 We will normalise all the numerical columns","metadata":{}},{"cell_type":"markdown","source":"Here I normalize the numerical columns and change the data type to save some space. Here we have used min max normalizer.","metadata":{}},{"cell_type":"code","source":"for col in train.columns:\n    if col not in ['customer_ID', 'target', 'B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68','S_2']:\n        train[col]=train[col]/(train[col].max()-train[col].min())\n    if col not in ['customer_ID', 'target']:\n        train[col]=train[col].astype(\"float16\")\ntrain[\"target\"]=train[\"target\"].astype(\"int8\")","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:48:35.154412Z","iopub.execute_input":"2022-08-07T12:48:35.154941Z","iopub.status.idle":"2022-08-07T12:49:30.274008Z","shell.execute_reply.started":"2022-08-07T12:48:35.154907Z","shell.execute_reply":"2022-08-07T12:49:30.272444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.isnull().values.sum()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:49:30.275880Z","iopub.execute_input":"2022-08-07T12:49:30.276303Z","iopub.status.idle":"2022-08-07T12:49:36.755290Z","shell.execute_reply.started":"2022-08-07T12:49:30.276264Z","shell.execute_reply":"2022-08-07T12:49:36.753882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here I check the size of dataframe to be downloaded.","metadata":{}},{"cell_type":"markdown","source":"Here we don't need to handle the skewness of the data as neural network can handle them automatically.(https://www.researchgate.net/publication/334309178_The_relationship_between_data_skewness_and_accuracy_of_Artificial_Neural_Network_predictive_model)","metadata":{}},{"cell_type":"code","source":"train.info()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:49:36.757290Z","iopub.execute_input":"2022-08-07T12:49:36.757800Z","iopub.status.idle":"2022-08-07T12:49:36.783447Z","shell.execute_reply.started":"2022-08-07T12:49:36.757742Z","shell.execute_reply":"2022-08-07T12:49:36.782115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I will train the model in other notebook to overcome the space constraint. I downloaded train dataframe as \"train.csv.gzip\"","metadata":{}},{"cell_type":"code","source":"train.to_csv(\"train.csv.gzip\",compression=\"gzip\",chunksize=1000)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T12:49:36.785514Z","iopub.execute_input":"2022-08-07T12:49:36.786038Z","iopub.status.idle":"2022-08-07T13:49:21.678567Z","shell.execute_reply.started":"2022-08-07T12:49:36.785980Z","shell.execute_reply":"2022-08-07T13:49:21.674520Z"},"trusted":true},"execution_count":null,"outputs":[]}]}