{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"---\n# Introduction\n\n---","metadata":{}},{"cell_type":"markdown","source":"**Problem Statement:**\n\n* Whether out at a restaurant or buying tickets to a concert, modern life counts on the convenience of a credit card to make daily purchases. It saves us from carrying large amounts of cash and also can advance a full purchase that can be paid over time.\n* How do card issuers know we’ll pay back what we charge? That’s a complex problem with many existing solutions—and even more potential improvements, to be explored in this competition.\n* Credit default prediction is central to managing risk in a consumer lending business. \n * Credit default prediction allows lenders to optimize lending decisions, which leads to a better customer experience and sound business economics. \n* Current models exist to help manage risk. But it's possible to create better models that can outperform those currently in use.\n\n* The objective of this competition is to predict the probability that a customer does not pay back their credit card balance amount in the future based on their monthly customer profile. \n * The target binary variable is calculated by observing 18 months performance window after the latest credit card statement, and if the customer does not pay due amount in 120 days after their latest statement date it is considered a default event.","metadata":{}},{"cell_type":"markdown","source":"---\n**Importing Libraries:**\n* To get started we will use Python for data pre-processing and data analysis.\n* Import python libraries as necessary to get started for data load and later import other libraries as needed\n---","metadata":{}},{"cell_type":"code","source":"import numpy as np \n# data processing, CSV file I/O (e.g. pd.read_csv)\nimport pandas as pd \n# data processing, CSV file I/O (e.g. dd.read_csv)\nimport dask\nimport dask.dataframe as dd\n# module finds all the pathnames matching a specified pattern\nimport glob \nimport os\n# Importing pyplot interface using matplotlib\nimport matplotlib.pyplot as plt \n# Importing seaborn library for interactive visualization\nimport seaborn as sns \n# Importing WordCloud for text data visualization\nfrom wordcloud import WordCloud\n# Importing matplotlib for plots\nimport matplotlib\n# Importing datetime for using datetime\nfrom datetime import datetime\n# Importing plotly for interactive plots\nimport plotly.express as px\n# Importing missingno for missing value plot\nimport missingno as msno\n# importing regex library for use of regex\nimport re","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T03:57:24.445051Z","iopub.execute_input":"2022-07-24T03:57:24.446085Z","iopub.status.idle":"2022-07-24T03:57:26.762362Z","shell.execute_reply.started":"2022-07-24T03:57:24.445975Z","shell.execute_reply":"2022-07-24T03:57:26.760960Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n# Dataset Load/Data Display\n---","metadata":{}},{"cell_type":"markdown","source":"**Dataset:**\n\n* **train_data.csv** - training data with multiple statement dates per customer_ID\n* **train_labels.csv** - target label for each customer_ID\n* **test_data.csv** - corresponding test data; objective is to predict the target label for each customer_ID\n* **sample_submission.csv** - a sample submission file in the correct format\n\n---","metadata":{}},{"cell_type":"markdown","source":"Let us check size of dataset CSV file","metadata":{}},{"cell_type":"code","source":"# calculate file size in KB, MB, GB\ndef convert_bytes(size):\n    \"\"\" Convert bytes to KB, or MB or GB\"\"\"\n    for x in ['bytes', 'KB', 'MB', 'GB', 'TB']:\n        if size < 1024.0:\n            return \"%3.1f %s\" % (size, x)\n        size /= 1024.0\n\n# display CSV file with size\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        csvfile=os.path.join(dirname, filename)\n        csvfilesize = os.path.getsize(csvfile)\n        filesize = convert_bytes(csvfilesize)\n        print(f'{csvfile} size is', filesize, 'bytes')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T03:57:26.764639Z","iopub.execute_input":"2022-07-24T03:57:26.765411Z","iopub.status.idle":"2022-07-24T03:57:26.780970Z","shell.execute_reply.started":"2022-07-24T03:57:26.765363Z","shell.execute_reply":"2022-07-24T03:57:26.779591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Loading dataset train_data.csv using dask\ntrain_dask_df = dd.read_csv('../input/amex-default-prediction/train_data.csv',blocksize=\"10MB\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T03:57:26.782750Z","iopub.execute_input":"2022-07-24T03:57:26.784613Z","iopub.status.idle":"2022-07-24T03:57:27.062603Z","shell.execute_reply.started":"2022-07-24T03:57:26.784564Z","shell.execute_reply":"2022-07-24T03:57:27.061479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# get shape of dataframe\nprint('Number of rows is:', train_dask_df.shape[0].compute())\n# print summary of dataframe\ntrain_dask_df.info()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T03:57:27.065707Z","iopub.execute_input":"2022-07-24T03:57:27.066379Z","iopub.status.idle":"2022-07-24T04:00:31.681819Z","shell.execute_reply.started":"2022-07-24T03:57:27.066332Z","shell.execute_reply":"2022-07-24T04:00:31.680562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\nConsidering large number of rows around 5.5 million in **train_data.csv** dataset, using nrows option to load first 100k rows from dataset file for EDA purpose.\n\n---","metadata":{}},{"cell_type":"code","source":"del train_dask_df","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:31.683962Z","iopub.execute_input":"2022-07-24T04:00:31.684907Z","iopub.status.idle":"2022-07-24T04:00:31.693422Z","shell.execute_reply.started":"2022-07-24T04:00:31.684852Z","shell.execute_reply":"2022-07-24T04:00:31.692438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Load train_data.csv dataset file using nrows=100000","metadata":{}},{"cell_type":"code","source":"# Loading dataset train_data.csv\ntrain_df = pd.read_csv('../input/amex-default-prediction/train_data.csv', nrows=100000)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:31.694989Z","iopub.execute_input":"2022-07-24T04:00:31.695405Z","iopub.status.idle":"2022-07-24T04:00:36.607608Z","shell.execute_reply.started":"2022-07-24T04:00:31.695364Z","shell.execute_reply":"2022-07-24T04:00:36.606436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# get shape of dataframe\nprint('Shape of dataset is:', train_df.shape)\n\n# print summary of dataframe\n#train_df.info(verbose=True)\ntrain_df.info()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:36.609240Z","iopub.execute_input":"2022-07-24T04:00:36.609713Z","iopub.status.idle":"2022-07-24T04:00:36.634808Z","shell.execute_reply.started":"2022-07-24T04:00:36.609667Z","shell.execute_reply":"2022-07-24T04:00:36.633585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observations:**\n\n* There are total 190 variables in train_data.csv dataset\n    * There are 185 variables(Columns) as dtype float64, 1 variable(Column) as dtype int64 and 4 variables(Columns) as dtype object","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What does data look like for train_data.csv?**\n\n---","metadata":{}},{"cell_type":"code","source":"# print last 10 rows of dataframe\ntrain_df.head(10)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:36.635964Z","iopub.execute_input":"2022-07-24T04:00:36.636495Z","iopub.status.idle":"2022-07-24T04:00:36.677427Z","shell.execute_reply.started":"2022-07-24T04:00:36.636450Z","shell.execute_reply":"2022-07-24T04:00:36.676123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print last 10 rows of dataframe\ntrain_df.tail(10)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:36.679024Z","iopub.execute_input":"2022-07-24T04:00:36.680052Z","iopub.status.idle":"2022-07-24T04:00:36.710788Z","shell.execute_reply.started":"2022-07-24T04:00:36.680007Z","shell.execute_reply":"2022-07-24T04:00:36.709569Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observations:**\n\n* customer_ID has encrypted value\n* S_2 variable has date as value\n* S_3 and few D variables appears to have missing value","metadata":{}},{"cell_type":"markdown","source":"---\nConsidering high number of variables, need to extract names of variable for easy access.\n\n---","metadata":{}},{"cell_type":"code","source":"# Lets extract columns names for easy access\nD_columns = [col for col in train_df.columns if 'D_' in col]\nprint (\"Total number of columns starting with D is\", len(D_columns))\nprint(\"###############################\")\n# segregate D Columns for easier access using regex\n# regex to capture column name with D_$$ (D_ and 2 digit number suffix)\nd_col1 = \"D_[0-9][0-9]$$\"\n# regex to capture column name with D_$$$ (D_ and 3 digit number suffix)\nd_col2 = \"D_[0-9][0-9][0-9]$$$\"\n# get first set of column name which matches regex for d_col1\nD_columns1 = [col for col in train_df.columns if re.match(d_col1,col)]\nprint (\"First set of columns starting with D is\", len(D_columns1))\nprint(D_columns1)\n# get second set of column name which matches regex for d_col2\nD_columns2 = [col for col in train_df.columns if re.match(d_col2,col)]\nprint (\"Second set of columns starting with D is\", len(D_columns2))\nprint(D_columns2)\nprint (\"Total Number of columns starting with D is\", len(D_columns2)+len(D_columns1))\nprint(\"###############################\")\nS_columns = [col for col in train_df.columns if 'S_' in col]\nprint (\"Total number of columns starting with S is\", len(S_columns))\nprint(S_columns)\nprint(\"###############################\")\nP_columns = [col for col in train_df.columns if 'P_' in col]\nprint (\"Total number of columns starting with P is\", len(P_columns))\nprint(P_columns)\nprint(\"###############################\")\nB_columns = [col for col in train_df.columns if 'B_' in col]\nprint (\"Total number of columns starting with B is\", len(B_columns))\nprint(B_columns)\nprint(\"###############################\")\nR_columns = [col for col in train_df.columns if 'R_' in col]\nprint (\"Toal number of columns starting with R is\", len(R_columns))\nprint(R_columns)\nprint(\"###############################\")\ntotal_columns = len(D_columns1)+len(D_columns2)+len(S_columns)+len(P_columns)+len(B_columns)+len(R_columns)\nprint(\"Total number of D, S, P, B, R variables(columns) per customer_ID  is\",total_columns)\nprint(\"###############################\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:36.715953Z","iopub.execute_input":"2022-07-24T04:00:36.716359Z","iopub.status.idle":"2022-07-24T04:00:36.733549Z","shell.execute_reply.started":"2022-07-24T04:00:36.716316Z","shell.execute_reply":"2022-07-24T04:00:36.732326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Q: What does data look like for D Variables?**\n\n---","metadata":{}},{"cell_type":"code","source":"# display data for columns starting with D\ntrain_df.filter(like='D_', axis=1)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:36.735088Z","iopub.execute_input":"2022-07-24T04:00:36.736393Z","iopub.status.idle":"2022-07-24T04:00:36.809363Z","shell.execute_reply.started":"2022-07-24T04:00:36.736342Z","shell.execute_reply":"2022-07-24T04:00:36.808070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observations:**\n\n* There are many D variables with missing value\n* There are many D variables with either 0 or 1 value","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What does data look like for S Variables?**\n\n---","metadata":{}},{"cell_type":"code","source":"# display data for columns starting with S\ntrain_df.filter(like='S_', axis=1)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:36.811093Z","iopub.execute_input":"2022-07-24T04:00:36.811558Z","iopub.status.idle":"2022-07-24T04:00:36.865632Z","shell.execute_reply.started":"2022-07-24T04:00:36.811482Z","shell.execute_reply":"2022-07-24T04:00:36.864409Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observations:**\n\n* S_2 variable has date value\n* S_3, S_9 and S_27 variable appears to have missing value","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What does data look like for P Variables?**\n\n---","metadata":{}},{"cell_type":"code","source":"# display data for columns starting with P\ntrain_df.filter(like='P_', axis=1)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:36.868536Z","iopub.execute_input":"2022-07-24T04:00:36.869288Z","iopub.status.idle":"2022-07-24T04:00:36.883053Z","shell.execute_reply.started":"2022-07-24T04:00:36.869248Z","shell.execute_reply":"2022-07-24T04:00:36.882252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observations:**\n\n* P_2 variable appears to have values towards 1\n* P_4 variable appears to have values towards 0","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What does data look like for B Variables?**\n\n---","metadata":{}},{"cell_type":"code","source":"# display data for columns starting with B\ntrain_df.filter(like='B_', axis=1)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:36.883887Z","iopub.execute_input":"2022-07-24T04:00:36.884162Z","iopub.status.idle":"2022-07-24T04:00:36.947549Z","shell.execute_reply.started":"2022-07-24T04:00:36.884135Z","shell.execute_reply":"2022-07-24T04:00:36.946731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observations:**\n\n* B_39 and B_42 appears to have missing value\n* B_31 appears to have majority of value as 1","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What does data look like for R Variables?**\n\n---","metadata":{}},{"cell_type":"code","source":"# display data for columns starting with R\ntrain_df.filter(like='R_', axis=1)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:36.948846Z","iopub.execute_input":"2022-07-24T04:00:36.949172Z","iopub.status.idle":"2022-07-24T04:00:36.993756Z","shell.execute_reply.started":"2022-07-24T04:00:36.949142Z","shell.execute_reply":"2022-07-24T04:00:36.992478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observations:**\n\n* R_9 and R_26 appears to have missing value\n* R_27 appears to have majority value as 1","metadata":{}},{"cell_type":"markdown","source":"---\nNeed to load **train_labels.csv** for customer_ID with target label for Default as True(1) and False(0) (Default=1 and Not Default=0)\n\n---","metadata":{}},{"cell_type":"code","source":"# Loading dataset train_labels.csv\ntrain_label_df = pd.read_csv('../input/amex-default-prediction/train_labels.csv')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:36.995613Z","iopub.execute_input":"2022-07-24T04:00:36.995922Z","iopub.status.idle":"2022-07-24T04:00:37.945984Z","shell.execute_reply.started":"2022-07-24T04:00:36.995894Z","shell.execute_reply":"2022-07-24T04:00:37.944781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# get shape of dataframe\nprint('Shape of dataset is:', train_label_df.shape)\n\n# print summary of dataframe\ntrain_label_df.info()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:37.947596Z","iopub.execute_input":"2022-07-24T04:00:37.947919Z","iopub.status.idle":"2022-07-24T04:00:38.012507Z","shell.execute_reply.started":"2022-07-24T04:00:37.947891Z","shell.execute_reply":"2022-07-24T04:00:38.011301Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observations:**\n\n* There are total 458,913 entries for target label with customer_ID\n* There is variable (column) customer_ID which has dtype as object and variable (column) target which has dtype as int64","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What does data look like for train_labels.csv?**\n\n---","metadata":{}},{"cell_type":"code","source":"# print first 10 rows of dataframe\ntrain_label_df.head(10)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:38.013856Z","iopub.execute_input":"2022-07-24T04:00:38.014469Z","iopub.status.idle":"2022-07-24T04:00:38.026566Z","shell.execute_reply.started":"2022-07-24T04:00:38.014430Z","shell.execute_reply":"2022-07-24T04:00:38.025372Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print last 10 rows of dataframe\ntrain_label_df.tail(10)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:38.028066Z","iopub.execute_input":"2022-07-24T04:00:38.029186Z","iopub.status.idle":"2022-07-24T04:00:38.044843Z","shell.execute_reply.started":"2022-07-24T04:00:38.029148Z","shell.execute_reply":"2022-07-24T04:00:38.042831Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Q: What is the distribution of target label?**\n\n---","metadata":{}},{"cell_type":"code","source":"# distribution count of target variable\ntrain_label_df['target'].value_counts()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:38.048502Z","iopub.execute_input":"2022-07-24T04:00:38.048890Z","iopub.status.idle":"2022-07-24T04:00:38.060038Z","shell.execute_reply.started":"2022-07-24T04:00:38.048857Z","shell.execute_reply":"2022-07-24T04:00:38.059012Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot percentage of distribution count for target variable\nfig = px.pie(train_label_df, names='target')\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:38.061249Z","iopub.execute_input":"2022-07-24T04:00:38.061810Z","iopub.status.idle":"2022-07-24T04:00:39.330371Z","shell.execute_reply.started":"2022-07-24T04:00:38.061771Z","shell.execute_reply":"2022-07-24T04:00:39.329241Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Sample dataset has target=1(Default) around 25.9% and target=0(Not Default) around 74.1% (data imbalance exist).","metadata":{}},{"cell_type":"markdown","source":"---\nUsing nrows option to load first 100k rows from **test_data.csv** dataset file.\n\n---","metadata":{}},{"cell_type":"code","source":"# Loading dataset test_data.csv\ntest_df = pd.read_csv('../input/amex-default-prediction/test_data.csv', nrows=100000)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:39.331786Z","iopub.execute_input":"2022-07-24T04:00:39.332127Z","iopub.status.idle":"2022-07-24T04:00:46.663261Z","shell.execute_reply.started":"2022-07-24T04:00:39.332095Z","shell.execute_reply":"2022-07-24T04:00:46.662122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# get shape of dataframe\nprint('Shape of dataset is:', test_df.shape)\n\n# print summary of dataframe\n#test_df.info(verbose=True)\ntest_df.info()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:46.664867Z","iopub.execute_input":"2022-07-24T04:00:46.665358Z","iopub.status.idle":"2022-07-24T04:00:46.687696Z","shell.execute_reply.started":"2022-07-24T04:00:46.665322Z","shell.execute_reply":"2022-07-24T04:00:46.686242Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* There are 185 variables(Columns) as dtype float64, 1 variable(Column) as dtype int64 and 4 variables(Columns) as dtype object, same structure as train_data.csv","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What does data look like for test_data.csv?**\n\n---","metadata":{}},{"cell_type":"code","source":"# print first 10 rows of dataframe\ntest_df.head(10)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:46.689505Z","iopub.execute_input":"2022-07-24T04:00:46.689987Z","iopub.status.idle":"2022-07-24T04:00:46.723022Z","shell.execute_reply.started":"2022-07-24T04:00:46.689940Z","shell.execute_reply":"2022-07-24T04:00:46.721821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\nLoading **sample_submission.csv** file just to get a glance :)\n\n---","metadata":{}},{"cell_type":"code","source":"# Loading sample_submission.csv\nsample_submission_df = pd.read_csv('../input/amex-default-prediction/sample_submission.csv')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:46.727035Z","iopub.execute_input":"2022-07-24T04:00:46.727410Z","iopub.status.idle":"2022-07-24T04:00:48.397098Z","shell.execute_reply.started":"2022-07-24T04:00:46.727374Z","shell.execute_reply":"2022-07-24T04:00:48.396248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# get shape of dataframe\nprint('Shape of dataset is:', sample_submission_df.shape)\n\n# print summary of dataframe\nsample_submission_df.info()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:48.398687Z","iopub.execute_input":"2022-07-24T04:00:48.399620Z","iopub.status.idle":"2022-07-24T04:00:48.514423Z","shell.execute_reply.started":"2022-07-24T04:00:48.399581Z","shell.execute_reply":"2022-07-24T04:00:48.513258Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Q: What does data look like for sample_submission.csv?**\n\n---","metadata":{}},{"cell_type":"code","source":"# print first 10 rows of dataframe\nsample_submission_df.head(10)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:48.516072Z","iopub.execute_input":"2022-07-24T04:00:48.516561Z","iopub.status.idle":"2022-07-24T04:00:48.528062Z","shell.execute_reply.started":"2022-07-24T04:00:48.516499Z","shell.execute_reply":"2022-07-24T04:00:48.526746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Data Definition/Data Description**\n\n---","metadata":{}},{"cell_type":"markdown","source":"The dataset contains aggregated profile features for each customer at each statement date.\n\nFeatures are anonymized and normalized, and fall into the following general categories:\n\n* **D_* = Delinquency variables**\n\n* **S_* = Spend variables**\n\n* **P_* = Payment variables**\n\n* **B_* = Balance variables**\n\n* **R_* = Risk variables**\n\nwith the following features being categorical:\n\n**['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']**","metadata":{}},{"cell_type":"code","source":"# check for correlation threshold between D, S, P, B and R variables\n# corr = train_df.corr(method='pearson')\n# threshold = abs(corr)\n# result = threshold[threshold>0.80]\n# pd.set_option('display.max_columns', None)\n# pd.set_option(\"max_rows\", None)\n# result","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:48.537839Z","iopub.execute_input":"2022-07-24T04:00:48.538195Z","iopub.status.idle":"2022-07-24T04:00:48.542885Z","shell.execute_reply.started":"2022-07-24T04:00:48.538164Z","shell.execute_reply":"2022-07-24T04:00:48.541579Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n# D_* = Delinquency variables\n\n---","metadata":{}},{"cell_type":"markdown","source":"---\n**Summary of Dataframe for Delinquency variables**\n\n---","metadata":{}},{"cell_type":"code","source":"#display summary of DataFrame for D variables\ndisplay(train_df[D_columns1].info(verbose=True))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:48.544681Z","iopub.execute_input":"2022-07-24T04:00:48.545773Z","iopub.status.idle":"2022-07-24T04:00:48.615842Z","shell.execute_reply.started":"2022-07-24T04:00:48.545725Z","shell.execute_reply":"2022-07-24T04:00:48.614612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#display summary of DataFrame for D variables\ndisplay(train_df[D_columns2].info(verbose=True))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:48.617407Z","iopub.execute_input":"2022-07-24T04:00:48.617823Z","iopub.status.idle":"2022-07-24T04:00:48.658262Z","shell.execute_reply.started":"2022-07-24T04:00:48.617788Z","shell.execute_reply":"2022-07-24T04:00:48.656961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* D_63, D_64, D_66, D_68, D_114, D_116, D_117, D_120, D_126 are variables with categorical value but dtype is float64 excluding dtype for D_63 and D_64 variable which is object, rest of the variable are of dtype float64","metadata":{}},{"cell_type":"markdown","source":"---\n\n**Descriptive statistics for Delinquency variables**\n\n---","metadata":{}},{"cell_type":"code","source":"#display descriptive statistics\ntrain_df[D_columns1].describe(include='all').T.sort_values('count', ascending=False)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:48.659542Z","iopub.execute_input":"2022-07-24T04:00:48.659894Z","iopub.status.idle":"2022-07-24T04:00:49.090497Z","shell.execute_reply.started":"2022-07-24T04:00:48.659863Z","shell.execute_reply":"2022-07-24T04:00:49.089387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#display descriptive statistics\ntrain_df[D_columns2].describe(include='all').T.sort_values('count', ascending=False)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:49.092270Z","iopub.execute_input":"2022-07-24T04:00:49.092611Z","iopub.status.idle":"2022-07-24T04:00:49.401327Z","shell.execute_reply.started":"2022-07-24T04:00:49.092574Z","shell.execute_reply":"2022-07-24T04:00:49.400254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Following D variables(columns) around 29 of them appears to have significant amount of missing values as per this sample dataset.\n\n    * \nD_42,D_43,D_46,D_48,D_49,D_50,D_53,D_56,D_61,D_62,D_66,D_73,D_76,D_77,D_82,D_88,D_87,D_105,D_106,D_108,D_110,D_111,D_132,D_136,D_138,D_135,D_134,D_137,D_142 \n* Majority of D Variables appears to have max value as 1.00\n* There are D variables with outliers\n* There are D variables with skewed distribution\n\n","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the missing value plot for D Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot of missing values\nmsno.bar(train_df[D_columns[:50]],figsize=(25,10),sort=\"ascending\", color=\"tomato\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:49.402796Z","iopub.execute_input":"2022-07-24T04:00:49.403158Z","iopub.status.idle":"2022-07-24T04:00:54.757158Z","shell.execute_reply.started":"2022-07-24T04:00:49.403127Z","shell.execute_reply":"2022-07-24T04:00:54.755998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows D variables which are having missing value and which are not having missing value\n\n    * e.g., D_87,D_88 or D_73 variables has significant amount of missing value","metadata":{}},{"cell_type":"code","source":"#plot of missing values\nmsno.bar(train_df[D_columns[50:]],figsize=(25,10),sort=\"ascending\", color=\"tomato\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:54.758903Z","iopub.execute_input":"2022-07-24T04:00:54.759612Z","iopub.status.idle":"2022-07-24T04:00:58.052963Z","shell.execute_reply.started":"2022-07-24T04:00:54.759557Z","shell.execute_reply":"2022-07-24T04:00:58.051761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows D variables which are having missing value and which are not having missing value\n\n    * e.g., D_108, D_110 or D_111 variables has significant amount of missing value","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the missing value relationship for D Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot of how missing values are related\nmsno.heatmap(train_df[D_columns],figsize=(75,60))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:00:58.054728Z","iopub.execute_input":"2022-07-24T04:00:58.055078Z","iopub.status.idle":"2022-07-24T04:01:10.653998Z","shell.execute_reply.started":"2022-07-24T04:00:58.055045Z","shell.execute_reply":"2022-07-24T04:01:10.652804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows all the D variable which has missing value and also how they are related to other D variable for missing value.\n\n    * e.g., missing value in D_49 variable is related to missing value in D_132 variable as they have relationship value of 1","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the correlation for D Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot correlation between D variables\ncorr = train_df[D_columns].corr(method='pearson')\nplt.figure(figsize=(75,60))\n# mask option can be used to not show values <=0.90\n#sns.heatmap(corr,vmax=.8,linewidth=.01, square = True, annot = True,cmap='YlGnBu',linecolor ='black',mask = (np.abs(corr) <= 0.90))\nsns.heatmap(corr,vmax=.8,linewidth=.01, square = True, annot = True,cmap='Oranges',linecolor ='black')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:01:10.655652Z","iopub.execute_input":"2022-07-24T04:01:10.656001Z","iopub.status.idle":"2022-07-24T04:01:45.943592Z","shell.execute_reply.started":"2022-07-24T04:01:10.655968Z","shell.execute_reply":"2022-07-24T04:01:45.942293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* D_42 has correlation of 1.0 with D_110 and D_111\n* D_58 has correlation of 0.92 with D_74 and 0.93 with D_75\n* D_62 has correlation of 1.0 with D_77\n* D_74 has correlation of 0.99 with D_75\n* D_73 has correlation of -1.0 with D_88\n* D_139 has correlation of 1.0 with D_143 and D_141\n* D_141 has correlation of 1.0 with D_143\n* D_132 has correlation of 0.92 with D_132\n* D_48 has correlation of 0.84 with D_55 and 0.86 with D_61\n* D_79 has correlation of 0.89 with D_131","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the distribution for D Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot distribution for D variables\nnrows=19\nncols=5\nfig, axes = plt.subplots(nrows, ncols, figsize=(30,50)) \nd_columns= ['D_39', 'D_41', 'D_42', 'D_43', 'D_44', 'D_45', 'D_46', 'D_47', 'D_48', 'D_49', 'D_50', 'D_51', 'D_52', 'D_53', 'D_54', 'D_55', 'D_56', 'D_58', 'D_59', 'D_60', 'D_61', 'D_62', 'D_63','D_65', 'D_66','D_68','D_69', 'D_70', 'D_71', 'D_72', 'D_73', 'D_74', 'D_75', 'D_76', 'D_77', 'D_78', 'D_79', 'D_80', 'D_81', 'D_82', 'D_83', 'D_84', 'D_86', 'D_87', 'D_88', 'D_89', 'D_91', 'D_92', 'D_93', 'D_94', 'D_96','D_102', 'D_103', 'D_104', 'D_105', 'D_106', 'D_107', 'D_108', 'D_109', 'D_110', 'D_111', 'D_112', 'D_113', 'D_114','D_115', 'D_116', 'D_117','D_118', 'D_119', 'D_120', 'D_121', 'D_122', 'D_123', 'D_124', 'D_125','D_126','D_127', 'D_128', 'D_129', 'D_130', 'D_131', 'D_132', 'D_133', 'D_134', 'D_135', 'D_136', 'D_137', 'D_138', 'D_139', 'D_140', 'D_141', 'D_142', 'D_143', 'D_144', 'D_145']\naxes = axes.flatten()   \ntrain_df_sample = train_df.sample(frac =.1)\nfor ax,col in zip(axes,d_columns):\n    ax.hist(train_df_sample[col],color=\"orange\")\n    ax.set_title(col)\n    \nplt.tight_layout()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:01:45.945151Z","iopub.execute_input":"2022-07-24T04:01:45.945526Z","iopub.status.idle":"2022-07-24T04:01:59.918787Z","shell.execute_reply.started":"2022-07-24T04:01:45.945482Z","shell.execute_reply":"2022-07-24T04:01:59.917583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows distribution for D variables and skewness in distribution\n\n    * There are many variables which appear to be having extreme end of value","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the outlier value plot for D Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot outlier for D variables\nnrows = 19\nncols = 5\nfig, axes = plt.subplots(figsize=(30,50)) \nd_columns= ['D_39', 'D_41', 'D_42', 'D_43', 'D_44', 'D_45', 'D_46', 'D_47', 'D_48', 'D_49', 'D_50', 'D_51', 'D_52', 'D_53', 'D_54', 'D_55', 'D_56', 'D_58', 'D_59', 'D_60', 'D_61', 'D_62', 'D_65', 'D_66','D_68','D_69', 'D_70', 'D_71', 'D_72', 'D_73', 'D_74', 'D_75', 'D_76', 'D_77', 'D_78', 'D_79', 'D_80', 'D_81', 'D_82', 'D_83', 'D_84', 'D_86', 'D_87', 'D_88', 'D_89', 'D_91', 'D_92', 'D_93', 'D_94', 'D_96','D_102', 'D_103', 'D_104', 'D_105', 'D_106', 'D_107', 'D_108', 'D_109', 'D_110', 'D_111', 'D_112', 'D_113', 'D_114','D_115', 'D_116', 'D_117','D_118', 'D_119', 'D_120', 'D_121', 'D_122', 'D_123', 'D_124', 'D_125','D_126','D_127', 'D_128', 'D_129', 'D_130', 'D_131', 'D_132', 'D_133', 'D_134', 'D_135', 'D_136', 'D_137', 'D_138', 'D_139', 'D_140', 'D_141', 'D_142', 'D_143', 'D_144', 'D_145']\ntrain_df_sample = train_df.sample(frac =.1)\nfor i, col in enumerate(d_columns):\n    ax=fig.add_subplot(nrows, ncols, i+1)\n    sns.boxplot(x=train_df_sample[col], ax=ax,palette=\"Oranges\")\n    \nfig.tight_layout()  \nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:01:59.920499Z","iopub.execute_input":"2022-07-24T04:01:59.921673Z","iopub.status.idle":"2022-07-24T04:02:10.284677Z","shell.execute_reply.started":"2022-07-24T04:01:59.921623Z","shell.execute_reply":"2022-07-24T04:02:10.283507Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observations:**\n\n* Above box plot shows variability for D variables and outliers for the same","metadata":{}},{"cell_type":"markdown","source":"---\n# S_* = Spend variables\n\n---","metadata":{}},{"cell_type":"markdown","source":"---\n**Summary of Dataframe for Spend variables**\n\n---","metadata":{}},{"cell_type":"code","source":"#display summary of DataFrame for S variables\ndisplay(train_df[S_columns].info(verbose=True))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:10.286368Z","iopub.execute_input":"2022-07-24T04:02:10.286730Z","iopub.status.idle":"2022-07-24T04:02:10.323998Z","shell.execute_reply.started":"2022-07-24T04:02:10.286697Z","shell.execute_reply":"2022-07-24T04:02:10.322951Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* S_2 is of dtype object with date value and rest are of dtype float64","metadata":{}},{"cell_type":"markdown","source":"---\n**Descriptive statistics for Spend variables**\n\n---","metadata":{}},{"cell_type":"code","source":"#display descriptive statistics for S variables\ntrain_df[S_columns].describe().T.sort_values('count', ascending=False)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:10.325680Z","iopub.execute_input":"2022-07-24T04:02:10.326008Z","iopub.status.idle":"2022-07-24T04:02:10.498010Z","shell.execute_reply.started":"2022-07-24T04:02:10.325976Z","shell.execute_reply":"2022-07-24T04:02:10.496818Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Following S variables(columns) around 4 of them appears to have higher missing values as per this sample dataset.\n\n    * S_3, S_7, S_27, S_9\n* There are S variables with outliers\n* There are S variables with skewed distribution","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the missing value plot for S Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot of missing values for S variables\nmsno.bar(train_df[S_columns],figsize=(20,8), sort=\"ascending\", color=(0.25,0.75,0.25))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:10.499540Z","iopub.execute_input":"2022-07-24T04:02:10.501082Z","iopub.status.idle":"2022-07-24T04:02:11.923092Z","shell.execute_reply.started":"2022-07-24T04:02:10.501033Z","shell.execute_reply":"2022-07-24T04:02:11.921881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows S variables which are having missing value and which are not having missing value\n\n    * e.g., S_9 or S_27 variable appears to have more missing value","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the missing value relationship for S Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot of how missing values for S variables are related\nmsno.heatmap(train_df[S_columns],figsize=(15,10))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:11.924400Z","iopub.execute_input":"2022-07-24T04:02:11.925264Z","iopub.status.idle":"2022-07-24T04:02:12.379035Z","shell.execute_reply.started":"2022-07-24T04:02:11.925220Z","shell.execute_reply":"2022-07-24T04:02:12.377916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows all the S variable which has missing value and also how they are related to other S variable for missing value.\n\n    * e.g., missing value in S_3 variable is related to missing value in S_7 variable as they have relationship value of 1","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the correlation for S Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot correlation between S variables\ncorr = train_df[S_columns].corr(method='pearson')\nplt.figure(figsize=(30,25))\n# mask option can be used to not show values <=0.90\n#sns.heatmap(corr,vmax=.8,linewidth=.01, square = True, annot = True,cmap='YlGnBu',linecolor ='black',mask = (np.abs(corr) <= 0.90))\nsns.heatmap(corr,vmax=.8,linewidth=.01, square = True, annot = True,cmap='Greens',linecolor ='black')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:12.380482Z","iopub.execute_input":"2022-07-24T04:02:12.380869Z","iopub.status.idle":"2022-07-24T04:02:14.599968Z","shell.execute_reply.started":"2022-07-24T04:02:12.380837Z","shell.execute_reply":"2022-07-24T04:02:14.598810Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* S_3 has correlation of 0.91 with S_7\n* S_22 has correlation of 0.94 with S_24","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the distribution for S Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot distribution for S variables\nfig, axes = plt.subplots(nrows = 7, ncols = 3, figsize=(30,20)) \ns_columns =['S_3', 'S_5', 'S_6', 'S_7', 'S_8', 'S_9', 'S_11', 'S_12', 'S_13', 'S_15', 'S_16', 'S_17', 'S_18', 'S_19', 'S_20', 'S_22', 'S_23', 'S_24', 'S_25', 'S_26', 'S_27']\naxes = axes.flatten()   \n#train_df_sample = train_df.sample(frac =.1)\nfor ax,col in zip(axes,s_columns):\n    ax.hist(train_df_sample[col], color=\"g\")\n    ax.set_title(col)\n    \nplt.tight_layout()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:14.601420Z","iopub.execute_input":"2022-07-24T04:02:14.602278Z","iopub.status.idle":"2022-07-24T04:02:18.232580Z","shell.execute_reply.started":"2022-07-24T04:02:14.602236Z","shell.execute_reply":"2022-07-24T04:02:18.231732Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows distribution for S variables and skewness in distribution\n\n    * There are many variables which appear to be having extreme end of value","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the outlier value plot for S Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot outlier for S variables\nnrows = 7\nncols = 3\nfig, axes = plt.subplots(figsize=(30,20)) \ns_columns =['S_3', 'S_5', 'S_6', 'S_7', 'S_8', 'S_9', 'S_11', 'S_12', 'S_13', 'S_15', 'S_16', 'S_17', 'S_18', 'S_19', 'S_20', 'S_22', 'S_23', 'S_24', 'S_25', 'S_26', 'S_27']\n#train_df_sample = train_df.sample(frac =.1)\nfor i, col in enumerate(s_columns):\n    ax=fig.add_subplot(nrows, ncols, i+1)\n    sns.boxplot(x=train_df_sample[col], ax=ax, palette=\"Set2\")\n    \nfig.tight_layout()  \nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:18.233725Z","iopub.execute_input":"2022-07-24T04:02:18.234622Z","iopub.status.idle":"2022-07-24T04:02:20.625290Z","shell.execute_reply.started":"2022-07-24T04:02:18.234588Z","shell.execute_reply":"2022-07-24T04:02:20.624413Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observations:**\n\n* Above box plot shows variability for S variables and outliers for the same","metadata":{}},{"cell_type":"markdown","source":"---\n# P_* = Payment variables\n\n---","metadata":{}},{"cell_type":"markdown","source":"---\n**Summary of Dataframe for Payment variables**\n\n---","metadata":{}},{"cell_type":"code","source":"#display summary of DataFrame for P variables\ndisplay(train_df[P_columns].info(verbose=True))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:20.626346Z","iopub.execute_input":"2022-07-24T04:02:20.626927Z","iopub.status.idle":"2022-07-24T04:02:20.643735Z","shell.execute_reply.started":"2022-07-24T04:02:20.626891Z","shell.execute_reply":"2022-07-24T04:02:20.642565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* All P variables are of dtype float64\n","metadata":{}},{"cell_type":"markdown","source":"---\n**Descriptive statistics for Payment variables**\n\n---","metadata":{}},{"cell_type":"code","source":"#display descriptive statistics for P variables\ntrain_df[P_columns].describe().T.sort_values('count', ascending=False)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:20.645481Z","iopub.execute_input":"2022-07-24T04:02:20.646189Z","iopub.status.idle":"2022-07-24T04:02:20.686304Z","shell.execute_reply.started":"2022-07-24T04:02:20.646144Z","shell.execute_reply":"2022-07-24T04:02:20.685191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n* All P variable appears to have outliers\n* P_2 variable appears to have slight skewed distribution\n* P_3 variable appears to have normal distribution","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the missing value plot for P Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot of missing values for P variables\nmsno.bar(train_df[P_columns],figsize=(8,5), sort=\"ascending\", color=(0.50,0.25,1.0))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:20.687571Z","iopub.execute_input":"2022-07-24T04:02:20.687883Z","iopub.status.idle":"2022-07-24T04:02:21.191879Z","shell.execute_reply.started":"2022-07-24T04:02:20.687855Z","shell.execute_reply":"2022-07-24T04:02:21.190566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows P variables which are having missing value and which are not having missing value\n\n    * e.g., P_3 or P_3 variable appears to have missing value","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the missing value relationship for P Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot of how missing values are related for P variables\nmsno.heatmap(train_df[P_columns],figsize=(8,5))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:21.193661Z","iopub.execute_input":"2022-07-24T04:02:21.195920Z","iopub.status.idle":"2022-07-24T04:02:21.472412Z","shell.execute_reply.started":"2022-07-24T04:02:21.195857Z","shell.execute_reply":"2022-07-24T04:02:21.471555Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows all the P variable which has missing value and also how they are related to other P variable for missing value.\n\n    * There are only 2 variable in P out of 3 variable which has missing value and they are not related","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the correlation for P Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot correlation between P variables\ncorr = train_df[P_columns].corr(method='pearson')\nplt.figure(figsize=(10,12))\nsns.heatmap(corr,vmax=.8,linewidth=.01, square = True, annot = True,cmap='Purples',linecolor ='black')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:21.473800Z","iopub.execute_input":"2022-07-24T04:02:21.474357Z","iopub.status.idle":"2022-07-24T04:02:21.728820Z","shell.execute_reply.started":"2022-07-24T04:02:21.474322Z","shell.execute_reply":"2022-07-24T04:02:21.727783Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* None of the P variables appears to be highly correlated.","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the distribution for P Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot distribution for P variables\nfig, axes = plt.subplots(nrows = 1, ncols = 3, figsize=(15,5)) \naxes = axes.flatten()   \n#train_df_sample = train_df.sample(frac =.1)\nfor ax,col in zip(axes,P_columns):\n    ax.hist(train_df_sample[col], color=\"purple\")\n    ax.set_title(col)\n    \nplt.tight_layout()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:21.730269Z","iopub.execute_input":"2022-07-24T04:02:21.730632Z","iopub.status.idle":"2022-07-24T04:02:22.257566Z","shell.execute_reply.started":"2022-07-24T04:02:21.730598Z","shell.execute_reply":"2022-07-24T04:02:22.256286Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* P_2 variable appears to have skewed distribution\n* P_3 variable appears to have normal distribution\n* P_4 variable appears to have bimodal distribution","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the outlier value plot for P Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot outlier for P variables\nnrows = 1\nncols = 3\nfig, axes = plt.subplots(figsize=(15,5)) \n#train_df_sample = train_df.sample(frac =.1)\nfor i, col in enumerate(P_columns):\n    ax=fig.add_subplot(nrows, ncols, i+1)\n    sns.boxplot(x=train_df_sample[col], ax=ax,palette=\"Purples\")\n    \nfig.tight_layout()  \nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:22.259160Z","iopub.execute_input":"2022-07-24T04:02:22.259933Z","iopub.status.idle":"2022-07-24T04:02:22.743978Z","shell.execute_reply.started":"2022-07-24T04:02:22.259885Z","shell.execute_reply":"2022-07-24T04:02:22.742775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above box plot shows variability for P variables and outliers for the same","metadata":{}},{"cell_type":"markdown","source":"---\n# B_* = Balance variables\n\n---","metadata":{}},{"cell_type":"markdown","source":"---\n**Summary of Dataframe for Balance variables**\n\n---","metadata":{}},{"cell_type":"code","source":"#display summary of DataFrame for B variables\ndisplay(train_df[B_columns].info(verbose=True))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:22.745359Z","iopub.execute_input":"2022-07-24T04:02:22.745718Z","iopub.status.idle":"2022-07-24T04:02:22.779943Z","shell.execute_reply.started":"2022-07-24T04:02:22.745685Z","shell.execute_reply":"2022-07-24T04:02:22.778788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* B_31 is of dtype int64, B_30 and B_38 as dtype float64 even though they both have categorical type value and rest are of dtype float64","metadata":{}},{"cell_type":"markdown","source":"---\n**Descriptive statistics for Balance variables**\n\n---","metadata":{}},{"cell_type":"code","source":"#display descriptive statistics for B variables\ntrain_df[B_columns].describe(include='all').T.sort_values('count', ascending=False)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:22.781710Z","iopub.execute_input":"2022-07-24T04:02:22.782156Z","iopub.status.idle":"2022-07-24T04:02:23.060635Z","shell.execute_reply.started":"2022-07-24T04:02:22.782101Z","shell.execute_reply":"2022-07-24T04:02:23.059574Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Following B variables(columns) around 4 of them appears to have significant amount of missing values as per this sample dataset.\n\n    * B_17, B_29, B_42, B_39\n* There are B variables with outliers\n* There are B variables with skewed distribution","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the missing value plot for B Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot of missing values for B variables\nmsno.bar(train_df[B_columns],figsize=(20,8), sort=\"ascending\", color=\"dodgerblue\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:23.061933Z","iopub.execute_input":"2022-07-24T04:02:23.062266Z","iopub.status.idle":"2022-07-24T04:02:25.919979Z","shell.execute_reply.started":"2022-07-24T04:02:23.062226Z","shell.execute_reply":"2022-07-24T04:02:25.918860Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows B variables which are having missing value and which are not having missing value\n\n    * e.g., B_39, B_42 or B_29 variable appears to have significant amount of missing value","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the missing value relationship for B Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot of how missing values are related for B variables\nmsno.heatmap(train_df[B_columns],figsize=(30,20))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:25.921131Z","iopub.execute_input":"2022-07-24T04:02:25.922071Z","iopub.status.idle":"2022-07-24T04:02:27.103765Z","shell.execute_reply.started":"2022-07-24T04:02:25.922035Z","shell.execute_reply":"2022-07-24T04:02:27.102597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows all the B variable which has missing value and also how they are related to other B variable for missing value.\n\n    * e.g., missing value in B_15 variable is related to missing value in B_25 variable as they have relationship value of 1","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the correlation for B Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot correlation between B variables\ncorr = train_df[B_columns].corr(method='pearson')\nplt.figure(figsize=(60,50))\n# mask option can be used to not show values <=0.90\n#sns.heatmap(corr,vmax=.8,linewidth=.01, square = True, annot = True,cmap='Blues',linecolor ='black',mask = (np.abs(corr) <= 0.90))\nsns.heatmap(corr,vmax=.8,linewidth=.01, square = True, annot = True,cmap='Blues',linecolor ='black')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:27.105448Z","iopub.execute_input":"2022-07-24T04:02:27.105902Z","iopub.status.idle":"2022-07-24T04:02:34.540966Z","shell.execute_reply.started":"2022-07-24T04:02:27.105858Z","shell.execute_reply":"2022-07-24T04:02:34.539673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* B_1 has correlation of 0.99 with B_37\n* B_1 has correlation of 1.0 with B_11\n* B_2 has correlation of 0.91 with B_33\n* B_2 has correlation of 0.85 with B_18\n* B_11 has correlation of 0.99 with B_37\n* B_12 has correlation of 0.91 with B_13\n* B_14 has correlation of 0.89 with B_15\n* B_7 has correlation of 1.0 with B_23","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the distribution for B Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot distribution for B variables\nfig, axes = plt.subplots(nrows = 8, ncols = 5, figsize=(30,35)) \naxes = axes.flatten()   \n#train_df_sample = train_df.sample(frac =.1)\nfor ax,col in zip(axes,B_columns):\n    ax.hist(train_df_sample[col],color=\"b\")\n    ax.set_title(col)\n    \nplt.tight_layout()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:34.542696Z","iopub.execute_input":"2022-07-24T04:02:34.543106Z","iopub.status.idle":"2022-07-24T04:02:40.876038Z","shell.execute_reply.started":"2022-07-24T04:02:34.543067Z","shell.execute_reply":"2022-07-24T04:02:40.874805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows distribution for B variables and skewness in distribution\n\n    * There are many variables which appear to be having extreme end of value","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the outlier value plot for B Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot outlier for B variables\nnrows = 8\nncols = 5\nfig, axes = plt.subplots(figsize=(30,35)) \n#train_df_sample = train_df.sample(frac =.1)\nfor i, col in enumerate(B_columns):\n    ax=fig.add_subplot(nrows, ncols, i+1)\n    sns.boxplot(x=train_df_sample[col], ax=ax, palette=\"Blues\")\n    \nfig.tight_layout()  \nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:40.877471Z","iopub.execute_input":"2022-07-24T04:02:40.877862Z","iopub.status.idle":"2022-07-24T04:02:45.601230Z","shell.execute_reply.started":"2022-07-24T04:02:40.877827Z","shell.execute_reply":"2022-07-24T04:02:45.600191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above box plot shows variability for B variables and outliers for the same","metadata":{}},{"cell_type":"markdown","source":"---\n# R_* = Risk variables\n\n---","metadata":{}},{"cell_type":"markdown","source":"---\n**Summary of Dataframe for Risk variables**\n\n---","metadata":{}},{"cell_type":"code","source":"#display summary of DataFrame for R variables\ndisplay(train_df[R_columns].info(verbose=True))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:45.602690Z","iopub.execute_input":"2022-07-24T04:02:45.603113Z","iopub.status.idle":"2022-07-24T04:02:45.632016Z","shell.execute_reply.started":"2022-07-24T04:02:45.603068Z","shell.execute_reply":"2022-07-24T04:02:45.631234Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* All R variable are of dtype float64\n* R_9 variable seems to be almost empty\n","metadata":{}},{"cell_type":"markdown","source":"---\n**Descriptive statistics for Risk variables**\n\n---","metadata":{}},{"cell_type":"code","source":"#display descriptive statistics for R variables\ntrain_df[R_columns].describe().T.sort_values('count', ascending=False)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:45.633241Z","iopub.execute_input":"2022-07-24T04:02:45.634193Z","iopub.status.idle":"2022-07-24T04:02:45.847653Z","shell.execute_reply.started":"2022-07-24T04:02:45.634160Z","shell.execute_reply":"2022-07-24T04:02:45.846375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Following R variables(columns) around 2 of them appears to have significant amount of missing values as per this sample dataset.\n\n    * R_26, R_9\n    \n* There are R variables with outliers\n* There are R variables with skewed distribution","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the missing value plot for R Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot of missing values for R variables\nmsno.bar(train_df[R_columns],figsize=(15,8),sort=\"ascending\", color=(0.75, 0.25, 0.25))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:45.849349Z","iopub.execute_input":"2022-07-24T04:02:45.850057Z","iopub.status.idle":"2022-07-24T04:02:48.033800Z","shell.execute_reply.started":"2022-07-24T04:02:45.850001Z","shell.execute_reply":"2022-07-24T04:02:48.032565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows R variables which are having missing value and which are not having missing value\n\n    * e.g., R_9 or R_26 variable appears to have quite a lot of missing value","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the missing value relationship for R Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot of how missing values are related for R variables\nmsno.heatmap(train_df[R_columns],figsize=(8,5))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:48.035291Z","iopub.execute_input":"2022-07-24T04:02:48.035689Z","iopub.status.idle":"2022-07-24T04:02:48.343424Z","shell.execute_reply.started":"2022-07-24T04:02:48.035656Z","shell.execute_reply":"2022-07-24T04:02:48.342340Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows all the R variable which has missing value and also how they are related to other R variable for missing value.\n\n    * * There are only 4 variable in R variable which has missing value and they are not related","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the correlation for R Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot correlation between R variables\ncorr = train_df[R_columns].corr(method='pearson')\nplt.figure(figsize=(60,50))\n# mask option can be used to not show values <=0.90\n#sns.heatmap(corr,vmax=.8,linewidth=.01, square = True, annot = True,cmap='Reds',linecolor ='black',mask = (np.abs(corr) <= 0.90))\nsns.heatmap(corr,vmax=.8,linewidth=.01, square = True, annot = True,cmap='Reds',linecolor ='black')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:48.344941Z","iopub.execute_input":"2022-07-24T04:02:48.345276Z","iopub.status.idle":"2022-07-24T04:02:52.282231Z","shell.execute_reply.started":"2022-07-24T04:02:48.345245Z","shell.execute_reply":"2022-07-24T04:02:52.281388Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* R_4 has correlation of 0.79 with R_2\n* R_5 has correlation of 0.8 with R_8","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the distribution for R Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot distribution for R variables\nfig, axes = plt.subplots(nrows = 6, ncols = 5, figsize=(30,25)) \naxes = axes.flatten()   \n#train_df_sample = train_df.sample(frac =.1)\nfor ax,col in zip(axes,R_columns):\n    ax.hist(train_df_sample[col], color=\"r\")\n    ax.set_title(col)\n    \nplt.tight_layout()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:52.283616Z","iopub.execute_input":"2022-07-24T04:02:52.284198Z","iopub.status.idle":"2022-07-24T04:02:56.486691Z","shell.execute_reply.started":"2022-07-24T04:02:52.284165Z","shell.execute_reply":"2022-07-24T04:02:56.485583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows distribution for R variables and skewness in distribution\n\n    * There are many variables which appear to be having extreme form of value","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the outlier value plot for R Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot outlier for R variables\nnrows = 7\nncols = 4\nfig, axes = plt.subplots(figsize=(30,25)) \n#train_df_sample = train_df.sample(frac =.1)\nfor i, col in enumerate(R_columns):\n    ax=fig.add_subplot(nrows, ncols, i+1)\n    sns.boxplot(x=train_df_sample[col], ax=ax, palette=\"Reds\")\n    \nfig.tight_layout()  \nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:02:56.488305Z","iopub.execute_input":"2022-07-24T04:02:56.489246Z","iopub.status.idle":"2022-07-24T04:03:00.031438Z","shell.execute_reply.started":"2022-07-24T04:02:56.489205Z","shell.execute_reply":"2022-07-24T04:03:00.030289Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above box plot shows variability for R variables and outliers for the same","metadata":{}},{"cell_type":"code","source":"del train_df_sample","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:03:00.033065Z","iopub.execute_input":"2022-07-24T04:03:00.033723Z","iopub.status.idle":"2022-07-24T04:03:00.039678Z","shell.execute_reply.started":"2022-07-24T04:03:00.033679Z","shell.execute_reply":"2022-07-24T04:03:00.038909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n# Categorical variables\n\n---","metadata":{}},{"cell_type":"markdown","source":"Following are categorical variables:\n\n**['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']**","metadata":{}},{"cell_type":"markdown","source":"---\n**Summary of Dataframe for Categorical variables**\n\n---","metadata":{}},{"cell_type":"code","source":"# display dataframe summary for categorical variables (columns)\ntrain_df[['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']].info()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:03:00.040922Z","iopub.execute_input":"2022-07-24T04:03:00.041758Z","iopub.status.idle":"2022-07-24T04:03:00.085878Z","shell.execute_reply.started":"2022-07-24T04:03:00.041704Z","shell.execute_reply":"2022-07-24T04:03:00.085092Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Converting B categorical variable dtype from float64 to object","metadata":{}},{"cell_type":"code","source":"#convert dtype for B categorical variable to object\ntrain_df = train_df.astype({\"B_30\": 'str', \"B_38\": 'str'})","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:03:00.087133Z","iopub.execute_input":"2022-07-24T04:03:00.087645Z","iopub.status.idle":"2022-07-24T04:03:00.290589Z","shell.execute_reply.started":"2022-07-24T04:03:00.087613Z","shell.execute_reply":"2022-07-24T04:03:00.289738Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Converting D categorical variable dtype from float64 to object","metadata":{}},{"cell_type":"code","source":"#convert dtype for D categorical variable to object\ntrain_df = train_df.astype({\"D_114\": 'str', \"D_116\": 'str', \"D_117\": 'str', \"D_120\": 'str', \"D_126\": 'str', \"D_66\": 'str', \"D_68\": 'str'})","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:03:00.291920Z","iopub.execute_input":"2022-07-24T04:03:00.292434Z","iopub.status.idle":"2022-07-24T04:03:00.857488Z","shell.execute_reply.started":"2022-07-24T04:03:00.292401Z","shell.execute_reply":"2022-07-24T04:03:00.856455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Summary of Dataframe for Categorical variables**\n\n---","metadata":{}},{"cell_type":"code","source":"# display dataframe summary for categorical variables (columns)\ntrain_df[['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']].info()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:03:00.858871Z","iopub.execute_input":"2022-07-24T04:03:00.859866Z","iopub.status.idle":"2022-07-24T04:03:01.447196Z","shell.execute_reply.started":"2022-07-24T04:03:00.859832Z","shell.execute_reply":"2022-07-24T04:03:01.445769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Q: What does data look like for Categorical variables?**\n\n---","metadata":{}},{"cell_type":"code","source":"# display sample data for categorical variables (columns)\ntrain_df[['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']]","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:03:01.448940Z","iopub.execute_input":"2022-07-24T04:03:01.449425Z","iopub.status.idle":"2022-07-24T04:03:01.484334Z","shell.execute_reply.started":"2022-07-24T04:03:01.449377Z","shell.execute_reply":"2022-07-24T04:03:01.483153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Descriptive statistics for Categorical variables**\n\n---","metadata":{}},{"cell_type":"code","source":"#display descriptive statistics for categorical variables\ntrain_df[['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']].describe(include='all')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:03:01.485829Z","iopub.execute_input":"2022-07-24T04:03:01.486187Z","iopub.status.idle":"2022-07-24T04:03:01.846753Z","shell.execute_reply.started":"2022-07-24T04:03:01.486154Z","shell.execute_reply":"2022-07-24T04:03:01.845573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* B_30 variable appears to have 4 unique categories with \"0.0\" having highest frequency of occurrence\n* B_38 variable appears to have 8 unique categories with \"2.0\" having highest frequency of occurence\n* D_114 variable appears to have 3 unique categories with \"1.0\" having highest frequency of occurence\n* D_114 variable appears to have 3 unique categories with \"1.0\" having highest frequency of occurence\n* D_116 variable appears to have 3 unique categories with \"0.0\" having highest frequency of occurence\n* D_117 variable appears to have 8 unique categories with \"-1.0\" having highest frequency of occurence\n* D_120 variable appears to have 3 unique categories with \"0.0\" having highest frequency of occurence\n* D_126 variable appears to have 4 unique categories with \"1.0\" having highest frequency of occurence\n* D_63 variable has 6 unique categories with \"CO\" having highest frequency of occurrence\n* D_64 variable has 4 unique categories with \"O\" having highest frequency of occurrence\n* D_66 variable appears to have significant amount of missing values in this sample dataset\n* D_68 variable appears to have 8 unique categories with \"6.0\" having highest frequency of occurence","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the distribution for Categorical Variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"# plot distribution of categorical variables\nfig,((ax1,ax2,ax3),(ax4,ax5,ax6),(ax7,ax8,ax9),(ax10,ax11,ax12)) = plt.subplots(4,3,figsize=(25,20))\nfig.suptitle('Distribution of Categorical Variable',fontsize=30)\n\nax1.pie(train_label_df['target'].value_counts(dropna=False), autopct='%1.1f%%',shadow=True, startangle=90, colors={'aquamarine','mediumseagreen'})\nax1.legend(labels = train_label_df['target'].value_counts(dropna=False).index, loc ='center left', bbox_to_anchor=(1, 0.5), title ='target',frameon = True)\n\nax2.pie(train_df['B_30'].value_counts(dropna=False), autopct='%1.1f%%',shadow=True, startangle=90,explode=(0.0,0.1,0.3,0.5), colors={'tab:cyan','tab:olive','tab:pink','tab:brown'})\nax2.legend(labels = train_df['B_30'].value_counts(dropna=False).index, loc ='center left', bbox_to_anchor=(1, 0.5), title ='B_30',frameon = True)\n\nax3.pie(train_df['B_38'].value_counts(dropna=False), autopct='%1.1f%%',shadow=True, startangle=90,explode=(0.0,0.0,0.0,0.1,0.2,0.3,0.5,0.9),colors={'tab:cyan','tab:olive','tab:pink','tab:brown','tab:gray','tab:purple','tab:red','tab:green'})\nax3.legend(labels = train_df['B_38'].value_counts(dropna=False).index, loc ='center left', bbox_to_anchor=(1, 0.5), title ='B_38',frameon = True)\n\nax4.pie(train_df['D_63'].value_counts(dropna=False), autopct='%1.1f%%',shadow=True, startangle=90, explode=(0.0,0.0,0.1,0.3,0.5,0.7))\nax4.legend(labels = train_df['D_63'].value_counts(dropna=False).index, loc ='center left', bbox_to_anchor=(1, 0.5),title ='D_63',frameon = True)\n\nax5.pie(train_df['D_64'].value_counts(dropna=False), autopct='%1.1f%%',shadow=True, startangle=90,explode=(0.0,0.0,0.0,0.3,0.5))\nax5.legend(labels = train_df['D_64'].value_counts(dropna=False).index, loc ='center left', bbox_to_anchor=(1, 0.5), title ='D_64',frameon = True)\n\nax6.pie(train_df['D_66'].value_counts(dropna=False), autopct='%1.1f%%',shadow=True, startangle=90,explode=(0.0,0.3,0.5))\nax6.legend(labels = train_df['D_66'].value_counts(dropna=False).index, loc ='center left', bbox_to_anchor=(1, 0.5), title ='D_66',frameon = True)\n\nax7.pie(train_df['D_68'].value_counts(dropna=False), autopct='%1.1f%%',shadow=True, startangle=90,explode=(0.0,0.0,0.0,0.1,0.2,0.3,0.5,0.9))\nax7.legend(labels = train_df['D_68'].value_counts(dropna=False).index, loc ='center left', bbox_to_anchor=(1, 0.5), title ='D_68',frameon = True)\n\nax8.pie(train_df['D_114'].value_counts(dropna=False), autopct='%1.1f%%',shadow=True, startangle=90)\nax8.legend(labels = train_df['D_114'].value_counts(dropna=False).index, loc ='center left', bbox_to_anchor=(1, 0.5), title ='D_114',frameon = True)\n\nax9.pie(train_df['D_116'].value_counts(dropna=False), autopct='%1.1f%%',shadow=True, startangle=90,explode=(0.0,0.3,0.5))\nax9.legend(labels = train_df['D_116'].value_counts(dropna=False).index, loc ='center left', bbox_to_anchor=(1, 0.5), title ='D_116',frameon = True)\n\nax10.pie(train_df['D_117'].value_counts(dropna=False), autopct='%1.1f%%',shadow=True, startangle=90,explode=(0.0,0.0,0.0,0.0,0.1,0.3,0.5,0.9))\nax10.legend(labels = train_df['D_117'].value_counts(dropna=False).index, loc ='center left', bbox_to_anchor=(1, 0.5), title ='D_117',frameon = True)\n\nax11.pie(train_df['D_120'].value_counts(dropna=False), autopct='%1.1f%%',shadow=True, startangle=90)\nax11.legend(labels = train_df['D_120'].value_counts(dropna=False).index, loc ='center left', bbox_to_anchor=(1, 0.5), title ='D_120',frameon = True)\n\nax12.pie(train_df['D_126'].value_counts(dropna=False), autopct='%1.1f%%',shadow=True, startangle=90,explode=(0.0,0.0,0.1,0.3))\nax12.legend(labels = train_df['D_126'].value_counts(dropna=False).index, loc ='center right', title ='D_126',frameon = True)\n\nplt.axis('equal')\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:03:01.848416Z","iopub.execute_input":"2022-07-24T04:03:01.849263Z","iopub.status.idle":"2022-07-24T04:03:04.143387Z","shell.execute_reply.started":"2022-07-24T04:03:01.849223Z","shell.execute_reply":"2022-07-24T04:03:04.140897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* target variable has 74.1% as target=0(Not Default) and 25.9% as target=1(Default)\n* B_30 variable has 85.0% of value as \"0.0\" value\n* B_38 variable has 34.5% of value as \"2.0\", 23.1% of value as \"3.0\" and 20.8% of value as \"1.0\"\n* D_63 variable has 73.8% of value as \"CO\", 17.4% of value as \"CR\" and 8.1% of value as \"CL\"\n* D_64 variable has 53.3% of value as \"O\", 27.4% of value as \"U\" and 14.5% of value as \"R\"\n* D_66 variable has 88.9% of value as nan\n* D_68 variable has 50.1% of value as \"6.0\", 21.8% of value as \"5.0\", 8.8% of value as \"4.0\" and 8.6% of value as \"3.0\"\n* D_114 variable has 60.6% of value as \"1.0\" and 36.2% of value as \"0.0\"\n* D_116 variable has 96.7% of value as \"0.0\"\n* D_117 variable has 26.1% of value as \"-1.0\", 21.3% of value as \"3.0\",20.7% of value as \"4.0\" and 11.6% of value as \"2.0\"\n* D_120 variable has 85.1% of value as \"0.0\" value and 11.6% of value as \"1.0\"\n* D_126 variable has 77.0% of value as \"1.0\" value and 16.% of value as \"0.0\"","metadata":{}},{"cell_type":"code","source":"del test_df\ndel sample_submission_df","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:03:04.144801Z","iopub.execute_input":"2022-07-24T04:03:04.145124Z","iopub.status.idle":"2022-07-24T04:03:04.150219Z","shell.execute_reply.started":"2022-07-24T04:03:04.145095Z","shell.execute_reply":"2022-07-24T04:03:04.149429Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n# Data Insights/Trends\n\n---","metadata":{}},{"cell_type":"markdown","source":"* Deliquency in general means violation of law and defaulting in credit card payment is considered violation of credit card agreement.\n\n* Factor(s) which can result in Deliquency may involve following background for respective customer\n\n    * Age\n    * Gender\n    * Education Level\n    * Marital Status\n    * Family Size\n    * Employment Status\n    * Income Level\n    * Geographic Location\n    \n* So lets do further analysis of D, S, P, B and R variables with respect to target label","metadata":{}},{"cell_type":"markdown","source":"Following are categorical variables:\n\n**['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']**","metadata":{}},{"cell_type":"markdown","source":"\n","metadata":{}},{"cell_type":"markdown","source":"---\nNeed to merge train_labels dataset with train_data dataset for target label.\n\n---","metadata":{}},{"cell_type":"code","source":"# Merge of train_df and train_label_df using key as customer_ID\ntrain_df_merged = pd.merge(train_df, train_label_df, how=\"inner\", on=[\"customer_ID\"])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:03:04.151401Z","iopub.execute_input":"2022-07-24T04:03:04.152067Z","iopub.status.idle":"2022-07-24T04:03:04.999754Z","shell.execute_reply.started":"2022-07-24T04:03:04.152017Z","shell.execute_reply":"2022-07-24T04:03:04.998580Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Convert S_2 variable dtype from object to datetime","metadata":{}},{"cell_type":"code","source":"# convert date to datetime type\ntrain_df_merged['S_2']= pd.to_datetime(train_df_merged['S_2'])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:03:05.001811Z","iopub.execute_input":"2022-07-24T04:03:05.002432Z","iopub.status.idle":"2022-07-24T04:03:05.035654Z","shell.execute_reply.started":"2022-07-24T04:03:05.002395Z","shell.execute_reply":"2022-07-24T04:03:05.034423Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print summary of merged dataframe\ntrain_df_merged.info()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:03:05.038537Z","iopub.execute_input":"2022-07-24T04:03:05.039007Z","iopub.status.idle":"2022-07-24T04:03:05.061427Z","shell.execute_reply.started":"2022-07-24T04:03:05.038960Z","shell.execute_reply":"2022-07-24T04:03:05.060004Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Merged dataset has one new variable (column) as target which represents Default (1) or Not Default(0) value","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What does data look like for merged dataset?**\n\n---","metadata":{}},{"cell_type":"code","source":"# print first 10 rows of merged dataframe\ntrain_df_merged.head(10)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:03:05.063024Z","iopub.execute_input":"2022-07-24T04:03:05.063464Z","iopub.status.idle":"2022-07-24T04:03:05.094937Z","shell.execute_reply.started":"2022-07-24T04:03:05.063419Z","shell.execute_reply":"2022-07-24T04:03:05.093737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print last 10 rows of merged dataframe\ntrain_df_merged.tail(10)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:03:05.096230Z","iopub.execute_input":"2022-07-24T04:03:05.096547Z","iopub.status.idle":"2022-07-24T04:03:05.130238Z","shell.execute_reply.started":"2022-07-24T04:03:05.096488Z","shell.execute_reply":"2022-07-24T04:03:05.129452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Lets check on correlation for D, S, P, B and R variables with respect to target variable**\n\n---","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the correlation for D variables with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot correlation for D variables with respect to target variable\nD_columns.append('target')\nprint(D_columns)\ncorr = train_df_merged[D_columns].corr(method='pearson')\nplt.figure(figsize=(75,60))\nsns.heatmap(corr,vmax=.8,linewidth=.01, square = True, annot = True,cmap='Oranges',linecolor ='black')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:03:05.131825Z","iopub.execute_input":"2022-07-24T04:03:05.132123Z","iopub.status.idle":"2022-07-24T04:03:36.631240Z","shell.execute_reply.started":"2022-07-24T04:03:05.132095Z","shell.execute_reply":"2022-07-24T04:03:36.629957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* D_41, D_42, D_43, D_44, D_45, D_47, D_48, D_51, D_55, D_58, D_61, D_62, D_70, D_74, D_75, D_77, D_78 appears to have favorable correlation (>=+-0.25) with target variable","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the correlation for S variables with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot correlation for S variables with respect to target variable\nS_columns.append('target')\nprint(S_columns)\ncorr = train_df_merged[S_columns].corr(method='pearson')\nplt.figure(figsize=(30,25))\nsns.heatmap(corr,vmax=.8,linewidth=.01, square = True, annot = True,cmap='Greens',linecolor ='black')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:03:36.643757Z","iopub.execute_input":"2022-07-24T04:03:36.644092Z","iopub.status.idle":"2022-07-24T04:03:39.670954Z","shell.execute_reply.started":"2022-07-24T04:03:36.644064Z","shell.execute_reply":"2022-07-24T04:03:39.669884Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* S_3, S_7, S_25 appears to have favorable correlation(>=+-0.25) with target variable","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the correlation for P variables with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot correlation for P variables with respect to target variable\nP_columns.append('target')\nprint(P_columns)\ncorr = train_df_merged[P_columns].corr(method='pearson')\nplt.figure(figsize=(10,12))\nsns.heatmap(corr,vmax=.8,linewidth=.01, square = True, annot = True,cmap='Purples',linecolor ='black')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:03:39.672277Z","iopub.execute_input":"2022-07-24T04:03:39.673172Z","iopub.status.idle":"2022-07-24T04:03:39.975935Z","shell.execute_reply.started":"2022-07-24T04:03:39.673139Z","shell.execute_reply":"2022-07-24T04:03:39.974656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* P_2, P_3 appears to have favorable correlation (>=+-0.25) with target variable","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the correlation for B variables with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot correlation for B variables with respect to target variable\nB_columns.append('target')\nprint(B_columns)\ncorr = train_df_merged[B_columns].corr(method='pearson')\nplt.figure(figsize=(60,50))\nsns.heatmap(corr,vmax=.8,linewidth=.01, square = True, annot = True,cmap='Blues',linecolor ='black')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:03:39.978410Z","iopub.execute_input":"2022-07-24T04:03:39.979077Z","iopub.status.idle":"2022-07-24T04:03:46.718755Z","shell.execute_reply.started":"2022-07-24T04:03:39.979039Z","shell.execute_reply":"2022-07-24T04:03:46.717573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* B_1, B_2, B_3, B_4, B_7, B_8, B_9, B_11, B_16, B_17, B_18, B_19, B_20, B_22, B_23, B_33, B_37 appears to have favorable correlation (>=+-0.25) with target variable","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the correlation for R variables with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot correlation for R variables with respect to target variable\nR_columns.append('target')\nprint(R_columns)\ncorr = train_df_merged[R_columns].corr(method='pearson')\nplt.figure(figsize=(60,50))\nsns.heatmap(corr,vmax=.8,linewidth=.01, square = True, annot = True,cmap='Reds',linecolor ='black')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:03:46.720386Z","iopub.execute_input":"2022-07-24T04:03:46.721010Z","iopub.status.idle":"2022-07-24T04:03:50.862136Z","shell.execute_reply.started":"2022-07-24T04:03:46.720964Z","shell.execute_reply":"2022-07-24T04:03:50.861083Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* R_1, R_2, R_3, R_27 appears to have favorable correlation(>=+-0.25) with target variable","metadata":{}},{"cell_type":"markdown","source":"---\n**Quick check on correlation for D, S, P, B and R variables with target variable using threshold of 0.25**\n\n---","metadata":{}},{"cell_type":"markdown","source":"For D Variable","metadata":{}},{"cell_type":"code","source":"# check on correlation for D variable with target variable using threshold of 0.25\ncorr = train_df_merged.corr(method='pearson')\nthreshold = abs(corr['target'])\ncorr_result = threshold[threshold>=0.25]\ncorr_result.filter(like='D_').sort_values(ascending=False)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:03:50.863391Z","iopub.execute_input":"2022-07-24T04:03:50.863728Z","iopub.status.idle":"2022-07-24T04:03:58.045721Z","shell.execute_reply.started":"2022-07-24T04:03:50.863698Z","shell.execute_reply":"2022-07-24T04:03:58.044890Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For S Variable","metadata":{}},{"cell_type":"code","source":"# check on correlation for S variable with target variable using threshold of 0.20\ncorr = train_df_merged.corr(method='pearson')\nthreshold = abs(corr['target'])\ncorr_result = threshold[threshold>=0.25]\ncorr_result.filter(like='S_').sort_values(ascending=False)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:03:58.047025Z","iopub.execute_input":"2022-07-24T04:03:58.047589Z","iopub.status.idle":"2022-07-24T04:04:05.143672Z","shell.execute_reply.started":"2022-07-24T04:03:58.047541Z","shell.execute_reply":"2022-07-24T04:04:05.142478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For P Variable","metadata":{}},{"cell_type":"code","source":"# check on correlation for P variable with target variable using threshold of 0.25\ncorr = train_df_merged.corr(method='pearson')\nthreshold = abs(corr['target'])\ncorr_result = threshold[threshold>=0.25]\ncorr_result.filter(like='P_')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:04:05.144920Z","iopub.execute_input":"2022-07-24T04:04:05.145244Z","iopub.status.idle":"2022-07-24T04:04:12.290344Z","shell.execute_reply.started":"2022-07-24T04:04:05.145213Z","shell.execute_reply":"2022-07-24T04:04:12.289119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For B Variable","metadata":{}},{"cell_type":"code","source":"# check on correlation for B variable with target variable using threshold of 0.25\ncorr = train_df_merged.corr(method='pearson')\nthreshold = abs(corr['target'])\ncorr_result = threshold[threshold>=0.25]\ncorr_result.filter(like='B_').sort_values(ascending=False)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:04:12.291894Z","iopub.execute_input":"2022-07-24T04:04:12.292289Z","iopub.status.idle":"2022-07-24T04:04:20.492632Z","shell.execute_reply.started":"2022-07-24T04:04:12.292205Z","shell.execute_reply":"2022-07-24T04:04:20.491522Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For R Variable","metadata":{}},{"cell_type":"code","source":"# check on correlation for R variable with target variable using threshold of 0.25\ncorr = train_df_merged.corr(method='pearson')\nthreshold = abs(corr['target'])\ncorr_result = threshold[threshold>=0.25]\ncorr_result.filter(like='R_').sort_values(ascending=False)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:04:20.494045Z","iopub.execute_input":"2022-07-24T04:04:20.495072Z","iopub.status.idle":"2022-07-24T04:04:27.482329Z","shell.execute_reply.started":"2022-07-24T04:04:20.495038Z","shell.execute_reply":"2022-07-24T04:04:27.481553Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For D, S, P, B and R Variable","metadata":{}},{"cell_type":"code","source":"# check on correlation for D, S, P, B and R variable with target variable using threshold of 0.25\ncorr = train_df_merged.corr(method='pearson')\nthreshold = abs(corr['target'])\ncorr_result = threshold[threshold>=0.25]\ncorr_result.sort_values(ascending=False)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:04:27.484011Z","iopub.execute_input":"2022-07-24T04:04:27.484841Z","iopub.status.idle":"2022-07-24T04:04:34.414676Z","shell.execute_reply.started":"2022-07-24T04:04:27.484795Z","shell.execute_reply":"2022-07-24T04:04:34.413569Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Summary for some of the key numeric variable which has correlation between them and favorable correlation (>=+-0.25) with target variable**\n\n---","metadata":{}},{"cell_type":"markdown","source":"Following D variables has high correlation(>=0.80) between them\n\n* D_42 has correlation of 1.0 with D_110 and D_111\n* D_58 has correlation of 0.92 with D_74 and 0.93 with D_75\n* D_62 has correlation of 1.0 with D_77\n* D_74 has correlation of 0.99 with D_75\n* D_73 has correlation of -1.0 with D_88\n* D_139 has correlation of 1.0 with D_143 and D_141\n* D_141 has correlation of 1.0 with D_143\n* D_132 has correlation of 0.92 with D_132\n* D_48 has correlation of 0.84 with D_55 and 0.86 with D_61\n* D_79 has correlation of 0.89 with D_131\n\nFollowing D variables appears to have favorable correlation (>=+-0.25) with target variable\n\n* D_48, D_61, D_44, D_55, D_75, D_58, D_74, D_62, D_77, D_70, D_47, **D_42**, D_43, D_78, D_41, D_45, D_51  ","metadata":{}},{"cell_type":"markdown","source":"Following D variable have higher missing value:\n\n* D_87, D_88, D_73, D_49, D_66, **D_42**, D_108, D_110, D_111, D_137, D_134, D_136, D_106,D_132, D_142 - 80%\n* D_53, D_82 - 70%\n* D_50, D_56 - 60%\n* D_77 , D_105 - 50%\n\nSelected D variables for further EDA\n\n* D_61, D_44, D_55, D_75, D_62, D_70, D_47, D_43, D_78, D_41, D_45, D_51\n\n---","metadata":{}},{"cell_type":"markdown","source":"Following S variables has high correlation(>=0.80) between them\n\n* S_3 has correlation of 0.91 with S_7\n* S_22 has correlation of 0.94 with S_24\n\nFollowing S variables appears to have favorable correlation (>=+-0.25) with target variable\n\n* S_7, S_3 \n\nFollowing S variable have higher missing value:\n\n* S_9 - 70%\n\nSelected S variables for further EDA\n\n* S_7\n\n\n---\n","metadata":{}},{"cell_type":"markdown","source":"Following P variables has high correlation(>=0.80) between them\n\n* None of the P variables appears to be highly correlated\n\nFollowing P variables appears to have favorable correlation (>=+-0.25) with target variable\n\n* P_2\n\nNone of the P variable have higher missing value\n\nSelected P variable for further EDA\n\n* P_2\n\n---","metadata":{}},{"cell_type":"markdown","source":"Following B variables has high correlation(>=0.80) between them\n\n* B_1 has correlation of 0.99 with B_37\n* B_1 has correlation of 1.0 with B_11\n* B_2 has correlation of 0.91 with B_33\n* B_2 has correlation of 0.85 with B_18\n* B_11 has correlation of 0.99 with B_37\n* B_12 has correlation of 0.91 with B_13\n* B_14 has correlation of 0.89 with B_15\n* B_7 has correlation of 1.0 with B_23\n\nFollowing B variables appears to have favorable correlation (>=+-0.25) with target variable\n\n* B_9, B_18, B_2, B_33, B_7, B_3, B_23, B_4, B_16, B_1, B_37, B_19, B_20, B_22, B_11, B_8, **B_17**\n\nFollowing B variable have higher missing value:\n\n* B_39, B_42, B_29 - 80%\n* **B_17** - 50%\n\nSelected B variables for further EDA\n\n* B_9, B_18, B_33, B_7, B_3, B_4, B_16, B_1, B_19, B_20, B_22, B_8\n\n---","metadata":{}},{"cell_type":"markdown","source":"Following R variables has high correlation(>=0.80) between them\n\n* R_4 has correlation of 0.79 with R_2\n* R_5 has correlation of 0.8 with R_8\n\nFollowing R variables appears to have favorable correlation (>=+-0.25) with target variable\n\n* R_1, R_3, R_27, R_2\n\nFollowing R variable have higher missing value:\n\n* R_9, R_26 - 80%\n\nSelected R variables for further EDA\n\n* R_1, R_3, R_27, R_2\n\n---","metadata":{}},{"cell_type":"markdown","source":"---\n**Lets look at distribution for selected D, S, P, B, R variables with respect to target variable**\n\n---","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the distribution for selected D variables with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot distribution for selected D variables with respect to target label\nnrows = 4\nncols = 3\nfig, axes = plt.subplots(figsize=(18,15)) \nd_columns=['D_61','D_44','D_55','D_75','D_62','D_70','D_47','D_43','D_78','D_41','D_45', 'D_51']\nfor i, col in enumerate(d_columns):\n    ax=fig.add_subplot(nrows, ncols, i+1)\n    sns.kdeplot(x=train_df_merged[col], hue=train_df_merged['target'], multiple=\"stack\",ax=ax)\n    \nfig.tight_layout()  \nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:04:34.416117Z","iopub.execute_input":"2022-07-24T04:04:34.416501Z","iopub.status.idle":"2022-07-24T04:04:43.590039Z","shell.execute_reply.started":"2022-07-24T04:04:34.416462Z","shell.execute_reply":"2022-07-24T04:04:43.589262Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Q: What is the distribution for selected S variables with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot distribution for selected S variables with respect to target label\nnrows = 1\nncols = 1\nfig, axes = plt.subplots(figsize=(15,8)) \ns_columns=['S_7']\nfor i, col in enumerate(s_columns):\n    ax=fig.add_subplot(nrows, ncols, i+1)\n    sns.kdeplot(x=train_df_merged[col], hue=train_df_merged['target'], multiple=\"stack\",palette=\"Accent\",ax=ax)\n    \nfig.tight_layout()  \nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:04:43.591085Z","iopub.execute_input":"2022-07-24T04:04:43.591996Z","iopub.status.idle":"2022-07-24T04:04:44.473647Z","shell.execute_reply.started":"2022-07-24T04:04:43.591958Z","shell.execute_reply":"2022-07-24T04:04:44.472578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Q: What is the distribution for selected P variable with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot distribution for selected P variable with respect to target label\nnrows = 1\nncols = 1\nfig, axes = plt.subplots(figsize=(15,8)) \np_columns=['P_2']\nfor i, col in enumerate(p_columns):\n    ax=fig.add_subplot(nrows, ncols, i+1)\n    sns.kdeplot(x=train_df_merged[col], hue=train_df_merged['target'], multiple=\"stack\",palette=\"crest\",ax=ax)\n    \nfig.tight_layout()  \nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:04:44.475603Z","iopub.execute_input":"2022-07-24T04:04:44.476666Z","iopub.status.idle":"2022-07-24T04:04:45.436047Z","shell.execute_reply.started":"2022-07-24T04:04:44.476618Z","shell.execute_reply":"2022-07-24T04:04:45.434887Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Q: What is the distribution for selected B variables with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot distribution for selected B variables with respect to target label\nnrows = 4\nncols = 3\nfig, axes = plt.subplots(figsize=(18,15)) \nb_columns=['B_9','B_18','B_33','B_7','B_3','B_4','B_16','B_1','B_19','B_20','B_22', 'B_8']\nfor i, col in enumerate(b_columns):\n    ax=fig.add_subplot(nrows, ncols, i+1)\n    sns.kdeplot(x=train_df_merged[col], hue=train_df_merged['target'], multiple=\"stack\",palette=\"ocean\",ax=ax)\n    \nfig.tight_layout()  \nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:04:45.437446Z","iopub.execute_input":"2022-07-24T04:04:45.437844Z","iopub.status.idle":"2022-07-24T04:04:53.728625Z","shell.execute_reply.started":"2022-07-24T04:04:45.437810Z","shell.execute_reply":"2022-07-24T04:04:53.727345Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Q: What is the distribution for selected R variables with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"#plot distribution for selected R variables with respect to target label\nnrows = 1\nncols = 4\nfig, axes = plt.subplots(figsize=(15,5)) \nr_columns=['R_1','R_3','R_27','R_2']\nfor i, col in enumerate(r_columns):\n    ax=fig.add_subplot(nrows, ncols, i+1)\n    sns.kdeplot(x=train_df_merged[col], hue=train_df_merged['target'], multiple=\"stack\",palette=\"summer\",ax=ax)\n    \nfig.tight_layout()  \nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:04:53.730224Z","iopub.execute_input":"2022-07-24T04:04:53.730684Z","iopub.status.idle":"2022-07-24T04:04:56.750373Z","shell.execute_reply.started":"2022-07-24T04:04:53.730638Z","shell.execute_reply":"2022-07-24T04:04:56.749289Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Lets look at trend for D_61, S_7, P_2, B_9 and R_1 variable with respect to target variable**\n\n---","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the trend for D_61 variable with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"# trend for D_61 variable\nplt.figure(figsize=(15,8))\nsns.lineplot(data = train_df_merged, x = 'S_2', y = 'D_61', hue= 'target', palette='spring')\nplt.xlabel('Date', size = 14)\nplt.ylabel('Deliquency Variable D_61', size = 14)\nplt.title('Trend for D_61 Variable', size = 16)\nplt.grid(visible = True, axis = 'y')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-24T04:04:56.751967Z","iopub.execute_input":"2022-07-24T04:04:56.752411Z","iopub.status.idle":"2022-07-24T04:05:19.877956Z","shell.execute_reply.started":"2022-07-24T04:04:56.752366Z","shell.execute_reply":"2022-07-24T04:05:19.876820Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows trend for Deliquency variable D_61 with respect to date S_2 and datapoints with target=1(Default) appears to have  higher Deliquency value.","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the trend for S_7 variable with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"# trend for S_7 variable\nplt.figure(figsize=(15,8))\nsns.lineplot(data = train_df_merged, x = 'S_2', y = 'S_7', hue= 'target', palette='winter')\nplt.xlabel('Date', size = 14)\nplt.ylabel('Spend Variable S_7', size = 14)\nplt.title('Trend for S_7 Variable', size = 16)\nplt.grid(visible = True, axis = 'y')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:05:19.879239Z","iopub.execute_input":"2022-07-24T04:05:19.879553Z","iopub.status.idle":"2022-07-24T04:05:42.874669Z","shell.execute_reply.started":"2022-07-24T04:05:19.879525Z","shell.execute_reply":"2022-07-24T04:05:42.873404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows trend for Spend variable S_7 with respect to date S_2 and datapoints with target=1(Default) appears to have higher Spend value.","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the trend for P_2 variable with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"# trend for P_2 variable\nplt.figure(figsize=(15,8))\nsns.lineplot(data = train_df_merged, x = 'S_2', y = 'P_2', hue= 'target', palette='ocean')\nplt.xlabel('Date', size = 14)\nplt.ylabel('Payment Variable P_2', size = 14)\nplt.title('Trend for P_2 Variable', size = 16)\nplt.grid(visible = True, axis = 'y')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:05:42.876330Z","iopub.execute_input":"2022-07-24T04:05:42.876802Z","iopub.status.idle":"2022-07-24T04:06:06.100397Z","shell.execute_reply.started":"2022-07-24T04:05:42.876759Z","shell.execute_reply":"2022-07-24T04:06:06.099550Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows trend for Payment variable P_2 with respect to date S_2 and datapoints with target=1(Default) appears to have lower Payment value.","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the trend for B_9 variable with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"# trend for B_9 variable\nplt.figure(figsize=(15,8))\nsns.lineplot(data = train_df_merged, x = 'S_2', y = 'B_9', hue= 'target', palette='summer')\nplt.xlabel('Date', size = 14)\nplt.ylabel('Balance Variable B_9', size = 14)\nplt.title('Trend for B_9 Variable', size = 16)\nplt.grid(visible = True, axis = 'y')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:06:06.101629Z","iopub.execute_input":"2022-07-24T04:06:06.102505Z","iopub.status.idle":"2022-07-24T04:06:29.709657Z","shell.execute_reply.started":"2022-07-24T04:06:06.102469Z","shell.execute_reply":"2022-07-24T04:06:29.708377Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows trend for Balance variable B_9 with respect to date S_2 and datapoints with target=1(Default) appears to have higher Balance value.","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the trend for R_1 variable with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"# trend for R_1 variable\nplt.figure(figsize=(15,8))\nsns.lineplot(data = train_df_merged, x = 'S_2', y = 'R_1', hue= 'target', palette='rainbow')\nplt.xlabel('Date', size = 14)\nplt.ylabel('Risk Variable R_1', size = 14)\nplt.title('Trend for R_1 Variable', size = 16)\nplt.grid(visible = True, axis = 'y')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:06:29.711157Z","iopub.execute_input":"2022-07-24T04:06:29.712067Z","iopub.status.idle":"2022-07-24T04:06:52.968994Z","shell.execute_reply.started":"2022-07-24T04:06:29.712022Z","shell.execute_reply":"2022-07-24T04:06:52.967776Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows trend for Risk variable R_1 with respect to date S_2 and datapoints with target=1(Default) appears to have higher Risk value.","metadata":{}},{"cell_type":"markdown","source":"---\n**Lets look at distribution for categorical variables with respect to target variable**\n\n---","metadata":{}},{"cell_type":"markdown","source":"Following are the variables with categorical data:\n\n* B_30, B_38, D_114, D_116, D_117, D_120, D_126, D_63, D_64, **D_66**, D_68\n\nNote: Will skip **D_66** variable as it has 88.9% of value as nan","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the distribution for B_30 variable with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"# plot distribution of B_30 variable with respect to target variable\nplt.figure(figsize=(15,8))\nsns.histplot(data=train_df_merged,x='B_30',hue='target', palette='ocean',element='bars')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:06:52.971214Z","iopub.execute_input":"2022-07-24T04:06:52.971938Z","iopub.status.idle":"2022-07-24T04:06:53.329696Z","shell.execute_reply.started":"2022-07-24T04:06:52.971887Z","shell.execute_reply":"2022-07-24T04:06:53.328464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows B_30 variable distribution with respect to target variable and \"0.0\" and \"1.0\" category has majority of distribution which appears to overlap with target variable with target=0(Not Default) as majority. \"1.0\" category has target=1(Default) as majority.","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the distribution for B_38 variable with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"# plot distribution of B_38 variable with respect to target variable\nplt.figure(figsize=(15,8))\nsns.histplot(data=train_df_merged,x='B_38',hue='target', palette='winter',element='bars')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:06:53.331842Z","iopub.execute_input":"2022-07-24T04:06:53.332692Z","iopub.status.idle":"2022-07-24T04:06:53.723364Z","shell.execute_reply.started":"2022-07-24T04:06:53.332632Z","shell.execute_reply":"2022-07-24T04:06:53.722237Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows B_38 variable distribution with respect to target variable and \"2.0\", \"1.0\" and \"3.0\" category has majority of distribution which appears to overlap for target variable with target=0(Not Default) as majority. \"4.0\" category is having target=1(Default) as majority with \"7.0\" category which appears to share both target=0(Not Default) and target=1(Default) equally.","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the distribution for D_63 variable with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"# plot distribution of D_63 variable with respect to target variable\nplt.figure(figsize=(15,8))\nsns.histplot(data=train_df_merged,x='D_63',hue='target', palette='twilight',element='bars')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:06:53.725037Z","iopub.execute_input":"2022-07-24T04:06:53.725379Z","iopub.status.idle":"2022-07-24T04:06:54.089442Z","shell.execute_reply.started":"2022-07-24T04:06:53.725348Z","shell.execute_reply":"2022-07-24T04:06:54.088274Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows D_63 variable distribution with respect to target variable and \"CO\", \"CR\" and \"CL\" category has majority of distribution which appears to overlap for target variable with target=0(Not Default) as majority.","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the distribution for D_64 variable with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"# plot distribution of D_64 variable with respect to target variable\nplt.figure(figsize=(15,8))\nsns.histplot(data=train_df_merged,x='D_64',hue='target', palette='plasma',element='bars')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:06:54.091445Z","iopub.execute_input":"2022-07-24T04:06:54.092251Z","iopub.status.idle":"2022-07-24T04:06:54.423132Z","shell.execute_reply.started":"2022-07-24T04:06:54.092210Z","shell.execute_reply":"2022-07-24T04:06:54.422284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows D_64 variable distribution with respect to target variable and \"O\", \"R\" and \"U\" category has majority of distribution which appears to overlap for target variable with target=0(Not Default) as majority.","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the distribution for D_68 variable with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"# plot distribution of D_68 variable with respect to target variable\nplt.figure(figsize=(15,8))\nsns.histplot(data=train_df_merged,x='D_68',hue='target', palette='magma',element='bars')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:06:54.425028Z","iopub.execute_input":"2022-07-24T04:06:54.425853Z","iopub.status.idle":"2022-07-24T04:06:54.813549Z","shell.execute_reply.started":"2022-07-24T04:06:54.425802Z","shell.execute_reply":"2022-07-24T04:06:54.812562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows D_68 variable distribution with respect to target variable and \"6.0\" and \"5.0\" category has majority of distribution which appears to overlap for target variable followed by \"3.0\", \"4.0\", \"2.0\" and \"1.0\" category with target=0(Not Default) as majority.","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the distribution for D_114 variable with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"# plot distribution of D_114 variable with respect to target variable\nplt.figure(figsize=(15,8))\nsns.histplot(data=train_df_merged,x='D_114',hue='target', palette='summer',element='bars')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:06:54.814894Z","iopub.execute_input":"2022-07-24T04:06:54.815239Z","iopub.status.idle":"2022-07-24T04:06:55.147339Z","shell.execute_reply.started":"2022-07-24T04:06:54.815199Z","shell.execute_reply":"2022-07-24T04:06:55.146327Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows D_114 variable distribution with respect to target variable and \"1.0\" category has majority of distribution which appears to overlap for target variable followed by \"0.0\" category with target=0(Not Default) as majority.","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the distribution for D_116 variable with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"# plot distribution of D_116 variable with respect to target variable\nplt.figure(figsize=(15,8))\nsns.histplot(data=train_df_merged,x='D_116',hue='target', palette='turbo',element='bars')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:06:55.148727Z","iopub.execute_input":"2022-07-24T04:06:55.149674Z","iopub.status.idle":"2022-07-24T04:06:55.541722Z","shell.execute_reply.started":"2022-07-24T04:06:55.149639Z","shell.execute_reply":"2022-07-24T04:06:55.540568Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows D_116 variable distribution with respect to target variable and \"0.0\" category has majority of distribution which appears to overlap for target variable with target=0(Not Default) as majority.","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the distribution for D_117 variable with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"# plot distribution of D_117 variable with respect to target variable\nplt.figure(figsize=(15,8))\nsns.histplot(data=train_df_merged,x='D_117',hue='target', palette='terrain',element='bars')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:06:55.543322Z","iopub.execute_input":"2022-07-24T04:06:55.544141Z","iopub.status.idle":"2022-07-24T04:06:55.987315Z","shell.execute_reply.started":"2022-07-24T04:06:55.544103Z","shell.execute_reply":"2022-07-24T04:06:55.986239Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows D_117 variable distribution with respect to target variable and \"-1.0\", \"4.0\" and \"3.0\" category has majority of distribution which appears to overlap for target variable followed by \"2.0\", \"5.0\", \"6.0\" and \"1.0\" category with target=0(Not Default) as majority in all of them.","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the distribution for D_120 variable with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"# plot distribution of D_120 variable with respect to target variable\nplt.figure(figsize=(15,8))\nsns.histplot(data=train_df_merged,x='D_120',hue='target', palette='rocket',element='bars')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:06:55.988876Z","iopub.execute_input":"2022-07-24T04:06:55.989281Z","iopub.status.idle":"2022-07-24T04:06:56.394883Z","shell.execute_reply.started":"2022-07-24T04:06:55.989249Z","shell.execute_reply":"2022-07-24T04:06:56.393846Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows D_120 variable distribution with respect to target variable and \"0.0\" category has majority of distribution which appears to overlap for target variable with target=0(Not Default) as majority followed by \"1.0\" which appears to share both target=0(Not Default) and target=1(Default) equally.","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the distribution for D_126 variable with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"# plot distribution of D_126 variable with respect to target variable\nplt.figure(figsize=(15,8))\nsns.histplot(data=train_df_merged,x='D_126',hue='target', palette='spring',element='bars')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:06:56.398477Z","iopub.execute_input":"2022-07-24T04:06:56.398879Z","iopub.status.idle":"2022-07-24T04:06:56.799092Z","shell.execute_reply.started":"2022-07-24T04:06:56.398844Z","shell.execute_reply":"2022-07-24T04:06:56.797959Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Above plot shows D_126 variable distribution with respect to target variable and \"1.0\" category has majority of distribution which appears to overlap for target variable with target=0(Not Default) as majority followed by \"0.0\" and \"-1.0\" category.","metadata":{}},{"cell_type":"markdown","source":"---\n**Lets try to look at correlation for categorical variables between them and with respect to target variable**\n\n---","metadata":{}},{"cell_type":"markdown","source":"Encoding categorical variables","metadata":{}},{"cell_type":"code","source":"# encode categorical columns \ntrain_df_merged=pd.get_dummies(train_df_merged,columns=['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64','D_68'])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:06:56.800691Z","iopub.execute_input":"2022-07-24T04:06:56.801004Z","iopub.status.idle":"2022-07-24T04:06:57.124641Z","shell.execute_reply.started":"2022-07-24T04:06:56.800976Z","shell.execute_reply":"2022-07-24T04:06:57.123493Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\nConsidering high number of encoded categorical variables, need to extract names of variable for easy access.\n\n---","metadata":{}},{"cell_type":"code","source":"# extract encoded column names\n# regex to capture column name with B_$$_ (excluding column name with suffix nan)\nb_col1 = \"B_[0-9][0-9]_[^nan]\"\n# regex to capture column name with D_$$_ (excluding column name with suffix nan)\nd_col1 = \"D_[0-9][0-9]_[^nan]\"\n# regex to capture column name with D_$$$_ (excluding column name with suffix nan)\nd_col2 = \"D_[0-9][0-9][0-9]_[^nan]\"\nB_cat_columns1 = [col for col in train_df_merged.columns if re.match(b_col1,col)]\nprint (\"Total number of cat columns starting with B is\", len(B_cat_columns1))\nprint(B_cat_columns1)\nprint(\"###############################\")\nD_cat_columns1 = [col for col in train_df_merged.columns if re.match(d_col1,col)]\nprint (\"Total number of first set of cat columns starting with D is\", len(D_cat_columns1))\nprint(D_cat_columns1)\nprint(\"###############################\")\nD_cat_columns2 = [col for col in train_df_merged.columns if re.match(d_col2,col)]\nprint (\"Toal number of second set of cat columns starting with D is\", len(D_cat_columns2))\nprint(D_cat_columns2)\nprint(\"###############################\")\ntotal_cat_columns = len(B_cat_columns1)+len(D_cat_columns1)+len(D_cat_columns2)\nprint(\"Total new cat columns\",total_cat_columns)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:06:57.126684Z","iopub.execute_input":"2022-07-24T04:06:57.127191Z","iopub.status.idle":"2022-07-24T04:06:57.141132Z","shell.execute_reply.started":"2022-07-24T04:06:57.127129Z","shell.execute_reply":"2022-07-24T04:06:57.139962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**Q: What is the correlation for encoded categorical variables?**\n\n---","metadata":{}},{"cell_type":"code","source":"# plot correlation between categorical variables (columns)\nnew_cat_columns=B_cat_columns1+D_cat_columns1+D_cat_columns2\nprint(new_cat_columns)\ncorr = train_df_merged[new_cat_columns].corr(method='pearson')\nplt.figure(figsize=(40,30))\nsns.heatmap(corr,vmax=.8,linewidth=.01, square = True, annot = True,cmap='Purples',linecolor ='black')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:06:57.142790Z","iopub.execute_input":"2022-07-24T04:06:57.143301Z","iopub.status.idle":"2022-07-24T04:07:05.817469Z","shell.execute_reply.started":"2022-07-24T04:06:57.143268Z","shell.execute_reply":"2022-07-24T04:07:05.816327Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* B_30_0.0 and B_30_1.0 has correlation of -0.96\n* D_63_CR and D_63_CO has correlation of -0.77\n* D_114_0.0 and D_114_1.0 has correlation of -0.93\n* D_120_0.0 and D_120_1.0 has correlation of -0.87\n* D_126_0.0 and D_126_1.0 has correlation of -0.81","metadata":{}},{"cell_type":"markdown","source":"---\n**Q: What is the correlation for encoded categorical variables with respect to target variable?**\n\n---","metadata":{}},{"cell_type":"code","source":"# plot correlation between categorical columns with respect to target variable\nnew_cat_columns=B_cat_columns1+D_cat_columns1+D_cat_columns2\nnew_cat_columns.append('target')\nprint(new_cat_columns)\ncorr = train_df_merged[new_cat_columns].corr(method='pearson')\nplt.figure(figsize=(40,30))\nsns.heatmap(corr,vmax=.8,linewidth=.01, square = True, annot = True,cmap='Blues',linecolor ='black')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:07:05.819303Z","iopub.execute_input":"2022-07-24T04:07:05.819734Z","iopub.status.idle":"2022-07-24T04:07:14.218064Z","shell.execute_reply.started":"2022-07-24T04:07:05.819691Z","shell.execute_reply":"2022-07-24T04:07:14.216929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Observation:**\n\n* Few category (encoded columns) in B_30, B_38, D_64, D_114 and D_120 variable appears to have favorable correlation with target variable.","metadata":{}},{"cell_type":"code","source":"# check on correlation for categorical variable with target variable using threshold of 0.15\ncorr = train_df_merged[new_cat_columns].corr(method='pearson')\nthreshold = abs(corr['target'])\ncorr_result = threshold[threshold>=0.15]\ncorr_result.sort_values(ascending=False)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:07:14.219439Z","iopub.execute_input":"2022-07-24T04:07:14.220362Z","iopub.status.idle":"2022-07-24T04:07:14.770524Z","shell.execute_reply.started":"2022-07-24T04:07:14.220328Z","shell.execute_reply":"2022-07-24T04:07:14.769331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n# Summary\n\n---","metadata":{}},{"cell_type":"markdown","source":"* Dataset is imbalanced with target=1(Default) as 25.9% and target=0(Not Default) as 74.1% \n* D_* Variable category has the highest number of variable 92 out of 189 variable (excluding customer_ID)\n* B_* Variable category has the second highest of variable as 40\n* R_* Variable category is the third highest with 28 variable\n* S_* Variable category has 22 number of variable\n* P_* Variable category has 3 variables\n* All the variable category has variables with missing value and outliers\n* All the variable category has variables with skewed or bimodal distribution\n* There are variable category (D,B and S) which has highly correlated variables.","metadata":{}},{"cell_type":"markdown","source":"---\n* **D_* = Delinquency variable category**\n\n---\n**Total Number of Variable:** 92\n\n* Number of Variable with dtype float64: 90\n\n* Number of Variable with dtype int64: 0\n\n* Number of Variable with dtype object: 2\n\n   * Number of Variable with Categorical Data: 9\n\n   * Number of Variable with Missing Value: 81\n\n---\n* **S_* = Spend variable category**\n\n---\n\n**Total Number of Variable:** 22\n\n* Number of Variable with dtype float64: 21\n\n* Number of Variable with dtype int64: 0\n\n* Number of Variable with dtype object: 1 (Date)\n\n    * Number of Variable with Categorical Data: None\n\n    * Number of Variable with Missing Value: 13\n\n---\n* **P_* = Payment variable category**\n\n---\n\n**Total Number of Variable:** 3\n\n* Number of Variable with dtype float64: 3\n\n* Number of Variable with dtype int64: 0\n\n* Number of Variable with dtype object: 0\n\n    * Number of Variable with Categorical Data: None\n\n    * Number of Variable with Missing Value: 2\n\n---\n* **B_* = Balance variable category**\n\n---\n\n**Total Number of Variable:** 40\n\n* Number of Variable with dtype float64: 39\n\n* Number of Variable with dtype int64: 1\n\n* Number of Variable with dtype object: 0\n\n    * Number of Variable with Categorical Data: 2\n\n    * Number of Variable with Missing Value: 19\n\n---\n* **R_* = Risk variable category**\n\n---\n\n**Total Number of Variable:** 28\n\n* Number of Variable with dtype float64: 28\n\n* Number of Variable with dtype int64: 0\n\n* Number of Variable with dtype object: 0\n\n    * Number of Variable with Categorical Data: None\n\n    * Number of Variable with Missing Value: 4\n\n---","metadata":{}},{"cell_type":"markdown","source":"Based on EDA, following variables can be considered for next steps\n\n* D_41, D_43, D_44, D_45, D_47, D_51, D_55, D_61, D_62, **D_64**, D_70, D_75, D_78, **D_114, D_120**\n\n* S_7\n\n* P_2\n\n* B_1, B_3, B_4, B_7, B_8, B_9, B_16, B_18, B_19, B_20, B_22, **B_30**, B_33, **B_38**\n\n* R_1, R_2, R_3, R_27","metadata":{}},{"cell_type":"markdown","source":"---\n# Next Steps\n\n---","metadata":{}},{"cell_type":"markdown","source":"* **For Model building and Prediction**\n\n    * Dataset is of large size and requires appropriate handling\n        * Usage of Dask or using compressed file format (Parquet or Feather)\n        * Using reduced size of training dataset is also an option\n    * Imbalance in training dataset will require handling\n        * Usage of SMOTE along with combination of oversampling and undersampling\n    * Variables with Missing value needs handling\n        * Variables with significant missing value (>= 80%) can be dropped\n        * Variables with missing value (20%) needs imputation\n    * Variables with high correlation (Multicollinearity) needs handling\n        * Variables (D, S, P, B, R) with high positive or negative correlation (>= 0.8) can be filtered\n    * Variables (D, S, P, B, R) having favorable correlation (>=+-0.25) with target variable should be preferred\n    * Categorical Variables requires encoding along with detailed correlation check with respect to target variable\n    * Feature Engineering mechanism appropriate for this dataset need to be explored\n    * Dimensionality Reduction via PCA can help reduce number of variables as needed\n    * Feature Importance mechanism can help select appropriate variable\n    * Standardization of variables will ensure same level of scale for model training/evaluation\n    \nhttps://www.kaggle.com/code/girishkumarsahu/american-express-default-prediction-ml-model","metadata":{}},{"cell_type":"markdown","source":"---\n**Thank you and Happy Learning.**\n\n---","metadata":{}},{"cell_type":"code","source":"thank_you_str=\"Thanks,Happy Learning,Collaboration,Thankyou,Keep Learning\"\n# create WordCloud with converted string\nwordcloud = WordCloud(width = 1000, height = 500, random_state=1, background_color='white', collocations=True).generate(thank_you_str)\nplt.figure(figsize=(20, 20))\nplt.imshow(wordcloud) \nplt.axis(\"off\")\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-24T04:07:14.772479Z","iopub.execute_input":"2022-07-24T04:07:14.773112Z","iopub.status.idle":"2022-07-24T04:07:15.277624Z","shell.execute_reply.started":"2022-07-24T04:07:14.773065Z","shell.execute_reply":"2022-07-24T04:07:15.276692Z"},"trusted":true},"execution_count":null,"outputs":[]}]}