{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
Unlocking Insights: EDA Practices of Top Kagglers
\n\n
\n \"image\"\n
\n
\n Photo from Unsplash\n
\n\n#
Introduction
\n\n## Goal\n\nWelcome to the first notebook in a series dedicated to unveiling the magic behind the top ranks in [American Express - Default Prediction\n](https://www.kaggle.com/competitions/amex-default-prediction) competition. As every data scientist knows, a competition isn't just won at the modeling stage – it starts with a deep and thoughtful Exploratory Data Analysis (EDA). \n\nThis notebook delves deep into the EDA practices that made a mark in the competition, guiding many to insightful findings. Whether you're an aspiring data enthusiast or an established data professional, there are gems here for everyone.\n\n## Data Fundamentals: Grasping the Basics\n\n***What is Tabular Data?***\n\nIn statistics, tabular data refers to data that is organized in a table with rows and columns. Within the table, the rows represent observations and the columns represent attributes for those observations. Tabular data refers to structured data that is organized in rows and columns, similar to what you'd find in spreadsheets or relational databases. Each row typically represents a unique record or item, and each column represents a specific attribute or field of that item. Tabular data is the simpler data you can feed to a machine learning model. It's called tabular because it can be represented in a table with rows and columns. You can think of these as Excel spreadsheets, but very common are files called CSV, which stands for comma-separated values, or TSV, which is tab-separated values. There are many other formats which have a tabular nature. Essentially, it's a file where data is organized in rows and columns. You find tables also in databases or when you have a collection of files that are all interconnected with keys that have relations with one another.\n\n\n***4 Types Of Data – Nominal, Ordinal, Discrete and Continuous***\n\n\"image\"\n\n\nThere are two types of data: **Qualitative** and **Quantitative data**, which are further classified into:\n\nThe data is classified into four categories:\n\n1. **Nominal data**\n\n1. **Ordinal data**\n\n1. **Discrete data**\n\n1. **Continuous data**\n\n**Qualitative or Categorical Data** is data that can’t be measured or counted in the form of numbers. These types of data are sorted by category, not by number. That’s why it is also known as Categorical Data. These data consist of audio, images, symbols, or text. The gender of a person, i.e., male, female, or others, is qualitative data. The other examples of qualitative data are :\n\n- What language do you speak\n- Favorite holiday destination\n- Opinion on something (agree, disagree, or neutral)\n- Colors\n\nThe Qualitative data are further classified into two parts :\n\n1. **Nominal Data:** Nominal Data is used to label variables without any order or quantitative value. The color of hair can be considered nominal data, as one color can’t be compared with another color. The name “nominal” comes from the Latin name “nomen,” which means “name.” With the help of nominal data, we can’t do any numerical tasks or can’t give any order to sort the data. These data don’t have any meaningful order; their values are distributed into distinct categories. Examples of Nominal Data :\n\n 1. Colour of hair (Blonde, red, Brown, Black, etc.)\n 1. Marital status (Single, Widowed, Married)\n 1. Nationality (Indian, German, American)\n 1. Gender (Male, Female, Others)\n 1. Eye Color (Black, Brown, etc.)\n\n1. **Ordinal Data:** Ordinal data have natural ordering where a number is present in some kind of order by their position on the scale. These data are used for observation like customer satisfaction, happiness, etc., but we can’t do any arithmetical tasks on them. Ordinal data is qualitative data for which their values have some kind of relative position. These kinds of data can be considered “in-between” qualitative and quantitative data. The ordinal data only shows the sequences and cannot use for statistical analysis. Compared to nominal data, ordinal data have some kind of order that is not present in nominal data. Examples of Ordinal Data :\n\n 1. When companies ask for feedback, experience, or satisfaction on a scale of 1 to 10\n 1. Letter grades in the exam (A, B, C, D, etc.)\n 1. Ranking of people in a competition (First, Second, Third, etc.)\n 1. Economic Status (High, Medium, and Low)\n 1. Education Level (Higher, Secondary, Primary)\n\n\n**Quantitative data** can be expressed in numerical values, making it countable and including statistical data analysis. These kinds of data are also known as Numerical data. It answers the questions like “how much,” “how many,” and “how often.” For example, the price of a phone, the computer’s ram, the height or weight of a person, etc., falls under quantitative data. Quantitative data can be used for statistical manipulation. These data can be represented on a wide variety of graphs and charts, such as bar graphs, histograms, scatter plots, boxplots, pie charts, line graphs, etc.\n\nExamples of Quantitative Data : \n\n- Height or weight of a person or object\n- Room Temperature\n- Scores and Marks (Ex: 59, 80, 60, etc.)\n- Time\n\nThe Quantitative data are further classified into two parts :\n\n1. **Discrete Data:** The term discrete means distinct or separate. The discrete data contain the values that fall under integers or whole numbers. The total number of students in a class is an example of discrete data. These data can’t be broken into decimal or fraction values. The discrete data are countable and have finite values; their subdivision is not possible. These data are represented mainly by a bar graph, number line, or frequency table.Examples of Discrete Data : \n\n 1. Total numbers of students present in a class\n 1. Cost of a cell phone\n 1. Numbers of employees in a company\n 1. The total number of players who participated in a competition\n 1. Days in a week\n \n1. **Continuous data** are in the form of fractional numbers. It can be the version of an android phone, the height of a person, the length of an object, etc. Continuous data represents information that can be divided into smaller levels. The continuous variable can take any value within a range. The key difference between discrete and continuous data is that discrete data contains the integer or whole number. Still, continuous data stores the fractional numbers to record different types of data such as temperature, height, width, time, speed, etc. Examples of Continuous Data : \n\n 1. Height of a person\n 1. Speed of a vehicle\n 1. “Time-taken” to finish the work \n 1. Wi-Fi Frequency\n 1. Market share price","metadata":{}},{"cell_type":"markdown","source":"#
Exploratory Data Analysis (EDA) Intro and Problem Formulation
\n","metadata":{}},{"cell_type":"code","source":"pip install seaborn==0.11.2","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pip install pandas==1.5.3","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom typing import Tuple, List\n\nimport seaborn as sns\n\nimport matplotlib.colors\n\nimport matplotlib.pyplot as plt\nimport matplotlib as mpl\nimport plotly.graph_objects as go\nimport plotly.figure_factory as ff\nfrom plotly.subplots import make_subplots\nimport plotly.express as px\nfrom plotly.offline import init_notebook_mode, iplot\nfrom plotly.colors import n_colors\nimport matplotlib.patches as mpatches","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-01T17:47:34.581532Z","iopub.execute_input":"2023-10-01T17:47:34.581963Z","iopub.status.idle":"2023-10-01T17:47:37.355816Z","shell.execute_reply.started":"2023-10-01T17:47:34.581928Z","shell.execute_reply":"2023-10-01T17:47:37.354921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.read_feather('../input/amexfeather/train_data.ftr')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-01T17:47:37.360507Z","iopub.execute_input":"2023-10-01T17:47:37.361294Z","iopub.status.idle":"2023-10-01T17:47:56.373632Z","shell.execute_reply.started":"2023-10-01T17:47:37.361266Z","shell.execute_reply":"2023-10-01T17:47:56.372381Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Competition's Goal\nIn this competition, the main task is to use historical data about a customer, which is collected month-by-month (like their spending habits, payment patterns, credit card balances, etc.) to estimate the likelihood that they will fail to repay what they owe on their credit card in the future. Here, the \"target binary variable\" is what participants of the competition are trying to predict. It's binary because it has two possible values: \"default\" (customer didn't pay) or \"no default\" (customer paid). The way this target is determined is by watching the customer's behavior for 18 months after their most recent credit card statement. If, during those 18 months of observation, the customer fails to make a payment within 120 days (4 months) from the date of their last statement, they are marked as having defaulted on their credit card balance. So, even if they make a payment on the 121st day or anytime after, for the purposes of this competition, they're categorized as having defaulted. \n\n## Analysis of Credit Card Statements per Customer\n\nUpon analyzing the distribution of credit card statements across customers, we observe the following pattern:\n\n1. A significant majority, precisely 80%, of our customers have 13 statements.\n1. The remaining 20% possess a varied number of statements, ranging from 1 to 12.\n\n**Insight:** The model cannot handle this variation in statement counts because it cannot handle different input sizes for each customer. An alternative approach to address this variability might be to consider only the latest statement or compute an average across all statements for each customer. Some machine learning models can indeed handle variable input sizes (like recurrent neural networks for sequences), while others require fixed-size inputs. ","metadata":{}},{"cell_type":"code","source":"nbr_customers = train[\"customer_ID\"].nunique()\nstatements_df = train.customer_ID.value_counts().value_counts().sort_index(ascending=False).reset_index().rename(columns={\"index\": \"Nbr of Statements per customer\", \"customer_ID\": \"Nbr of customers\"})\nstatements_df[\"% of customers\"] = np.round((statements_df[\"Nbr of customers\"] / nbr_customers)*100, 2)\nstatements_df.style.background_gradient(axis=0, cmap='YlOrBr', subset=[\"Nbr of customers\", \"% of customers\"])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-01T17:47:56.375039Z","iopub.execute_input":"2023-10-01T17:47:56.375729Z","iopub.status.idle":"2023-10-01T17:47:57.921734Z","shell.execute_reply.started":"2023-10-01T17:47:56.375689Z","shell.execute_reply":"2023-10-01T17:47:57.920425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's find out when these customers got their last statement. The histogram of the last statement dates shows that every train customer got his last statement in March of 2018. The first four Saturdays (March 3, 10, 17, 24) have more statements than an average day.\n\n## Missing Values\n","metadata":{}},{"cell_type":"code","source":"# Returns a Series giving the count of null values in each column of the DataFrame.\n# It's a handy way to quickly check if your data has any missing values and how many there are in each column.\nmissing_values = train.isnull().sum()\n# It converts the Series to a DataFrame using .to_frame().\n# Then, it resets the index of the DataFrame so that the original index (which was the column names from the DataFrame) becomes a column \n# in the DataFrame, and a new numerical index is created using .reset_index().\n# Finally, it sorts the DataFrame by the \"Number of null values\" column in descending order (so the column with the most missing values will be at the top)\nmissing_values_df = missing_values.to_frame().reset_index().rename(columns={\"index\": \"Feature\", 0: \"Number of missing values\"}).sort_values([\"Number of missing values\"], ascending=False).reset_index(drop=True)\nmissing_values_df[\"% Missing values\"] = np.round((missing_values_df[\"Number of missing values\"] / train.shape[0])*100, 2)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-01T17:47:57.925289Z","iopub.execute_input":"2023-10-01T17:47:57.925725Z","iopub.status.idle":"2023-10-01T17:48:02.138356Z","shell.execute_reply.started":"2023-10-01T17:47:57.925694Z","shell.execute_reply":"2023-10-01T17:48:02.137611Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"{missing_values_df[ missing_values_df['% Missing values'] > 89].shape[0]} out of {missing_values_df.shape[0]}, ({np.round((missing_values_df[ missing_values_df['% Missing values'] > 89].shape[0] / missing_values_df.shape[0] ) *100, 2)}%), features in the dataset have missing values in more than 89% of their rows.\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-01T17:48:02.139933Z","iopub.execute_input":"2023-10-01T17:48:02.140720Z","iopub.status.idle":"2023-10-01T17:48:02.149473Z","shell.execute_reply.started":"2023-10-01T17:48:02.140681Z","shell.execute_reply":"2023-10-01T17:48:02.148269Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"missing_values_df.head(40).style.background_gradient(axis=0, cmap='YlOrBr', subset=[\"% Missing values\"])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-01T17:48:02.151128Z","iopub.execute_input":"2023-10-01T17:48:02.151630Z","iopub.status.idle":"2023-10-01T17:48:02.174069Z","shell.execute_reply.started":"2023-10-01T17:48:02.151588Z","shell.execute_reply":"2023-10-01T17:48:02.173385Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Insights:**\n\n- There are many columns with missing values: Dropping all columns which have missing values is not a sensible strategy.\n- There are many rows with missing values: Dropping all rows which have missing values is not a sensible strategy.\n- Many decision-tree based algorithms can deal with missing values. If we choose such a model, we don't need to change the missing values.\nNeural networks and other estimators cannot deal with missing values. If we choose such a model, we need to impute values. See this [guide](https://www.kaggle.com/code/parulpandey/a-guide-to-handling-missing-values-in-python) for an overview of the many imputation options.\n","metadata":{}},{"cell_type":"code","source":"train = train.groupby('customer_ID').tail(1).set_index('customer_ID')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-01T17:48:02.174969Z","iopub.execute_input":"2023-10-01T17:48:02.176020Z","iopub.status.idle":"2023-10-01T17:48:04.917914Z","shell.execute_reply.started":"2023-10-01T17:48:02.175992Z","shell.execute_reply":"2023-10-01T17:48:04.916706Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_df=train.reset_index().groupby(\"S_2\")[\"customer_ID\"].nunique().reset_index()\n\nfig=go.Figure()\n\nfig.add_trace(\n go.Scatter(\n x=plot_df[\"S_2\"], \n y=plot_df[\"customer_ID\"], \n mode=\"lines\",\n line=dict(color=\"#6082B6\", width=3), \n hovertemplate = \"\"\n )\n)\n\nlarge_title_format = \"Frequency of Customer Statements\"\nsmall_title_format = \"\"\n\nfig.update_layout(title = large_title_format + \"
\" + small_title_format,\n hovermode=\"x unified\", width=800,height=500,\n xaxis_title='Statement Date', yaxis_title=\"Number of Statements Issued\", font = dict(color = '#7b6b59'),\n plot_bgcolor='#fff')\nfig.show()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-01T17:48:04.919671Z","iopub.execute_input":"2023-10-01T17:48:04.920388Z","iopub.status.idle":"2023-10-01T17:48:05.554098Z","shell.execute_reply.started":"2023-10-01T17:48:04.920343Z","shell.execute_reply":"2023-10-01T17:48:05.552992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mpl.rcParams.update(mpl.rcParamsDefault)\n\nfig1 = plt.figure(figsize=(2,2),facecolor='white')\n\nax1 = fig1.add_subplot(1,1,1)\nfont = 'monospace'\nax1.text(0.6, 0.8, \"Key figures of the training set\",color='#7b6b59',fontsize=18, fontweight='bold', fontfamily=font, ha='center')\n\n\nax1.text(0, 0.4, \"{:,d}\".format(train.shape[0]), color='#6082B6', fontsize=24, fontweight='bold', fontfamily=font, ha='center')\nax1.text(0, 0.100, \"# of customers\",color='#7b6b59',fontsize=15, fontweight='light', fontfamily=font,ha='center')\n\n\nax1.text(1.1, 0.4, \"{}\".format(train.shape[1]), color='#6082B6', fontsize=24, fontweight='bold', fontfamily=font, ha='center')\nax1.text(1.1, 0.100, \"# of features\",color='#7b6b59',fontsize=15, fontweight='light', fontfamily=font, ha='center')\n\n\nax1.set_yticklabels('')\nax1.tick_params(axis='y',length=0)\nax1.tick_params(axis='x',length=0)\nax1.set_xticklabels('')\n\nfor direction in ['top','right','left','bottom']:\n ax1.spines[direction].set_visible(False)\n\nfig1.subplots_adjust(top=0.9, bottom=0.2, left=0, hspace=1)\n\nfig1.patch.set_linewidth(3)\nfig1.patch.set_edgecolor('#6082B6')\nfig1.patch.set_facecolor('white')\nax1.set_facecolor('white')\n\nplt.show()\nplt.close()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-01T17:48:05.555736Z","iopub.execute_input":"2023-10-01T17:48:05.556681Z","iopub.status.idle":"2023-10-01T17:48:05.729041Z","shell.execute_reply.started":"2023-10-01T17:48:05.556643Z","shell.execute_reply":"2023-10-01T17:48:05.727960Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"delinquency_features = [c for c in train.columns if c.startswith('D_')]\nspend_features = [c for c in train.columns if c.startswith('S_')]\npayment_features = [c for c in train.columns if c.startswith('P_')]\nbalance_features = [c for c in train.columns if c.startswith('B_')]\nrisk_features = [c for c in train.columns if c.startswith('R_')]\n\nprint(f\"Total number of Delinquency variables: {len(delinquency_features)}\")\nprint(f\"Total number of Spend variables: {len(spend_features)}\")\nprint(f\"Total number of Payment variables: {len(payment_features)}\")\nprint(f\"Total number of Balance variables: {len(balance_features)}\")\nprint(f\"Total number of Risk variables: {len(risk_features)}\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-01T17:48:05.730114Z","iopub.execute_input":"2023-10-01T17:48:05.730403Z","iopub.status.idle":"2023-10-01T17:48:05.738394Z","shell.execute_reply.started":"2023-10-01T17:48:05.730379Z","shell.execute_reply":"2023-10-01T17:48:05.737306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"labels=['Delinquency', 'Spend','Payment','Balance','Risk']\nvalues= [len(delinquency_features), len(spend_features),len(payment_features), len(balance_features),len(risk_features)]\n\n\n## First Plot\nfig = go.Figure(data=[go.Pie(\n labels=labels, \n values=values,\n hole=.4\n)])\n\nfig.update_traces(marker=dict(colors=['#a43725', '#e0d5bd', '#beb29e', '#E6b6a4', '#c07156']))\n\nlarge_title_format = \"Feature Distribution\"\nsmall_title_format = \"\"\n\nlayout = dict(\n title = large_title_format + \"
\" + small_title_format,\n font = dict(color = '#7b6b59'),\n showlegend = True,\n margin = dict(t=80,pad=0),\n\n)\nfig[\"layout\"].update(layout)\n\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-01T17:48:05.739829Z","iopub.execute_input":"2023-10-01T17:48:05.740103Z","iopub.status.idle":"2023-10-01T17:48:05.781933Z","shell.execute_reply.started":"2023-10-01T17:48:05.740079Z","shell.execute_reply":"2023-10-01T17:48:05.780907Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"titles=['Delinquency '+str(i).split('_')[1] if i.startswith('D') else 'Spend '+str(i).split('_')[1] \n if i.startswith('S') else 'Payment '+str(i).split('_')[1] if i.startswith('P') \n else 'Balance '+str(i).split('_')[1] if i.startswith('B') else \n 'Risk '+str(i).split('_')[1] for i in train.columns[:-1]]\ncat_cols=['Balance 30', 'Balance 38', 'Delinquency 63', 'Delinquency 64', 'Delinquency 66', 'Delinquency 68',\n 'Delinquency 114', 'Delinquency 116', 'Delinquency 117', 'Delinquency 120', 'Delinquency 126', 'Target']\n\ntitles.append('Target')\ntrain.columns=titles","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-10-01T17:48:05.783228Z","iopub.execute_input":"2023-10-01T17:48:05.784174Z","iopub.status.idle":"2023-10-01T17:48:05.791153Z","shell.execute_reply.started":"2023-10-01T17:48:05.784143Z","shell.execute_reply":"2023-10-01T17:48:05.790146Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
Target Variable Analysis
\n\n- Plot the distribution of the target variable (pie chart or bar chart since it's binary).\n- Calculate the base rate (i.e., what percentage of clients default).\n","metadata":{}},{"cell_type":"code","source":"# Count occurrences of each target value\ntarget_counts = train[\"Target\"].value_counts()\ntotal_counts = len(train)\n\n# Calculate percentages\npercentages = (target_counts / total_counts * 100).round(2)\n\n# Create the bar plot\nfig = go.Figure()\n\n\n# Add bar plot for target distribution\n# Customizing the legend entries using the 'name' attribute\nfig.add_trace(go.Bar(\n x=['0'],\n y=[target_counts[0]],\n text=f\"{percentages[0]}%\",\n textposition='auto',\n marker_color='#6082B6',\n name='Non-Default / Paid' # Custom legend entry\n))\n\nfig.add_trace(go.Bar(\n x=['1'],\n y=[target_counts[1]],\n text=f\"{percentages[1]}%\",\n textposition='auto',\n marker_color='#933A16',\n name='Default' # Custom legend entry\n))\n\n\nlarge_title_format = \"Target Distribution\"\nsmall_title_format = \"\"\n\nlayout = dict(\n title = large_title_format + \"
\" + small_title_format,\n font = dict(color = '#7b6b59'),\n xaxis=dict(showticklabels=False),\n margin = dict(t=80,pad=0),\n plot_bgcolor='#fff',\n\n)\nfig[\"layout\"].update(layout)\n\n\nfig.show()\n","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We start by reading the labels for the training data. There are neither missing values nor duplicated customer_IDs. Of the 458913 customer_IDs, 74 % have a label of 0 (good customer, no default) and 26 % have a label of 1 (bad customer, default).\n\nWe know that the good customers have been subsampled by a factor of 20; this means that in reality there are 6.8 million good customers. 98 % of the customers are good; 2 % are bad.\n\n**Insight:**\n\n- **The classes are imbalanced.** A StratifiedKFold for cross-validation is recommended. Because the classes are imbalanced, accuracy would be a bad metric to evaluate a classifier. The competition metric is a mix of area under the roc curve (auc) and recall.\n","metadata":{}},{"cell_type":"markdown","source":"#
EDA for Categorical Features
\n\nThe distribution of a categorical variable refers to the frequency (or proportion) of each category or level within that variable. Unlike numerical variables, which are characterized by a range of values, categorical variables consist of distinct groups or categories. Understanding the distribution means understanding how often each category occurs in your dataset. The **Distribution of Categorical Variables** is summarized using:\n\n- Visual display: pie chart or bar chart, supplemented by\n- Numerical measures: frequency table of category counts and percentages.\n\nDescriptive Statistics:\n\n1. **Frequency Distribution:** Count the number of occurrences for each category. This simply counts the number of occurrences for each category within the variable.\n\n1. **Relative Frequency (or Proportion) Distribution - Percentage Count:** Get the percentage distribution for each category. This represents the proportion or percentage of each category relative to the whole dataset.\n\nVisualization:\n\n1. **Bar Chart:** For each categorical variable, use a bar chart to visualize the frequency distribution. Bar charts can be used in many ways, **one of the common use is to visualize the data distribution of categorical variables in data.** X-axis being the unique category values and Y-axis being the frequency of each value.\n\n1. **Stacked Bar Chart / Mosaic Plot:** Show the distribution of the target variable (default or not) with respect to each category. This will help you understand if certain categories have higher or lower default rates. A stacked bar chart divides a bar for each category of one variable into segments that represent the distribution of a second categorical variable. Example: Let's consider the categorical variable \"Employment Type\" with categories like \"Full-time\", \"Part-time\", \"Self-employed\", etc. If you want to see how defaults (your target variable with categories \"Default\" and \"No Default\") are distributed across each employment type, you'd:\n\n - Plot a bar for each employment type.\n - Divide or \"stack\" each bar based on the number of defaults and non-defaults.\n\n1. **Grouped bar charts** are another effective way to compare the distribution of a categorical variable across different levels of another categorical variable (like your target variable).\n\n 1. **Comparing Absolute Frequencies:** Each colored bar within a group will show the absolute number of occurrences for each category of the target variable (e.g., \"Default\" vs. \"No Default\") for a particular category of your feature variable. By comparing the heights of the colored bars side by side, you can see which groups have more defaults in absolute terms.\n \n 1. **Comparing Across Categories:** You can also compare across the main categories of the feature variable. For instance, if you're looking at \"Employment Type,\" you can compare the \"Default\" bars for \"Full-time\", \"Part-time\", and \"Self-employed\" directly since they're placed side by side.\n\n 1. **Visualizing Disparities:** If there's a stark difference in the heights of bars for a particular category (like a much taller \"Default\" bar for \"Self-employed\" compared to \"No Default\"), it indicates a higher number of defaults for that group.\n\n 1. **Potential Caveats:** As with the stacked bar charts, be cautious about interpreting raw counts. A category might have higher defaults just because it has more data points overall. Always consider looking at proportions or default rates for a more accurate picture. To address this concern when using grouped bar charts:\n\n 1. **Supplementary Analysis:** It's often helpful to complement these charts with another analysis or visualization that does focus on proportions or rates. For instance, a table next to the chart that lists default rates by category can be very insightful.\n\n 1. **Alternative Visualization:** Consider creating a separate chart that visualizes the proportion of defaults in each category. This could be a simple bar chart where the height of the bar represents the default rate for each category.\n\n 1. **Annotation:** If you have a key insight from another analysis (like the actual default rate for a category), you can annotate the grouped bar chart with this information. For example, above the \"Self-employed\" bars, you might note \"Default rate: 20%\".\n\n 1. **Educate the Audience:** If presenting to stakeholders, it's essential to emphasize that they're looking at absolute numbers and highlight the importance of also considering proportions.\n\n 1. **Diversify Visualizations:** Sometimes, using multiple visualization types for the same data can provide a more comprehensive understanding. While the grouped bar chart showcases absolute numbers, another chart (like a pie chart or a simple proportion bar chart) can present the proportional breakdown for clearer risk assessment.\n\n 1. **Ease of Interpretation:** Grouped bar charts are especially useful when you have a few distinct categories to compare. If there are too many categories or if the differences are subtle, the chart may become cluttered and harder to interpret than stacked bar charts.\n \n 1. **Trends and Patterns:** You might observe certain trends, like certain categories consistently having higher defaults across various features. For instance, if you observe that younger age groups show a higher default rate across various employment types, you might conclude that age plays a more dominant role in determining defaults than employment type.\n\nTo sum up, grouped bar charts, while presenting data in absolute terms, can still provide valuable insights into the distribution of your target variable across different categories. As always, the key lies in thoughtful interpretation and considering the broader context. In essence, while grouped bar charts are useful for visualizing and comparing raw counts across categories, it's crucial to be aware of their limitations and to complement them with additional analyses or visualizations that capture proportions or rates to convey a more accurate picture of risk.\n\n\n**Conclusions and Insights You Can Draw:**\n\n1. **Identifying High-Risk Categories:** If a certain category has a much higher proportion of defaults than others, that category might be considered higher risk. For instance, if \"Self-employed\" individuals have a high default rate, lending policies might be reviewed for this group.\n - You can easily see if, for instance, \"Self-employed\" individuals have a higher proportion of defaults compared to \"Full-time\" employees.\n - The absolute height of the bar can show you how many people belong to each employment category, while the segments give you a proportionate view of defaults within each category.\n\n1. **Feature Importance for Modeling:** Categories with clear differences in default rates can be indicative of important features that a machine learning model should not overlook.\n\n1. **Potential Bias Identification:** If a certain category is disproportionately leading to defaults, it might be worth investigating if there's any data bias or external factors causing this pattern.\n\n1. **Policy Implications:** For financial institutions, understanding which groups are more likely to default can lead to tailored financial literacy programs, different engagement strategies, or alternative payment plans for those groups.\n\n\nThe raw count (or frequency) of occurrences in a category can sometimes be misleading. If a particular category has a high number of defaults, but also a high number of non-defaults, then it may not necessarily indicate higher risk; it might just indicate that the category itself is more common in the dataset.\n\nTo address this:\n\n\n\n1. **Focus on Proportions or Rates:** Rather than just looking at raw counts, consider the default rate (i.e., the proportion of defaults within the category). For instance, if \"Self-employed\" individuals comprise 200 defaults out of a total of 1,000 self-employed individuals, the default rate is 20%. If \"Full-time\" employees have 500 defaults out of a total of 5,000 full-time employees, their default rate is 10%. Even though the absolute number of defaults is higher for full-time employees, the risk, as represented by the default rate, is higher for the self-employed group.\n\n1. **Visual Tools:** This is where tools like stacked bar charts are handy. While the height of the bar might represent the total count, the segmented (or stacked) portions represent the proportion of each category (default vs. non-default). It's easy to see and compare proportions across categories.\n\n1. **Dive Deeper with Segmentation:** If a category has a high frequency overall, consider breaking it down further if possible. For example, \"Self-employed\" can be further segmented into \"Freelancers,\" \"Business Owners,\" \"Contractors,\" etc. This might reveal more specific patterns or risk profiles.\n\n1. **Contextualize with Other Features:** It's also important to see how this category interacts with other features. Perhaps \"Self-employed\" individuals with a certain credit score range, or from a particular region, are more prone to defaulting than others.\n\nIn conclusion, while raw counts provide a starting point, it's vital to dive deeper and consider proportions, utilize appropriate visualizations, and perform further analyses to draw meaningful conclusions about risk.\n\n**Broadly speaking, if the distributions for target=0 and target=1 differ significantly for a given feature, it suggests that the feature carries information about the target variable, and it might be useful for predictive modeling.** When the distributions for the two target values are distinct or significantly different for a specific feature, it indicates that there's a relationship or association between that feature and the outcome. In other words, knowing the value of the feature might give some insight or increase the likelihood of predicting the target correctly.","metadata":{}},{"cell_type":"code","source":"# This code visualizes the distribution of cat_cols (which presumably represents categorical columns) with respect to the target variable (Target).\n# Each category value is shown in subplots, and for each category value, two bars are displayed: \n# one for target 1 (Default) and another for target 0 (Paid). The colors and styles are also customized.\npal, color=['#6082B6', '#933A16'], ['#6082B6', '#933A16']\nrgb=['rgba'+str(matplotlib.colors.to_rgba(i,0.7)) for i in pal]\n\n# Creates a subplot grid with 4 rows and 3 columns. \n# The titles of the subplots are taken from the variable cat_cols, excluding its last element. \nfig = make_subplots(\n rows=4, \n cols=3, \n subplot_titles=cat_cols[:-1], \n vertical_spacing=0.1\n)\n\nrow=0\nc=[1,2,3]*5\nplot_df=train[cat_cols]\n# Loop to Add Bars to Subplots\n# This loop runs through each column in cat_cols (excluding the last one) \nfor i,col in enumerate(cat_cols[:-1]):\n \n if i%3==0:\n row+=1\n \n plot_df[col]=plot_df[col].astype(object).fillna(\"N/A\")\n \n df=plot_df.groupby(col)['Target'].value_counts(dropna=False).rename('count').reset_index()\n \n all_categories = df[col].unique()\n \n # Getting values for Target=1\n df_target_1 = df[df.Target == 1].set_index(col)['count'].reindex(all_categories).fillna(0).reset_index()\n df_target_1[col]=df_target_1[col].values.astype(str)\n\n \n # Getting values for Target=0\n df_target_0 = df[df.Target == 0].set_index(col)['count'].reindex(all_categories).fillna(0).reset_index()\n df_target_0[col]=df_target_0[col].values.astype(str)\n\n \n # Adds a bar for the target 1 (Default) and another for target 0 (Paid) to the subplots.\n fig.add_trace(\n go.Bar(\n x=df_target_1[col],\n y=df_target_1['count'],\n marker_color=rgb[1], \n marker_line=dict(color=pal[1],width=2), \n hovertemplate='Value %{x} Frequency = %{y}',\n name='Default', \n showlegend=(True if i==0 else False)\n ),\n row=row, \n col=c[i]\n )\n \n fig.add_trace(\n go.Bar(\n x=df_target_0[col], \n y=df_target_0['count'],\n marker_color=rgb[0],\n marker_line=dict(color=pal[0],width=2),\n hovertemplate='Value %{x} Frequency = %{y}',\n name='Paid', showlegend=(True if i==0 else False)\n ),\n row=row, \n col=c[i]\n )\n if i%3==0:\n fig.update_yaxes(title='Frequency',row=row,col=c[i])\n\nlarge_title_format = \"Distribution of Categorical Variables\"\nsmall_title_format = \"\"\n\ntitle_format = {\n 'text': large_title_format + \"
\" + small_title_format,\n 'xanchor': 'center', # anchor point\n 'x': 0.5 # x-coordinate in fractional coordinates, 0.5 represents the center\n}\n\nfig.update_layout(\n legend=dict(orientation=\"h\",yanchor=\"bottom\",y=1.03,xanchor=\"right\",x=0.2),\n barmode='group',\n height=1500,\n width=900,\n title=title_format,\n font = dict(color = '#7b6b59'),\n plot_bgcolor='#fff'\n)\n\nfig.show()\n","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"According to the data description, there are eleven categorical features. We plot histograms for target=0 and target=1. For the features which have missing values, the missing values are represented by the rightmost bar of the histogram.\n\n**Insight:**\n\n- Every feature has at most eight categories (including a nan category). One-hot encodings are feasible.\n- The distributions for target=0 and target=1 differ. This means that every feature might give some information about the target.","metadata":{}},{"cell_type":"code","source":"pal, color=['#6082B6', '#933A16'], ['#6082B6', '#933A16']\nrgb=['rgba'+str(matplotlib.colors.to_rgba(i,0.7)) for i in pal]\n\nfig = make_subplots(rows=4, cols=3, subplot_titles=cat_cols[:-1], vertical_spacing=0.1)\n\nrow = 0\nc = [1,2,3] * 5\nplot_df = train[cat_cols]\n\nfor i, col in enumerate(cat_cols[:-1]):\n \n if i % 3 == 0:\n row += 1\n \n plot_df[col] = plot_df[col].astype(object).fillna(\"N/A\")\n \n # Grouping by category and target to get counts\n df = plot_df.groupby(col)['Target'].value_counts(dropna=False).rename('count').reset_index()\n \n # Getting total counts per category\n df_total = plot_df[col].value_counts().rename('total').reset_index()\n df_total.columns = [col, 'total']\n \n # Merging to get total counts alongside individual target counts\n df = df.merge(df_total, on=col)\n \n # Calculating percentage\n df['percentage'] = (df['count'] / df['total']) * 100\n \n all_categories = df[col].unique()\n for target in [1, 0]:\n \n df_target = df[df.Target == target].set_index(col)['count'].reindex(all_categories).fillna(0).reset_index()\n df_percentage = df[df.Target == target].set_index(col)['percentage'].reindex(all_categories).fillna(0).reset_index()\n df_target[col]=df_target[col].values.astype(str)\n \n hover_text = [f'Value: {cat}
Percentage: {perc:.2f}%' for cat, perc in zip(df_target[col], df_percentage['percentage'])]\n \n fig.add_trace(\n go.Bar(\n x=df_target[col],\n y=df_target['count'], # Using actual counts for y-values\n hovertext=hover_text, # Add custom hover text here\n marker_color=rgb[target], \n marker_line=dict(color=pal[target], width=2), \n hoverinfo='text', # Use the custom hover text\n name='Default' if target == 1 else 'Paid', \n showlegend=(True if i == 0 else False)\n ),\n row=row, \n col=c[i]\n )\n\n if i % 3 == 0:\n fig.update_yaxes(title='Frequency', row=row, col=c[i])\n\n \nlarge_title_format = \"Distribution of Categorical Variables\"\nsmall_title_format = \"\"\n\ntitle_format = {\n 'text': large_title_format + \"
\" + small_title_format,\n 'xanchor': 'center',\n 'x': 0.5\n}\n\nfig.update_layout(\n legend=dict(orientation=\"h\", yanchor=\"bottom\", y=1.03, xanchor=\"right\", x=0.2),\n barmode='stack',\n height=1500,\n width=900,\n title=title_format,\n font=dict(color='#7b6b59'),\n plot_bgcolor='#fff'\n)\n\nfig.show()\n","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
Density Plots, EDA for Numerical Features
\n\n1. **Density Plots:** Gives a smoothed version of the histogram, useful for understanding the shape of the distribution.\n1. **Correlation Heatmap:** To understand the relationship between multiple numerical features.\n1. **Histograms:** To see the distribution of data.\n1. **Box Plots:** Helps in identifying outliers and understanding the distribution.\n\n#### What is a histogram? \n\nA histogram is a chart that plots the distribution of a numeric variable’s values as a series of bars. Each bar typically covers a range of numeric values called a bin or class; a bar’s height indicates the frequency of data points with a value within the corresponding bin. Histograms are good for showing general distributional features of dataset variables. You can see roughly where the peaks of the distribution are, whether the distribution is skewed or symmetric, and if there are any outliers. In order to use a histogram, we simply require a variable that takes continuous numeric values. This means that the differences between values are consistent regardless of their absolute values. For example, even if the score on a test might take only integer values between 0 and 100, a same-sized gap has the same meaning regardless of where we are on the scale: the difference between 60 and 65 is the same 5-point size as the difference between 90 to 95. To construct a histogram:\n\n1. The data range is divided into a set of bins.\n1. The data points are sorted into each bin based on their value.\n1. The number of data points that fall into each bin are then represented as vertical bars, with the height of each bar corresponding to the number of data points (frequency) in that bin.\n \nKey characteristics of a histogram:\n\n1. **Bins:** These are intervals into which the data range is divided. The width and number of bins can significantly affect the resulting shape and interpretation of the histogram.\n\n1. **Frequency:** This is the number of data points in each bin. The height of each bar represents this frequency.\n\n1. **Shape:** The overall appearance of the histogram can give insights into the underlying distribution of the data. For instance, a histogram might appear bell-shaped, indicating a normal distribution, or it might be skewed to the left or right.\n\n1. **No Gaps:** Unlike bar charts, which represent categorical data and can have gaps between bars, histograms typically have no gaps between bars since they represent continuous, numerical data.\n\n#### Differences between a bar chart and a histogram:\n\n1. **Type of Data:** Histograms are used for continuous numerical data, while bar charts are used for categorical data or discrete numerical data.\n\n1. **Gaps:** Bar charts can have gaps between bars, while histograms typically don't.\n\n1. **Order:** In bar charts, bars can be arranged in any order, whereas histograms are ordered based on the value of the data\n\n#### Density function\n\nA **density function** describes the likelihood or probability of a particular value or range of values of a random variable. It provides a way to model continuous random variables and their distributions. There are two primary types of density functions:\n\n- **Probability Density Function (PDF)**: For continuous random variables.\n- **Probability Mass Function (PMF)**: For discrete random variables.\n\n- Here, let's focus on the **Probability Density Function (PDF)** since you inquired about \"density function\": For a continuous random variable X, the PDF, often denoted as f(x), is a function that describes the likelihood of X taking on a particular value. However, because X is continuous, the probability of X being exactly any specific value is technically zero. Instead, we often look at the probability that X falls within a specific interval.\n\n- A density function provides a mathematical model for how the probabilities are distributed across different values of a random variable. By understanding the shape and properties of a density function, we can make predictions, inferences, and conclusions about the underlying process or system that produces the data.\n\n#### Probability Density Function (PDF)\n\n\n**What Is a Probability Density Function (PDF)?** The probability density function (PDF) is a statistical expression that defines the probability that some outcome will occur. In this function, the probability is the percentage of a dataset's distribution that falls between two criteria. The probability density function measures continuous variables—stock and investment returns are generally not continuous random variables; they are discrete. However, most financial analysts assume that returns and prices are continuous so that they can model performance and analyze risks. What Does a Probability Density Function (PDF) Tell Us? A probability density function (PDF) describes how likely it is to observe some outcome resulting from a data-generating process. A PDF can tell us which values are most likely to appear vs. the less likely outcomes. This will change depending on the shape and characteristics of the PDF. To plot the Probability Density Function (PDF) of a dataset, you generally have two scenarios:\n\n- **Known Distribution:** If you know or assume that your data follows a particular theoretical distribution (like normal, exponential, etc.), you can use the mathematical formula for that distribution's PDF. For example, the PDF of a normal distribution is a bell-shaped curve defined by a specific formula involving its mean and standard deviation.\n\n- **Unknown Distribution:** If you don't have a specific distributional assumption for your data, you can estimate its PDF using techniques like Kernel Density Estimation (KDE). If you don't want to assume a specific distribution, you can estimate the PDF using Kernel Density Estimation (KDE).In the KDE approach, the resulting curve gives an estimate of the data's PDF without assuming it follows any specific known distribution. Finally, remember that while the shape of the estimated PDF (from KDE) provides insights into the data's distribution, it is influenced by factors like kernel choice and bandwidth. Adjusting these parameters can change the appearance of the estimated PDF.\n\n#### Kernel density estimate (KDE) plot\n\nA **kernel density estimate (KDE) plot** is a method for visualizing the distribution of observations in a dataset, analogous to a histogram. KDE represents the data using a continuous probability density curve in one or more dimensions. A density estimate or density estimator is just a fancy word for a guess: We are trying to guess the **density function f that describes well the randomness of the data.** A kernel is a smoothing function used to estimate the probability density function of a random variable. Essentially, KDE is a way to produce a smoothed version of a histogram.\n\nHere's how KDE works:\n\n1. **Place a Kernel on Each Point:** For each data point in the dataset, place a kernel (usually a bell-shaped curve like a Gaussian).\n\n1. **Sum All Kernels:** Sum up the value of the kernel function for every point to produce a continuous, smooth curve. This means that at any given x-value, you're adding up the contributions from each kernel at that x-value.\n\n1. **Normalize**: The area under the resulting curve is normalized to 1, making it a valid probability density function.\n\nThe key aspects to understand about the kernel in KDE are:\n\n1. **Type of Kernel:** There are various types of kernel functions that can be used in KDE, including Gaussian, Epanechnikov, Exponential, and others. The Gaussian kernel is the most common and looks like a normal distribution curve.\n\n1. **Bandwidth (h):** The bandwidth is a crucial parameter in KDE. It determines the width of the kernel placed at each data point. A small bandwidth will produce a curve that is wiggly and adheres closely to the data points, capturing more details but also potentially noise. A large bandwidth will produce a smoother curve but might oversmooth and miss some nuances in the data distribution. Choosing an appropriate bandwidth is vital for producing a meaningful KDE.\n\nA KDE plot provides a smooth and continuous representation of data distribution, whereas a histogram provides a binned representation. The advantage of KDE is that it can provide a clearer picture of the data distribution without being sensitive to the choice of bins (as in a histogram). However, the choice of kernel and bandwidth in KDE can significantly affect the resulting plot. Choosing an appropriate kernel function for Kernel Density Estimation (KDE) largely depends on the nature of the data, the specific application, and sometimes even personal or domain-specific preferences. However, there are some general guidelines and considerations that might help:\n\n1. **Data Characteristics:** Consider the characteristics of your data. If you have prior knowledge about the underlying distribution or specific properties of the data, that might guide your choice. For example, if you know the data is strictly positive, you might not want a kernel that produces estimates below zero (like the Gaussian kernel might).\n\n1. **Commonly Used Kernels:** For many general applications, the Gaussian (normal) kernel is the default choice because of its mathematical properties and the central limit theorem, which suggests that many natural processes have outputs that are roughly normally distributed.\n\n\n***A histogram aims to approximate the underlying probability density function that generated the data by binning and counting observations. Kernel density estimation (KDE) presents a different solution to the same problem. Rather than using discrete bins, a KDE plot smooths the observations with a Gaussian kernel, producing a continuous density estimate.*** A histogram puts all samples between the boundaries of each bin will fall into the bin. It doesn't differentiate whether the value falls close the left, to the right or the center of the bin. A kde plot, on the other hand, takes each individual sample value and draws a small gaussian bell curve over it. Then, all bell curves are summed together to form the final curve. A bell curve has some width, making the kde curve a bit wider than the histogram. In general, a kdeplot supposes the underlying distribution is quite smooth and goes slowly to zero near the edges.\n\n- The seaborn library uses gaussian kernel to plot the Kernel Density Estimation `kdeplot(data)`\n\n","metadata":{}},{"cell_type":"markdown","source":"### EDA of Delinquency variables","metadata":{}},{"cell_type":"code","source":"def get_the_kde_plot(\n fig_nbr_rows: int, \n fig_nbr_columns: int, \n fig_size: Tuple[int, int], \n scope_cols: List[str], \n plot_df: pd.DataFrame, \n title: str, font_size: int\n) -> None:\n \"\"\"\n Create a KDE plot for the given columns in the dataframe.\n\n Parameters:\n fig_nbr_rows (int): Number of rows in the figure grid.\n fig_nbr_columns (int): Number of columns in the figure grid.\n fig_size (Tuple[int, int]): Size of the entire figure.\n scope_cols (List[str]): Columns to be plotted.\n plot_df (pd.DataFrame): DataFrame containing the data to be plotted.\n title (str): Title of the plot.\n font_size (int): Font size of the title.\n \n Returns:\n None. Displays the plot.\n \"\"\" \n fig, axes = plt.subplots(fig_nbr_rows, fig_nbr_columns, figsize = fig_size)\n\n row = 0\n col = [i for i in range(0, fig_nbr_columns)] * fig_nbr_rows\n\n for i, column in enumerate(scope_cols[:-1]):\n if (i != 0) & (i%fig_nbr_columns == 0):\n row+=1\n \n sns.kdeplot(\n x=column, \n hue=\"Target\",\n hue_order=[1,0], \n palette=[\"#933A16\", \"#6082B6\"], \n label=[\"Default\", \"Paid\"], \n data=plot_df[scope_cols],\n fill=True, \n linewidth=2, \n legend=False, \n ax= axes[row,col[i]] if fig_nbr_rows > 1 else axes[i],\n warn_singular=False\n )\n if fig_nbr_rows > 1:\n axes[row,col[i]].tick_params(left=False,bottom=False)\n axes[row,col[i]].set(title='\\n\\n{}'.format(column), xlabel='', ylabel=(\"Density\" if i%fig_nbr_columns == 0 else ''))\n else:\n axes[i].tick_params(left=False,bottom=False)\n axes[i].set(title='\\n\\n{}'.format(column), xlabel='', ylabel=(\"Density\" if i%fig_nbr_columns == 0 else ''))\n \n empty_plots = (fig_nbr_rows * fig_nbr_columns) - (len(list(plot_df[scope_cols].columns)) - 1)\n\n for i in range(fig_nbr_columns - empty_plots, fig_nbr_columns):\n axes[fig_nbr_rows - 1,i].set_visible(False)\n \n if fig_nbr_rows > 1:\n handles, _ = axes[0,0].get_legend_handles_labels() \n else:\n handles, _ = axes[0].get_legend_handles_labels() \n \n fig.legend(labels=['Default','Paid'], handles=reversed(handles), ncol=2, bbox_to_anchor=(0.18, 1))\n sns.despine(bottom=True, trim=True)\n plt.tight_layout(rect=[0, 0.2, 1, 0.99])\n\n fig.suptitle(title, fontsize = font_size, x = 0.5, y = 1, color=\"#7b6b59\")\n plt.show()\n plt.close()\n ","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cols=[col for col in train.columns if (col.startswith(('D','T'))) & (col not in cat_cols[:-1] + [\"Spend 2\"])]\nplot_df=train[cols]\nget_the_kde_plot(22, 4, (25, 120), cols, plot_df, \"Distribution of Delinquency Variables\", 28)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cols=[col for col in train.columns if (col.startswith(('B','T'))) & (col not in cat_cols[:-1])]\nplot_df=train[cols]\nget_the_kde_plot(8, 5, (16, 32), cols, plot_df, \"Distribution of Balance Variables\", 22)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cols=[col for col in train.columns if (col.startswith(('R','T'))) & (col not in cat_cols[:-1])]\nplot_df=train[cols]\nget_the_kde_plot(6, 5, (16, 24), cols, plot_df, \"Distribution of Risk Variables\", 22)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cols=[col for col in train.columns if (col.startswith(('S','T'))) & (col not in cat_cols[:-1] + [\"Spend 2\"])]\nplot_df=train[cols]\nget_the_kde_plot(5, 5, (16, 20), cols, plot_df, \"Distribution of Spend Variables\", 22)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cols=[col for col in train.columns if (col.startswith(('P','T'))) & (col not in cat_cols[:-1])]\nplot_df=train[cols]\nget_the_kde_plot(1, 3, (16, 5), cols, plot_df, \"Distribution of Payment Variables\", 22)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The Probability Density Function (PDF) provides a visual representation of the distribution of a continuous random variable. By analyzing the PDF, you can draw several conclusions about the underlying data:\n\n1. **Central Tendency:** The location of the peak(s) of the PDF can give an indication of the central tendency of the data, often representing the mode of the distribution.\n\n1. **Spread/Variability:** The width of the PDF provides insights into the variability or spread of the data. A wider distribution indicates more variability, while a narrower distribution indicates data points are clustered closely around the mean.\n\n1. **Symmetry/Skewness:** If the PDF is symmetric around its central value, the distribution is symmetric. If one tail is longer than the other, the distribution is skewed. A longer right tail indicates positive skewness, while a longer left tail indicates negative skewness. The KDE can give insights into the skewness of the data — whether the distribution is symmetric, right-skewed, or left-skewed. It can also provide a sense of the \"heaviness\" of the tails of the distribution, which can be important in risk assessment in fields like finance.\n\n1. **Kurtosis:** The \"tailedness\" or the extremity of the tails of the PDF can provide insights into the kurtosis of the distribution. Heavy tails might suggest a higher likelihood of outliers.\n\n1. **Multimodality:** Multiple peaks in the PDF suggest that the data might have multiple modes, which could indicate the presence of distinct groups or clusters within the data.\n\n1. **Outliers:** Any small bumps or spikes in the tails of the PDF might suggest potential outliers or rare events.\n\n1. **Known Distributions:** By comparing the shape of the PDF to known distributions (e.g., normal, exponential, etc.), you can hypothesize about the underlying process generating the data. For instance, a bell-shaped PDF might suggest the data is normally distributed. One of the most immediate insights gained from a KDE is the shape of the data's distribution. You can quickly determine if the data is normally distributed, bimodal (two peaks), multimodal (multiple peaks), or has some other form.\n\n1. **Domain-Specific Insights:** In some fields, the shape of the distribution can provide specific insights. For instance, in finance, a \"fat-tailed\" distribution might indicate a higher than expected likelihood of extreme market events.\n\n1. **Comparison Between Groups:** If you have PDFs of multiple groups or categories, comparing their shapes can provide insights into differences between the groups. For instance, comparing the income distributions of two countries might reveal differences in equality.\n\n1. **Data Transformation Clues:** If you're aiming to apply techniques that assume normality, the PDF can guide you on necessary data transformations. For instance, if the data is log-normally distributed, taking the logarithm might make it more normally distributed.\n\n1. **Identify Gaps:** Sometimes, the PDF might reveal clear gaps or regions where data is sparse, indicating potential areas of missing data or simply areas of low probability.\n\nIt's worth noting that while the PDF provides a lot of insights, it's often beneficial to couple it with other statistical measures (like mean, median, standard deviation, skewness, kurtosis, etc.) and plots (like box plots, CDFs, etc.) to get a comprehensive understanding of the data's distribution. Also, always consider the context and domain when interpreting the PDF; what's meaningful in one scenario or field might be different in anothe\n\nKDE plots are especially useful when comparing the distributions of two or more groups on the same axis. For instance, if you're analyzing a dataset of test scores for two different teaching methods, overlaying the two KDEs can immediately show differences in distributions. When visualizing Kernel Density Estimation (KDE) plots of continuous variables for a binary classification problem, such as credit card default prediction, comparing the two distributions (one for each target value: default vs. no default) can yield various insights. Here's what you might conclude from such plots:\n\n1. **Feature Discrimination:** If the two distributions (default vs. no default) are distinctly separate for a given feature, it indicates that the feature is potentially very useful in discriminating between the two classes. A feature with overlapping distributions might be less informative, though it could still offer some value in combination with other features.\n\n1. **Thresholds for Decision Making:** If a clear separation exists between the two distributions, you can identify threshold values of the feature that might indicate higher risk. For example, if the distribution of 'credit limit' is significantly lower for defaulters than non-defaulters, a low credit limit might be a risk indicator.\n\n1. **Risk Patterns:** Some features might show that as their value increases or decreases, the likelihood of default changes. For instance, if a KDE plot of 'monthly expenditure' shows that higher expenditures correlate with higher default rates, it suggests a pattern that can inform risk assessment.\n\n1. **Data Quality Issues:** Anomalies in the KDE plots, such as unexpected spikes, might indicate issues with data quality, recording errors, or outliers.\n\n1. **Multimodal Distributions:** If any of the continuous features show multiple peaks (modes) in their KDE plots, it suggests underlying sub-groups within the data. Investigating these sub-groups might reveal specific behaviors or patterns. For instance, a multimodal distribution in 'age' might indicate different default behaviors across age groups.\n\n1. **Skewness & Symmetry:** If the distributions are skewed (either left or right), it might indicate that certain ranges of the feature values are more prone to defaults. For example, if the distribution of 'balance amounts' is right-skewed for defaulters, it indicates that people with higher balances are fewer but have a higher tendency to default.\n\n1. **Comparative Analysis:** For some features, you might find that the distribution for defaulters closely mirrors that for non-defaulters but is simply shifted up or down the value axis. This could indicate that while the general pattern of behavior is the same for both groups, the scale of the behavior (e.g., amount spent, frequency of use) differs.\n\n1. **Feature Engineering Ideas:** Observing the patterns and behaviors in KDE plots can inspire ideas for feature engineering. For instance, if you find that defaults are higher for very low and very high credit limits but lower for mid-range limits, you might engineer a feature that captures this non-linear relationship.\n\n1. **Validation of Hypotheses:** The KDE plots can help validate or refute domain-specific hypotheses. For instance, if there's a belief that people with very recent credit inquiries are more likely to default, the KDE plots can provide evidence for or against this belief.\n\nWhen analyzing KDE plots (or any visualization, for that matter), it's crucial to approach the analysis with a combination of domain knowledge, statistical understanding, and a dose of skepticism. Always consider external factors, potential biases in the data, and the broader context when drawing conclusions.\n\n\n","metadata":{}},{"cell_type":"markdown","source":"\n#
Correlation heatmap, EDA for Numerical Features
\n\n\nBefore we dive in into distributions of the individual features let's check are there any highly correlated ones.\n\n**What is Correlation Heatmap?**\n\nCorrelation heatmaps are a type of plot that visualize the strength of relationships between numerical variables. Correlation plots are used to understand which variables are related to each other and the strength of this relationship. A correlation plot typically contains a number of numerical variables, with each variable represented by a column. The rows represent the relationship between each pair of variables. The values in the cells indicate the strength of the relationship, with positive values indicating a positive relationship and negative values indicating a negative relationship. Correlation heatmaps can be used to find potential relationships between variables and to understand the strength of these relationships. In addition, correlation plots can be used to identify outliers and to detect linear and nonlinear relationships. The color-coding of the cells makes it easy to identify relationships between variables at a glance. Correlation heatmaps can be used to find both linear and nonlinear relationships between variables.\n\nWarmer colors (like red) typically represent positive correlations, cooler colors (like blue) represent negative correlations, and colors close to white or gray indicate little to no correlation.\n\nFor numerical variables, the most commonly used correlation coefficient is the Pearson correlation coefficient. It measures the linear relationship between two numerical variables. The coefficient ranges from -1 to 1:\n\n- 1: Perfect positive correlation.\n- -1: Perfect negative correlation.\n- 0: No linear correlation.\n\n\n**Benefits of a Correlation Heatmap:**\n\n1. **Visual Overview**: It offers a quick visual overview of the relationships among multiple variables, enabling you to spot potential patterns, outliers, or areas of interest.\n\n1. **Identifying Multicollinearity:** Before building regression models, it's crucial to check for multicollinearity (when two or more variables are highly correlated). A heatmap can help quickly identify these variables.\n\n1. **Feature Selection:** When building models, it might be beneficial to remove variables that are highly correlated, as they carry similar information.\n\n1. **Insights for Further Analysis:** Identifying which variables are strongly correlated can guide further analyses, hypothesis tests, or feature engineering.\n\nHowever, remember:\n\n1. **Correlation ≠ Causation:** Just because two variables are correlated doesn't mean one causes the other.\n\n1. **Linearity Limitation:** Pearson's correlation coefficient measures only linear relationships. Nonlinear relationships might not be well-represented by the coefficient. In such cases, other correlation measures like Spearman's rank correlation might be more appropriate\n\n\nIn statistics, the **Pearson correlation coefficient (PCC)** is a correlation coefficient that measures linear correlation between two sets of data. The Pearson correlation coefficient is a descriptive statistic, meaning that it summarizes the characteristics of a dataset. Specifically, it describes the strength and direction of the linear relationship between two quantitative variables. The Pearson correlation coefficient is a descriptive statistic, meaning that it summarizes the characteristics of a dataset. Specifically, it describes the strength and direction of the linear relationship between two quantitative variables.\n\n\"image\"\n\"image\"\n\n**Pearson correlation coefficient using `corr()`**\n\nThe Pearson correlation coefficient can be easily calculated in Pandas using corr(). The corr() function is used to compute pairwise correlation coefficients on Pandas dataframe values, and can either calculate them as individual pairs (i.e. top speed and price), or as pairs across an entire dataframe.\n\nSince Pearson correlation coefficient is so widely used by statisticians and data scientists, the corr() function is pre-configured with default values to return the Pearson correlation coefficient. However, you can change this to use the similar Spearman rank correlation (or Spearman’s r), or the Kendall Tau correlation coefficient, if you think they better suit your data.\n\n\nThe `seaborn.heatmap()` function is just a visualization tool. It displays whatever matrix you provide to it. When it comes to correlations, whether the relationship is linear or not depends on the type of correlation coefficient you compute, not the heatmap itself.\n\nCommonly used correlation coefficients include:\n\n1. **Pearson Correlation Coefficient:** This measures the linear relationship between two numerical variables. When you use the corr() method from a pandas DataFrame, it computes the Pearson correlation by default. If your heatmap is based on this, then yes, it reflects linear relationships.\n\n1. **Spearman Rank Correlation:** This measures the monotonic relationship between two variables. It's based on the rank values of the variables rather than their actual values. It's useful when the relationship might be nonlinear but still monotonic, or when dealing with ordinal data.\n\n1. **Kendall's Tau:** Like Spearman, this is a rank-based correlation coefficient. It measures the strength of the association between two variables.\n\n\"image\"\n\nTo visualize non-linear relationships in a heatmap using `seaborn`, you'd first compute the desired correlation matrix (e.g., Spearman's correlation) and then plot it:\n\n\n\n***What is multicolinearity and why is it bad?***\n\nMulticolinearity happens when two or more features in a model are correlated with one another. For example, if I want to build a regression model to predict the likelihood of catching a particular disease and I include inputs “history of disease A in the past 12 months” and “history of any disease in the past 12 months,” you can see how a positive outcome in the first input directly correlates to a positive outcome in the second. This creates a case called multi-colinearity. When this occurs, the integrity of the model is compromised because the two inputs are not independent of each other and thus the model cannot distinguish the significance of either in predicting the target outcome.\n\n**Conclusions and Insights You Can Draw:**\n\n1. There are several highly correlated Delinquency variables, with a few pairs perfectly positively correlated at 1.0. There are also a number of missing correlations, due to null values in the data. Below are the relationships between some of the most correlated Delinquency variables.\n\n","metadata":{}},{"cell_type":"code","source":"def get_the_correlation_heatmap_plot(plot_df: pd.DataFrame, fig_size: Tuple[int, int], title: str, title_fontsize: int, cbar: bool = False) -> None:\n \"\"\"Generates a heatmap visualizing the pairwise correlation of columns in a DataFrame.\n\n Parameters:\n plot_df (pd.DataFrame): DataFrame for which the correlations will be computed.\n fig_size (Tuple[int, int]): Size of the figure to be plotted (width, height).\n title (str): Title of the heatmap.\n title_fontsize (int): The fontsize of the title\n cbar (bool, optional): Whether to display a color bar on the heatmap. Default is False.\n\n Returns:\n None: The function displays the heatmap.\n \"\"\"\n corr=plot_df.corr()\n\n # It creates a boolean array of the same shape as corr filled with True\n # It returns the upper triangle of the array (it makes the lower triangle False)\n mask = np.triu(np.ones_like(corr, dtype = bool))[1:,:-1]\n\n # This line slices the corr DataFrame to exclude its first row and last column. \n # This aligns it with the mask created in the previous step.\n corr=corr.iloc[1:,:-1].copy()\n\n fig, ax = plt.subplots(figsize=fig_size) \n sns.heatmap(\n corr, \n mask=mask, \n vmin=-1, \n vmax=1,\n center=0, \n annot=True, \n fmt=\".2f\", \n cmap=\"coolwarm\", \n annot_kws={\n \"fontsize\":10,\n \"fontweight\":\"bold\"\n },\n cbar=cbar\n )\n ax.tick_params(left=False,bottom=False)\n ax.set_xticklabels(ax.get_xticklabels(), rotation=45, horizontalalignment='right',fontsize=12)\n ax.set_yticklabels(ax.get_yticklabels(), fontsize=12)\n plt.title(title, fontsize=title_fontsize, color=\"#7b6b59\")\n\n plt.show()\n plt.close()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"delinquency_cols = [col for col in train.columns if (col.startswith((\"D\"))) & (col not in cat_cols)]\nplot_df=train[delinquency_cols]\nget_the_correlation_heatmap_plot(plot_df, (50,50), \"Correlations between Delinquency Variables\", 38)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Τhere are several highly correlated Delinquency variables, with a few pairs perfectly positively correlated at 1.0. There are also a number of missing correlations, particularly in `Delinquency 87`, due to null values in the data.","metadata":{}},{"cell_type":"code","source":"spend_cols = [col for col in train.columns if (col.startswith((\"S\"))) & (col not in cat_cols)]\nplot_df=train[spend_cols]\nget_the_correlation_heatmap_plot(plot_df, (16,12), \"Correlations between Spend Variables\", 18, True)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"payment_cols = [col for col in train.columns if (col.startswith((\"P\"))) & (col not in cat_cols)]\nplot_df=train[payment_cols]\nget_the_correlation_heatmap_plot(plot_df, (12,4 ), \"Correlations between Payment Variables\", 16)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"balance_cols = [col for col in train.columns if (col.startswith((\"B\"))) & (col not in cat_cols)]\nplot_df=train[balance_cols]\nget_the_correlation_heatmap_plot(plot_df, (24,22), \"Correlations between Balance Variables\", 28)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"risk_cols = [col for col in train.columns if (col.startswith((\"R\"))) & (col not in cat_cols)]\nplot_df=train[risk_cols]\nget_the_correlation_heatmap_plot(plot_df, (26,30), \"Correlations between Risk Variables\", 28)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" #
Correlation with the target
\n\n**Correlation and Causation**\n\n1. **Correlation:** A measure of the extent of interdependence between variables.\n1. **Causation:** the relationship between cause and effect between two variables.\n\nIt is important to know the difference between these two. Correlation does not imply causation. Determining correlation is much simpler than determining causation as causation may require independent experimentation. While causation and correlation can exist simultaneously, correlation does not imply causation. Causation means one thing causes another—in other words, action A causes outcome B. On the other hand, correlation is simply a relationship where action A relates to action B—but one event doesn’t necessarily cause the other event to happen.\n\n\"image\"\n\n\nTwo or more variables considered to be related, in a statistical context, if their values change so that as the value of one variable increases or decreases so does the value of the other variable (although it may be in the opposite direction).\n\nFor example, for the two variables \"hours worked\" and \"income earned\" there is a relationship between the two if the increase in hours worked is associated with an increase in income earned. If we consider the two variables \"price\" and \"purchasing power\", as the price of goods increases a person's ability to buy these goods decreases (assuming a constant income).\n\nCorrelation is a statistical measure (expressed as a number) that describes the size and direction of a relationship between two or more variables. A correlation between variables, however, does not automatically mean that the change in one variable is the cause of the change in the values of the other variable.\n\nCausation indicates that one event is the result of the occurrence of the other event; i.e. there is a causal relationship between the two events. This is also referred to as cause and effect.\n\nTheoretically, the difference between the two types of relationships are easy to identify — an action or occurrence can cause another (e.g. smoking causes an increase in the risk of developing lung cancer), or it can correlate with another (e.g. smoking is correlated with alcoholism, but it does not cause alcoholism). In practice, however, it remains difficult to clearly establish cause and effect, compared with establishing correlation.\n\n\n\n**Pearson Correlation**\n\nThe Pearson Correlation measures the linear dependence between two variables X and Y. The resulting coefficient is a value between -1 and 1 inclusive, where:\n\n- 1: Perfect positive linear correlation.\n- 0: No linear correlation, the two variables most likely do not affect each other.\n- -1: Perfect negative linear correlation.\n\n**Pearson Correlation is the default method of the function “corr”.**\n\nExamining the correlation of each feature with the target variable can be very useful. Here's why it's beneficial and how we can achieve it:\n\n1. **Feature Selection:** Features that have a higher correlation with the target variable might be more informative for prediction. Examining correlations can help prioritize which features to use in a model or even guide feature selection methods. In predictive modeling, we're interested in variables that have a relationship (linear or otherwise) with the target, regardless of whether one causes the other. A variable that's correlated with the target can be a good predictor, helping the model make more accurate predictions.\n\n1. **Understanding Data Structure:** Knowing how different features relate to the target and to each other provides insights into the data's structure. This can guide data preprocessing, feature engineering, and model selection. By understanding which variables are strongly correlated with the target, we gain insights into potential relationships and patterns in the data. This can guide hypothesis generation and further research.\n\n1. **Dimensionality Reduction:** If you have many features, correlation analysis can help identify the most relevant ones, allowing you to focus on those and potentially exclude others, reducing the complexity of your model.\n\n1. **Statistical Significance:** While calculating correlations, it's also essential to evaluate the statistical significance of these correlations. High correlation coefficients may not always be statistically significant, especially when the sample size is small. Observing correlations can lead to hypotheses about potential causal relationships, which can then be investigated further using more robust causal inference methods or experiments.\n\n1. **Informing Stakeholders:** Correlation plots and metrics are easily interpretable and can be shared with stakeholders to provide a high-level understanding of which variables might be important.\n\n1. **Problem Framing:** Understanding which variables correlate with the target can help in framing the problem, deciding on the modeling approach, and determining which additional data might be helpful.\n\nCorrelation, especially Pearson correlation, captures linear relationships. Some variables might have a non-linear relationship with the target and might still be valuable despite a low correlation coefficient. Remember, while correlation indicates a linear relationship between variables, it does not imply causation. Furthermore, features that have low or zero correlation might still be important in the context of other variables, so simply discarding them based on low correlation can sometimes be premature. It's always beneficial to combine this analysis with domain knowledge and other feature selection methods. Remember, while correlation does not mean causation, it does indicate a relationship. When we're building predictive models, we're often looking for relationships, not necessarily causal ones. For example, if we find that customers who visited a website frequently in the past week are more likely to make a purchase, it's useful information for prediction—even if frequent website visits don't cause the purchase. However, if we were looking to intervene and boost sales, understanding causation would be critical. For instance, if the real cause of increased sales is a popular product release, merely driving more website visits might not increase sales.\n\n**Correlation and Feature importance**\n\nCorrelation with the target variable and feature importance from a model are distinct concepts, though both provide insights into the relationship between features and the target variable. Here's a breakdown of the differences:\n\n**Correlation with Target:**\n\n1. Nature: Measures linear (Pearson correlation) or monotonic (Spearman correlation) relationships between individual features and the target variable.\n1. Scope: Considers each feature in isolation without accounting for potential interactions with other features.\n1. Limitation: Just because a feature is not correlated with the target doesn't mean it's not important. The relationship could be complex or non-linear, which might not be captured by simple correlation metrics.\n\n**Feature Importance from a Model:**\n\n1. Nature: Derives importance based on how frequently a feature is used to split the data in tree-based models (like Decision Trees, Random Forests, or Gradient Boosted Trees) or based on coefficients in linear models, among other methods.\n1. Scope: Takes into account the relationship between features and the target in the context of other features. It inherently captures feature interactions, especially in tree-based models.\n1. Limitation: The interpretation can be model-specific. For example, feature importance in a random forest might differ from that in a boosted tree model. Additionally, correlated features can have distributed importances, where one feature might \"hog\" the importance that could have been shared with another related feature.\n\n**Which to Use?**\n\n- **Early EDA:** Correlation with the target is quick to compute and can provide early insights during exploratory data analysis (EDA).\n\n- **Model Building:** Feature importance from models can help refine and optimize feature selection, especially when you're working with a large set of features.\n\n- **Interpretation:** If you're trying to explain a model's predictions, feature importance can provide more nuanced insights, especially when used with tools like SHAP (SHapley Additive exPlanations) or permutation importance.\n\nIn practice, both correlation and feature importance can be valuable at different stages of a machine learning project. While correlation gives you a fast, model-agnostic overview, feature importance can offer deeper insights tailored to the specifics of the model you're using.\n\n","metadata":{}},{"cell_type":"code","source":"temp=dict(\n layout=go.Layout(\n font=dict(family=\"Franklin Gothic\", size=12), \n height=500, \n width=1000)\n)\n\n# Computes the pairwise Pearson correlation of all columns in the dataframe train.\n# 178 x 178, number of features = 190 but we excluded the categorical features\ncorr=train.corr()\n# Filters the correlations to only those with the column Target,\n# then sorts them in descending order, and\n# excludes the correlation of Target with itself\ncorr=corr['Target'].sort_values(ascending=False)[1:]\n\n# Uses Seaborn to create a reversed red color palette with 135 different shades.\npal=sns.color_palette(\"Reds_r\",135).as_hex()\n# Converts the hexadecimal colors in pal to RGB format (with added alpha transparency) using Matplotlib.\nrgb=['rgba'+str(matplotlib.colors.to_rgba(i,0.7)) for i in pal]\n\nfig = go.Figure()\n\nfig.add_trace(\n go.Bar(\n x=corr[corr>=0],\n y=corr[corr>=0].index, \n marker_color=rgb,\n orientation='h', \n marker_line=dict(color=pal,width=2), \n name='',\n hovertemplate='%{y} correlation with target: %{x:.3f}',\n showlegend=False\n )\n)\n\npal=sns.color_palette(\"Blues\",100).as_hex()\nrgb=['rgba'+str(matplotlib.colors.to_rgba(i,0.7)) for i in pal]\n\n# Adds the positively correlated features as horizontal bars to the figure using the colors defined earlier.\n# Sets the orientation of the bars to horizontal, specifies hover information, \n# and ensures that these bars won't appear in a legend.\nfig.add_trace(\n go.Bar(\n x=corr[corr<0], \n y=corr[corr<0].index, \n marker_color=rgb[25:], \n orientation='h', \n marker_line=dict(color=pal[25:],width=2), \n name='',\n hovertemplate='%{y} correlation with target: %{x:.3f}',\n showlegend=False\n )\n)\n\nfig.update_layout(\n template=temp,\n title=\"Feature Correlations with Target\",\n xaxis_title=\"Correlation\", \n margin=dict(l=150),\n height=3000, \n width=700, \n hovermode='closest' #ensures that the closest data point to the hover position will be shown in the hover label.\n)\nfig.show()\n","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are several strong correlations with the target variable. `Payment 2` is the most negatively correlated with the probability of defaulting with a correlation of -0.67, while `Delinquency 48` is the most positively correlated overall at 0.61. `Delinquency 87` is also missing from the correlations above due to the proportion of null values. ","metadata":{}},{"cell_type":"markdown","source":" #
Random Uniform Noise Detected
\n \n**Context:**\n\nMoney has this unique property that values although highly cardinal it is still discrete with an increment of $0.01.\n\nIt can be expected that some debit cards would have a balance of $0.00 or credit cards maxed out the credit limit. Therefore spikes should appear in the money related feature distribution plots. It should be observed one way or another, even though the data has been anonymized & normalized (as per competition description).\n\nLet's pick a random balance feature `Balance 2` and look at its distribution:\n","metadata":{}},{"cell_type":"code","source":"x = pd.read_csv('../input/amex-default-prediction/train_data.csv', usecols=['B_2'])","metadata":{"execution":{"iopub.status.busy":"2023-10-01T18:03:28.516202Z","iopub.execute_input":"2023-10-01T18:03:28.517264Z","iopub.status.idle":"2023-10-01T18:08:38.883201Z","shell.execute_reply.started":"2023-10-01T18:03:28.517227Z","shell.execute_reply":"2023-10-01T18:08:38.880931Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"1. **Function:**\n\n - `sns.displot():` This is Seaborn's function to plot distributions. It's a figure-level function, meaning it creates a full figure.\n\n1. **Parameters:**\n\n - `train['Balance 2']`: This specifies the data to be plotted, which in this case is the `Balance 2` column from the `train` dataframe.\n - `bins=500`: This divides the range of the data into 500 bins or intervals, and then counts the number of data points in each bin. The count in each bin is then represented by the height of the bar in the histogram. The larger the number of bins, the finer the granularity of the histogram, but if too large, it might also introduce noise.\n - `kde=False`: This specifies that the kernel density estimate (KDE) plot should not be overlaid on top of the histogram. By default, `displot` can also draw a KDE plot, but this parameter turns that feature off, so only the histogram is shown.\n\nThe output will be a histogram that shows the distribution of values in the `Balance 2` column, with the data divided into 500 bins. The x-axis will represent the range of values in `Balance 2`, and the y-axis will represent the count of data points in each bin.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(12, 8))\nsns.displot(x['B_2'], bins=500, kde=False)\nplt.show()\nplt.close()","metadata":{"execution":{"iopub.status.busy":"2023-10-01T18:08:38.886505Z","iopub.execute_input":"2023-10-01T18:08:38.886910Z","iopub.status.idle":"2023-10-01T18:08:43.522559Z","shell.execute_reply.started":"2023-10-01T18:08:38.886881Z","shell.execute_reply":"2023-10-01T18:08:43.521406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's zoom in into these 2 spikes.\n\nWe can see that the spike is in range in (1, 1.01) and not a single value. I guess the max value for this feature in raw form is 1 and random uniform value from [0, 0.01] was added for each observation.\n\n","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(12, 8))\nsns.distplot(x.loc[x[\"B_2\"]>0.99,\"B_2\"],bins=500,kde=False)\nplt.show()\nplt.close()","metadata":{"execution":{"iopub.status.busy":"2023-10-01T18:08:43.523929Z","iopub.execute_input":"2023-10-01T18:08:43.524352Z","iopub.status.idle":"2023-10-01T18:08:44.426222Z","shell.execute_reply.started":"2023-10-01T18:08:43.524296Z","shell.execute_reply":"2023-10-01T18:08:44.425382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(12, 8))\nsns.distplot(x.loc[(x.B_2>0.8) & (x.B_2<0.83),'B_2'],bins=500,kde=False)\nplt.show()\nplt.close()","metadata":{"execution":{"iopub.status.busy":"2023-10-01T18:10:28.517005Z","iopub.execute_input":"2023-10-01T18:10:28.517833Z","iopub.status.idle":"2023-10-01T18:10:29.397770Z","shell.execute_reply.started":"2023-10-01T18:10:28.517786Z","shell.execute_reply":"2023-10-01T18:10:29.396536Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The second spike confirms the idea of [0, 0.01] random uniform noise.\n\n**The other way to look at it...**\n\n99.9% values are unique. This goes against the nature of how maney is distributed in bank accounts. This confirms the hypothesis of random noise injection.\n\n","metadata":{}},{"cell_type":"code","source":"len(x['B_2'].unique())/len(x['B_2'])\n","metadata":{"execution":{"iopub.status.busy":"2023-10-01T18:10:55.595251Z","iopub.execute_input":"2023-10-01T18:10:55.595877Z","iopub.status.idle":"2023-10-01T18:10:56.680041Z","shell.execute_reply.started":"2023-10-01T18:10:55.595842Z","shell.execute_reply":"2023-10-01T18:10:56.679157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.distplot(x.loc[x.B_2<0.1,'B_2'],bins=500,kde=False)\nplt.show()\nplt.close()","metadata":{"execution":{"iopub.status.busy":"2023-10-01T18:11:22.926258Z","iopub.execute_input":"2023-10-01T18:11:22.927023Z","iopub.status.idle":"2023-10-01T18:11:23.921801Z","shell.execute_reply.started":"2023-10-01T18:11:22.926987Z","shell.execute_reply":"2023-10-01T18:11:23.920645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Monetary Variables\n\n\nThe classification of money as a **discrete** or **continuous** variable can depend on the context in which it's being considered. Let's break it down:\n\n1. In Practical Terms (**Everyday Transactions**): Money can be considered both, but physical money like banknotes and coins are definitely discrete. You can’t pay 1.243 USD. You can only pay 1.24 USD. That’s because the difference between two sums of money can be 1 cent at most. Money is typically treated as a discrete variable because it is quantized into specific units (e.g., pennies, cents, etc.). For example, in the U.S., you cannot have an amount smaller than 0.01USD for standard transactions. Hence, when considering everyday transactions or accounting, money is discrete. Money, especially in the context of most real-world transactions, behaves like a discrete variable. Let's use the U.S. dollar as an example: The smallest unit of the U.S. currency is the penny, which is worth 0.01 USD. Therefore, when dealing with standard transactions, all amounts must be a multiple of this smallest unit. This means you can have amounts like 0.01 USD, 0.02 USD, 0.03 USD, and so on, but not amounts like 0.015 USD or 0.025 USD in typical transactions.\n\n 1. **Cardinality refers to the number of unique values a variable can take.**\n - **High cardinality** means there's a wide range of unique values.\n - **Low cardinality** means there are very few unique values.\n \n Money has high cardinality because, even within a small range, there are many possible values. For example, between 1 USD and 2 USD, you have 100 unique values.\n\n1. In Theoretical or **Modeling Contexts**: Money can be treated as a continuous variable. For instance, when modeling economic phenomena, interest rates, or other financial metrics, it might be more convenient to treat money as continuous to allow for more nuanced calculations and predictions. In these scenarios, you might work with fractional pennies or cents, even if such values aren't practical in a real-world transaction.\n\n1. **In Computing and Data Representation:** Even if treated as continuous in models, money's representation in computing systems can become discrete due to limitations in floating-point arithmetic. That's one reason financial systems often use integer arithmetic (counting cents rather than fractional dollars) to avoid rounding errors.\n\nIn summary, while money in practical, everyday scenarios is discrete, it can be treated as continuous in certain theoretical or modeling contexts. Always choose the representation that's most appropriate for the specific problem or analysis at hand.\n\n## What Is Uniform Distribution?\n\n\nIn statistics, uniform distribution refers to a type of probability distribution in which all outcomes are equally likely. A uniform distribution, also called a rectangular distribution, is a probability distribution that has constant probability.\n\nThis distribution is defined by two parameters, a and b:\n\n- a is the minimum.\n- b is the maximum.\n\nLike all probability distributions for continuous random variables, the area under the graph of a random variable is always equal to 1.\n\n\"image\"\n\n\n## Anonymizing numerical data\n\n\nAnonymizing numerical data is essential to protect individuals' privacy when sharing datasets, especially in situations where the data can be personally identifiable or sensitive. Among several methods and techniques to anonymize numerical data one common technique is: \n\n1. **Random Noise Addition:** Random noise addition is a technique where you add a randomly generated value (noise) to each data point to obscure the original value, thus enhancing data privacy.\n\n - Add or subtract a small random number from each data point to mask the original value.\n - The noise should be generated from a known distribution, often a Gaussian (normal) distribution.\n - Ensure the noise doesn't drastically change the nature of the data.\n \n\n## Random Uniform Noise\n\n\n**Why Add Noise?**\n\nBy adding random noise to data, individual data points get perturbed, making it difficult to identify or associate the information with any specific entity or individual. This helps in preserving privacy while still retaining, to an extent, the statistical properties of the dataset.\n\n**Types of Noise:** There are several distributions from which you can generate noise. Two of the most commonly used are:\n\n1. **Gaussian (or Normal) Noise:** This is noise generated from a normal distribution. The amount of noise added would be a random value typically centered around zero and with a defined standard deviation.\n\n1. **Uniform Noise:** This is noise generated from a uniform distribution. For each data point, a random value is chosen from a fixed range, like [-a, a]. The uniform noise can take any value from this range with equal probability. Adding Uniform Noise: Let's discuss adding uniform noise in more detail. When you add uniform noise from a range [-a, a], you're effectively adding a random amount between -a and a to each data point. For example, if a = 2, then you're adding anywhere from -2 to 2 to each data entry. Steps to Add Uniform Noise:\n\n 1. Determine the range of the noise. Decide how much perturbation is acceptable for your dataset. This will define the values of -a and a.\n\n 1. For each numerical data point in your dataset, generate a random number from the uniform distribution defined by [-a, a].\n \n 1. Add the generated random number to the data point.\n \n 1. Repeat for all data points in your dataset.\n\n**Considerations:**\n\n1. The choice of the range [-a, a] is crucial. If the range is too small, the original data points remain almost identifiable, defeating the purpose of adding noise. If the range is too large, the perturbed data might lose a lot of its original utility and meaning.\n\n1. Uniform noise addition can alter the statistical properties of your data, such as mean and variance. Depending on the range of noise and the distribution of your original data, these changes can be significant or minor.\n\n1. While noise addition can provide some level of privacy, it's not foolproof, especially if an attacker has additional information or other means of cross-referencing data. It's essential to evaluate the risk based on the specific context of the data and its potential uses.\n\nIn summary, adding uniform noise can be a simple yet effective method for data anonymization, but care should be taken to ensure that the perturbed data retains its utility and that privacy requirements are met.\n\n\n**Detecting uniform noise from a distribution plot (like a histogram or KDE) of a random variable can be challenging, especially if you don't have access to the original unperturbed data for comparison. However, there are some characteristics you might look for:**\n\n1. **Uniform \"Background\" Noise:** If the original data had very discrete or specific values (e.g., only integers or specific categories) and a significant amount of uniform noise is added, then in a histogram, you might notice a \"background\" level of noise — a nearly uniform count across many bins, upon which the original data's pattern might be superimposed.\n\n**The Nature of Uniform Noise:**\n\nUniform noise, when added to data, tends to spread data points evenly across a certain range. For instance, if you add uniform noise from the range [-0.5, 0.5] to a set of data points, you are effectively \"jittering\" or \"shaking\" those data points randomly, but within that specific range.\n\n**Discrete Data:**\n\nImagine a dataset where values are very discrete, such as the number of times a person visits a store in a month. The values might be integers like 0, 1, 2, 3, etc. If you were to create a histogram of this data, you'd see bars at these integer positions, and zero (or near zero) counts in between these integers.\n\n**Adding Uniform Noise:**\n\nNow, let's add uniform noise from the range [-0.5, 0.5] to this discrete data. Some of the original values will be reduced by up to 0.5, some will be increased by up to 0.5, and some will change by some value in between. This means that a value that was originally 2 could now be anywhere between 1.5 and 2.5.\n\n**Effect on the Histogram:**\n\nAfter adding the noise, if you plot a histogram of this \"noisy\" data, you won't just see bars at the integer positions. Instead:\n\nThe bars at the integer positions will likely be shorter than before because some of the data points have been \"pushed\" into the neighboring bins by the noise.\nYou'll see new bars appearing between the integers. These bars represent the data points that have been \"jittered\" away from their original integer positions by the noise.\nThe bars between the integers will be of roughly equal height, creating a sort of \"background\" level in the histogram. This is because the noise is uniform and spreads data points evenly in its range.\nThis consistent, roughly equal-height \"background\" is superimposed on the original pattern of the data (the spikes at integer positions). It's like the original, clean data is trying to \"peak through\" this added noise.\n\n**Why it's called \"Background\" Noise:**\n\nThe term \"background\" is used here to give an intuitive sense of something that's pervasive and ever-present, like a backdrop. In this context, because the noise spreads data points somewhat evenly between discrete values, it gives rise to a consistent \"background\" level in a histogram.\n\n**In Conclusion:**\n\nWhen you see a histogram with expected spikes (from the original, clean data) combined with an unexpected, nearly uniform distribution in between those spikes, it can be an indicator of added uniform noise. This is especially noticeable when the original data had very discrete or specific values.\n\n","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}