{"cells":[{"metadata":{},"cell_type":"markdown","source":"\nTable of Contents \n\n1. [Introduction](#1)\n1. [Missing Data Imputation](#2)\n 1. [Complete Case Analysis](#21)\n 1. [Mean / Median / Mode Imputation](#22)\n 1. [Random Sample Imputation](#23)\n 1. [Replacement by Arbitrary Value](#24)\n 1. [End of Distribution Imputation](#25)\n1. [Categorical Encoding](#3)\n 1. [Classic Encoders](#31)\n * [One Hot Encoder](#311)\n * [Count and Frequency Encoder](#312)\n * [Binary Encoder](#313)\n * [Ordinal Encoder](#314)\n * [BaseN Encoder](#315)\n * [Hashing Encoder](#316)\n * [Sum Encoder](#317)\n 1. [Contrast Encoders](#32)\n * [Helmert (reverse)](#321)\n * [Backward Difference](#322)\n * [Polynomial](#323)\n 1. [Bayesian Encoders](#33) \n * [Target encoding / Mean encoding](#331)\n * [Weight of Evidence](#332)\n * [Rare Label encoding](#333)\n * [LeaveOneOut](#334)\n * [James-Stein](#335)\n * [M-estimator](#336)\n1. [Gaussian Transformation](#4)\n 1. [Logarithm transformation](#41)\n 1. [Reciprocal transformation](#42)\n 1. [Square root transformation](#43)\n 1. [Exponential transformation](#44)\n 1. [Yeo-Johnson transformation](#45)\n 1. [Box-Cox transformation](#46)\n1. [Discretisation](#5)\n 1. [Equal width discretisation](#51)\n 1. [Equal Frequency discretisation](#52)\n 1. [Discretisation using decision trees](#53)\n1. [Outlier Engineering](#6)\n 1. [Outlier Detection & Removal](#61)\n * [Outlier Detection & Removal Using IQR(Inter Quartile Range](#611)\n * [Outlier Detection & Removal using Percentile](#612) \n * [Outlier Detection & Removal using Z score](#613) \n * [Outlier Detection using Box Plot](#614)\n * [Outlier Detection using Scatter Plot](#615) \n 1. [Treating outliers as missing values](#62)\n 1. [Top / bottom / zero coding](#63)\n 1. [Discretisation](#64)\n1. [Feature Scaling](#7)\n 1. [Standardisation](#71)\n 1. [Min-Max Scaling](#72)\n 1. [Maximum Absolute Scaling](#73)\n 1. [Robust Scaling](#74)\n 1. [Mean normalisation](#75) \n 1. [Quantile Transformer Scaler](#76) \n 1. [Power Transformer Scaler](#77) \n \n1. [Date and Time Engineering](#8)\n 1. [Introduction](#81)\n 1. [Date-Related Features](#82)\n 1. [Time-Related Features](#83)\n 1. [Lag Features](#84)\n 1. [Rolling Window](#85)\n 1. [Expanding Window](#86)\n 1. [Domain-Specific ](#87)\n1. [Read more](#9) \n 1. [References](#111) ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
\n# 1. Introduction\n\n\n## What is Feature Engineering?\n\n### Feature Engineering is a process of extracting useful features from existing raw data using maths,statistics and domain knowledge.\n\n\n#### Feature Engineering is one of the most important steps to complete before starting a Machine Learning analysis.Most of the basic Feature Engineering techniques consist of finding inconsistencies in the data and of creating new features by combining/diving existing ones. Creating the best possible Machine Learning/Deep Learning model can certainly help to achieve good results, but choosing the right features in the right format to feed in a model can by far boost performances leading to the following benefits:\n\n* Enable us to achieve good model performances using simpler Machine Learning models.\n* Using simpler Machine Learning models, increases the transparency of our model, therefore making easier for us to understand how is making its predictions.\n* Reduced need to use Ensemble Learning techniques.\n* Reduced need to perform Hyperparameters Optimization.\n\n#### In this kernel notebook I am going to cover an exhaustive Feature Engineering topics. I hope you like this kernel and appreciate to leave your suggestions /comments and please do UPVOTE if you like this notebook.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 2. Missing Data Imputation
\n\n\n### Data in real world are rarely clean and homogeneous. Typically this is because of the following reasons\n\n1. Corrupt data\n2. Failure to load information\n3. Incomplete extraction\n4. Noisy, and inconsistent\n5. Incomplete data\n\nSo it is an important task of a Data scientist to prepossess the data by filling missing values because making the right decision on how to handle it generates robust data models. It is important to be handled as they could lead to wrong prediction or classification for any given model being used. The goal of this article is to cover the basic techniques for handling missing values in a dataset.\n\nReal-world data often has missing values.\nFirstly we cannot simply ignore missing values in a dataset. We must handle them in some way for the very practical reason that most algorithms do not accept missing values.\n\n\"Common sense\" is not sensible here.From my experience the 2 most commonly recommended ways of dealing with missing data actually are not accurate .\n\nThey are:\n\n1. Dropping observations that have missing values\n2. Imputing the missing values based on other observations\n\nDropping missing values is sub-optimal because when we drop observations, we drop information.\n\nThe fact that the value was missing may be informative in itself.We need to understand business deeper to uncover why this information is missing in real world problems.In real time problems we need to make predictions even if some of the features are missing !!!.\n\n\nImputing missing values is sub-optimal because the value was originally missing but you filled it in, which always leads to a loss in information, no matter how sophisticated our imputation method is.\n\n\"Missingness\" is almost always informative in itself, and we should tell our algorithm if a value was missing.Even if we build a model to impute our values, we are not adding any real information. You’re just reinforcing the patterns already provided by other features.\n\nBasically, **there are three categories of missing data**. We assume that each record or observation can be divided into an \"observable component\" and an \"unobservable component\". We also assume that the records are independent and identically distributed.\n\n1. **MCAR (Missing Completely At Random)** where the pattern of missinginess is statistically independent of the data record. Example: you have a data set on a piece of paper and you spill coffee on the paper destroying part of the data.\n\n2. **MAR (Missing At Random)** where the probability distribution of the pattern of missingness is functionally dependent upon the observable component in the record. MCAR is a special case of MAR. Example: you have a question on a survey asking if the survey participant is a drug addict and another question which asks if the survey participant has less than one alcoholic drink per year. Assume the answer to the alcoholic drink question is always observable, then the probability that someone fails to answer the drug addict question is most likely functionally dependent upon their answer to the alcoholic drink question.\n\n3. **MNAR (Missing Not at Random)** which is defined as the case which is NOT MAR. In the MNAR case, you can have situations where both the drug addict and alcoholic drink questions are absent in the same record. Another example, is a case where the question is: \"What is your gender?\" Suppose that females are less likely to answer this question than males. This is another example of an MNAR question because the probability that the answer is observable is conditionally dependent upon the unobservable component of the data record.\n\nNow that we know the basic terminology,I like to share some strategies and recommendations\n\n### Important Strategies:\n\n1. We should never insert mean, mode, median, max, min or anything else for missing values. That is, avoid deterministic imputation even though it is widely used and available in most software packages. It underestimates and distorts the statistical regularities (e.g., underestimates variance is one example) present in the data sample. \n\n2. If the data records are MCAR Then you can delete records with missing data.\n\n3. If the data records are MCAR, then sometimes you can stochastically impute the missing values rather than deterministically impute them. So this means that if you specify the marginal probability distribution of a missing value as Gaussian with some known mean and some known variance then you can sample from that distribution to impute values into the data set. We need to be careful and do some additional research and analysis on data ,understand the business completely and take a judicious decision.\n\n4. If the data is MAR then an algorithm such as Expectation Maximization can be used to handle the missing observations.\n\n5. If the data is MNAR we can include binary indicators in the data record which explicitly identify when a variable is not observable. The challenge with this approach is that a highly nonlinear model needs to be designed to properly integrate this information in an appropriate manner. This might work in a machine learning algorithm where the binary indicators \"disconnect\" the influence of predictors which are not observable. Consequently, the MNAR theory (i.e., the theory of the joint distribution of the complete data record and missing data pattern) is instantiated in the learning machine's probabilistic model of its statistical environment.\n\n\n### Missing numeric data\n\nFor missing numeric data, we should flag and fill the values.\n\n1. Flag the observation with an indicator variable of missingness.\n2. Secondly fill the original missing value with 0 just to meet the technical requirement of no missing values.\n\nBy using this technique of flagging and filling, we are essentially allowing the algorithm to estimate the optimal constant for missingness, instead of just filling it in with the mean.\n\n### Missing categorical data\n\nThe best way to handle missing data for categorical features is to simply label them as ’Missing’!\n\n1. We are essentially adding a new class for the feature.\n2. This tells the algorithm that the value was missing.\n3. This also gets around the technical requirement for no missing values.\n\n#### For detailed understanding of missing value treatment please visit my other kernel https://www.kaggle.com/pavansanagapati/simple-tutorial-how-to-handle-missing-data","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 1. Complete Case Analysis
\n\n**Complete-case analysis (CCA)**, also called list-wise deletion of cases, consists in simply discarding observations where values in any of the variables are missing. Complete Case Analysis means literally analysing only those observations for which there is information in all of the variables (Xs). \nCCA can be applied to both categorical and numerical variables.\n\n**Assumptions**\nCCA works well when the data are missing completely at random (MCAR). In this case, excluding observations with missing data would be the same as randomly excluding some observations from the dataset, given that the missing data are totally at random. \n\n**Advantages**\n* Easy to implement\n* The same set of data is used for all analyses (no data manipulation)\n* Preserves variable distribution (if data is MCAR, then the distribution of the variables of the reduced dataset should match the distribution in the original dataset)\n\n**Disadvantages**\n* It can exclude a large fraction of the original sample, which are potentially informative for the analysis\n* CCA will be biased if the complete cases systematically differ from the original sample (e.g. when the missing information is in fact MAR or NMAR: not missing at random).\n\nIn practice, CCA may be an acceptable method when the amount of missing information is small. Unfortunately, there is no rule of thumb to determine how much missing data is small or negligible.\n\nIn many real life datasets, the amount of missing data is never small, and therefore CCA is typically never an option. \n\n## Cornell Birdcall Identification Challenge\n\nOver 10,000 bird species occur in the world, and they can be found in nearly every environment, from untouched rainforests to suburbs and even cities. Birds play an essential role in nature. They are high up in the food chain and integrate changes occurring at lower levels. As such, birds are excellent indicators of deteriorating habitat quality and environmental pollution. However, it is often easier to hear birds than see them.\n\nWith proper sound detection and classification, researchers could automatically intuit factors about an area’s quality of life based on a changing bird population.\n\nThere are already many projects underway to extensively monitor birds by continuously recording natural soundscapes over long periods. However, as many living and nonliving things make noise, the analysis of these datasets is often done manually by domain experts. These analyses are painstakingly slow, and results are often incomplete. Data science may be able to assist, so researchers have turned to large crowdsourced databases of focal recordings of birds to train AI models. Unfortunately, there is a domain mismatch between the training data (short recording of individual birds) and the soundscape recordings (long recordings with often multiple species calling at the same time) used in monitoring applications. This is one of the reasons why the performance of the currently used AI models has been subpar.\n\n### Objective\n\nTo identify a wide variety of bird vocalizations in soundscape recordings. Due to the complexity of the recordings, they contain weak labels. There might be anthropogenic sounds (e.g., airplane overflights) or other bird and non-bird (e.g., chipmunk) calls in the background, with a particular labeled bird species in the foreground. Bring new ideas to build effective detectors and classifiers for analyzing complex soundscape recordings.\n\nSo let us use the dataset of Cornell Lab of Ornithology’s Center for Conservation Bioacoustics (CCB) to do a complete exploratory data analysis and finding the insights about data and based on the findings come up with AI model that can achieve the above objective.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Import Libraries\nimport pandas as pd\nimport numpy as np\nimport matplotlib as mpl\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.ensemble import AdaBoostClassifier\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.model_selection import train_test_split\n# Load Data\ntrain = pd.read_csv('../input/birdsong-recognition/train.csv')\ntest = pd.read_csv('../input/birdsong-recognition/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.info(),test.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"total = train.isnull().sum().sort_values(ascending=False)\npercent = (train.isnull().sum()/train.isnull().count()).sort_values(ascending=False)\nmissing_data = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\nf, ax = plt.subplots(figsize=(15, 6))\nplt.xticks(rotation='90')\nsns.barplot(x=missing_data.index, y=missing_data['Percent'])\nplt.xlabel('Features', fontsize=15)\nplt.ylabel('Percent of Missing Values', fontsize=15)\nplt.title('Percentage of Missing Data by Feature', fontsize=15)\nmissing_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's go ahead and have a look at how many observations we would drop\nprint('Total bird records with values in all variables: ', train.dropna().shape[0])\nprint('Total bird records: ', train.shape[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In fact, we have complete information for only close 25% of our records in the above dataset. Thus, CCA would not be an option for this dataset.\n## 2. Mean / Median / Mode Imputation
\n\nThis strategy can be applied on a feature which has numeric data like the age of a person or the ticket fare. We can calculate the mean, median or mode of the feature and replace it with the missing values. This is an approximation which can add variance to the data set. But the loss of the data can be negated by this method which yields better results compared to removal of rows and columns. Replacing with the above three approximations are a statistical approach of handling the missing values. This method is also called as leaking the data while training. Another way is to approximate it with the deviation of neighbouring values. This works better if the data is linear.\n\nMEAN: Suitable for continuous data without outliers","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load Data\ntrain = pd.read_csv('../input/titanic/train.csv')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"total = train.isnull().sum().sort_values(ascending=False)\npercent = (train.isnull().sum()/train.isnull().count()).sort_values(ascending=False)\nmissing_data = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])\nf, ax = plt.subplots(figsize=(15, 6))\nplt.xticks(rotation='90')\nsns.barplot(x=missing_data.index, y=missing_data['Percent'])\nplt.xlabel('Features', fontsize=15)\nplt.ylabel('Percent of Missing Values', fontsize=15)\nplt.title('Percentage of Missing Data by Feature', fontsize=15)\nmissing_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Let us take Age feature and find the total no of records it has null values\ndf = train\ndf['Age'].isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df['Age'].replace(np.NaN,df['Age'].mean()).head(15)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you see above in the fifth row the mean value is replaced in place of NaN.\n\n**MEDIAN :** Suitable for continuous data with outliers","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_median = train\ndf_median['Age'].fillna(df_median['Age'].median(),inplace=True)\ndf_median['Age']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Mode:** \n\nFor categorical feature we can select to fill in the missing values with the most common value(mode) as illustrated below.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load Data\ntrain = pd.read_csv('../input/birdsong-recognition/train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# As per the categorical features which have missing values I have choosen \"background\" as feature for Mode imputation\n\ndata_cat=train\ndata_cat['background'].fillna(data_cat['background'].mode()[0], inplace=True)\ndata_cat['background'].head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3. Random Sample Imputation
\n\nImputation is the act of replacing missing data with statistical estimates of the missing values. The goal of any imputation technique is to produce a complete dataset that can then be then used for machine learning.\n\nRandom sampling imputation is in principle similar to mean/median imputation, in the sense that it aims to preserve the statistical parameters of the original variable, for which data is missing.\n\nRandom sampling consist of taking a random observation from the pool of available observations of the variable, and using that randomly extracted value to fill the NA. In Random Sampling one takes as many random observations as missing values are present in the variable.\n\nBy random sampling observations of the variable for those instances where data is available, we guarantee that the mean and standard deviation of the variable are preserved.\n\n**Assumptions**\n\n* Random sample imputation assumes that the data are missing completely at random (MCAR). If this is the case, it makes sense to substitute the missing values, by values extracted from the original variable distribution. \n* From a probabilistic point of view, values that are more frequent (like the mean or the median) will be selected more often (because there are more of them to select from), but other less frequent values will be selected as well. Thus, the variance of the variable is preserved. \n* The rationale is to replace the population of missing values with a population of values with the same distribution of the variable.\n\n**Advantages**\n* Easy to implement\n* Fast way of obtaining complete datasets\n* Preserves the variance of the variable\n\n**Limitations**\n* Randomness\n\nRandomness may not seem much of a concern when replacing missing values for data competitions, where the whole batch of missing values is replaced once and then the dataset is scored and that is the end of the problem. However, in business scenarios the situation is very different. \n\nReplacement of missing values by random sample, although similar in concept to replacement by the median or mean, is not as widely used in the data science community as the mean/median imputation, presumably because of the element of randomness.However, it is a valid approach, with advantages over mean/median imputation as it preserves the distribution of the variable.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nimport warnings\nwarnings.filterwarnings('ignore')\ntrain = pd.read_csv('../input/titanic/train.csv', usecols = ['Age', 'Fare', 'Survived'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's separate into training and testing set\n\nX_train, X_test, y_train, y_test = train_test_split(train, train.Survived, test_size=0.3,random_state=0)\nX_train.shape, X_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's make a function to create 3 variables from Age 1-filling NA with median, 2- random sampling or 3- zeroes\n\ndef impute_na(df, variable, median):\n df[variable+'_median'] = df[variable].fillna(median)\n df[variable+'_zero'] = df[variable].fillna(0)\n \n # random sampling\n df[variable+'_random'] = df[variable]\n # extract the random sample to fill the na\n random_sample = X_train[variable].dropna().sample(df[variable].isnull().sum(), random_state=0)\n # pandas needs to have the same index in order to merge datasets\n random_sample.index = df[df[variable].isnull()].index\n df.loc[df[variable].isnull(), variable+'_random'] = random_sample","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"median = X_train.Age.mean()\nimpute_na(X_train, 'Age', median)\nX_train.head(20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let us see the distribution of the Age variable after filling NA with random value\nfig = plt.figure()\nax = fig.add_subplot(111)\nX_train['Age'].plot(kind='kde', ax=ax)\nX_train.Age_random.plot(kind='kde', ax=ax, color='red')\nlines, labels = ax.get_legend_handles_labels()\nax.legend(lines, labels, loc='best')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let us see distribution of the Age variable after filling NA with median value\nfig = plt.figure()\nax = fig.add_subplot(111)\nX_train['Age'].plot(kind='kde', ax=ax)\nX_train.Age_median.plot(kind='kde', ax=ax, color='red')\nlines, labels = ax.get_legend_handles_labels()\nax.legend(lines, labels, loc='best')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let us see distribution of the Age variable after filling NA with zero value\nfig = plt.figure()\nax = fig.add_subplot(111)\nX_train['Age'].plot(kind='kde', ax=ax)\nX_train.Age_zero.plot(kind='kde', ax=ax, color='red')\nlines, labels = ax.get_legend_handles_labels()\nax.legend(lines, labels, loc='best')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4. Replacement by Arbitrary Value
\n\nReplacing the NA by artitrary values should be used when there are reasons to believe that the NA are not missing at random. In situations like this, we would not like to replace with the median or the mean, and therefore make the NA look like the majority of our observations.\nInstead, we want to flag them. We want to capture the missingness somehow by following the below approaches\n\n1) adding an additional binary variable to indicate whether the value is missing (1) or not (0)\n2) replacing the NA by a value at a far end of the distribution\n\nIt consists of replacing the NA by an arbitrary value. \n\nThe problem consists in deciding which arbitrary value to choose.\n\n**Advantages**\n\n* Easy to implement\n* Captures the importance of missingess if there is one\n\n**Disadvantages**\n\n* Distorts the original distribution of the variable\n* If missingess is not important, it may mask the predictive power of the original variable by distorting its distribution\n* Hard to decide which value to use If the value is outside the distribution it may mask or create outliers\n\nWhen variables are captured by third parties, like credit agencies, they place arbitrary numbers already to signal this missingness. It is a common practice in real life data collections.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nimport warnings\nwarnings.filterwarnings('ignore')\ntrain = pd.read_csv('../input/titanic/train.csv', usecols = ['Age', 'Fare', 'Survived'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's separate into training and testing set\n\nX_train, X_test, y_train, y_test = train_test_split(train, train.Survived, test_size=0.3,random_state=0)\nX_train.shape, X_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Let us define a function with replacement of NA value with an arbitrary value as 0 and 100\ndef impute_na(df, variable):\n df[variable+'_zero'] = df[variable].fillna(0)\n df[variable+'_hundred']= df[variable].fillna(100)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's replace the NA with the median value in the training set\nimpute_na(X_train, 'Age')\nX_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The arbitrary value has to be determined for each variable specifically. For example, for this dataset, the choice of replacing NA in age by 0 or 100 are valid, because none of those values are frequent in the original distribution of the variable, and they lie at the tails of the distribution.\n\nHowever, if we were to replace NA in fare, those values are not good any more, because we can see that fare can take values of up to 500. So we might want to consider using 500 or 1000 to replace NA instead of 100.\nAs you can see this is totally arbitrary. And yet, it is used in the industry.\n\nTypical values chose by companies are -9999 or 9999, or similar.\n\n## 5. End of Distribution Imputation
\n\nOn occasions, one has reasons to suspect that missing values are not missing at random. And if the value is missing, there has to be a reason for it. Therefore, we would like to capture this information.\n\nAdding an additional variable indicating missingness may help with this task. However, the values are still missing in the original variable, and they need to be replaced if we plan to use the variable in machine learning.\n\nSometimes, we may also not want to increase the feature space by adding a variable to capture missingness.\nSo what can we do instead?\n\nWe can replace the NA, by values that are at the far end of the distribution of the variable.\n\nThe rationale is that if the value is missing, it has to be for a reason, therefore, we would not like to replace missing values for the mean and make that observation look like the majority of our observations. Instead, we want to flag that observation as different, and therefore we assign a value that is at the tail of the distribution, where observations are rarely represented in the population.\n\n**Advantages**\n\n* Easy to implement\n* Captures the importance of missingess if there is one\n\n**Disadvantages**\n\n* Distorts the original distribution of the variable\n* If missingess is not important, it may mask the predictive power of the original variable by distorting its distribution\n* If the number of NA is big, it will mask true outliers in the distribution\n* If the number of NA is small, the replaced NA may be considered an outlier and pre-processed in a subsequent step of feature engineering\n\nThis method is used in finance companies. When capturing the financial history of customers, if some of the variables are missing, the company does not like to assume that missingness is random. Therefore, a different treatment is provided to replace them, by placing them at the end of the distribution.\n\nLet us see how we can implement this imputation below","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/titanic/train.csv', usecols = ['Age', 'Fare', 'Survived'])\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's separate into training and testing set\nX_train, X_test, y_train, y_test = train_test_split(train, train.Survived, test_size=0.3,random_state=0)\nX_train.shape, X_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.Age.hist(bins=50)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# far end of the distribution\nX_train.Age.mean()+3*X_train.Age.std()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let us see if there are a few outliers for Age, according to its distribution these outliers will be masked when we replace NA by values at the far end \nsns.boxplot('Age', data=train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def impute_na(df, variable, median, extreme):\n df[variable+'_far_end'] = df[variable].fillna(extreme)\n df[variable].fillna(median, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's replace the NA with the median value in the training and testing sets\nimpute_na(X_train, 'Age', X_train.Age.median(), X_train.Age.mean()+3*X_train.Age.std())\nX_train.head(20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# As you can see an accumulation of values around the median for the median imputation\nX_train.Age.hist(bins=50)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Now finally let us see an accumulation of values at the far end imputation\nX_train.Age_far_end.hist(bins=50)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Far end imputation now indicates that there are no outliers in the variable as shown below\nsns.boxplot('Age_far_end', data=X_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Categorical Encoding
\n\nBetter encoding of categorical data can mean better model performance.\n\nThe categorical-encoder is the most extensive Python package for categorical variable encoding, including some common procedures like one hot encoding and weight of evidence, as well as more complex ways of encoding variables like BaseN and feature hashing. \n\nUse Category Encoders to improve model performance when you have nominal or ordinal data that may provide value.\n\n**So before get into details of encoders let us first understand how many types of data and what are they?\n**\n\nThere are four data measurement scales in research and statistics.They are \n\n* Nominal Data\n* Ordinal Data\n* Interval Data\n* Ratio Data\n\nThe Nominal and Ordinal data types are classified under categorical\n\nInterval and ratio data are classified under numerical\n\nCategorical data is a data type that not quantitative i.e. does not have a number. Therefore, both nominal and ordinal data are non-quantitative, which may mean a string of text or date. \n\n**What is Nominal Data? **\n\nNominal data is defined as data that is used for naming or labelling variables, without any quantitative value. It is sometimes called “named” data - a meaning coined from the word nominal. There is usually no intrinsic ordering to nominal data. For example, Race is a nominal variable having a number of categories, but there is no specific way to order from highest to lowest and vice versa.\n\n**What is Ordinal Data? **\n\nOrdinal data is a type of categorical data with an order. The variables in ordinal data are listed in an ordered manner. The ordinal variables are usually numbered, so as to indicate the order of the list. However, the numbers are not mathematically measured or determined but are merely assigned as labels for opinions.\n\n\nNominal data is a group of non-parametric variables, while Ordinal data is a group of non-parametric ordered variables. Although, they are both non-parametric variables, what differentiates them is the fact that ordinal data is placed into some kind of order by their position.\n\n**For example, very hot, hot, cold, very cold, warm are all nominal data when considered individually. But when placed on a scale and arranged in a given order (very hot, hot, warm, cold, very cold), they are regarded as ordinal data.**\n\n\n**Cardinality:**\"Cardinality\" means the number of unique values in a column.\n\nFor nominal columns we use OneHotEncoder, Hashing, LeaveOneOut, and Target encoding. Avoid OneHot for high cardinality columns and decision tree-based algorithms.\n\nFor ordinal columns try Ordinal (Integer), Binary, OneHot, LeaveOneOut, and Target. Helmert, Sum, BackwardDifference and Polynomial are less likely to be helpful, but if you have time or theoretic reason you might want to try them.\n\n\n**Important Note** : We should classify data as one of seven types to make better models faster. Here are the seven data types:\n\n* Useless — useless for machine learning algorithms, that is — discrete\n* Nominal — groups without order — discrete\n* Binary — either/or — discrete\n* Ordinal — groups with order — discrete\n* Count — the number of occurrences — discrete\n* Time — cyclical numbers with a temporal component — continuous\n* Interval — positive and/or negative numbers without a temporal component — continuous\n\n\n**Classic Encoders**\nThe first group of five classic encoders can be seen on a continuum of embedding information in one column (Ordinal) up to k columns (OneHot). These are very useful encodings for machine learning practitioners to understand.\n\n* **OneHot** — one column for each value to compare vs. all other values. Nominal, ordinal.\n* **Binary** — convert each integer to binary digits. Each binary digit gets one column. Some info loss but fewer dimensions. Ordinal.\n* **Ordinal** — convert string labels to integer values 1 through k. Ordinal.\n* **BaseN** — Ordinal, Binary, or higher encoding. Nominal, ordinal. Doesn’t add much functionality. Probably avoid.\n* **Hashing** — Like OneHot but fewer dimensions, some info loss due to collisions. Nominal, ordinal.\n* **Sum** — Just like OneHot except one value is held constant and encoded as -1 across all columns.\n\n**Contrast Encoders**\nThe five contrast encoders all have multiple issues that I argue make them unlikely to be useful for machine learning. They all output one column for each value found in a column. Their stated intents are below.\n\n* **Helmert (reverse)** — The mean of the dependent variable for a level is compared to the mean of the dependent variable over all previous levels.\n* **Backward Difference** — the mean of the dependent variable for a level is compared with the mean of the dependent variable for the prior level.\n* **Polynomial — orthogonal polynomial contrasts**. The coefficients taken on by polynomial coding for k=4 levels are the linear, quadratic, and cubic trends in the categorical variable.\n\n**Bayesian Encoders**\nThe Bayesian encoders use information from the dependent variable in their encodings. They output one column and can work well with high cardinality data.\n\n* Target — use the mean of the DV, must take steps to avoid overfitting/ response leakage. Nominal, ordinal. For classification tasks.\n* LeaveOneOut — similar to target but avoids contamination. Nominal, ordinal. For classification tasks.\n* WeightOfEvidence — \n* James-Stein \n* M-estimator - Simplified target encoder.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 1. Classic Encoders
\n## One Hot Encoder
\n\nOne hot encoding, consists of replacing the categorical variable by different boolean variables, which take value 0 or 1, to indicate whether or not a certain category / label of the variable was present for that observation.\n\nEach one of the boolean variables are also known as dummy variables or binary variables.\n\nFor example, from the categorical variable \"Gender\", with labels 'female' and 'male', we can generate the boolean variable \"female\", which takes 1 if the person is female or 0 otherwise. We can also generate the variable male, which takes 1 if the person is \"male\" and 0 otherwise. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\n\ntrain_df = pd.read_csv('../input/titanic/train.csv', usecols=['Sex'])\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\nNow let us perform one hot encoding","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.get_dummies(train_df).head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# For a clear understanding let us visualise like below\npd.concat([train_df, pd.get_dummies(train_df)], axis=1).head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\nAs you can see above that we need only 1 of the 2 dummy variables to represent the original categorical variable Sex. Any of the 2 will suffice, and it doesn't matter which one we select, since they are equivalent.Therefore, to encode a categorical variable with 2 labels, we need 1 dummy variable. \n\n**So to encode categorical variable with k labels, we need k-1 dummy variables.**\nLet us see how to achieve this","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.get_dummies(train_df, drop_first=True).head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let us look at another example feature like Embarkment","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df=pd.read_csv('../input/titanic/train.csv', usecols=['Embarked'])\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let us check the number of unique emabrked labels\ntrain_df.Embarked.unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Now let us get the complete set of dummy variables for embarked feature\n\npd.get_dummies(train_df).head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Now let us get k-1 dummy variables\n\npd.get_dummies(train_df, drop_first=True).head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Also to mention we may have some missing values in this feature so it is better to include an additional dummy variable to indicate whether there was missing data\n\npd.get_dummies(train_df, drop_first=True, dummy_na=True).head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Now let us find out how many observations we have for each variable (i.e., each category)\n\npd.get_dummies(train_df, drop_first=True, dummy_na=True).sum(axis=0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Both pandas and sklearn will provide a whole set of dummy variables from a categorical variable. This is, instead of returning k-1 binary variables, they will return k, with the option in pandas of dropping the first binary variable and obtain k-1.\n\n**When should you use k and when k-1?**\n\nWhen the original variable is binary, that is, when the original variable has only 2 labels, then you should create one and only one binary variable.\nWhen the original variable has more than 2 labels, the following is important:\n\n**One hot encoding into k-1:**\n\nOne hot encoding into k-1 binary variables takes into account that we can use 1 less dimension and still represent the whole information: if the observation is 0 in all the binary variables, then it must be 1 in the final (removed) binary variable. As an example, for the variable gender encoded into male, if the observation is 0, then it has to be female. We do not need the additional female variable to explain that. \nOne hot encoding with k-1 binary variables should be used in linear regression, to keep the correct number of degrees of freedom (k-1). The linear regression has access to all of the features as it is being trained, and therefore examines altogether the whole set of dummy variables. This means that k-1 binary variables give the whole information about (represent completely) the original categorical variable to the linear regression.\nAnd the same is true for all machine learning algorithms that look at ALL the features at the same time during training. For example, support vector machines and neural networks as well. And clustering algorithms.\n\n**One hot encoding into k dummy variables**\n\nHowever, tree based models select at each iteration only a group of features to make a decision. This is to separate the data at each node. Therefore, the last category, the one that was removed in the one hot encoding into k-1 variables, would only be taken into account by those splits or even trees, that use the entire set of binary variables at a time. And this would rarely happen, because each split usually uses 1-3 features to make a decision. So, tree based methods will never consider that additional label, the one that was dropped. Thus, if the categorical variables will be used in a tree based learning algorithm, it is good practice to encode it into k binary variables instead of k-1.\nFinally, if you are planning to do feature selection, you will also need the entire set of binary variables (k) to let the machine learning model select which ones have the most predictive power.\n\nTo understand the above concept in detail let me show you by training a model on titanic data with **categorical variables re-encoded with One Hot Encoding.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.ensemble import AdaBoostClassifier\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.model_selection import train_test_split","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df=pd.read_csv('../input/titanic/train.csv')\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Now let us a copy of the above dataset, in which we encode the categorical variables using One Hot Encoder\n\ntrain_df_OneHotEncoder = pd.concat([train_df[['Pclass', 'Age', 'SibSp','Parch', 'Survived']], # Choosen the numerical variables \n pd.get_dummies(train_df.Sex, drop_first=True), # Sex as explained above which is binary categorical variable\n pd.get_dummies(train_df.Embarked, drop_first=True)], # Embarked as explained above has k categories in categorical\n axis=1)\n\ntrain_df_OneHotEncoder.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train, X_test, y_train, y_test = train_test_split(train_df_OneHotEncoder[['Pclass', 'Age', 'SibSp',\n 'Parch', 'male', 'Q', 'S']].fillna(0),\n train_df_OneHotEncoder.Survived,\n test_size=0.3,\n random_state=0)\nX_train.shape, X_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's build a random forest model with the above data\n\nrf = RandomForestClassifier(n_estimators=200, random_state=39, max_depth=3)\nrf.fit(X_train, y_train)\nprint('Train set')\npred = rf.predict_proba(X_train)\nprint('Random Forests roc-auc: {}'.format(roc_auc_score(y_train, pred[:,1])))\nprint('Test set')\npred = rf.predict_proba(X_test)\nprint('Random Forests roc-auc: {}'.format(roc_auc_score(y_test, pred[:,1])))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Now let us build a AdaBoost classifier\n\nada = AdaBoostClassifier(n_estimators=200, random_state=44)\nada.fit(X_train, y_train)\nprint('Train set')\npred = ada.predict_proba(X_train)\nprint('AdaBoost roc-auc: {}'.format(roc_auc_score(y_train, pred[:,1])))\nprint('Test set')\npred = ada.predict_proba(X_test)\nprint('AdaBoost roc-auc: {}'.format(roc_auc_score(y_test, pred[:,1])))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Finally with logistic regression\nlogit = LogisticRegression(random_state=44)\nlogit.fit(X_train, y_train)\nprint('Train set')\npred = logit.predict_proba(X_train)\nprint('Logistic Regression roc-auc: {}'.format(roc_auc_score(y_train, pred[:,1])))\nprint('Test set')\npred = logit.predict_proba(X_test)\nprint('Logistic Regression roc-auc: {}'.format(roc_auc_score(y_test, pred[:,1])))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As per the above roc auc scores it is evident that Logistic regression and Adaboost show the highest performance using the titanic dataset.\n\n\n**One Hot encoding Pros & Cons**\n\n**Pros**\n\n- Straightforward to implement\n- Makes no assumption\n- Keeps all the information of the categorical variable\n\n**Cons**\n\n- Does not add any information that may make the variable more predictive\n- If the variable has loads of categories, then One Hot Encoder increases the feature space dramatically\nTo illustrate this let us take an example of another variable in the same dataset and see what actually it means","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's load the variable Cabin of the titanic dataset\n\ntrain_df=pd.read_csv('../input/titanic/train.csv', usecols = ['Cabin'])\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Now let's inspect the number of unique labels in Cabin feature\nprint('Number of unique labels in Cabin Feature: {}'.format(len(train_df.Cabin.unique())))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Now let us see how many features we can create if we did One Hot Encoder for Cabin feature\nCabin_OneHotEncoder = pd.get_dummies(train_df.Cabin)\nCabin_OneHotEncoder.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Cabin_OneHotEncoder.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\nIf we performed One Hot Encoding in the variable Cabin that contains 148 different labels, we would end up with 147 variables, where originally there was one. If we have a few categorical variables like this, we would end up with huge datasets. **Therefore, One Hot Encoder is not always the best option to encode categorical variables.**\n\n**If our datasets have a few multi-label variables, we will end up very soon with datasets with thousands of columns or more. And this may make training of our algorithms slow.**\n\nIn addition, many of these dummy variables may be similar to each other, since it is not unusual for 2 or more variables to share the same combinations of 1 and 0s.\n## Count and Frequency Encoder
\nAnother way to refer to variables that have a multitude of categories, is to call them variables with high cardinality.If a categorical variable contains multiple labels, then by re-encoding them using one hot encoding, we will expand the feature space dramatically.\n\nOne approach is to replace each label of the categorical variable by the count, this is the amount of times each label appears in the dataset .\nOr the frequency, this is the percentage of observations within that category. The 2 are equivalent.\n\n**Pros:**\n* Simple\n* Does not expand the feature space\n**Cons:**\n* If 2 labels appear the same amount of times in the dataset, that is, contain the same number of observations, they will be merged: may loose valuable information\n* Adds somewhat arbitrary numbers, and therefore weights to the different labels, that may not be related to their predictive power\n\nLet's see how this works:\n\t","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\nfrom sklearn.model_selection import train_test_split\n\ndata = pd.read_csv('../input/titanic/train.csv', usecols=['Embarked', 'Survived'])\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's have a look at how many labels\n\nfor col in data.columns[1:]:\n print(col, ': ', len(data[col].unique()), ' labels')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train, X_test, y_train, y_test = train_test_split(data[['Embarked']], data.Survived,\n test_size=0.3,\n random_state=0)\nX_train.shape, X_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's obtain the counts for each one of the labels in variable Embarked and capture this in a dictionary that we can use to re-map the labels\n\nX_train.Embarked.value_counts().to_dict()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# lets look at X_train so we can compare then the variable re-coding\n\nX_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# And now let's replace each label in X2 by its count.Firstly we make a dictionary that maps each label to the counts\nX_frequency_map = X_train.Embarked.value_counts().to_dict()\n\n# and now we replace X2 labels both in train and test set with the same map\nX_train.Embarked = X_train.Embarked.map(X_frequency_map)\nX_test.Embarked = X_test.Embarked.map(X_frequency_map)\n\nX_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Where in the original dataset, for the record 1(857) above in the variable Embarked before it was 'S', now it was replaced by the count 457. And so on for the rest of the categories.\n\n**Important Note**\n\nIf a category is present in the test set, that was not present in the train set, this method will generate missing data in the test set. This is why it is extremely important to handle rare categories.\nThen we can combine rare label replacement plus categorical encoding with counts like this: we may choose to replace the 10 most frequent labels by their count, and then group all the other labels under one label (for example \"Rare\"), and replace \"Rare\" by its count, to account for what I just mentioned.\n\nI want you to keep in mind that There is no rule of thumb to indicate which method you should use to encode categorical variables. It is mostly up to what makes sense for the data, and it also depends on what you are trying to achieve. In general, we value more model predictive power, whereas in business scenarios we want to capture and understand the information, and generally, we want to transform variables in a way that it makes 'Business sense'. Some of our common sense and a lot of conversation with the business people that understand the data well will be required to encode categorical labels.\n## Binary Encoder
\n\nBinary can be thought of as a hybrid of one-hot and hashing encoders. Binary creates fewer features than one-hot, while preserving some uniqueness of values in the the column. It can work well with higher dimensionality ordinal data.\n\nHere’s how it works:\n\n* The categories are encoded by OrdinalEncoder if they aren’t already in numeric form.\n* Then those integers are converted into binary code, so for example 5 becomes 101 and 10 becomes 1010\n* Then the digits from that binary string are split into separate columns. So if there are 4–7 values in an ordinal column then 3 new columns are created: one for the first bit, one for the second, and one for the third.\n* Each observation is encoded across the columns in its binary form.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport category_encoders as ce \nfrom sklearn.preprocessing import LabelEncoder\npd.options.display.float_format = '{:.2f}'.format \n\ndf = pd.read_csv('../input/titanic/train.csv', usecols=['Embarked', 'Survived'])\nX = df.drop('Survived', axis = 1)\ny = df.drop('Embarked', axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"binary_encoder = ce.BinaryEncoder(cols = ['Embarked'])\nbinary_encoder.fit_transform(X, y)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"With only three levels, the information embedded becomes muddled. There are many collisions and the model can’t glean much information from the features. Just one-hot encode a column if it only has a few values.\n\nIn contrast, binary really shines when the cardinality of the column is higher — with the 50 US states, for example.\n\n**Binary encoding creates fewer columns than one-hot encoding. It is more memory efficient. It also reduces the chances of dimensionality problems with higher cardinality.**\nMost similar values overlap with each other across many of the new columns. This allows many machine learning algorithms to learn the values similarity. Binary encoding is a decent compromise for ordinal data with high cardinality.\nFor nominal data a hashing algorithm with more fine-grained control usually makes more sense. \n## Ordinal Encoder
\n\nOrdinalEncoder converts each string value to a whole number. The first unique value in your column becomes 1, the second becomes 2, the third becomes 3, and so on.\n\nWhat the actual value was prior to encoding does not affect what it becomes when you fit_transform with OrdinalEncoder. The first value could have been 10 and the second value could have been 3. Now they will be 1 and 2, respectively.\n\nIf the column contains nominal data, stopping after you use OrdinalEncoder is a bad idea. Your machine learning algorithm will treat the variable as continuous and assume the values are on a meaningful scale. Instead, if you have a column with values car, bus, and truck you should first encode this nominal data using OrdinalEncoder. Then encode it again using one of the methods appropriate to nominal data that we’ll explore below.\n\nIn contrast, if your column values are truly ordinal, that means that the integer assigned to each value is meaningful. Assignment should be done with intention. Say your column had the string values “First”, “Third”, and “Second” in it. Those values should be mapped to the corresponding integers by passing OrdinalEncoder a list of dicts.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ordinal_encoder = ce.OrdinalEncoder(cols = ['Embarked'])\nordinal_encoder.fit_transform(X, y['Survived'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## BaseN Encoder
\nWhen the BaseN base = 1 it is basically the same as one hot encoding. When base = 2 it is basically the same as binary encoding. The main reason for BaseN’s existence is to possibly make grid searching easier. You could use BaseN with scikit-learn’s gridsearchCV.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"BaseN_encoder = ce.BaseNEncoder(cols = ['Embarked'])\nBaseN_encoder.fit_transform(X, y)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The default base for BaseNEncoder is 2, which is the equivalent of BinaryEncoder\n## Hashing Encoder
\n\nHashingEncoder implements the hashing trick. It is similar to one-hot encoding but with fewer new dimensions and some info loss due to collisions. The collisions do not significantly affect performance unless there is a great deal of overlap. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"Hashing_encoder = ce.HashingEncoder(cols = ['Embarked'])\nHashing_encoder.fit_transform(X, y)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The n_components parameter controls the number of expanded columns. The default is eight columns. In our example column with three values the default results in five columns full of 0s.\nIf you set n_components less than k you’ll have a small reduction in the value provided by the encoded data. You’ll also have fewer dimensions.\nYou can pass a hashing algorithm of your choice to HashingEncoder; the default is md5. It’s worth trying HashingEncoder for nominal and ordinal data if you have high cardinality features. \n## Sum Encoder
","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"Sum_encoder = ce.SumEncoder(cols = ['Embarked'])\nSum_encoder.fit_transform(X, y)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2. Contrast Encoders
\n\n## Helmert (reverse)
\nHelmert coding is the mirror image of difference coding: instead of comparing each level of categorical variable to the mean of the previous level, it is compared to the mean of the subsequent levels.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ce_helmert = ce.HelmertEncoder(cols = ['Embarked'])\nce_helmert.fit_transform(X, y)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Backward Difference
\nIn backward difference coding, the mean of the dependent variable for a level is compared with the mean of the dependent variable for the prior level. This type of coding may be useful for a nominal or an ordinal variable.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ce_backward = ce.BackwardDifferenceEncoder(cols = ['Embarked'])\nce_backward.fit_transform(X, y)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Polynomial
","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ce_poly = ce.PolynomialEncoder(cols = ['Embarked'])\nce_poly.fit_transform(X, y)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3. Bayesian Encoders
\nThe Bayesian encoders use information from the dependent variable in their encodings. They output one column and can work well with high cardinality data.\n## Target encoding / Mean encoding
\n\nUse the mean of the DV, must take steps to avoid overfitting/ response leakage. Nominal, ordinal. For classification tasks.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ce_target = ce.TargetEncoder(cols = ['Embarked'])\nce_target.fit(X, y)\nce_target.transform(X, y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Target with higher smoothing\nce_target_leaf = ce.TargetEncoder(cols = ['Embarked'], smoothing = 10)\nce_target_leaf.fit(X, y)\nce_target_leaf.transform(X, y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Target with lower smoothing\nce_target_leaf = ce.TargetEncoder(cols = ['Embarked'], smoothing = .10)\nce_target_leaf.fit(X, y)\nce_target_leaf.transform(X, y)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Weight of Evidence
\nWeight of Evidence (WoE) describes the relationship between a predictor and a binary dependent variable. Information Value (IV) is the measurement of that relationship’s power. Based on its role, IV can be used as a base for attributes selection.\n\n\nLogistic regression model is one of the most commonly used statistical technique for solving binary classification problem. It is acceptable in almost all the domains. These two concepts - **weight of evidence (WOE)** and **information value (IV) **evolved from the same logistic regression technique. These two terms have been in existence in credit scoring world for more than 4-5 decades. They have been used as a benchmark to screen variables in the credit risk modeling projects such as **probability of default**. They help to explore data and screen variables. It is also used in marketing analytics project such as customer attrition model, campaign response model etc.\n\n ![](\"https://i.imgur.com/5cKHNKJ.jpg\")
\n\n### Weight of Evidence (WOE)\n\nThe weight of evidence tells the predictive power of an independent variable in relation to the dependent variable.\n\nSince it evolved from credit scoring world, it is generally described as a measure of the separation of good and bad customers. \"Bad Customers\" refers to the customers who defaulted on a loan. and \"Good Customers\" refers to the customers who paid back loan.\n\n![](\"https://i.imgur.com/hrTnEXv.jpg\")
\n\nMany people do not understand the terms goods/bads as they are from different background than the credit risk. It's good to understand the concept of WOE in terms of events and non-events. It is calculated by taking the natural logarithm (log to base e) of division of % of non-events and % of events.\n\n![](\"https://i.imgur.com/dfosGgc.jpg\")
\n\nWOE is typically used as part of the credit scorecard development .The development process consists of four main parts: variable transformations, model training using logistic regression, model validation, and scaling.\n\n![](\"https://i.imgur.com/lbC7lfU.jpg\")
\n\n### Variable Transformations\n\nIf you torture the data long enough, it will confess to anything. A standard scorecard model, based on logistic regression, is an additive model; hence, special variable transformations are required. The commonly adopted transformations – fine classing, coarse classing, and either dummy coding or weight of evidence (WOE) transformation – form a sequential process providing a model outcome that is both easy to implement and explain to the business. Additionally, these transformations assist in converting non-linear relationships between independent variables and the dependent variable into a linear relationship – the customer behaviour often requested by the business.\n\n#### Fine classing\n\nApplied to all continuous variables and those discrete variables with high cardinality. This is the process of initial binning into typically between 20 and 50 fine granular bins.\n\n**To summarize create 10/20 bins/groups for a continuous independent variable and then calculates WOE and IV of the variable\n\n#### Coarse classing\n\nWhere a binning process is applied to the fine granular bins to merge those with similar risk and create fewer bins, usually up to ten. The purpose is to achieve simplicity by creating fewer bins, each with distinctively different risk factors while minimising the information loss. However, to create a robust model that is resilient to overfitting, each bin should contain a sufficient number of observations from the total account (5% is the minimum recommended by most practitioners). These opposing goals can be achieved through an optimisation in the form of optimal binning that maximises a variable’s predictive power during the coarse classing process. Optimal binning utilises the same statistical measures used during variable selection, such as information value, Gini and chi-square statistics. The most popular measure is, again, information value, although combination of two or more measures is often beneficial. The missing values, if they contain predictive information, should be a separate class or merged to bin with similar risk factors.\n\n**To summarize combine adjacent categories with similar WOE scores\n\n#### Dummy coding\n\nThe process of creating binary (dummy) variables for all coarse classes except the reference class. This approach may present issues as the extra variables requires more memory and processing resources, and occasionally overfitting may arise because of the reduced degrees of freedom.\n\n#### Weight of evidence (WOE) transformation\n\nThe alternative, more favoured, approach to dummy coding that substitutes each coarse class with a risk value, and in turn collapses the risk values into a single numeric variable. The numeric variable describes the relationship between an independent variable and a dependent variable. The WOE framework is well suited for logistic regression modelling as both are based on log-odds calculation. In addition, WOE transformation standardises all independent variables, hence, the parameters in a subsequent logistic regression can be directly compared. \n\n**Weight of Evidence (WOE)** helps to transform a continuous independent variable into a set of groups or bins based on similarity of dependent variable distribution i.e. number of events and non-events.\n\n**For continuous independent variables** : First, create bins (categories / groups) for a continuous independent variable and then combine categories with similar WOE values and replace categories with WOE values. Use WOE values rather than input values in your model.\n\n**For categorical independent variables** : Combine categories with similar WOE and then create new categories of an independent variable with continuous WOE values. In other words, use WOE values rather than raw categories in your model. The transformed variable will be a continuous variable with WOE values. It is same as any continuous variable.\n\n#### Why combine categories with similar WOE?\n\nIt is because the categories with similar WOE have almost same proportion of events and non-events. In other words, the behavior of both the categories is same.\n\n#### Rules related to WOE\n- Each category (bin) should have at least 5% of the observations.\n- Each category (bin) should be non-zero for both non-events and events.\n- The WOE should be distinct for each category. Similar groups should be aggregated. \n- The WOE should be monotonic, i.e. either growing or decreasing with the groupings.\n- Missing values are binned separately.\n\n#### Number of Bins (Groups)\n\nIn general, 10 or 20 bins are taken. Ideally, each bin should contain at least 5% cases. The number of bins determines the amount of smoothing - the fewer bins, the more smoothing. If someone asks you ' \"why not to form 1000 bins?\" The answer is the fewer bins capture important patterns in the data, while leaving out noise. Bins with less than 5% cases might not be a true picture of the data distribution and might lead to model instability.\n\n#### Handle Zero Event/ Non-Event\n\nIf a particular bin contains no event or non-event, you can use the formula below to ignore missing WOE. We are adding 0.5 to the number of events and non-events in a group.\n\nAdjustedWOE = ln (((Number of non-events in a group + 0.5) / Number of non-events)) / ((Number of events in a group + 0.5) / Number of events))\n\n\n#### How to check correct binning with WOE\n\n1. The WOE should be monotonic i.e. either growing or decreasing with the bins. You can plot WOE values and check linearity on the graph.\n\n2. Perform the WOE transformation after binning. Next, we run logistic regression with 1 independent variable having WOE values. If the slope is not 1 or the intercept is not ln(% of non-events / % of events) then the binning algorithm is not good.\n\nBoth dummy coding and WOE transformation give the similar results. The choice which one to use mainly depends on personal preferences.\n\nIn general optimal binning, dummy coding and weight of evidence transformation are, when carried out manually, time-consuming processes. \n\n**WOE Advantage**:\n\nThe advantages of WOE transformation are\n\n**1. Handles missing values**\n\n**2. Handles outliers**\n\n**3. The transformation is based on logarithmic value of distributions. This is aligned with the logistic regression output function**\n\n**4. No need for dummy variables**\n\n**5. By using proper binning technique, it can establish monotonic relationship (either increase or decrease) between the independent and dependent variable**\n\n**WOE Disadvantage**:\n\nThe main disadvantage of WOE transformation is \n\n**- in only considering the relative risk of each bin, without considering the proportion of accounts in each bin. The information value can be utilised instead to assess the relative contribution of each bin.**\n\n### Information Value (IV)\n\nInformation value is one of the most useful technique to select important variables in a predictive model. It helps to rank variables on the basis of their importance. The IV is calculated using the following formula :\n ![](\"https://i.imgur.com/r6ACeFN.jpg\")
\n\n ** IV statistic in credit scoring can be interpreted as follows. **\n\n![](\"https://i.imgur.com/cZx3taD.jpg\")
\n \nIf the IV statistic is:\n\n- Less than 0.02, then the predictor is not useful for modeling (separating the Goods from the Bads)\n- 0.02 to 0.1, then the predictor has only a weak relationship to the Goods/Bads odds ratio\n- 0.1 to 0.3, then the predictor has a medium strength relationship to the Goods/Bads odds ratio\n- 0.3 to 0.5, then the predictor has a strong relationship to the Goods/Bads odds ratio.\n- 0.5, suspicious relationship (Check once)\n\n#### Important Points\n\n- Information value increases as bins / groups increases for an independent variable. We have to be careful when there are more than **20 bins** as some bins may have a very few number of events and non-events.\n\n- Information value should not be used as a feature selection method when you are building a classification model other than binary logistic regression (for eg. random forest or SVM) as it's designed for binary logistic regression model only.","execution_count":null},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!pip install chart-studio","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"!pip install woe","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"# Import packages\nimport pandas as pd\nimport numpy as np\nimport pandas.core.algorithms as algos\nfrom pandas import Series\nimport scipy.stats.stats as stats\nimport re\nimport traceback\nimport string\nimport os\nimport woe\nfrom woe.eval import plot_ks\nprint(os.listdir(\"../input\"))\nimport matplotlib.pyplot as plt\n%matplotlib inline\nfrom pylab import rcParams\nrcParams['figure.figsize'] = 14, 8\nimport warnings\nwarnings.filterwarnings('ignore')\nimport chart_studio.plotly.plotly as py\nimport chart_studio.plotly\nmax_bin = 20\nforce_bin = 3","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.read_csv('../input/uci-credit-carefrom-python-woe-pkg/UCI_Credit_Card.csv',sep=',')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Define a binning function for continuous independent variables","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def mono_bin(Y, X, n = max_bin):\n df1 = pd.DataFrame({\"X\": X, \"Y\": Y})\n justmiss = df1[['X','Y']][df1.X.isnull()]\n notmiss = df1[['X','Y']][df1.X.notnull()]\n r = 0\n while np.abs(r) < 1:\n try:\n d1 = pd.DataFrame({\"X\": notmiss.X, \"Y\": notmiss.Y, \"Bucket\": pd.qcut(notmiss.X, n)})\n d2 = d1.groupby('Bucket', as_index=True)\n r, p = stats.spearmanr(d2.mean().X, d2.mean().Y)\n n = n - 1 \n except Exception as e:\n n = n - 1\n\n if len(d2) == 1:\n n = force_bin \n bins = algos.quantile(notmiss.X, np.linspace(0, 1, n))\n if len(np.unique(bins)) == 2:\n bins = np.insert(bins, 0, 1)\n bins[1] = bins[1]-(bins[1]/2)\n d1 = pd.DataFrame({\"X\": notmiss.X, \"Y\": notmiss.Y, \"Bucket\": pd.cut(notmiss.X, np.unique(bins),include_lowest=True)}) \n d2 = d1.groupby('Bucket', as_index=True)\n \n d3 = pd.DataFrame({},index=[])\n d3[\"MIN_VALUE\"] = d2.min().X\n d3[\"MAX_VALUE\"] = d2.max().X\n d3[\"COUNT\"] = d2.count().Y\n d3[\"EVENT\"] = d2.sum().Y\n d3[\"NONEVENT\"] = d2.count().Y - d2.sum().Y\n d3=d3.reset_index(drop=True)\n \n if len(justmiss.index) > 0:\n d4 = pd.DataFrame({'MIN_VALUE':np.nan},index=[0])\n d4[\"MAX_VALUE\"] = np.nan\n d4[\"COUNT\"] = justmiss.count().Y\n d4[\"EVENT\"] = justmiss.sum().Y\n d4[\"NONEVENT\"] = justmiss.count().Y - justmiss.sum().Y\n d3 = d3.append(d4,ignore_index=True)\n \n d3[\"EVENT_RATE\"] = d3.EVENT/d3.COUNT\n d3[\"NON_EVENT_RATE\"] = d3.NONEVENT/d3.COUNT\n d3[\"DIST_EVENT\"] = d3.EVENT/d3.sum().EVENT\n d3[\"DIST_NON_EVENT\"] = d3.NONEVENT/d3.sum().NONEVENT\n d3[\"WOE\"] = np.log(d3.DIST_EVENT/d3.DIST_NON_EVENT)\n d3[\"IV\"] = (d3.DIST_EVENT-d3.DIST_NON_EVENT)*np.log(d3.DIST_EVENT/d3.DIST_NON_EVENT)\n d3[\"VAR_NAME\"] = \"VAR\"\n d3 = d3[['VAR_NAME','MIN_VALUE', 'MAX_VALUE', 'COUNT', 'EVENT', 'EVENT_RATE', 'NONEVENT', 'NON_EVENT_RATE', 'DIST_EVENT','DIST_NON_EVENT','WOE', 'IV']] \n d3 = d3.replace([np.inf, -np.inf], 0)\n d3.IV = d3.IV.sum()\n \n return(d3)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Define a binning function for categorical independent variables","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def char_bin(Y, X):\n \n df1 = pd.DataFrame({\"X\": X, \"Y\": Y})\n justmiss = df1[['X','Y']][df1.X.isnull()]\n notmiss = df1[['X','Y']][df1.X.notnull()] \n df2 = notmiss.groupby('X',as_index=True)\n \n d3 = pd.DataFrame({},index=[])\n d3[\"COUNT\"] = df2.count().Y\n d3[\"MIN_VALUE\"] = df2.sum().Y.index\n d3[\"MAX_VALUE\"] = d3[\"MIN_VALUE\"]\n d3[\"EVENT\"] = df2.sum().Y\n d3[\"NONEVENT\"] = df2.count().Y - df2.sum().Y\n \n if len(justmiss.index) > 0:\n d4 = pd.DataFrame({'MIN_VALUE':np.nan},index=[0])\n d4[\"MAX_VALUE\"] = np.nan\n d4[\"COUNT\"] = justmiss.count().Y\n d4[\"EVENT\"] = justmiss.sum().Y\n d4[\"NONEVENT\"] = justmiss.count().Y - justmiss.sum().Y\n d3 = d3.append(d4,ignore_index=True)\n \n d3[\"EVENT_RATE\"] = d3.EVENT/d3.COUNT\n d3[\"NON_EVENT_RATE\"] = d3.NONEVENT/d3.COUNT\n d3[\"DIST_EVENT\"] = d3.EVENT/d3.sum().EVENT\n d3[\"DIST_NON_EVENT\"] = d3.NONEVENT/d3.sum().NONEVENT\n d3[\"WOE\"] = np.log(d3.DIST_EVENT/d3.DIST_NON_EVENT)\n d3[\"IV\"] = (d3.DIST_EVENT-d3.DIST_NON_EVENT)*np.log(d3.DIST_EVENT/d3.DIST_NON_EVENT)\n d3[\"VAR_NAME\"] = \"VAR\"\n d3 = d3[['VAR_NAME','MIN_VALUE', 'MAX_VALUE', 'COUNT', 'EVENT', 'EVENT_RATE', 'NONEVENT', 'NON_EVENT_RATE', 'DIST_EVENT','DIST_NON_EVENT','WOE', 'IV']] \n d3 = d3.replace([np.inf, -np.inf], 0)\n d3.IV = d3.IV.sum()\n d3 = d3.reset_index(drop=True)\n \n return(d3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def data_vars(df1, target):\n \n stack = traceback.extract_stack()\n filename, lineno, function_name, code = stack[-2]\n vars_name = re.compile(r'\\((.*?)\\).*$').search(code).groups()[0]\n final = (re.findall(r\"[\\w']+\", vars_name))[-1]\n \n x = df1.dtypes.index\n count = -1\n \n for i in x:\n if i.upper() not in (final.upper()):\n if np.issubdtype(df1[i], np.number) and len(Series.unique(df1[i])) > 2:\n conv = mono_bin(target, df1[i])\n conv[\"VAR_NAME\"] = i\n count = count + 1\n else:\n conv = char_bin(target, df1[i])\n conv[\"VAR_NAME\"] = i \n count = count + 1\n \n if count == 0:\n iv_df = conv\n else:\n iv_df = iv_df.append(conv,ignore_index=True)\n \n iv = pd.DataFrame({'IV':iv_df.groupby('VAR_NAME').IV.max()})\n iv = iv.reset_index()\n return(iv_df,iv)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final_iv, IV = data_vars(df,df.target)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Information Value","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"final_iv","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"IV.sort_values('IV',ascending=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the above Information Value (IV) the following features are found to be good predictors have a strong relationship to the Goods/Bads odds ratio.\n \n VAR_NAME\tIV\n\tPAY_3\t0.409001\n\tPAY_4\t0.355175\n\tPAY_5\t0.329335\n\tPAY_6\t0.281748\n LIMIT_BAL\t0.175361\n PAY_AMT1\t0.142889\n PAY_AMT2\t0.128998\n PAY_AMT3\t0.113012\n \n The below features are found to be suspicious in nature and too good to be true .\n VAR_NAME IV\n PAY_0\t0.684208\n\tPAY_2\t0.540881\n \nNow let us look at another example taking the housing prices dataset as shown below","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Using the category encoder library\nce_WOE = ce.WOEEncoder(cols = ['Embarked'])\nce_WOE.fit(X, y) \nce_WOE.transform(X, y) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's load again the titanic dataset\n\ndata = pd.read_csv('../input/titanic/train.csv', usecols=['Cabin', 'Survived'])\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's first fill NA values with an additional label\n\ndata.Cabin.fillna('Missing', inplace=True)\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Cabin has indeed a lot of labels, here for simplicity, I will capture the first letter of the cabin, but the procedure could be done as well without any prior variable manipulation\n\nlen(data.Cabin.unique())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Now we extract the first letter of the cabin\ndata['Cabin'] = data['Cabin'].astype(str).str[0]\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# check the labels\ndata.Cabin.unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#The calculation of the WoE to replace the labels should be done considering the ONLY the training set, and then expanded it to the test set. \n# Let's divide into train and test set\n\nX_train, X_test, y_train, y_test = train_test_split(data[['Cabin', 'Survived']], data.Survived, test_size=0.3,\n random_state=0)\nX_train.shape, X_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Now we calculate the probability of target=1 \nX_train.groupby(['Cabin'])['Survived'].mean()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's make a dataframe with the above calculation\n\nprob_df = X_train.groupby(['Cabin'])['Survived'].mean()\nprob_df = pd.DataFrame(prob_df)\nprob_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# and now the probability of target = 0 and we add it to the dataframe\n\nprob_df = X_train.groupby(['Cabin'])['Survived'].mean()\nprob_df = pd.DataFrame(prob_df)\nprob_df['Died'] = 1-prob_df.Survived\nprob_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Since the log of zero is not defined, let's set this number to something small and non-zero\n\nprob_df.loc[prob_df.Survived == 0, 'Survived'] = 0.00001\nprob_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Finally it is time to calculate the Weight of Evidence (WoE)\n\nprob_df['WoE'] = np.log(prob_df.Survived/prob_df.Died)\nprob_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let us create a dictionary to re-map the variable\n\nprob_df['WoE'].to_dict()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Now we make a dictionary to map the orignal variable to the WoE but we capture the dictionary in a variable\n\nordered_labels = prob_df['WoE'].to_dict()\n# Replace the labels with the above label for WoE\n\nX_train['Cabin_ordered'] = X_train.Cabin.map(ordered_labels)\nX_test['Cabin_ordered'] = X_test.Cabin.map(ordered_labels)\nX_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plot the original variable\nimport matplotlib.pyplot as plt\n%matplotlib inline\nfig = plt.figure()\nfig = X_train.groupby(['Cabin'])['Survived'].mean().plot()\nfig.set_title('Normal relationship between variable and target')\nfig.set_ylabel('Survived')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plot the transformed result: the monotonic variable\n\nfig = plt.figure()\nfig = X_train.groupby(['Cabin_ordered'])['Survived'].mean().plot()\nfig.set_title('Monotonic relationship between variable and target')\nfig.set_ylabel('Survived')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can see in the above plot, there is now a monotonic relationship between the variable Cabin and probability of survival. The higher the Cabin number, the more likely the person was to survive.Monotonic does not mean strictly linear. Monotonic means that it increases constantly, or it decreases constantly.\n## Rare Label encoding
\n**Labels that occur rarely**\n\nCategorical variables are those which values are selected from a group of categories, also called labels. It is not unusual that different labels appear in the dataset with different frequencies. Some categories may appear a lot in the dataset, whereas some other categories appear only in a few number of observations.\n\nConsider for example a dataset with information about loan applicants for the Lending Club data exercise where one of the variables is \"city where the applicant lives\". We can imagine that cities like 'New York' may appear a lot in the data set just because New York has a huge population. On the other hand, smaller towns like 'Leavenworth' will appear only on a few occasions (if at all, population < 2000 people), just because the population there is very small.\n\nCategorical variables in business datasets very often contain a few dominant labels that account for the majority of the observations and a large number of labels that appear only seldomly. \n\n**Are Rare Labels in a categorical variable a problem?**\n\nRare values can add a lot of information at the time of making an assessment or none at all. For example, consider a stockholder meeting where each person can vote in proportion to their number of shares. One of the shareholders owns 50% of the stock, and the other 999 shareholders own the remaining 50%. The outcome of the vote is largely influenced by the shareholder who holds the majority of the stock. The remaining shareholders may have an impact collectively, but they have virtually no impact individually. \n\nThe same occurs in real life datasets. The label that is over-represented in the dataset tends to dominate the outcome, and those that are under-represented may have no impact individually, but could have an impact if considered collectively.\n\nMore specifically,\n\n- Rare values in categorical variables tend to cause over-fitting, particularly in tree based methods.\n\n- A big number of infrequent labels adds noise, with little information, therefore causing over-fitting.\n\n- Rare labels may be present in training set, but not in test set, therefore causing over-fitting to the train set.\n\n- Rare labels may appear in the test set, and not in the train set. Thus, the machine learning model will not know how to evaluate it. \n\n\n**Note** Sometimes rare values, like outliers, are worth having a look at, in particular, when we are looking for rare instances (e.g., when the target is highly unbalanced). For example, if we are building a model to predict fraudulent applications, which are in essence rare, then a rare value in a certain variable, may be indeed important. This rare value could be telling us that the observation is most likely a fraudulent application, and therefore we would choose not to ignore it.\nSo, let's go ahead and see the effect of Rare Values in Categorical variables on Machine Learning models.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.ensemble import AdaBoostRegressor\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.ensemble import GradientBoostingRegressor\n\nfrom sklearn.metrics import mean_squared_error, r2_score\nfrom sklearn.model_selection import train_test_split","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data = pd.read_csv('../input/titanic/train.csv', usecols=['Embarked', 'Survived'])\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's check at the different number of labels within each variable\ncols_to_use = ['Embarked']\n\nfor col in cols_to_use:\n print('Variable: ', col, ' Number of Labels: ', len(data[col].unique()))\n \nprint('Total passengers: ', len(data))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's plot how frequently appears each label within a variable in the dataset\n\ntotal_passengers = len(data)\n\nfor col in cols_to_use:\n # count the number of observations per label and divide by total \n # number of cars\n temp_df = pd.Series(data[col].value_counts() / total_passengers)\n \n # make plot with the above percentages\n fig = temp_df.sort_values(ascending=False).plot.bar()\n fig.set_xlabel(col)\n fig.set_ylabel('Percentage of Passengers')\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that for each variables, 2 labels appear in more than 10% of the cars in the dataset, and one label appear in less than 10% the passengers. **These are infrequent labels or Rare Values and could cause over-fitting.**\n\n### How is the target, \"time to pass testing\", related to these categories?","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# I will work first the the variable Embarked\n\n# Let's calculate again the frequency of the different categories/labels in Embarked\n\ntemp_df = pd.Series(data['Embarked'].value_counts() / total_passengers).reset_index()\ntemp_df.columns = ['Embarked', 'Percentage of Passengers']\ntemp_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Now let's calculate the mean \"time to pass testing\" for each label in Embarked\n\ndata.groupby(['Embarked'])['Survived'].mean().reset_index()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## LeaveOneOut
\nIt is very similar to target but avoids contamination. Nominal, ordinal. For classification tasks.\nLeave One Out encoding essentially calculates the mean of the target variables for all the records containing the same value for the categorical feature variable in question. The encoding algorithm is slightly different between training and test data set. For training data set, the record under consideration is left out, hence the name Leave One Out. The encoding is as follows for certain value of a certain categorical variable.\n\nci = (Σj != i tj / (n – 1 + R)) x (1 + εi) where\nci = encoded value for ith record\ntj = target variable value for jth record\nn = number of records with the same categorical variable value\nR = regularization factor\nεi = zero mean random variable with normal distribution N(0, s)\n\nFor validation data or prediction data set, the definition is slightly different. We don’t need to leave the current record out and we don’t need the randomness factor. It’s simpler definition is as below\n\nci = (Σj tj / (n + R)) where\nci = encoded value for ith record\ntj = target variable value for jth record\nn = number of records with the same categorical variable value\nR = regularization factor\n\nFor validation data set, we don’t leave the current record out. As we will see later, all the statistics about target variable for each value of each categorical variable as calculated for the training data set is saved and then used for validation and prediction data set.\n\nThe factor R acts as a regularizer. When the support is low for a particular value of variable i.e when n is low, then the encoded value is not reliable. Regularization factor R remedies the problem. It also ensures that the denominator is always positive for training data set.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ce_leave = ce.LeaveOneOutEncoder(cols = ['Embarked'])\nce_leave.fit(X, y) \nce_leave.transform(X, y) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## James-Stein
\n\n**James-Stein estimator.**\n\n For feature value `i`, James-Stein estimator returns a weighted average of:\n 1. The mean target value for the observed feature value `i`.\n 2. The mean target value (regardless of the feature value).\n This can be written as::\n JS_i = (1-B)*mean(y_i) + B*mean(y)\n The question is, what should be the weight `B`?\n If we put too much weight on the conditional mean value, we will overfit.\n If we put too much weight on the global mean, we will underfit.\n The canonical solution in machine learning is to perform cross-validation.\n However, Charles Stein came with a closed-form solution to the problem.\n The intuition is: If the estimate of `mean(y_i)` is unreliable (`y_i` has high variance),\n we should put more weight on `mean(y)`. Stein put it into an equation as::\n B = var(y_i) / (var(y_i)+var(y))\n The only remaining issue is that we do not know `var(y)`, let alone `var(y_i)`.\n Hence, we have to estimate the variances. But how can we reliably estimate the\n variances, when we already struggle with the estimation of the mean values?!\n There are multiple solutions:\n 1. If we have the same count of observations for each feature value `i` and all\n `y_i` are close to each other, we can pretend that all `var(y_i)` are identical.\n This is called a pooled model.\n 2. If the observation counts are not equal, it makes sense to replace the variances\n with squared standard errors, which penalize small observation counts::\n SE^2 = var(y)/count(y)\n This is called an independent model.\n James-Stein estimator has, however, one practical limitation - it was defined\n only for normal distributions. If you want to apply it for binary classification,\n which allows only values {0, 1}, it is better to first convert the mean target value\n from the bound interval <0,1> into an unbounded interval by replacing mean(y)\n with log-odds ratio::\n log-odds_ratio_i = log(mean(y_i)/mean(y_not_i))\n This is called binary model. The estimation of parameters of this model is, however,\n tricky and sometimes it fails fatally. In these situations, it is better to use beta\n model, which generally delivers slightly worse accuracy than binary model but does\n not suffer from fatal failures.\n \n Parameters\n ----------\n verbose: int\n integer indicating verbosity of the output. 0 for none.\n cols: list\n a list of columns to encode, if None, all string columns will be encoded.\n drop_invariant: bool\n boolean for whether or not to drop encoded columns with 0 variance.\n return_df: bool\n boolean for whether to return a pandas DataFrame from transform (otherwise it will be a numpy array).\n handle_missing: str\n options are 'return_nan', 'error' and 'value', defaults to 'value', which returns the prior probability.\n handle_unknown: str\n options are 'return_nan', 'error' and 'value', defaults to 'value', which returns the prior probability.\n model: str\n options are 'pooled', 'beta', 'binary' and 'independent', defaults to 'independent'.\n randomized: bool,\n adds normal (Gaussian) distribution noise into training data in order to decrease overfitting (testing data are untouched).\n sigma: float\n standard deviation (spread or \"width\") of the normal distribution.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ce_James = ce.JamesSteinEncoder(cols = ['Embarked'])\nce_James.fit(X, y) \nce_James.transform(X, y) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## M-estimator
\n\n**M-probability estimate of likelihood.**\n This is a simplified version of target encoder, which goes under names like m-probability estimate or\n additive smoothing with known incidence rates. In comparison to target encoder, m-probability estimate\n has only one tunable parameter (`m`), while target encoder has two tunable parameters (`min_samples_leaf`\n and `smoothing`).\n \n Parameters\n ----------\n verbose: int\n integer indicating verbosity of the output. 0 for none.\n cols: list\n a list of columns to encode, if None, all string columns will be encoded.\n drop_invariant: bool\n boolean for whether or not to drop encoded columns with 0 variance.\n return_df: bool\n boolean for whether to return a pandas DataFrame from transform (otherwise it will be a numpy array).\n handle_missing: str\n options are 'return_nan', 'error' and 'value', defaults to 'value', which returns the prior probability.\n handle_unknown: str\n options are 'return_nan', 'error' and 'value', defaults to 'value', which returns the prior probability.\n randomized: bool,\n adds normal (Gaussian) distribution noise into training data in order to decrease overfitting (testing data are untouched).\n sigma: float\n standard deviation (spread or \"width\") of the normal distribution.\n m: float\n this is the \"m\" in the m-probability estimate. Higher value of m results into stronger shrinking.\n M is non-negative.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ce_M_Estimator = ce.MEstimateEncoder(cols = ['Embarked'])\nce_M_Estimator.fit(X, y) \nce_M_Estimator.transform(X, y) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Gaussian Transformation
\n\nSome machine learning models like linear and logistic regression assume that the variables are normally distributed. Others benefit from \"Gaussian-like\" distributions, as in such distributions the observations of X available to predict Y vary across a greater range of values. Thus, Gaussian distributed variables may boost the machine learning algorithm performance.\n\nIf a variable is not normally distributed, sometimes it is possible to find a mathematical transformation so that the transformed variable is Gaussian.\n\nHow can we transform variables so that they follow a Gaussian distribution?\n\nThere are a few straightforward methods to transform variables so that they follow a Gaussian distribution. None of them is better than the other, they rather depend on the original distribution of the variables. They are:\n\n* Logarithmic Transformation\n* Reciprocal Transformation\n* Square Root Transformation\n* Exponential Transformation \n* Boxcox Transformation\n\nLet us take the titanic dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport pylab \nimport scipy.stats as stats\n# load the numerical variables of the Titanic Dataset\n\ntrain_data = pd.read_csv('../input/titanic/train.csv', usecols = ['Age', 'Fare', 'Survived'])\ntrain_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# first fill the missing data of the variable age, with a random sample of the variable\n\ndef impute_na(data, variable):\n # function to fill na with a random sample\n df = data.copy()\n \n # random sampling\n df[variable+'_random'] = df[variable]\n \n # extract the random sample to fill the na\n random_sample = df[variable].dropna().sample(df[variable].isnull().sum(), random_state=0)\n \n # pandas needs to have the same index in order to merge datasets\n random_sample.index = df[df[variable].isnull()].index\n df.loc[df[variable].isnull(), variable+'_random'] = random_sample\n \n return df[variable+'_random']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# fill nul values for Age\ntrain_data['Age'] = impute_na(train_data, 'Age')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To visualise the distribution of the variable, Age in this case, we plot a histogram to visualise a bell-shape, and the Q-Qplot. Remember that if the variable is normally distributed, we should see a 45 degree straight line of the values over the theoretical quantiles. See the lecture \"Variable Distribution\" on section 4 for a description of Q-Q plots.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plot the histograms to have a quick look at the distributions and plot Q-Q plots to visualise if the variable is normally distributed\n\ndef diagnostic_plots(df, variable):\n # function to plot a histogram and a Q-Q plot\n # side by side, for a certain variable\n \n plt.figure(figsize=(15,6))\n plt.subplot(1, 2, 1)\n df[variable].hist()\n\n plt.subplot(1, 2, 2)\n stats.probplot(df[variable], dist=\"norm\", plot=pylab)\n\n plt.show()\n \ndiagnostic_plots(train_data, 'Age')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\t\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"\nThe variable Age is almost normally distributed, except for some observations on the lower value tail of the distribution. Note the slight skew to the left in the histogram, and the deviation from the straight line towards the lower values ont he Q-Q- plot. In the following cells, I will apply the above mentioned transformations and compare the distributions of the transformed Age.\n\n## 1. Logarithmic Transformation
\nThe log transformation is, arguably, the most popular among the different types of transformations used to transform skewed data to approximately conform to normality. If the original data follows a log-normal distribution or approximately so, then the log-transformed data follows a normal or near normal distribution.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data['Age_log'] = np.log(train_data.Age)\ndiagnostic_plots(train_data, 'Age_log')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2. Reciprocal transformation
\t","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data['Age_reciprocal'] = 1 / train_data.Age\ndiagnostic_plots(train_data, 'Age_reciprocal')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The reciprocal transformation was also not useful to transform Age into a variable normally distributed.\n\n## 3. Square root transformation
","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data['Age_sqr'] =train_data.Age**(1/2)\ndiagnostic_plots(train_data, 'Age_sqr')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The square root transformation is a bit more succesful that the previous2 transformations, however, the variable is still not Gaussian, and this does not represent an improvement towards normality respect the original distribution of Age.\n## 4. Exponential transformation
","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data['Age_exp'] = train_data.Age**(1/1.2) \ndiagnostic_plots(train_data, 'Age_exp')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The exponential transformation is the best of all the transformations above, at the time of generating a variable that is normally distributed. Comparing the histogram and Q-Q plot of the exponentially transformed Age with the original distribution, I would say that by visual inspection the transformed variable follows more closely a Gaussian distribution. \nShould I transform the variable?\nIt depends on what we are trying to achieve. If this was a situation in a business setting, I would use the original variable without transformation to train the model, as this would represent a simpler situation at the time of asking developers to implement the model in real life, and also it will be easier to interpret. If on the other hand this was an exercise to win a data science competition, I would opt to use the variable that gives me the highest performance.\n## 5. Yeo-Johnson Transformation
\t\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data['Age_yeojohnson'], param = stats.yeojohnson(train_data.Age) \nprint('Optimal λ: ', param)\ndiagnostic_plots(train_data, 'Age_yeojohnson')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6. Box-Cox Transformation
\nThe Box-Cox transformation is defined as: \n\nT(Y)=(Y exp(λ)−1)/λ\n\nwhere Y is the response variable and λ is the transformation parameter. \nλ varies from -5 to 5. In the transformation, all values of λ are considered and the optimal value for a given variable is selected.\n\nBriefly, for each λ (the transformation tests several λs), the correlation coefficient of the Probability Plot (Q-Q plot below, correlation between ordered values and theoretical quantiles) is calculated. The value of λ corresponding to the maximum correlation on the plot is then the optimal choice for λ.In python, we can evaluate and obtain the best λ with the stats.boxcox function from the package scipy.\n\nLet's have a look.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data['Age_boxcox'], param = stats.boxcox(train_data.Age) \nprint('Optimal λ: ', param)\ndiagnostic_plots(train_data, 'Age_boxcox')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n# 5. Discretisation\n\n**Discretisation** refers to sorting the values of the variable into bins or intervals, also called buckets. There are multiple ways to discretise variables:\n\n\n\n**Discretisation approaches:**\n\nThere are several approaches to transform continuous variables into discrete ones. This process is also known as binning, with each bin being each interval.\n**\nDiscretisation methods fall into 2 categories: supervised and unsupervised. \n**\n**Unsupervised methods** do not use any information, other than the variable distribution, to create the contiguous bins in which the values will be placed. \n\n**Supervised methods** typically use target information in order to create the bins or intervals.\n\n**Unsupervised discretisation methods**\n\n* Equal width binning\n* Equal frequency binning\n\n**Supervised discretisation methods**\n\n* Discretisation using decision trees\n\n\n\nData binning, which is also known as bucketing or discretization, is a technique used in data processing and statistics. Binning can be used for example, if there are more possible data points than observed data points. \n\nAn example is to bin the body heights of people into intervals or categories. Let us assume, we take the heights of 30 people. The length values can be between - roughly guessing - 1.30 metres to 2.50 metres. Theoretically, there are 120 different cm values possible, but we can have at most 30 different values from our sample group. One way to group them could be to put the measured values into bins ranging from 1.30 - 1.50 metres, 1.50 - 1.70 metres, 1.70 - 1.90 metres and so on. This means that the original data values, will be assigned to a bin into wich they fit according to their size. The original values will be replaced by values representing the corresponding intervals. \n\nBinning is a form of quantization.Bins do not necessarily have to be numerical, they can be categorical values of any kind, like \"dogs\", \"cats\", \"hamsters\", and so on.\n\nBinning is also used in image processing, binning. It can be used to reduce the amount of data, by combining neighboring pixel into single pixels. kxk binning reduces areas of k x k pixels into single pixel.\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 1. Equal Width Discretisation
\t\nEqual width binning divides the scope of possible values into N bins of the same width.The width is determined by the range of values in the variable and the number of bins we wish to use to divide the variable.\n\nwidth = (max value - min value) / N\n\nFor example if the values of the variable vary between 0 and 100, we create 5 bins like this: width = (100-0) / 5 = 20. The bins thus are 0-20, 20-40, 40-60, 80-100. The first and final bins (0-20 and 80-100) can be expanded to accommodate outliers (that is, values under 0 or greater than 100 would be placed in those bins as well).\nThere is no rule of thumb to define N. Typically, we would not want more than 10.\n\nI will demonstrate how to perform equal width binning using the Titanic dataset.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n%matplotlib inline\nfrom sklearn.model_selection import train_test_split\nequal_width_data = pd.read_csv('../input/titanic/train.csv', usecols = ['Age', 'Fare', 'Survived'])\nequal_width_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's separate into train and test set\n\nX_train, X_test, y_train, y_test = train_test_split(equal_width_data[['Age', 'Fare', 'Survived']], equal_width_data.Survived, test_size=0.3,\n random_state=0)\nX_train.shape, X_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# replace NA in both train and test sets\n\nX_train['Age'] = impute_na(equal_width_data, 'Age')\nX_test['Age'] = impute_na(equal_width_data, 'Age')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let us remind ourselves of the distribution of Age\nequal_width_data.Age.hist()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"How does survival correlate with Age?\n\nWe can determine the relation between survival and Age by plotting the mean survival per Age. In this case, I will calculate the Survival rate per each year of Age. See below.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure()\nfig = equal_width_data.groupby(['Age'])['Survived'].mean().plot()\nfig.set_title('Normal relationship between Age and Survived')\nfig.set_ylabel('Survived')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\nBy looking at the plot, we get an intuition that younger people (children) were more likely to survive (see higher survival rates at ages below 10 and 10-15). However, it looks like children of 10 years of age, had a very low survival chance. This does not make a lot of sense and most likely indicates that our 10 year old sample is not big enough and then the survival rate is underestimated. Let's see below.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure()\nfig = equal_width_data.groupby(['Age'])['Survived'].count().plot()\nfig.set_title('Number of people per year age bin')\nfig.set_ylabel('Survived')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we suspected, there were very few children at the age of 10 on the Titanic. If none of them or only half of the 10 year old children survived, we would be led to think that being 10 years old decreased your chances of survival, when in reality, there were only 2 children of the age that were not lucky enough to survive, therefore leading to underestimation of survival rate.\n\nBy grouping Age into bins, we can get a better view of the survival rate depending on the Age of the passenger. Let's see below.\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let us capture the range of the variable age to begin with\n\nAge_range = X_train.Age.max() - X_train.Age.min()\nAge_range","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Now let us capture the lower and upper boundaries\n\nmin_value = int(np.floor(X_train.Age.min()))\nmax_value = int(np.ceil(X_train.Age.max()))\n\n# let's round the bin width\ninter_value = int(np.round(Age_range/10))\n\nmin_value, max_value, inter_value","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let us capture the interval limits, so we can pass them to the pandas cut function to generate the bins\n\nintervals = [i for i in range(min_value, max_value+inter_value, inter_value)]\nintervals","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's make labels to label the different bins\nlabels = ['Bin_'+str(i) for i in range(1,len(intervals))]\nlabels","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create Binned age groups\n\n# create one column with labels\nX_train['Age_disc_labels'] = pd.cut(x = X_train.Age, bins=intervals, labels=labels, include_lowest=True)\n\n# and one with bin boundaries\nX_train['Age_disc'] = pd.cut(x = X_train.Age, bins=intervals, include_lowest=True)\n\nX_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see in the above output how by discretising using equal width, we placed each Age observation within one interval /bin. For example, age 51 was placed in the 48-56 interval, whereas age 14.5 was placed into the 8-16 interval.\nBecause we discretised the variable using equal width intervals instead of equal frequency, there won't necessarily be the same amount of passengers in each of the intervals. See below.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.groupby('Age_disc')['Age'].count()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.groupby('Age_disc')['Age'].count().plot.bar()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The majority of people on the Titanic were between 16-40 years of age.\n\nNow, we can discretise Age in the test set, using the same interval boundaries that we calculated for the train set. See below.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"X_test['Age_disc_labels'] = pd.cut(x = X_test.Age, bins=intervals, labels=labels, include_lowest=True)\nX_test['Age_disc'] = pd.cut(x = X_test.Age, bins=intervals, include_lowest=True)\n\nX_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# If the distributions in train and test set are similar, we should expect similar distribution of observations in the different intervals in the train and test set.\n\nt1 = X_train.groupby(['Age_disc'])['Survived'].count() / np.float(len(X_train))\nt2 = X_test.groupby(['Age_disc'])['Survived'].count() / np.float(len(X_test))\ntemp = pd.concat([t1,t2], axis=1)\ntemp.columns = ['train', 'test']\ntemp.plot.bar()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\nIndeed, the proportion of passengers within each bin is roughly the same.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Now let's observe the relationship between age and surival again, using the discrete Age transformed variable\n\nfig = plt.figure()\nfig = X_train.groupby(['Age_disc'])['Survived'].mean().plot(figsize=(12,6))\nfig.set_title('Normal relationship between variable and target')\nfig.set_ylabel('Survived')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure()\nfig = X_train.groupby(['Age_disc'])['Survived'].count().plot(figsize=(12,6))\nfig.set_title('Number of Passengers within each Age bin')\nfig.set_ylabel('No of Passengers')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In this case, by dividing Age into bins, we removed some of the noise plotted in the previous graph (a few cells ago) using the untransformed Age. Using the discrete variable Age, we observe as expected that children (<16) had the highest survival chance.\n\nWhat happens to older people is less clear from the discrete variable. It looks like people between 32-40 and 48-56 are more likely to survive, than people from 40-48. This might or might not be true and more research would be needed to determine if this was the case, what the reason behind it was. In addition, it looks like being extremely old, was very favourable for survival (look at the 72-80 bucket). However, most likely, these are consequences of an arbitrary binning strategy. There are very few people in the bucket 72-80, and thus, the fact that 1 of them survived inflates (overestimates) the survival rate for that bucket. And the same is true for the remaining of the bins, the lowest the amount of observations within buckets, the highest the risk of over or underestimating the target (survival in this case). So we begin to see some of the consequences of this binning strategy.\n\n## 2. Equal Frequency Discretisation
\n\nEqual frequency binning divides the scope of possible values of the variable into N bins, where each bin carries the same amount of observations. This is particularly useful for skewed variables as it spreads the observations over the different bins equally. Typically, we find the interval boundaries by determining the quantiles.\n\nEqual frequency discretisation using quantiles consists of dividing the continuous variable into N quantiles, N to be defined by the user. There is no rule of thumb to define N. However, if we think of the discrete variable as a categorical variable, where each bin is a category, we would like to keep N (the number of categories) low (typically no more than 10).\n\nEqual frequency binning is straightforward to implement and by spreading the values of the observations more evenly it may help boost the algorithm's performance. On the other hand, this arbitrary binning may also disrupt the relationship with the target on occasions. Therefore, whenever possible, it will bring value to examine whether this type of binning is the right strategy, and it will depend on the variable and the algorithm that we want to use to make the predictions.\n\nI will demonstrate how to perform equal frequency binning using the Titanic dataset.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n%matplotlib inline\nfrom sklearn.model_selection import train_test_split\nequal_freq_data= pd.read_csv('../input/titanic/train.csv', usecols = ['Age', 'Fare', 'Survived'])\nequal_freq_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's separate into train and test set\n\nX_train, X_test, y_train, y_test = train_test_split(equal_freq_data[['Age', 'Fare', 'Survived']],equal_freq_data.Survived, test_size=0.3, random_state=0)\nX_train.shape, X_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# replace NA in both train and test sets\n\nX_train['Age'] = impute_na(equal_freq_data, 'Age')\nX_test['Age'] = impute_na(equal_freq_data, 'Age')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's remind ourselves of the original distribution\n\nequal_freq_data.Age.hist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# We will divide Age in 5 bins. I use the qcut (quantile cut) function from pandas and I indicate that I want 4 cutting points, thus 5 bins. retbins= True indicates that I want to capture the limits of each interval (so I can then use them to cut the test set)\n\nAge_disccretised, intervals = pd.qcut(equal_freq_data.Age, 4, labels=None, retbins=True, precision=3, duplicates='raise')\npd.concat([Age_disccretised, equal_freq_data.Age], axis=1).head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see in the above output how by discretising using quantiles, we placed each Age observation within one interval. For example, age 22 was placed in the 20-28 interval, whereas age 38 was placed into the 28-28 interval. We can visualise the interval cut points below.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"intervals","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And because we generated the bins using the quantile cut method, we should have roughly the same amount of observations per bin. See below.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Calculate number of passengers per bin\ntemp = pd.concat([Age_disccretised, equal_freq_data.Age], axis=1)\ntemp.columns = ['Age_discretised', 'Age']\ntemp.groupby('Age_discretised')['Age'].count()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# We can also add labels instead of having the interval boundaries, to the bins, as follows:\n\nAge_disccretised, intervals = pd.qcut(equal_freq_data.Age, 4, labels=['Q1', 'Q2', 'Q3', 'Q4'], retbins=True, precision=3, duplicates='raise')\nAge_disccretised.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# We will divide into 10 quantiles for the rest of the exercise. I will leave the quantile boundary and generate labels as well for comparison\n\n# create 10 labels, one for each quantile\n\nlabels = ['Q'+str(i+1) for i in range(0,10)]\nprint(labels)\n\n# bins with labels\nX_train['Age_disc_label'], bins = pd.qcut(x=X_train.Age, q=10, labels=labels, retbins=True, precision=3, duplicates='raise')\n\n# bins with boundaries\nX_train['Age_disc'], bins = pd.qcut(x=X_train.Age, q=10, retbins=True, precision=3, duplicates='raise')\n\n\nX_train.head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\nAge 51 was placed into the 49-80 bucket, which is also the last bucket (Q10) in our distribution, as it contains the majority of the elder people. The age of 8 was placed into the 0-13 bucket, which is the first interval (Q1) of the discretised variable. And below the interval limits.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"\nTo divide the test set, we will use the cut method of pandas, and we will pass the interval edges above. \nAs with all feature engineering methods, the limits of the intervals should be set on the training set, and then propagated to the test set. We should not calculate the quantiles in the test set. We should use the limits estimated using the train set data. If the distributions are similar, we should expect to have roughly the same amount of observations in each age bucket in the test set as well.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"X_test['Age_disc_label'] = pd.cut(x = X_test.Age, bins=bins, labels=labels)\nX_test['Age_disc'] = pd.cut(x = X_test.Age, bins=bins)\n\nX_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's check that we have equal frequency (equal number of observations per bin)\nX_test.groupby('Age_disc')['Age'].count()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Discretisation by quantiles guarantees same number of observations in each partition/bin/interval**\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"t1 = X_train.groupby(['Age_disc'])['Survived'].count() / np.float(len(X_train))\nt1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"t2 = X_test.groupby(['Age_disc'])['Survived'].count() / np.float(len(X_test))\nt2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"temp = pd.concat([t1,t2], axis=1)\ntemp.columns = ['train', 'test']\ntemp","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"temp.plot.bar()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's observe the relationship between age and survival again, using the discrete Age transformed variable\n\nfig = plt.figure()\nfig = X_train.groupby(['Age_disc'])['Survived'].mean().plot(figsize=(12,6))\nfig.set_title('Normal relationship between variable and target')\nfig.set_ylabel('Survived')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Using discretisation we capture more clearly the survival rate according to the age of the passenger. We can see that children (<18) were the most likely to survive, and people in the early 30s as well, potentially mother with children, whereas people between 20-30 and older than 35 were the less likely to survive the tragedy.\n\nNow that we have a discrete variable, we can go ahead and preprocess it as categorical variable, to squeeze a bit more performance out of the machine learning algorithm. Why not ordering the bins according to survival rate?\n\n**Combine discretisation with label ordering according to target**\n\nYou can revise the lectures on engineering categorical variables to re-cap on how to preprocess labels.\n\nFor this demonstration I will assign ordinal numbers to the different bins, according to the survival rate per bin.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# order the intervals according to survival rate\nordered_labels = X_train.groupby(['Age_disc_label'])['Survived'].mean().sort_values().index\n\n# number the intervals according to survival rate\nordinal_label = {k:i for i, k in enumerate(ordered_labels, 0)} \n\n# remap the intervals to the encoded variable\nX_train['Age_disc_ordered'] = X_train.Age_disc_label.map(ordinal_label)\nX_test['Age_disc_ordered'] = X_test.Age_disc_label.map(ordinal_label)\n\nX_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plot the monotonic relationship\nfig = plt.figure()\nfig = X_train.groupby(['Age_disc_ordered'])['Survived'].mean().plot()\nfig.set_title('Monotonic relationship between discretised Age and target')\nfig.set_ylabel('Survived')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n## 3. Discretisation using decision trees\n\nDiscretisation with Decision Trees consists of using a decision tree to identify the optimal splitting points that would determine the bins or contiguous intervals: \nFirst, it trains a decision tree of limited depth (2, 3 or 4) using the variable we want to discretise to predict the target.\nThe original variable values are then replaced by the probability returned by the tree. The probability is the same for all the observations within a single bin, thus replacing by the probability is equivalent to grouping the observations within the cut-off decided by the decision tree.\n\n**Advantages**\n\n* The probabilistic predictions returned decision tree are monotonically related to the target.\n* The new bins show decreased entropy, this is the observations within each bucket / bin are more similar to themselves than to those of other buckets / bins.\n* The tree finds the bins automatically\n\n**Disadvantages**\n\n* It may cause over-fitting\n* More importantly, some tuning of tree parameters need to be done to obtain the optimal splits (e.g., depth, minimum number of samples in one partition, maximum number of partitions, and a minimum information gain). This it can be time consuming.\n\nBelow, I will demonstrate how to perform discretisation with decision trees using the Titanic dataset.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n%matplotlib inline\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.tree import DecisionTreeClassifier, export_graphviz\nfrom sklearn.model_selection import cross_val_score\n# load the numerical variables of the Titanic Dataset\ndata_decision_tree = pd.read_csv('../input/titanic/train.csv', usecols = ['Age', 'Fare', 'Survived'])\ndata_decision_tree.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's separate into train and test set\n\nX_train, X_test, y_train, y_test = train_test_split(data_decision_tree[['Age', 'Fare', 'Survived']],\n data_decision_tree.Survived, test_size=0.3,\n random_state=0)\nX_train.shape, X_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train['Age'] = impute_na(data_decision_tree, 'Age')\nX_test['Age'] = impute_na(data_decision_tree, 'Age')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let us now build Classification tree using Age to predict Survived\n\ntree_model = DecisionTreeClassifier(max_depth=2)\ntree_model.fit(X_train.Age.to_frame(), X_train.Survived)\nX_train['Age_tree'] = tree_model.predict_proba(X_train.Age.to_frame())[:,1]\nX_train.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# monotonic relationship with target\n\nfig = plt.figure()\nfig = X_train.groupby(['Age_tree'])['Survived'].mean().plot()\nfig.set_title('Monotonic relationship between discretised Age and target')\nfig.set_ylabel('Survived')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Number of passengers per probabilistic bucket / bin\n\nX_train.groupby(['Age_tree'])['Survived'].count().plot.bar()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Median age within each bucket originated by the tree\n\nX_train.groupby(['Age_tree'])['Age'].median().plot.bar()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Now let us see the Age limits buckets generated by the tree by capturing the minimum and maximum age per each probability bucket, we get an idea of the bucket cut-offs\n\npd.concat( [X_train.groupby(['Age_tree'])['Age'].min(),\n X_train.groupby(['Age_tree'])['Age'].max()], axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Conclusion**: Thus, the decision tree generated the buckets: 0-11, 12-15, 16-63 and 46-80, with probabilities of survival of .51, .81, .37 and .1 respectively.\n# 6. Outlier Engineering
\n\n#### What is an Outlier?\n\nAn outlier is a data point in a data set that is distant from all other observations. A data point that lies outside the overall distribution of the dataset.\n\n#### Should outliers be removed?\nDepending on the context, outliers either deserve special attention or should be completely ignored. Take the example of revenue forecasting: if unusual spikes of revenue are observed, it's probably a good idea to pay extra attention to them and figure out what caused the spike. In the same way, an unusual transaction on a credit card is usually a sign of fraudulent activity, which is what the credit card issuer wants to prevent. So in instances like these, it is useful to look for and investigate further outlier values.\n\nIf outliers are however, due to mechanical error, measurement error or anything else that can't be generalised, it’s a good idea to filter out these outliers before feeding the data to the modeling algorithm. \n\n#### What defines an outlier?\n* Data point that falls outside of 1.5 times of an interquartile range above the 3rd quartile and below the 1st quartile\n* Data point that falls outside of 3 standard deviations. we can use a z score and if the z score falls outside of 2 standard deviation\n\n#### Why do outliers exist in a dataset?\n* Variability in the data\n* An experimental measurement error\n\n#### Impact of an Outlier\n* It causes various problems during our statistical analysis\n* It may cause a significant impact on the mean and the standard deviation\n\n#### How to find an outlier?\nOutlier analysis and anomaly detection are a huge field of research devoted to optimise methods and create new algorithms to reliably identify outliers. There are a huge number of ways optimised to detect outliers in different situations. These are mostly targeted to identify outliers when those are the observations that we indeed want to focus on, for example for fraudulent credit card activity.\n\nUsing the following methods we can detect an outlier\n* IQR interquantile range\n* z score\n* Scatter plots\n* Box plot\n\n\nNow let us see how we can detect an outlier using the above methods and remove them where neccessary. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
\n## 1. Outlier Detection & Removal","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
\n## Outlier Detection & Removal using IQR(Inter Quartile Range)\n\n\n\nThe **interquartile range (IQR)** is a measure of variability, based on dividing a data set into quartiles.\n\nQuartiles divide a rank-ordered data set into four equal parts. The values that divide each part are called the first, second, and third quartiles; and they are denoted by Q1, Q2, and Q3, respectively.\n\nQ1 is the \"middle\" value in the first half of the rank-ordered data set.\n\nQ2 is the median value in the set.\n\nQ3 is the \"middle\" value in the second half of the rank-ordered data set.\n\nThe interquartile range is equal to Q3 minus Q1.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\n# To display the total number columns present in the dataset\npd.set_option('display.max_columns', None)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let us load the titanic dataset\n\ndata = pd.read_csv('../input/titanic/train.csv')\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Now let us visualise median and quantiles for Age feature.Below illustration shows 25%, 50% and 75% are nothing but 25th quantile, median and 75th quantile respectively","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data.Age.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Let's calculate the boundaries outside which sit the outliers assuming Age follows a Gaussian distribution","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"Upper_boundary_limit = data.Age.mean() + 3* data.Age.std()\nLower_boundary_limit = data.Age.mean() - 3* data.Age.std()\n\nUpper_boundary_limit, Lower_boundary_limit","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### The upper boundary for Age is 73-74 years. The lower boundary is meaningless as there can't be negative age. This value could be generated due to the lack of normality of the data.\n\n#### Now let's use the IQR(Inter Quantile Range) to calculate the boundaries\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"IQR = data.Age.quantile(0.75) - data.Age.quantile(0.25)\n\nLower_quantile_lower = data.Age.quantile(0.25) - (IQR * 1.5)\nUpper_quantile_lower = data.Age.quantile(0.75) + (IQR * 1.5)\n\nUpper_quantile_lower, Lower_quantile_lower, IQR","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### The boundary using 1.5 times the interquantile range coincides roughly with the boundary determined using the Gaussian distribution (64 vs 71 years).\n\nNow lets look for extreme outliers ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"IQR = data.Age.quantile(0.75) - data.Age.quantile(0.25)\n\nLower_quantile = data.Age.quantile(0.25) - (IQR * 3)\nUpper_quantile = data.Age.quantile(0.75) + (IQR * 3)\n\nUpper_quantile, Lower_quantile, IQR","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### The boundary value using the 3 times the interquantile is a bit high according to normal human life expectancy, particularly in the days of the Titanic.\n\n#### Now let us find out whether there are any outliers according to the above boundaries:\n\n#### Before that let us first remove the passengers with missing data for Age feature.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data = data.dropna(subset=['Age'])\n\ntotal_passengers = np.float(data.shape[0])\n\nprint('Passengers older than 73 years old (Gaussian approach): {}'.format(data[data.Age > 73].shape[0] / total_passengers))\nprint('Passengers older than 65 years (IQR): {}'.format(data[data.Age > 65].shape[0] / total_passengers))\nprint('Passengers older than 91 years (IQR, extreme): {}'.format(data[data.Age >= 91].shape[0] / total_passengers))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Approximately ~1-2 percent of the passengers were extremely old age.\n\n#### Now finally here are the outliers ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data[(data.AgeUpper_quantile_lower)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### So clearly it is indicative that the above are the outliers . Based on the above observation it is evident that the majority of the outliers did not survive.\n\n#### Now let us remove the outliers ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data_with_no_outlier = data[(data.Age>Lower_quantile_lower)&(data.Age
\n## Outlier Detection & Removal using Percentile\n\nOutlier detection using percentile approach is requires loading and sorting the entire data set. This can be CPU and memory intensive, depending on the number of measurements. Averages don’t require keeping the entire data set in memory during the calculation, and their values don't need to be sorted. This makes percentiles relatively difficult to calculate over a large time span for systems with many measurements.\n\nHowever for learning purpose I am including this method as well.Lets explore this in detail below.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.read_csv(\"../input/house-prices-advanced-regression-techniques/train.csv\")\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In this method there is no fixed approach for setting the upper and lower threshold and based on purely the problem statement and the dataset we have.In this example above we will be exploring the sale price of the house and define the upper and lower threshold values as 95% and 5% respectively.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"upper_threshold = df['SalePrice'].quantile(0.95)\nupper_threshold","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lower_threshold = df['SalePrice'].quantile(0.05)\nlower_threshold","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let us determine the outliers ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df[(df.SalePriceupper_threshold)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So we have got around 144 outliers .Now its time to eliminate these outliers if we think the defined thresholds makes sense for us.\n\nLet us now remove the outliers","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data_with_no_outlier_percentile_approach = df[(df.SalePrice>lower_threshold)&(df.SalePrice
\n## Outlier Detection & Removal using Z score\n\n\n### Z score indicates how many standard deviation away a data point is.\n\nLet us relook at the titanic problem again and calculate **Mean** and **Standard Deviation** for the Age feature","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data_zscore = pd.read_csv('../input/titanic/train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_zscore.Age.mean(),data_zscore.Age.std()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In this case mean is 29.69 and standard deviation is 14.52. \nNow let us calculate the Z score based on the above formula","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data_zscore['zscore'] = ( data_zscore.Age - data_zscore.Age.mean() ) / data_zscore.Age.std()\ndata_zscore.head(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Above for first record with age 22, z score is -0.53. This means 22 is -0.53 standard deviation away from mean","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"(22-29.69)/14.509433962264152","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n#### Get data points that has z score higher than 3 or lower than -3. Another way of saying same thing is get data points that are more than 3 standard deviation away","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data_zscore[(data_zscore.zscore>3) | (data_zscore.zscore<-3)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### So above are the list of final outliers with data points more than 3 standard deviations away.\n\n#### Now let us remove these outliers ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_no_outliers_zscore = data_zscore[(data_zscore.zscore>-3) & (data_zscore.zscore<3)]\ndf_no_outliers_zscore.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### So finally we were able to remove two outliers using this Z score approach\n
\n## Outlier Detection & Removal using Box Plot\n\n\nA **boxplot** is a standardized way of displaying the distribution of data based on a five number summary (“minimum”, first quartile (Q1), median, third quartile (Q3), and “maximum”). It can tell you about your outliers and what their values are. \n\nMedian (Q2/50th Percentile): the middle value of the dataset.\n\nFirst quartile (Q1/25th Percentile): the middle number between the smallest number (not the “minimum”) and the median of the dataset.\n\nThird quartile (Q3/75th Percentile): the middle value between the median and the highest value (not the “maximum”) of the dataset.\n\nInter Quartile range (IQR): 25th to the 75th percentile.\nWhiskers (shown in above picture)\n\nOutliers (shown as circles in above picture)\n\n“maximum”: Q3 + 1.5*IQR\n\n“minimum”: Q1 -1.5*IQR\n\n\nLet us take the example of titanic dataset to show a simple box plot","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data_box_plot = pd.read_csv('../input/titanic/train.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Let us first look at the distribution of age feature using histogram as shown below","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = data_box_plot.Age.hist(bins=50)\nfig.set_title('Age Distribution')\nfig.set_xlabel('Age')\nfig.set_ylabel('Number of Passengers')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let us look at the boxplot of age feature","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = data_box_plot.boxplot(column='Age')\nfig.set_title('')\nfig.set_xlabel('Survived')\nfig.set_ylabel('Age')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n## Outlier Detection & Removal using Scatter Plot\n\nA **scatter plot** (also called a scatterplot, scatter graph, scatter chart, scattergram, or scatter diagram) is a type of plot or mathematical diagram using Cartesian coordinates to display values for typically two variables for a set of data.\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport seaborn as sns\ntrain = pd.read_csv(\"../input/house-prices-advanced-regression-techniques/train.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(nrows=2, ncols=2, figsize=(14,10))\nOverallQual_scatter_plot = pd.concat([train['SalePrice'],train['GarageArea']],axis = 1)\nsns.regplot(x='GarageArea',y = 'SalePrice',data = OverallQual_scatter_plot,scatter= True, fit_reg=True, ax=ax1)\nTotalBsmtSF_scatter_plot = pd.concat([train['SalePrice'],train['TotalBsmtSF']],axis = 1)\nsns.regplot(x='TotalBsmtSF',y = 'SalePrice',data = TotalBsmtSF_scatter_plot,scatter= True, fit_reg=True, ax=ax2)\nGrLivArea_scatter_plot = pd.concat([train['SalePrice'],train['GrLivArea']],axis = 1)\nsns.regplot(x='GrLivArea',y = 'SalePrice',data = GrLivArea_scatter_plot,scatter= True, fit_reg=True, ax=ax3)\nGarageArea_scatter_plot = pd.concat([train['SalePrice'],train['BsmtFinSF1']],axis = 1)\nsns.regplot(x='BsmtFinSF1',y = 'SalePrice',data = GarageArea_scatter_plot,scatter= True, fit_reg=True, ax=ax4)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n## 2. Treating outliers as missing values\nThere are basically three methods for treating outliers in a data set. \n* Remove outliers as a means of trimming the data set. \n* Replacing the values of outliers or reducing the influence of outliers through outlier weight adjustments. \n* Estimate the values of outliers using robust techniques.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
\n## 3. Top / Bottom / Zero coding\n**Top or bottom coding** are also known as Winsorisation or outlier capping. The procedure involves capping the maximum and minimum values at a predefined value. This predefined value can be arbitrary, or it can be derived from the variable distribution.\n\nHow can we derive the maximum and minimum values? \n\nIf the variable is normally distributed we can cap the maximum and minimum values at the mean plus or minus 3 times the standard deviation. If the variable is skewed, we can use the inter-quantile range proximity rule or cap at the top and bottom percentiles.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
\n## 4. Discretisation\n\nDiscretisation handles outliers automatically, as outliers are sorted into the terminal bins, together with the other higher or lower value observations. The best approaches are equal frequency and tree based discretisation.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 7. Feature Scaling
\n\n**Feature scaling** in machine learning is one of the most critical steps during the pre-processing of data before creating a machine learning model. Scaling can make a difference between a weak machine learning model and a better one.\n\nThe most common techniques of feature scaling are Normalization and Standardization.\n\n**Normalisation**\n\nOne method utilised to bring all the variables to a more homogeneous scale is normalisation. Normalisation is synonym of centering the distribution. This means subtracting the mean of the variable to each observation. This procedure will \"center\" the new distribution at zero (the new mean of the variable will now be zero).\n\n**Standarisation**\n\nStandarisation is also used to bring all the variables to a similar scale. Standarisation means centering the variable at zero, and standarising the variance at 1. The procedure involves subtracting the mean of each observation and then dividing by the standard deviation:\n\nz = (x - x_mean) / std\n\n\n**Why do we need scaling?**\n\nMachine learning algorithm just sees number — if there is a vast difference in the range say few ranging in thousands and few ranging in the tens, and it makes the underlying assumption that higher ranging numbers have superiority of some sort. So these more significant number starts playing a more decisive role while training the model.\nThe machine learning algorithm works on numbers and does not know what that number represents. A weight of 10 grams and a price of 10 dollars represents completely two different things — which is a no brainer for humans, but for a model as a feature, it treats both as same.\n\nAnother reason why feature scaling is applied is that few algorithms like Neural network gradient descent converge much faster with feature scaling than without it.\n\n\nOne more reason is saturation, like in the case of sigmoid activation in Neural Network, scaling would help not to saturate too fast.\n\n**When to do scaling?**\n\nFeature scaling is essential for machine learning algorithms that calculate distances between data. If not scale, the feature with a higher value range starts dominating when calculating distances, as explained intuitively in the “why?” section.\n\nThe ML algorithm is sensitive to the “relative scales of features,” which usually happens when it uses the numeric values of the features rather than say their rank.\n\nIn many algorithms, when we desire faster convergence, scaling is a MUST like in Neural Network.\n\nSince the range of values of raw data varies widely, in some machine learning algorithms, objective functions do not work correctly without normalization. For example, the majority of classifiers calculate the distance between two points by the distance. If one of the features has a broad range of values, the distance governs this particular feature. Therefore, the range of all features should be normalized so that each feature contributes approximately proportionately to the final distance.\n\nEven when the conditions, as mentioned above, are not satisfied, you may still need to rescale your features if the ML algorithm expects some scale or a saturation phenomenon can happen. Again, a neural network with saturating activation functions (e.g., sigmoid) is a good example.\n\nRule of thumb we may follow here is an algorithm that computes distance or assumes normality, scales your features.\nSome examples of algorithms where feature scaling matters are:\n\n* K-nearest neighbors (KNN) with a Euclidean distance measure is sensitive to magnitudes and hence should be scaled for all features to weigh in equally.\n* K-Means uses the Euclidean distance measure here feature scaling matters.\n* Scaling is critical while performing Principal Component Analysis(PCA). PCA tries to get the features with maximum variance, and the variance is high for high magnitude features and skews the PCA towards high magnitude features.\n* We can speed up gradient descent by scaling because θ descends quickly on small ranges and slowly on large ranges, and oscillates inefficiently down to the optimum when the variables are very uneven.\n\nAlgorithms that do not require normalization/scaling are the ones that rely on rules. They would not be affected by any monotonic transformations of the variables. Scaling is a monotonic transformation. Examples of algorithms in this category are all the tree-based algorithms — CART, Random Forests, Gradient Boosted Decision Trees. These algorithms utilize rules (series of inequalities) and do not require normalization.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom sklearn.preprocessing import StandardScaler,MinMaxScaler,MaxAbsScaler,RobustScaler,Normalizer\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.model_selection import train_test_split\nimport matplotlib.pyplot as plt\n%matplotlib inline\n# load the numerical variables of the Titanic Dataset\ndata = pd.read_csv('../input/titanic/train.csv', usecols = ['Pclass', 'Age', 'Fare', 'Survived'])\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's look at missing data\ndata.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's separate into training and testing set\nX_train, X_test, y_train, y_test = train_test_split(data[['Pclass', 'Age', 'Fare']],\n data.Survived, test_size=0.3,\n random_state=0)\nX_train.shape, X_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's fill first the missing data\n\nX_train.Age.fillna(X_train.Age.median(), inplace=True)\nX_test.Age.fillna(X_train.Age.median(), inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1. Standardisation
\nStandardScaler from scikit-learn removes the mean and scales the data to unit variance. \n\nThe Standard Scaler assumes data is normally distributed within each feature and scales them such that the distribution centered around 0, with a standard deviation of 1.\nCentering and scaling happen independently on each feature by computing the relevant statistics on the samples in the training set. If data is not normally distributed, this is not the best Scaler to use.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"scaler = StandardScaler() # create an object\nX_train_scaled = scaler.fit_transform(X_train) # fit the scaler to the train set, and then transform it","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#let's have a look at the scaled training dataset: mean and standard deviation\nprint('means (Pclass, Age and Fare): ', X_train_scaled.mean(axis=0))\nprint('std (Pclass, Age and Fare): ', X_train_scaled.std(axis=0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's look at the transformed min and max values\nprint('Min values (Pclass, Age and Fare): ', X_train_scaled.min(axis=0))\nprint('Max values (Pclass, Age and Fare): ', X_train_scaled.max(axis=0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's look at the distribution of the transformed variable Age\nplt.hist(X_train_scaled[:,1], bins=20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's look at how transformed age looks like compared to the original variable\nsns.jointplot(X_train.Age, X_train_scaled[:,1], kind='kde')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2. Min-Max Scaling
\n\nTransform features by scaling each feature to a given range. This estimator scales and translates each feature individually such that it is in the given range on the training set, e.g., between zero and one. This Scaler shrinks the data within the range of -1 to 1 if there are negative values. We can set the range like [0,1] or [0,5] or [-1,1].\nThis Scaler responds well if the standard deviation is small and when a distribution is not Gaussian. This Scaler is sensitive to outliers.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"minmaxscaler = MinMaxScaler() # create an object\nX_train_scaled = minmaxscaler.fit_transform(X_train) # fit the scaler to the train set, and then transform it","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#let's have a look at the scaled training dataset: mean and standard deviation\nprint('means (Pclass, Age and Fare): ', X_train_scaled.mean(axis=0))\nprint('std (Pclass, Age and Fare): ', X_train_scaled.std(axis=0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's look at the transformed min and max values\nprint('Min values (Pclass, Age and Fare): ', X_train_scaled.min(axis=0))\nprint('Max values (Pclass, Age and Fare): ', X_train_scaled.max(axis=0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's look at how transformed age looks like compared to the original variable\nsns.jointplot(X_train.Age, X_train_scaled[:,1], kind='kde')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3. Maximum Absolute Scaling
\nScale each feature by its maximum absolute value. This estimator scales and translates each feature individually such that the maximal absolute value of each feature in the training set is 1.0. It does not shift/center the data and thus does not destroy any sparsity.\n\nOn positive-only data, this Scaler behaves similarly to Min Max Scaler and, therefore, also suffers from the presence of significant outliers.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"maxscaler = MaxAbsScaler() # create an object\nX_train_scaled = maxscaler.fit_transform(X_train) # fit the scaler to the train set, and then transform it","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#let's have a look at the scaled training dataset: mean and standard deviation\nprint('means (Pclass, Age and Fare): ', X_train_scaled.mean(axis=0))\nprint('std (Pclass, Age and Fare): ', X_train_scaled.std(axis=0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's look at the transformed min and max values\nprint('Min values (Pclass, Age and Fare): ', X_train_scaled.min(axis=0))\nprint('Max values (Pclass, Age and Fare): ', X_train_scaled.max(axis=0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's look at how transformed age looks like compared to the original variable\nsns.jointplot(X_train.Age, X_train_scaled[:,1], kind='kde')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4. Robust Scaling
\nThis Scaler is robust to outliers. If our data contains many outliers, scaling using the mean and standard deviation of the data won’t work well.\n\nThis Scaler removes the median and scales the data according to the quantile range (defaults to IQR: Interquartile Range). The IQR is the range between the 1st quartile (25th quantile) and the 3rd quartile (75th quantile). The centering and scaling statistics of this Scaler are based on percentiles and are therefore not influenced by a few numbers of huge marginal outliers. Note that the outliers themselves are still present in the transformed data. If a separate outlier clipping is desirable, a non-linear transformation is required.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"robustscaler = RobustScaler() # create an object\nX_train_scaled = robustscaler.fit_transform(X_train) # fit the scaler to the train set, and then transform it","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#let's have a look at the scaled training dataset: mean and standard deviation\nprint('means (Pclass, Age and Fare): ', X_train_scaled.mean(axis=0))\nprint('std (Pclass, Age and Fare): ', X_train_scaled.std(axis=0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's look at the transformed min and max values\nprint('Min values (Pclass, Age and Fare): ', X_train_scaled.min(axis=0))\nprint('Max values (Pclass, Age and Fare): ', X_train_scaled.max(axis=0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's look at how transformed age looks like compared to the original variable\nsns.jointplot(X_train.Age, X_train_scaled[:,1], kind='kde')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5. Mean Normalisation
","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"normalizer = Normalizer() # create an object\nX_train_scaled = normalizer.fit_transform(X_train) # fit the scaler to the train set, and then transform it","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#let's have a look at the scaled training dataset: mean and standard deviation\nprint('means (Pclass, Age and Fare): ', X_train_scaled.mean(axis=0))\nprint('std (Pclass, Age and Fare): ', X_train_scaled.std(axis=0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's look at the transformed min and max values\nprint('Min values (Pclass, Age and Fare): ', X_train_scaled.min(axis=0))\nprint('Max values (Pclass, Age and Fare): ', X_train_scaled.max(axis=0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's look at how transformed age looks like compared to the original variable\nsns.jointplot(X_train.Age, X_train_scaled[:,1], kind='kde')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6. Quantile Transformer Scaler
\nTransform features using quantiles information.\n\nThis method transforms the features to follow a uniform or a normal distribution. Therefore, for a given feature, this transformation tends to spread out the most frequent values. It also reduces the impact of (marginal) outliers: this is, therefore, a robust pre-processing scheme.\n\nThe cumulative distribution function of a feature is used to project the original values. Note that this transform is non-linear and may distort linear correlations between variables measured at the same scale but renders variables measured at different scales more directly comparable. This is also sometimes called as Rank scaler.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import QuantileTransformer\nquantileTransformer = QuantileTransformer()\nX_train_scaled = quantileTransformer.fit_transform(X_train) # fit the scaler to the train set, and then transform it","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#let's have a look at the scaled training dataset: mean and standard deviation\nprint('means (Pclass, Age and Fare): ', X_train_scaled.mean(axis=0))\nprint('std (Pclass, Age and Fare): ', X_train_scaled.std(axis=0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's look at the transformed min and max values\nprint('Min values (Pclass, Age and Fare): ', X_train_scaled.min(axis=0))\nprint('Max values (Pclass, Age and Fare): ', X_train_scaled.max(axis=0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's look at how transformed age looks like compared to the original variable\nsns.jointplot(X_train.Age, X_train_scaled[:,1], kind='kde')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 7. Power Transformer Scaler
\nThe power transformer is a family of parametric, monotonic transformations that are applied to make data more Gaussian-like. This is useful for modeling issues related to the variability of a variable that is unequal across the range (heteroscedasticity) or situations where normality is desired.\n\nThe power transform finds the optimal scaling factor in stabilizing variance and minimizing skewness through maximum likelihood estimation. Currently, Sklearn implementation of PowerTransformer supports the Box-Cox transform and the Yeo-Johnson transform. The optimal parameter for stabilizing variance and minimizing skewness is estimated through maximum likelihood. Box-Cox requires input data to be strictly positive, while Yeo-Johnson supports both positive or negative data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import PowerTransformer\npowerTransformer = PowerTransformer()\nX_train_scaled = powerTransformer.fit_transform(X_train) # fit the scaler to the train set, and then transform it","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#let's have a look at the scaled training dataset: mean and standard deviation\nprint('means (Pclass, Age and Fare): ', X_train_scaled.mean(axis=0))\nprint('std (Pclass, Age and Fare): ', X_train_scaled.std(axis=0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's look at the transformed min and max values\nprint('Min values (Pclass, Age and Fare): ', X_train_scaled.min(axis=0))\nprint('Max values (Pclass, Age and Fare): ', X_train_scaled.max(axis=0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's look at how transformed age looks like compared to the original variable\nsns.jointplot(X_train.Age, X_train_scaled[:,1], kind='kde')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 8. Date and Time Engineering
\nTime is the most essential concept in any business.There is a lot of nuance to time series data that we need to consider when we’re working with datasets that are time-sensitive.Existing time series forecasting models undoubtedly work well in most cases, but they do have certain limitations. \n\n\nThere’s no one-size-fits-all approach here. There’ll be projects, such as demand forecasting or click prediction when you would need to rely on supervised learning algorithms.And there’s where feature engineering for time series comes to the fore. This has the potential to transform your time series model from just a good one to a powerful forecasting model.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"
\n## 1. Introduction","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\ndata = pd.read_csv('../input/birdsong-recognition/train.csv')\ndata.dtypes","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2. Date-Related Features
\nWe can find out the birds pattern for weekdays and weekends based on historical data. Thus, having information about the day, month, year, etc. can be useful for forecasting the value","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\ndata = pd.read_csv('../input/birdsong-recognition/train.csv')\ndata['date'] = pd.to_datetime(data['date'],format='%Y-%m-%d', errors='coerce')\n\ndata['year']=data['date'].dt.year \ndata['month']=data['date'].dt.month \ndata['day']=data['date'].dt.day\ndata['dayofweek_num']=data['date'].dt.dayofweek \ndata['dayofweek_name']=data['date'].dt.weekday\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3. Time-Related Features
\nWe can similarly extract more granular features if we have the time stamp. For instance, we can determine the hour or minute of the day when the data was recorded and compare the trends between the business hours and non-business hours.\n\nIf we are able to extract the ‘hour’ feature from the time stamp, we can make more insightful conclusions about the data. \n\nExtracting time-based features is very similar to what we did above when extracting date-related features. We start by converting the column to DateTime format and use the .dt accessor. Here’s how to do it in Python:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\ndata = pd.read_csv('../input/birdsong-recognition/train.csv')\ndata['time'] = pd.to_datetime(data['date'],format='%Y-%m-%d', errors='coerce')\n\ndata['time'] = pd.to_datetime(data['time'],format='%H:%M')\n\ndata['Hour'] = data['time'].dt.hour \ndata['minute'] = data['time'].dt.minute \n\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Similarly, we can extract a number of features from the date column. Here’s a complete list of features that we can generate:\n\n## 4. Lag Features
\nWhen working on a time series problem can anyone imagine that we can also use the target variable for feature engineering!Strange right but we can :)\n\nConsider this – you are predicting the sale price for a house. So, the previous prices of house price is important to make a prediction, right? In other words, the value at time t is greatly affected by the value at time t-1. The past values are known as lags, so t-1 is lag 1, t-2 is lag 2, and so on.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\ndata = pd.read_csv('../input/house-prices-advanced-regression-techniques/train.csv')\ndata['YrSold'] = pd.to_datetime(data['YrSold'],format='%Y')\n\ndata['lag_1'] = data['SalePrice'].shift(1)\ndata = data[['YrSold', 'lag_1', 'SalePrice']]\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here, we were able to generate lag one feature for our series. But why lag one? Why not five or seven? To answer this let us understand it better below.\n\nThe lag value we choose will depend on the correlation of individual values with its past values.\n\nIf the series has a weekly trend, which means the value last Monday can be used to predict the value for this Monday, you should create lag features for seven days. Getting the drift?\n\nWe can create multiple lag features as well! Let’s say we want lag 1 to lag 7 – we can let the model decide which is the most valuable one. So, if we train a linear regression model, it will assign appropriate weights (or coefficients) to the lag features:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\ndata = pd.read_csv('../input/house-prices-advanced-regression-techniques/train.csv')\ndata['YrSold'] = pd.to_datetime(data['YrSold'],format='%Y')\n\ndata['lag_1'] = data['SalePrice'].shift(1)\ndata['lag_2'] = data['SalePrice'].shift(2)\ndata['lag_3'] = data['SalePrice'].shift(3)\ndata['lag_4'] = data['SalePrice'].shift(4)\ndata['lag_5'] = data['SalePrice'].shift(5)\ndata['lag_6'] = data['SalePrice'].shift(6)\ndata['lag_7'] = data['SalePrice'].shift(7)\ndata = data[['YrSold', 'lag_1', 'lag_2', 'lag_3', 'lag_4', 'lag_5', 'lag_6', 'lag_7', 'SalePrice']]\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is more than one way of determining the lag at which the correlation is significant. For instance, we can use the ACF (Autocorrelation Function) and PACF (Partial Autocorrelation Function) plots.\n\nACF: The ACF plot is a measure of the correlation between the time series and the lagged version of itself\nPACF: The PACF plot is a measure of the correlation between the time series with a lagged version of itself but after eliminating the variations already explained by the intervening comparisons\n\nFor our particular example, here are the ACF and PACF plots:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from statsmodels.graphics.tsaplots import plot_acf,plot_pacf\nplot_acf(data['SalePrice'], lags=10)\nplot_pacf(data['SalePrice'], lags=10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The partial autocorrelation function shows a high correlation with the first lag and lesser correlation with the second and third lag. The autocorrelation function shows a slow decay, which means that the future values have a very high correlation with its past values.\n\n**Note :** The number of times you shift, the same number of values will be reduced from the data. You would see some rows with NaNs at the start. That’s because the first observation has no lag. You’ll need to discard these rows from the training data.\n\n## 5. Rolling Window
\nHow about calculating some statistical values based on past values? This method is called the **rolling window method** because the window would be different for every data point.\n\n\nSince this looks like a window that is sliding with every next point, the features generated using this method are called the ‘rolling window’ features.\n\nNow the question we need to address – how are we going to perform feature engineering here? Let’s start simple. We will select a window size, take the average of the values in the window, and use it as a feature. Let’s implement it in Python:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\ndata = pd.read_csv('../input/house-prices-advanced-regression-techniques/train.csv')\ndata['YrSold'] = pd.to_datetime(data['YrSold'],format='%Y')\n\ndata['rolling_mean'] = data['SalePrice'].rolling(window=7).mean()\ndata = data[['YrSold', 'rolling_mean', 'SalePrice']]\ndata.head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Similarly, you can consider the sum, min, max value, etc. (for the selected window) as a feature and try it out on your own machine.\n## 6. Expanding Window
\nThis is simply an advanced version of the rolling window technique. In the case of a rolling window, the size of the window is constant while the window slides as we move forward in time. Hence, we consider only the most recent values and ignore the past values.\n\nThe idea behind the expanding window feature is that it takes all the past values into account.\n\nAs shown below expanding window function how it works:\n\nAs you can see, with every step, the size of the window increases by one as it takes into account every new value in the series. This can be implemented easily in Python by using the expanding() function. Let’s code this using the same data:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\ndata = pd.read_csv('../input/house-prices-advanced-regression-techniques/train.csv')\ndata['YrSold'] = pd.to_datetime(data['YrSold'],format='%Y')\n\ndata['Expanding_Mean'] = data['SalePrice'].expanding(2).mean()\ndata = data[['YrSold', 'SalePrice','Expanding_Mean']]\ndata.head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 7. Domain-Specific
\nHaving a good understanding of the problem statement, clarity of the end objective and knowledge of the available data is essential to engineer domain-specific features for the model.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 9. Read more
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"\n## 1. References
\n\n* https://towardsdatascience.com/smarter-ways-to-encode-categorical-data-for-machine-learning-part-1-of-3-6dca2f71b159\n* https://mux.com/blog/using-percentiles-to-identify-outliers/\n* https://towardsdatascience.com/understanding-boxplots-5e2df7bcbd51\n* https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5548942/#:~:text=There%20are%20basically%20three%20methods,outliers%20through%20outlier%20weight%20adjustments.\n* https://www.trainindata.com/post/feature-engineering-comprehensive-overview\n* https://towardsdatascience.com/feature-engineering-techniques-9a57e4901545\n* https://towardsdatascience.com/all-about-feature-scaling-bcc0ad75cb35\n* https://www.analyticsvidhya.com/blog/2019/12/6-powerful-feature-engineering-techniques-time-series/","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# I hope by now you have a fair good understanding of what is Feature Engineering is all about.\n\n# Greatly appreciate to leave your comments /feedback and if you like this kernel notebook please do UPVOTE","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}