{"cells":[{"metadata":{"_uuid":"54287a280bbadb6c4090da672d59fc9b922f83d6"},"cell_type":"markdown","source":"# **Acknowledgements**\nTo the following whose kernels I used extensively for this kernel:\n* Manav Sehgal - \"Titanic Data Science Solutions\" - https://www.kaggle.com/startupsci/titanic-data-science-solutions/notebook\n* Pedro Marcelino - \"Comprehensive data exploration with Python\" - https://www.kaggle.com/pmarcelino/comprehensive-data-exploration-with-python\n* Serigne - \"Stacked Regressions to predict House Prices\" - https://www.kaggle.com/serigne/stacked-regressions-top-4-on-leaderboard\n* Laurens ten Cate - \"Top 2% of LeaderBoard - Advanced FE\"    https://www.kaggle.com/laurenstc/top-2-of-leaderboard-advanced-fe\n\nI highly recommend exploring these kernels to get a more indepth understanding of their respective approaches and also useful links."},{"metadata":{"_uuid":"f6ffc02d9ec694cea5345a23dd8c5ee8782132f9"},"cell_type":"markdown","source":"# Me\n\nI'm new to data science and this is my second challenge and Notebook.  So please excuse inefficient code as I learn:\n* MathPlotLib abd Seaborn - I've spent ages learning about these libraries as I developed this Notebook\n* Stats (e.g. testing assumptions) - learnt a lot about that developing this too - thanks to Pedro (see acknowledgements)\n* Python\n"},{"metadata":{"_uuid":"3d3c177bcf54a43d3573216a2df9ca4b2e8c3339"},"cell_type":"markdown","source":"# **Workflow Stages**\n***\n\n**1. Question or problem definition**\n\n    This competition challenges you to predict the final price of each home.\n    \n**2. Acquire training and testing data**\n\n    Provided - 79 variables - excluding the 'Id' and predictor variable 'SalePrice'\n\n**3. Data Analysis**\n\n    3.1. Analyse by describing the data\n   \n        * Explore features (meta data)\n        * Initial examination of the variables, segments, types and meaning\n\n    3.2. Explore the target feature - SalePrice\n   \n        3.2.1. Univariate study - Focus on the target feature and try to know a little bit more about it.\n    \n        3.2.2. Bivariate study - Try to understand how the target feature relates to other features\n       \n            * Examine numerical features\n            * Examine categorical features\n\n        3.2.3. Multivariate study\n   \n           We'll use heatmaps and correlations to examine the interlationships between all the features\n\n**4. Data Cleaning and Pre-Processing - prepare and cleanse data before modelling**\n\n    4.1 Outliers\n    4.2 Statistical transformations\n    \n**5. Feature Engineering**\n\n    5.1 Concatenation\n    5.2 NA's\n    5.3 Incorrect values\n    5.4 Label Encoding and Feature Conversion\n    5.5 Further Statistical transformation\n    5.6 Column removal\n    5.7 Creating features\n    5.8 Dummies\n    5.9 In-depth outlier detection\n    5.10 Overfit prevention\n    5.11 Baseline model\n   \n**6. Feature Selection**\n   * Filter methods\n       * baseline coefficients\n   * Embedded methods\n       * L2: Ridge Regression\n       * L1: Lasso regression\n           * In-depth coefficient analysis\n       * Elasticnet\n       * XGBoost\n       * SVR\n       * LightGBM\n\n**7. Ensemble methods**\n   * Stacked generalizations\n   * Averaging\n   * standard\n   * weighted\n\n**8. Prediction**\n\n   * Supply or submit the results\n\n    4.2. Drop features with large numbers of null values\n    4.3. Drop features due to overlapping feature meaning\n    4.4. Create / derive new features from existing\n    4.5. Impute missing values by considering each feature one by one\n    4.6. Transforming numerical features which are actually categorical\n    4.7. Label encoding - some features may contain some information in their ordering set\n    4.8. Log transformation of skewed values (consider / compare Box-Cox and Arcsine transformations)\n    4.9. Get dummy features for categoprical features\n\n\n**6. Feature Selection**\n\n\n**6. Model, predict and solve the problem**\n\n    6.1 Choose base models\n    6.2 Cross validate the models\n    6.3 Stack/Ensemble the models\n\n**7. Visualize, report, and present the problem solving steps and final solution.**\n\n\n"},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"6ee10869e22918b9df1f56781e8ef7321281fb38"},"cell_type":"code","source":"# Data analysis and wranging\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport math\nimport sys\nimport os\nfrom decimal import *\nimport warnings\n\n# Set ipython's max row, max columns and display width display settings\npd.set_option('display.max_row', 1000)\npd.set_option('display.max_columns', 100)\npd.set_option('display.width', 400)\npd.set_option('display.float_format', lambda x: '{:.3f}'.format(x)) #Limiting floats output to 3 decimal points\n\n# Visualisation\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom matplotlib.ticker import FuncFormatter\n%matplotlib inline\n\n# machine learning\nfrom scipy.stats import norm, skew\nimport scipy.stats as stats\nfrom scipy.stats import chi2_contingency\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.svm import SVC, LinearSVC\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.naive_bayes import GaussianNB\nfrom sklearn.linear_model import Perceptron\nfrom sklearn.linear_model import SGDClassifier\nfrom sklearn.tree import DecisionTreeClassifier\nfrom sklearn.ensemble import ExtraTreesClassifier\n\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.model_selection import KFold, cross_val_score\nfrom sklearn.model_selection import cross_val_predict\nfrom sklearn.preprocessing import RobustScaler\nfrom sklearn.pipeline import make_pipeline\n\n# Feature selection\nfrom sklearn.feature_selection import RFE\n\n# Create the function to enable us to stop deprecated function warnings\ndef fxn():\n    warnings.warn(\"deprecated\", DeprecationWarning)\n    \ndef ignore_warn(*args, **kwargs):\n    pass\nwarnings.warn = ignore_warn #ignore annoying warning (from sklearn and seaborn)\n\ndef feature_null_analysis(df_desc, df, drop_theshold): \n    # Create lists of information we want to see\n    feat_names = list(df)\n    feat_dtype = list(df.dtypes)\n    feat_default = ['Mode: ' + df[feat].mode()[0] if df[feat].dtype == 'object' else 'Median: ' + str(round(df[feat].median(),2)) for feat in list(df)]\n    feat_default_perc = ['%/total: ' + str(round(((train[feat].value_counts().iloc[0] / train.shape[0]) * 100),2))\n                             if train[feat].dtype == 'object' else '' for feat in list(train)]\n    feat_nulls = df.isnull().sum()\n    feat_nullperc = df.isnull().mean() * 100\n    feat_dropind = ['Y' if val >= drop_theshold else 'N' for val in feat_nullperc]\n    \n    # Combine the info into one soreted list\n    feat_analysis_all = sorted(list(zip(feat_names,feat_dtype,feat_default,feat_default_perc,feat_nulls,feat_nullperc, feat_dropind))\n                               ,key=lambda x: x[4], reverse=True)\n    feat_analysis_nulls = [feat for feat in feat_analysis_all if feat[5] > 0]  # features with nulls\n    feat_droplist = [feat[0] for feat in feat_analysis_all if feat[5] >= 15]  # features recommended to drop\n    \n    # print the results\n    print_feature_null_analysis('features', feat_analysis_nulls)\n    \n    # Pass back the list of features recommended to drop to make it easier to drop them\n    # return feat_analysis_nulls, feat_droplist\n\ndef print_feature_null_analysis(df_desc, feat_analysis):\n    # print the analysis\n    print('\\n{: >{width}}'.format(df_desc, width=2 * PRINT_WIDTH))\n    print('{: >{width}}'.format('+++++++++', width=2 * PRINT_WIDTH))\n    print('{: <{width}}{: <{width}}{: <{width}}{: <{width}}{: <{width}}{: <{width}}{: <{width}}'.\n          format('Feature','DType','Mode/Median','Perc. Rows = Mode','No. Nulls','Perc. Nulls','Drop? (Y/N)', width=PRINT_WIDTH))\n    print('{: <{width}}{: <{width}}{: <{width}}{: <{width}}{: <{width}}{: <{width}}{: <{width}}'.\n          format('=======','=====','===========','=================','=========','===========','=====', width=PRINT_WIDTH))\n    \n    for feat in feat_analysis:\n        print('{: <{width}}{: <{width}}{: <{width}}{: <{width}}{: <{width}}{: <{width}.2F}{: <{width}}'.\n              format(feat[0],str(feat[1]),feat[2],feat[3],feat[4],feat[5],feat[6], width=PRINT_WIDTH))\n\ndef drop_features(df_desc, df, feats_to_drop):\n    print('\\nBefore shape for {}: {}'.format(df_desc, df.shape))\n    df.drop(feats_to_drop, axis=1, inplace=True)\n    print('\\nAfter shape for {}: {}'.format(df_desc, df.shape))\n    feat_analysis_nulls, feat_droplist = feature_null_analysis(df_desc, df, NULL_PERC_DROP_PERC)  # As we've dropped one or more features we need to recreate the analysis\n    return feat_analysis_nulls, FEATURES_DROPPED + feats_to_drop\n\n# def drop_target_feature(df):\n#     target_feature_data = df[TARGET_FEATURE]\n#     df.drop(TARGET_FEATURE, axis=1, inplace=True)\n#     return target_feature_data\n\nNULL_PERC_DROP_PERC = 15  # Set the threshold percentage of nulls in a column - will determine if it's recommended to drop\nPRINT_WIDTH = 20  # Print parameter","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"df896014db2e1fb76f1cba5816360280b078592a"},"cell_type":"markdown","source":"# 1. Question or problem definition\n\nThis competition challenges you to predict the final price of each home."},{"metadata":{"_uuid":"4cb86432544cbff0014f637915ab5070fdc28cea"},"cell_type":"markdown","source":"# 2. Acquire training and test data"},{"metadata":{"trusted":false,"_uuid":"bb5e1dca62c846534755454488ed2e53f7f4c201"},"cell_type":"code","source":"# Check files in input directory - Windows\n# input_dir = 'C:/Users/.........'\n# Check files in input directory - Jupyter\ninput_dir = '../input/'\nl = list(os.listdir(input_dir))\nfor f in l:\n    print(f)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"ccc89b38a278c40d846c3134776a000357dbe820"},"cell_type":"code","source":"# Import Train and Test data\ntrain = pd.read_csv('../input/train.csv')\ntest = pd.read_csv('../input/test.csv')\n\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"eaf72f9d79bb1da5894d89883086224c96fec720"},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"60c77988b13ca4fbb20e8f068cc1eb0294f49053"},"cell_type":"code","source":"# Save then drop the Id column from Train and Test - not needed for modelling/prediction\n\n# Save Id\ntrain_id = train['Id']\ntest_id = train['Id']\n\n# Print shapes before drop\nprint('\\nBefore shape for {}: {}'.format('train', train.shape))\nprint('\\nBefore shape for {}: {}'.format('test', test.shape))\n\n# Drop Id\ntrain.drop('Id', axis=1, inplace=True)\ntest.drop('Id', axis=1, inplace=True)\n\n# Print shapes after drop\nprint('\\nAfter shape for {}: {}'.format('train', train.shape))\nprint('\\nAfter shape for {}: {}'.format('test', test.shape))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d8e4c85875d1282528b836c841796885c367d765"},"cell_type":"markdown","source":"# 3. **Explore features - get to know the data**\n\n## 3.1 Analyse by describing the data\n\n**Create an Excel spreadhseet**\nList the features and analyse them - indicating if you think they're gong to be influential; include the following:\n\n* Variable - Variable name\n* Type - Identification of the variables' type. There are two possible values for this field: 'numerical' or 'categorical'. By 'numerical' we mean variables for which the values are numbers, and by 'categorical' we mean variables for which the values are categories.\n* Segment - Identification of the variables' segment. We can define three possible segments:\n    * Building - a variable that relates to the physical characteristics of the building (e.g. 'OverallQual')\n    * Space - a variable that reports space properties of the house (e.g. 'TotalBsmtSF')\n    * Location - a variable that gives information about the place where the house is located (e.g. 'Neighborhood')\n* Expectation - Our expectation about the variable influence in 'SalePrice'. We can use a categorical scale with 'High', 'Medium' and 'Low' as possible valuee\n* Conclusion - Our conclusions about the importance of the variable, after we give a quick look at the data. We can keep with the same categorical scale as in 'Expectation'\n* Comments - Any general comments that occured to us e.g. opportunity to derive a new feature (e.g. split or combine others)\n\n**Which features are categorical?**\n\nThese values classify the samples into sets of similar samples. Within categorical features are the values nominal, ordinal, ratio, or interval based? Among other things this helps us select the appropriate plots for visualization.\n\n* Nominal: 'MSSubClass', 'MSZoning', 'Street', 'Alley', 'LandContour', 'Utilities', 'LotConfig', 'Neighborhood', 'Condition1', 'Condition2', 'BldgType', 'HouseStyle', 'RoofStyle', 'RoofMatl', 'Exterior1st', 'Exterior2nd', 'MasVnrType', 'Foundation', 'Heating', 'CentralAir', 'Electrical', 'GarageType', 'MiscFeature', 'SaleType', 'SaleCondition'\n\n\n* Ordinal: 'OverallQual', 'OverallCond', 'ExterQual', 'ExterCond', 'BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinType2', 'HeatingQC', 'KitchenQual', 'Functional', 'FireplaceQu', 'GarageFinish', 'GarageQual', 'GarageCond', 'PoolQC', 'Fence', 'LotShape', 'LandSlope', 'PavedDrive'\n\n\n* Interval - Discreet: 'YearBuilt', 'YearRemodAdd' (=same as YearBuilt if no remodelling), 'YrSold', 'MoSold', 'GarageYrBlt'\n\n**Which features are numerical?**\n\nThese values change from sample to sample. Within numerical features are the values discrete, continuous, or timeseries based? Among other things this helps us select the appropriate plots for visualization.\n\n* Nominal (Discreet): 'BsmtFullBath', 'BsmtHalfBath', 'FullBath', 'HalfBath', 'BedroomAbvGr', 'KitchenAbvGr', 'TotRmsAbvGrd', 'Fireplaces', 'GarageCars'\n\n\n* Ratio - Continuous: 'LotFrontage', 'LotArea', 'MasVnrArea', 'BsmtFinSF1', 'BsmtFinSF2', 'BsmtUnfSF', 'TotalBsmtSF', '1stFlrSF', '2ndFlrSF', 'LowQualFinSF', 'GrLivArea',  'GarageArea', 'WoodDeckSF', 'OpenPorchSF', 'EnclosedPorch', '3SsnPorch', 'ScreenPorch', 'PoolArea', 'MiscVal', 'SalePrice'\n\n**Which are the alphanumeric features?**\n\nLotConfig, BldgType, HouseStyle are all alphanumeric features.\n\n\n**Which features may contain errors or typos?**\nThis is hard for such a large dataset with so many features - but there don't appear to be any obvious features which could contain typos such as a Name or a Title feature.\n\nCreating the XLS will help become more familiar with the data and it's meaning - which is half the battle."},{"metadata":{"_uuid":"73dc8aa99009539b5465994e3f4e7780f1714783"},"cell_type":"markdown","source":"## 3.2 Identify features with excessive null values\n\nThe process of dealing with nulls is heavily dependant on the meaning of the data and it's relevance to the model. However, if there are greater than ~15% (parameterised in the routine below) of a feature's values which are nulls I drop the feature.\n\n* feature_null_analysis - creates a list which details relevant information to help identify features to drop if they have a high null count. It also details the mode or median value for those columns\n* print_feature_null_analysis - prints the analysis\n* drop_features - drops those features passed into it and maintains a list of features dropped across repeated executions of his proc"},{"metadata":{"scrolled":false,"trusted":false,"_uuid":"187702f0bfbf556ac0a0786da9b28e335c65acbd"},"cell_type":"code","source":"# Analyse and print the features for Null values\nfeature_null_analysis('train', train, NULL_PERC_DROP_PERC)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fbfd8b15e28d622d276aecbca0277e05c51ca046"},"cell_type":"markdown","source":"## 3.3. Explore the independent variable - SalePrice\n\n\n### 3.3.1 Univariate Analysis"},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"84b963a193ef68c2d0a25f988bbe9e8fbb07628d"},"cell_type":"code","source":"train['SalePrice'].describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fc6aaa60e6f89970a1a720984390b94f6f780acd"},"cell_type":"markdown","source":"\n\nNo warnings signs here - min above zero - be interesting to see if we have outliers which could impact our analysis and model accuracy.\n\n"},{"metadata":{"trusted":false,"_uuid":"b5c105c0f67a4eba8c4224a369580b1b44570c80"},"cell_type":"code","source":"sns.set(style=\"whitegrid\", palette='bright')\nfig, ax = plt.subplots(figsize=(12,6))\nwith warnings.catch_warnings():\n    warnings.simplefilter(\"ignore\")\n    fxn()\n    ax.xaxis.set_major_formatter(FuncFormatter(lambda x, _: '${:,.0F}'.format(x))) \n    p = sns.distplot(train['SalePrice'] , fit=stats.norm, ax=ax)\n    ax.set(xlabel='Sale Price', ylabel='Density')\n    sns.despine(trim=True)\n    p.set(xlim=(0, None), ylim=(0,None))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eecd140913702d3e936e8f7dc8adc4802f240ed8"},"cell_type":"markdown","source":"Positively skewed with a few significant outliers at around $500-800k."},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"a6c3cc860dbe613c46840252c189841dbeb5fe58"},"cell_type":"code","source":"# Get the fitted parameters used by the function plus skewness and kurtosis\nmu, sigma = stats.norm.fit(train['SalePrice'])\nsales_skew = train['SalePrice'].skew()\nsales_kurtosis = train['SalePrice'].kurtosis()\nprint( '\\n mu(Avg) = {:.2f}, sigma(SD) = {:.2f}, skew = {:.2F} and kurtosis = {:.2F}\\n'.format(mu, sigma, sales_skew, sales_kurtosis))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"107bf24c3ff8d7b31a3d23f248993a6cca847aed"},"cell_type":"markdown","source":"##  3.3.2 Bivariate analysis\n\n**Relationship with selected numerical variables**"},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"0bb2fefefb888e559faae5a265a2eae926bcc40b"},"cell_type":"code","source":"numeric_features = train.select_dtypes(exclude=['object']).columns.tolist()\n\ngrid_cols = 2\npair_plots = 1\ngrid_rows = math.ceil(len(numeric_features) / ((grid_cols // pair_plots)))\nfig, axes = plt.subplots(grid_rows, grid_cols, figsize=(20,80))\n\naxes = axes.ravel()\ncolix = 0\nplt.subplots_adjust(top = 0.99, bottom=0.01, hspace=0.4, wspace=0.1)\nwith warnings.catch_warnings():\n    warnings.simplefilter(\"ignore\")\n    fxn()\n    for i in range(len(numeric_features)):\n        ax = axes[i]\n        for label in (ax.get_xticklabels() + ax.get_yticklabels()):\n            label.set_fontname('Arial')\n            label.set_fontsize(12)\n        ax.set_ylabel('SalePrice', fontsize=12)    \n        ax.set_xlabel(numeric_features[colix], fontsize=12)\n        ax.tick_params(axis='x', rotation=70)\n        \n        g = sns.regplot(x=numeric_features[colix], y='SalePrice', ax=ax, data=train, scatter_kws={'s':3})\n        colix += 1","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5722896b682602bcadf6fadd60637b641c566562"},"cell_type":"markdown","source":"Looking at the regression plots above there are a few positive linear reltionships of varying stregnths:\n* TotalBsmtSF\n* 1stFlrFSF\n* GrLivArea\n* Garage Area"},{"metadata":{"_uuid":"64e5c461fe365d56460ba39b5ea7bd54fea611cb"},"cell_type":"markdown","source":"**Relationship with selected categorical \"Space\" segment variables**\n\n* Ignore features with a > 15% number of nulls ( PoolQC, MiscFeature, Alley, Fence, FirePlaceQu)"},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"4f4cff3d1a920e35ee845b6cda65a85ee380c247"},"cell_type":"code","source":"categorical_features = train.select_dtypes(include=['object']).columns.tolist()\n\nfeat_cols = ['MSSubClass', 'MSZoning', 'Street', 'LotShape', 'LandContour', 'Utilities', 'LotConfig', 'LandSlope', 'Neighborhood', 'Condition1', 'Condition2',\n           'BldgType', 'HouseStyle', 'RoofStyle', 'RoofMatl', 'Exterior1st', 'Exterior2nd', 'MasVnrType', 'ExterQual', 'ExterCond', 'Foundation', 'BsmtQual', \n           'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinType2', 'Heating', 'HeatingQC', 'CentralAir', 'Electrical', 'KitchenQual', 'Functional',\n           'GarageType', 'GarageFinish', 'GarageQual', 'GarageCond', 'PavedDrive', 'SaleType', 'SaleCondition']\n\ngrid_cols = 4\npair_plots = 2\ngrid_rows = len(categorical_features) // ((grid_cols // pair_plots))\nfig, axes = plt.subplots(grid_rows, grid_cols, figsize=(20,80))\naxes = axes.ravel()\ndv = 'SalePrice'\ncolix = 0\nplt.subplots_adjust(top = 0.99, bottom=0.01, hspace=0.4, wspace=0.2)\n\nwith warnings.catch_warnings():\n    warnings.simplefilter(\"ignore\")\n    fxn()\n    for i in range(0,grid_rows * grid_cols,2):\n        g = sns.boxplot(x=categorical_features[colix], y=dv, data=train, ax=axes[i])\n        p = sns.countplot(x=categorical_features[colix], data=train, ax=axes[i+1])\n        g.set_xticklabels(g.get_xticklabels(), rotation=70, fontsize = 12)\n        p.set_xticklabels(g.get_xticklabels(), rotation=70, fontsize = 12)\n        if categorical_features[colix] in ['YearBuilt', 'YearRemodAdd']:\n            g.set_xticklabels(g.get_xticklabels(), visible=False)\n            p.set_xticklabels(p.get_xticklabels(), visible=False)\n        colix += 1","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8b65b76b07b500b2ad288eb01acdc9ab45e82f50"},"cell_type":"markdown","source":"**Observations:**\n\nMost category distributions are heavily skewed to a single or a few values and should be considered for removal\n\n* QverallQual - show a clear exponential distribution\n* YearBuilt and YearRemodelled - show a recent increase in sales price bur the count plot indicates a surge in recent years of both units sold and remodelled\n* MoSold - it's clear the most popular time for selling is in the summer (May through July) -  July sales seem to attain higher sales prices\n\n**Decisions:**\n\nKeep columns:\n* OverallQual\n* MoSold\n\nOther columns can be included for now - but test the impact on accuracy if they are selectively removed."},{"metadata":{"_uuid":"45cf400c3d4a61ebb3e9977ba653c57a0a6b66de"},"cell_type":"markdown","source":"## 3.4. Multivariate Analysis\n\n### 3.4.1 Produce a Heatmap to examine the relative coefficicients between features\n"},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"d549a64db798436e43af6a17f49c3c7c43725802"},"cell_type":"code","source":"corrmat = train.corr()\nfig, ax = plt.subplots(figsize=(12, 9))\nsns.heatmap(corrmat, vmax=0.8, square=False);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1686e00bddbb505f81b61ce930e75b0c92501dba"},"cell_type":"markdown","source":"**Observations:**\n\nOverallQual and GrLivArea are strongly correlated with SalePrice.  GarageCars (and GarageArea) too."},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"1fab0ceacf59c7d7ab784060745a3fcfa9fbe801"},"cell_type":"code","source":"# Saleprice correlation matrix\nk = 10 #number of variables for heatmap\ncols = corrmat.nlargest(k, 'SalePrice')['SalePrice'].index\ncm = np.corrcoef(train[cols].values.T)\nsns.set(font_scale=1.25)\nfig, ax = plt.subplots(figsize=(10, 5))\nhm = sns.heatmap(cm, cbar=True, annot=True, square=True, fmt='.2f', annot_kws={'size': 10}, yticklabels=cols.values, xticklabels=cols.values)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e4e0b6f28bf855bc69c3f30c765ac2c0b18ce2ee"},"cell_type":"markdown","source":"**Observations:**\n   * GarageCars and GarageArea are strongly correlated - not a huge surprise as the number of cars you can fit in a garage is dependent on the size of the garage\n   * TotalBsmtSF is strongly correlated to 1stFlrSF - again undertandable unless you have either a massive basement or a exceptionally large 1st floor\n  \n**Decisons:**\n   * We can drop GarageArea - as it's covered by GarageCars\n   * We cab drop 1stFlrSF - as it's covered by TotalBsmtSF"},{"metadata":{"_uuid":"1d6b88d0b2e520c30ba1cd35a9b2fa0cb1a60ce1"},"cell_type":"markdown","source":"### 3.4.2 Scatter plots between the target feature (SalePrice) and selected independent features"},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"8609f6499f6b70d3b5dd9e6ee4efb70ef3ebe469"},"cell_type":"code","source":"iv_cols = ['OverallQual', 'GrLivArea', 'GarageCars', 'TotalBsmtSF', 'FullBath', 'TotRmsAbvGrd','YearBuilt']\ndv = train['SalePrice']\ngrid_cols = 4\npair_plots = 1\ncolix = 0\ngrid_rows = math.ceil(len(iv_cols) / ((grid_cols // pair_plots)))\nfig, axes = plt.subplots(grid_rows, grid_cols, figsize=(25,8))\naxes = axes.ravel()\nplt.subplots_adjust(top = 0.99, bottom=0.01, hspace=0.4, wspace=0.5)\n\nwith warnings.catch_warnings():\n    warnings.simplefilter(\"ignore\")\n    fxn()\n    for i in range(len(iv_cols)):\n        g = sns.regplot(x=iv_cols[colix], y=dv, ax=axes[i], data=train, scatter_kws={'s':1}, order=2)\n        colix += 1","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1f267033b47f7d8e03933e6e5afc7dcfe90a8239"},"cell_type":"markdown","source":"**Observations:**\n   * TotalRmsAbvGrd and GrLivArea - appears to be a straight line indicating, unsurprisingly, there is a strong relationship between the number of rooms and the space available\n   * YearBuilt and OverallQual - also show some degree of exponentiality especially at the top, most recent years.  It could indicate the maxing out of prices in the area - folks just won't pay more than a certain threshold for those neighborhoods\n   * TotalBsmtSF is strongly correlated to 1stFlrSF - again, asindicated in the heatmap above, undertandable unless you have either a massive basement or a exceptionally large 1st floor\n\nSome features have a very similar corrleation and overlap:\n* TotalBsmtSF and 1stFlrSF have exactly the same correlation and are very likely to be aligend because the basement size will likely dictate the size of the 1st floor\n* The key Garage variable is GarageCars, so we'll get rid of the others\n* We'll treat all the Basement features in a similar way to the Garage features for the same reason (keeping TotalBsmtSF)\n\n**Decisions:**\n* Drop all Garage festures except GarageCars\n* Drop all Basement features except TotalBsmtSF\n* Drop 1stFlrSF as it's covered by TotalBsmtSF\n"},{"metadata":{"_uuid":"cf42f597a2420b0b4849d79b4dfc8b944cf9d3f7"},"cell_type":"markdown","source":"# 4. 4. Data Cleaning and Pre-Processing (Prepare and cleanse data before modelling)\n\nWe have collected several assumptions and decisions regarding our datasets and solution requirements. So far we did not have to change a single feature or value to arrive at these. Let us now execute our decisions and assumptions for correcting, creating, and completing goals as part of this and feature engineering activities."},{"metadata":{"_uuid":"9d54e8872f560ee59ef8a68c3e215ec439a01d02"},"cell_type":"markdown","source":"## 4.1 Outliers \n\n### 4.1.1 Outliers - Univariate analysis\n\nThe primary concern here is to establish a threshold that defines an observation as an outlier. To do so, we'll standardize the data. In this context, data standardization means converting data values to have mean of 0 and a standard deviation of 1."},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"17ac91590005a4b0d962361743a03df86f48b388"},"cell_type":"code","source":"#standardizing data\nsaleprice_scaled = StandardScaler().fit_transform(train['SalePrice'][:,np.newaxis]);\nlow_range = saleprice_scaled[saleprice_scaled[:,0].argsort()][:10]\nhigh_range= saleprice_scaled[saleprice_scaled[:,0].argsort()][-10:]\nprint('outer range (low) of the distribution:')\nprint(low_range)\nprint('\\nouter range (high) of the distribution:')\nprint(high_range)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bffd31b5dc5c95296b1a51f503782d75461ab78b"},"cell_type":"markdown","source":"**Observations:**\n\nLower range close to zero\nUpper range indicates a couple of large outliers (the 7+ values)\n\n**Decisions:**\n\nLeave alone for now"},{"metadata":{"_uuid":"89d9384b2c1330b04a3315e8fee8763c5f51523b"},"cell_type":"markdown","source":"### 4.1.2 Outliers - Bivariate analysis\n\n#### 4.1.2.1 GrLivArea"},{"metadata":{"scrolled":false,"trusted":false,"_uuid":"ee1765252114dd3e9d4829ac458b98f2a6a01143"},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(20,10))\nwith warnings.catch_warnings():\n    warnings.simplefilter(\"ignore\")\n    fxn()\n    g = sns.scatterplot(train['GrLivArea'], train['SalePrice'], palette=\"Set1\", hue=train['Neighborhood'], legend='full', s=100)\n    sns.regplot(train['GrLivArea'], train['SalePrice'], data=train, scatter=False)  # Add a regression line to the scatterplot\n    # Put the legend out of the figure\n    _ = plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"82932a54a7b78e71c980c20916c9e1524fdfb882"},"cell_type":"markdown","source":"**Investigate the outliers**\n\n**Obervations:**\n* We see the two outliers at the top, with SalePrices of over 700,000\n* We also see two outliers with a large GrLivArea, over 4000 sq ft.  Odd given their respective SalePrice which you'd expect to be higher\n* They all have above average number of rooms (TotRmsAbvGrd)\n\n**Decisions:**\n* The higher priced houses seem to followign the trend, so we'll leave them alone\n* The lower priced houses with higher GrLivArea's seem like anomalies. Both are neighborhoods locates in Ames, near the university (https://www.zillow.com/ames-ia/) - so from the data and this brief analysis of the location I can't see why their prices would be so low.  So, let's investigate the outliers to double check on their details and if no surprises delete them.\n"},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"6d49a30ff788a8c30fd13722d125e0762411f060"},"cell_type":"code","source":"outliers_df = train.query('GrLivArea > 4500 | SalePrice > 700000').sort_values('SalePrice')\noutliers_df.sort_values('SalePrice', ascending=False)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"a3e1b10fc7ab5d0340afaed1783a5aab7996c899"},"cell_type":"code","source":"# Deleting Outliers\nprint('\\nBefore shape for {}: {}'.format('train', train.shape))\ntrain = train.drop(train[(train['GrLivArea'] > 4500) & (train['SalePrice'] < 200000)].index)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"53fe7904ffd61b10f275008a27680a8df0b69400"},"cell_type":"code","source":"# Check deletion\nprint('\\nAfter shape for {}: {}'.format('train', train.shape))\ntrain.sort_values(by = 'GrLivArea', ascending = False)[:2]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0c5f3e12eb80a99cf4fc25a280dbff92d876f30c"},"cell_type":"markdown","source":"#### 4.1.2.2 TotalBsmtSF"},{"metadata":{"scrolled":false,"trusted":false,"_uuid":"89a28b1eb7f3e71e160659051ef4f050b0ab2b8e"},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(20,10))\nwith warnings.catch_warnings():\n    warnings.simplefilter(\"ignore\")\n    fxn()\n    g = sns.scatterplot(train['TotalBsmtSF'], train['SalePrice'], palette=\"Set1\", hue=train['Neighborhood'], legend='full', s=100)\n    sns.regplot(train['TotalBsmtSF'], train['SalePrice'], data=train, scatter=False)  # Add a regression line to the scatterplot\n# Put the legend out of the figure\n_ = plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c096888b77d7652508e962122fa7b35eb08ee4c7"},"cell_type":"markdown","source":"**Investigate the outliers**"},{"metadata":{"_uuid":"07a0361a25afe281f07fa5bf3988a00e77d12e01"},"cell_type":"markdown","source":"**Observations:**\n* Fairly even number of observations either sie of the regression line\n* Some strange occurences with TotalBsmtSF > 3000 - but they look broadly on trend\n\n**Decisions:**\n* Do nothing"},{"metadata":{"_uuid":"cd2a4614d02dfb4cbac2da2929ecd393ad93beb4"},"cell_type":"markdown","source":"## 4.2 Target Variable\n\n### 4.2.1 Analysis\n\nMore analysis on SalePrice.\n\nLet's remind ourselves of the distribution."},{"metadata":{"trusted":false,"_uuid":"26fc3223c6a8e8ffcd38cfc562627fcb9f5b48bf"},"cell_type":"code","source":"sns.set(style=\"whitegrid\", palette='bright')\nfig, ax = plt.subplots(figsize=(12,6))\nwith warnings.catch_warnings():\n    warnings.simplefilter(\"ignore\")\n    fxn()\n    ax.xaxis.set_major_formatter(FuncFormatter(lambda x, _: '${:,.0F}'.format(x))) \n    p = sns.distplot(train['SalePrice'] , fit=stats.norm, ax=ax)\n    ax.set(xlabel='Sale Price', ylabel='Density')\n    sns.despine(trim=True)\n    p.set(xlim=(0, None), ylim=(0,None))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"419a4f8fc9f39005d1bdb8de834c8c37f4196a7b"},"cell_type":"markdown","source":"Now lets get the QQ-plot.\n\nThe Q-Q plot, or quantile-quantile plot, is a graphical tool to help us assess if a set of data plausibly came from some theoretical distribution \nsuch as a Normal or exponential. For example, if we run a statistical analysis that assumes our dependent variable is Normally distributed, \nwe can use a Normal Q-Q plot to check that assumption. It’s just a visual check, not an air-tight proof, so it is somewhat subjective.\nBut it allows us to see at-a-glance if our assumption is plausible, and if not, how the assumption is violated and what data points contribute \nto the violation.\nA Q-Q plot is a scatterplot created by plotting two sets of quantiles against one another. If both sets of quantiles came from the same distribution, \nwe should see the points forming a line that’s roughly straight. Here’s an example of a Normal Q-Q plot when both sets of quantiles truly come\nfrom Normal distributions."},{"metadata":{"trusted":false,"_uuid":"4ff08425d127818d5bfe6ec9735f70a3f96a2768"},"cell_type":"code","source":"# Get the QQ-plot\nfig = plt.figure(figsize=(12,6))\nres = stats.probplot(train['SalePrice'], plot=plt)\n# sns.despine(trim=True)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1194ef8b2e2ae78d54b9c45c07089ad2f0290d5f"},"cell_type":"markdown","source":"### 4.2.2 Log transformation of skewed values\n\nIf SalePrice were a normal distribution then we'd see a straight line.  We already know it's right skewed and this visual is a good way to examine it and also compare it.  For our model we want to make it more normal \n"},{"metadata":{"trusted":false,"_uuid":"0f1bbe8b677e03b955d7d94b73133781518aef98"},"cell_type":"code","source":"#We use the numpy fuction log1p which  applies log(1+x) to all elements of the column\ntrain[\"SalePrice\"] = np.log1p(train[\"SalePrice\"])\n\n#Check the new distribution \nfig, ax = plt.subplots(figsize=(12,6))\nwith warnings.catch_warnings():\n    warnings.simplefilter(\"ignore\")\n    fxn()\n    # ax.xaxis.set_major_formatter(FuncFormatter(lambda x, _: '${:,.0F}'.format(x))) \n    p = sns.distplot(train['SalePrice'] , fit=stats.norm, ax=ax)\n    ax.set(xlabel='Sale Price', ylabel='Density')\n    sns.despine(trim=True)\n    # p.set(xlim=(0, None), ylim=(0,None))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"dbec47f860c5f0e104cfa294786437ec9c902e83"},"cell_type":"code","source":"# Get the QQ-plot\nfig = plt.figure(figsize=(12,6))\nres = stats.probplot(train['SalePrice'], plot=plt)\n# sns.despine(trim=True)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0830947b2810b862162dd4210677aeac459c66b0"},"cell_type":"markdown","source":"# 5. Feature Engineering\n\n    5.1 Concatenation\n    5.2 NA's\n    5.3 Incorrect values\n    5.4 Label Encoding and Feature Conversion\n    5.5 Further Statistical transformation\n    5.6 Column removal\n    5.7 Creating features\n    5.8 Dummies\n    5.9 In-depth outlier detection\n    5.10 Overfit prevention\n    5.11 Baseline model"},{"metadata":{"_uuid":"429df2b87f0d0733bb7723f4a80296671662d746"},"cell_type":"markdown","source":"## 5.1 Concatenation\n\nSave the target variable then drop it from the train datset. Set up the train and test feature df's then concatenate them so we can apply changes to features across both in one statement."},{"metadata":{"trusted":false,"_uuid":"68e85b48d9ec355bff12e883fcfd58996bf05602"},"cell_type":"code","source":"y = train['SalePrice'].reset_index(drop=True)\ntrain_features = train.drop(['SalePrice'], axis=1)\ntest_features = test\nprint('train_features shape: ', train_features.shape)\nprint('train_features shape: ', test_features.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"f0aeeca064644834fd900b509cb1ee35ac969d7e"},"cell_type":"code","source":"features = pd.concat([train_features, test_features]).reset_index(drop=True)\nprint('features shape: ', features.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"72993d511f54b9efd3e30c66b749320d9776532d"},"cell_type":"markdown","source":"## 5.2 NA's\n\n      * How prevalent is missing data?\n      * Is there a pattern?\n\nWe need to consider remedial actions to deal with missing values which may include dropping columns or rows.\n\nExamine NA's by cateogory and impute them as best we can"},{"metadata":{"trusted":false,"_uuid":"e76ac839acca2719b95c6060aba9c14da749e46d"},"cell_type":"code","source":"# Analyse and print the features for Null values\nfeature_null_analysis('features', features, NULL_PERC_DROP_PERC)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"73141d8d91e9fff00ccd39d1f58b1d812afdcc0b"},"cell_type":"markdown","source":"* Functional: The documentation (https://ww2.amstat.org/publications/jse/v19n3/decock.pdf) says that we should assume \"Typ\", so lets impute that - it's the most common value\n* Electrical: The documentation doesn't give any information but obviously every house has this so let's impute the most common value: \"SBrkr\" - it't the most common value\n* KitchenQual: Similar to Electrical, most common value: \"TA\"\n* Exterior 1 and Exterior 2: Let's use the most common one here\n* SaleType: Similar to electrical, let's use most common value"},{"metadata":{"trusted":false,"_uuid":"e89dcdfe12d89f6d7787d8c40c23d14a66d477ab"},"cell_type":"code","source":"features['Functional'] = features['Functional'].fillna(features['Functional'].mode()[0])\nfeatures['Electrical'] = features['Electrical'].fillna(features['Electrical'].mode()[0])\nfeatures['KitchenQual'] = features['KitchenQual'].fillna(features['KitchenQual'].mode()[0])\nfeatures['Exterior1st'] = features['Exterior1st'].fillna(features['Exterior1st'].mode()[0])\nfeatures['Exterior2nd'] = features['Exterior2nd'].fillna(features['Exterior2nd'].mode()[0])\nfeatures['SaleType'] = features['SaleType'].fillna(features['SaleType'].mode()[0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bf7399c86768af9524ca2f56ed2c526ee143d3cd"},"cell_type":"markdown","source":"**PoolQC**\n\nCheck out observations where the property has a Pool but PoolQC is set to NA"},{"metadata":{"trusted":false,"_uuid":"5de64c25a45c455dfacbc2a0f7eee132bf262374"},"cell_type":"code","source":"features.query('PoolArea > 0 & PoolQC.isnull()', engine='python')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fcf94182f705e242d38ad7b6f7922154eca0ffc1"},"cell_type":"markdown","source":"From the data description we can impute the PoolQC from the OverallCond of the house:\n\nOverallCond: Rates the overall condition of the house\n\n       10\tVery Excellent\n       9\tExcellent\n       8\tVery Good\n       7\tGood\n       6\tAbove Average\t\n       5\tAverage\n       4\tBelow Average\t\n       3\tFair\n       2\tPoor\n       1\tVery Poor\n2418 - 4 - Below Average\n2501 - 6 - Above Average\t\n2597 - 3 - Fair\n       \nPoolQC: Pool quality\n\t\t\n       Ex\tExcellent\n       Gd\tGood\n       TA\tAverage/Typical\n       Fa\tFair\n       NA\tNo Pool\n\nSo we set PoolQC as follows:\n2418 - Fa\n2501 - Gd\n2597 - Fa \n"},{"metadata":{"trusted":false,"_uuid":"06976c70639c6630a10db1dd3e6a18fad43cb06a"},"cell_type":"code","source":"features.loc[2418, 'PoolQC'] = 'Fa'\nfeatures.loc[2501, 'PoolQC'] = 'Gd'\nfeatures.loc[2597, 'PoolQC'] = 'Fa'","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e54e7813f8dcf35710e038658ed048ae16e44e89"},"cell_type":"markdown","source":"**Garage Features**\n\nLet's check for houses with detached garages with all/some null garage features"},{"metadata":{"trusted":false,"_uuid":"6108176233dea74eb9e2e854471a1016ff590e21"},"cell_type":"code","source":"features.query('GarageType == \"Detchd\" & GarageYrBlt.isnull()', engine='python')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4d9f1d036c14aa2144b80ef13b4df91b71ff3e18"},"cell_type":"markdown","source":"So there are houses with garages that are detached but that have NaN's for all other Garage variables. Let's impute these manually too."},{"metadata":{"trusted":false,"_uuid":"37aa779756f8a866658a87e5557d41baf0f65edc"},"cell_type":"code","source":"features.loc[2124, 'GarageYrBlt'] = features.loc[2124, 'YearRemodAdd']\nfeatures.loc[2574, 'GarageYrBlt'] = features.loc[2574, 'YearRemodAdd']\n\nfeatures.loc[2124, 'GarageFinish'] = features['GarageFinish'].mode()[0]\nfeatures.loc[2574, 'GarageFinish'] = features['GarageFinish'].mode()[0]\n\nfeatures.loc[2574, 'GarageCars'] = features['GarageCars'].median()\n\nfeatures.loc[2124, 'GarageArea'] = features['GarageArea'].median()\nfeatures.loc[2574, 'GarageArea'] = features['GarageArea'].median()\n\nfeatures.loc[2124, 'GarageQual'] = features['GarageQual'].mode()[0]\nfeatures.loc[2574, 'GarageQual'] = features['GarageQual'].mode()[0]\n\nfeatures.loc[2124, 'GarageCond'] = features['GarageCond'].mode()[0]\nfeatures.loc[2574, 'GarageCond'] = features['GarageCond'].mode()[0]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4840873f7af2c43978887db6c5a306692c34389d"},"cell_type":"markdown","source":"**Basement Features**\n\n* BsmtQual\n* BsmtCond\n* BsmtExposure\n* BsmtFinType1\n* BsmtFinType2\n* BsmtFinSF1\n* BsmtFinSF2\n* BsmtUnfSF\n* TotalBsmtSF"},{"metadata":{"trusted":false,"_uuid":"c1203e7f74d874f4d8f9059cd542ce681811a41d"},"cell_type":"code","source":"# Create a new df focusing on where any basement figure are null\nbasement_features = ['BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinType2', 'BsmtFinSF1', 'BsmtFinSF2', 'BsmtUnfSF', 'TotalBsmtSF']\nbasement_df = features[basement_features]\nbasement_df_nulls = basement_df[basement_df.isnull().any(axis=1)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"5a87d71803c1ca5217755736e983880729e4da14"},"cell_type":"code","source":"#now select just the rows that have less then 5 NA's, meaning there is incongruency in the row\nbasement_df_nulls[(basement_df_nulls.isnull()).sum(axis=1) < 5]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"2c5369fdc8013a3178647efafba734ee3ded6420"},"cell_type":"code","source":"features.loc[332, 'BsmtFinType2'] = 'ALQ' #since SF2 smaller than SF1\nfeatures.loc[947, 'BsmtExposure'] = 'No' \nfeatures.loc[1485, 'BsmtExposure'] = 'No'\nfeatures.loc[2038, 'BsmtCond'] = 'TA'\nfeatures.loc[2183, 'BsmtCond'] = 'TA'\nfeatures.loc[2215, 'BsmtQual'] = 'Po' #v small basement so let's do Poor.\nfeatures.loc[2216, 'BsmtQual'] = 'Fa' #similar but a bit bigger.\nfeatures.loc[2346, 'BsmtExposure'] = 'No' #unfinished bsmt so prob not.\nfeatures.loc[2522, 'BsmtCond'] = 'Gd' #cause ALQ for bsmtfintype1\n# Analyse and print the features for Null values\nfeature_null_analysis('features', features, NULL_PERC_DROP_PERC)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f4f4a093786f264594b0dfacf9a7a5c73426cfa1"},"cell_type":"markdown","source":"**MSZoning**\n\nMSZoning: Identifies the general zoning classification of the sale.\n\t\t\n       A\tAgriculture\n       C\tCommercial\n       FV\tFloating Village Residential\n       I\tIndustrial\n       RH\tResidential High Density\n       RL\tResidential Low Density\n       RP\tResidential Low Density Park \n       RM\tResidential Medium Density\n       \nLet's find the mode for each MSSubClass (type of dwelling) and set NA values to the mode."},{"metadata":{"trusted":false,"_uuid":"938676bbdd8e616af239fc43344010e04ea3b500"},"cell_type":"code","source":"features.groupby('MSSubClass')['MSZoning'].apply(lambda x: x.mode()[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"f29f81a2e2d7c3cc4fea025c0a4d551a162eab0f"},"cell_type":"code","source":"features['MSZoning'] = features.groupby('MSSubClass')['MSZoning'].transform(lambda x: x.fillna(x.mode()[0]))\n# Analyse and print the features for Null values\nfeature_null_analysis('features', features, NULL_PERC_DROP_PERC)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"872bf7f466451b4a43a0e6cb7b51a5bd579d3212"},"cell_type":"markdown","source":"**MasVnrType & MasVnrArea**\n\nMasVnrType: Masonry veneer type\n\n       BrkCmn\tBrick Common\n       BrkFace\tBrick Face\n       CBlock\tCinder Block\n       None\tNone\n       Stone\tStone\n\nThere are a similar number of NA's - let check that they are the same observations and also look at the one observation which isn't."},{"metadata":{"trusted":false,"_uuid":"19e78f3d314c83c0d89bd1523c3a54cea12595df"},"cell_type":"code","source":"features.loc[features['MasVnrType'].isnull() & features['MasVnrArea'].notnull()]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"441737b81211352147b73f7be11ccd65185dd2b3"},"cell_type":"markdown","source":" The most common MasVnrType in the neighborhood 'Mitchel' is None. So, we'll set it to the most common overall, 'BrkFace;"},{"metadata":{"trusted":false,"_uuid":"ec6c6322b7880234be24e53e512d4b20b51b7696"},"cell_type":"code","source":"features.groupby('Neighborhood')['MasVnrType'].apply(lambda x: x.mode()[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"809fd498efdd5e753bc572ed5d82c3063d1e2527"},"cell_type":"code","source":"features.loc[2608, 'MasVnrType'] = 'BrkFace'\n# Analyse and print the features for Null values\nfeature_null_analysis('features', features, NULL_PERC_DROP_PERC)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c2c2e8370076d84e0531be7d2f4a0a3dd35f0583"},"cell_type":"markdown","source":"For the remainder of the categorical variables we'll set the values to 'None'"},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"95d2224715aac4bdc6bf368167a68781f4fe3fca"},"cell_type":"code","source":"categorical_features = list(features.select_dtypes(include='object'))\nfeatures.update(features[categorical_features].fillna('None'))\n# Analyse and print the features for Null values\nfeature_null_analysis('features', features, NULL_PERC_DROP_PERC)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"892489b99a74fc00fb4ce0482ca73f00046b2377"},"cell_type":"markdown","source":"**LotFrontage**\n\nSince the area of each street connected to the house property most likely have a similar area to other houses in its neighborhood , we can fill in missing values by the median LotFrontage of the neighborhood."},{"metadata":{"trusted":false,"_uuid":"c162e13651c0731a03ce837ac7e072e101a96b20"},"cell_type":"code","source":"features.groupby('Neighborhood')['LotFrontage'].apply(lambda x: x.median())","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"5944ffc3fdae580fe8fb648eb67e44c34d70fff4"},"cell_type":"code","source":"features['LotFrontage'] = features.groupby('Neighborhood')['LotFrontage'].transform(lambda x: x.median())\n# Analyse and print the features for Null values\nfeature_null_analysis('features', features, NULL_PERC_DROP_PERC)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"08b8d4ace21a50f1763064eb2ab6808bf39de1e5"},"cell_type":"markdown","source":"**GarageYrBlt**\n"},{"metadata":{"trusted":false,"_uuid":"403039221cf6cd0e003cd52f20b70d61c243e834"},"cell_type":"code","source":"garage_features = ['GarageType','GarageYrBlt','GarageFinish','GarageCars','GarageArea','GarageQual','GarageCond']\nfeatures.loc[features['GarageYrBlt'].isnull(), list(garage_features)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"b0471e6f663aafb8f785607ed23220fdbc3f662a"},"cell_type":"code","source":"# Check for any anomalies\nfeatures.loc[features['GarageYrBlt'].isnull() & features['GarageArea'] > 0]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"012d5a96fc584e247f0f455e7d10345fe18de775"},"cell_type":"markdown","source":"**MasVnrArea**\n"},{"metadata":{"trusted":false,"_uuid":"6dae296560dfb5125fee278a04e4aade40c6e2b9"},"cell_type":"code","source":"features[(features['MasVnrArea'].isnull())]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"84eedf9cf34cadde2a18c35c510cb0af6ee03056"},"cell_type":"markdown","source":"No anomalies,so we can impute the value 0 assuming that the properties do not have this feature."},{"metadata":{"trusted":false,"_uuid":"23c5a059b21862ea647ce0e4f86d97fc39e39c51"},"cell_type":"code","source":"numeric_features = list(features.select_dtypes(exclude='object'))\nfeatures.update(features[numeric_features].fillna(0))\n# Analyse and print the features for Null values\nfeature_null_analysis('features', features, NULL_PERC_DROP_PERC)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bded1dfb7aea1602a3ddd6132331cd5f9aa9c5a2"},"cell_type":"markdown","source":"## 5.3 Incorrect Values\n\nExamine min/max values to check validiy of data.\n"},{"metadata":{"trusted":false,"_uuid":"b31fe29cba29ea7210bbc2f215fb8f69c13d3b5f"},"cell_type":"code","source":"features.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3027ba2cd35dea3fdb924b4407312b9e53d4cba5"},"cell_type":"markdown","source":"Looking at the min and max of each variable there are some errors in the data.\n\n* GarageYrBlt - the max value is 2207, this is obviously wrong since the data is only until 2010.\n\nThe rest of the data looks fine. Let's inspect this row a bit more carefully and impute an approximate correct value."},{"metadata":{"trusted":false,"_uuid":"702d2ad85bb3940306d9e7620371039ad42c9729"},"cell_type":"code","source":"features[features['GarageYrBlt'] > 2100]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"fda5852b58cc7f4c1df1d12de368db4c50ea0492"},"cell_type":"code","source":"features.loc[2590, 'GarageYrBlt'] = 2007  # Assuming the garage was added when the house was remodelled and someone made a typo entering 2207 instead of 2007","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bb52e9f29e5858571f4a2c2d3b6f52eb63247cc2"},"cell_type":"markdown","source":"## 5.4 Further Statistical transformation - Transform Skewed Features\n\nFurther Statistical transformation:\n\n* Transfom skewed (numeric) features - we want to consider all numneric features but ignore numeric categorical variables:\n    * OverallQual\n    * OverallCond\n\n    5.1 Concatenation\n    5.2 NA's\n    5.3 Incorrect values\n    5.4 Further Statistical transformation\n    5.5 Label Encoding and Feature Conversion\n    5.6 Column removal\n    5.7 Creating features\n    5.8 Dummies\n    5.9 In-depth outlier detection\n    5.10 Overfit prevention\n    5.11 Baseline model"},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"21b6f298146de653ea6a638acc43878698efbae1"},"cell_type":"code","source":"categorical_numeric_features = ['OverallQual', 'OverallCond', 'MSSubClass']\nall_numeric_features = list(features.select_dtypes(exclude='object'))\n\n# Remove them from the list of numerics\ngen_numeric_features = [feat for feat in all_numeric_features if feat not in categorical_numeric_features]\n\n# get the skew measures\nskew_features = features[gen_numeric_features].apply(lambda x: skew(x)).sort_values(ascending=False)\nskew_features","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bd90d45f0a987e7e2b514d500772cc5a7b947405"},"cell_type":"markdown","source":"From Laurens ten Cate- use the boxcox1p transformation here because he tried the log transform first but a lot of skew remained in the data. So use boxcox1p over normal boxcox because boxcox can't handle zero values.\n\nI tried log1p and obtained slightly better skew figures for some but other were much worse - so I stuck with the method Laurens used."},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"d9851d1f90999c65df30d1c8999594a879f85531"},"cell_type":"code","source":"from scipy.special import boxcox1p\nfrom scipy.stats import boxcox_normmax\n\nhigh_skew = skew_features[skew_features > 0.5]\n\n# Log1p doesn't give as good results - try it!\n# for feat in high_skew.index:\n#     features[feat] = np.log1p(features[feat])\n\nfor feat in high_skew.index:\n    features[feat] = boxcox1p(features[feat], boxcox_normmax(features[feat]+1))\n\nskew_features2 = features[gen_numeric_features].apply(lambda x: skew(x)).sort_values(ascending=False)\nskew_features2","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"90268805b35076d0865f0f7d690776287078b975"},"cell_type":"markdown","source":"## 5.5 Label Encoding and Feature Conversion\n\nWe need to consider:\n\n* Numeric categorical features which have no order implied by the numbers - so we convert to an object data type so when we create dummy features these are included in scope\n* Numeric categorical features whch DO have order - so we leave these alone\n* Non-numeric categorical features which DO have an order implied by the numbers - so we encode them so order is retained\n\nWe can use the analysis we did in section 3.1 (the work we did to capture each feature in an Excel spreadsheet and categorise them). So we have the following:\n\nNumeric with no order implied - so we convert from numeric to an Object:\n* 'MSSubClass'\n\nNumeric with order implied - so we leave alone:\n* 'OverallQual', 'OverallCond'\n\nNon-numeric with order implied so we encode:\n* 'ExterQual', 'ExterCond', 'BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinType2', 'HeatingQC', 'KitchenQual', 'Functional', 'FireplaceQu', 'GarageFinish', 'GarageQual', 'GarageCond', 'PoolQC', 'Fence', 'LotShape', 'LandSlope', 'PavedDrive'"},{"metadata":{"trusted":false,"_uuid":"ca71691d2f38598ec618a5d17180fea5470ddf89"},"cell_type":"code","source":"# Convert numeric categorical features to Object dtype\nfeatures_to_convert = ['MSSubClass']\nfeatures[features_to_convert].dtypes\n\nfor feat in features_to_convert:\n    print('{} before - {}'.format(feat,features[feat].dtype), end=' -> ')\n    features[feat] = features[feat].astype('object')\n    print('{} after - {}'.format(feat,features[feat].dtype))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"c8384eac03d055a86f7262f86a6e03e2e2188fd2"},"cell_type":"code","source":"# Convert ordinal categorical \nfrom sklearn.preprocessing import LabelEncoder\n\nordinal_catorgorical_features = ['ExterQual', 'ExterCond', 'BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinType2', 'HeatingQC', 'KitchenQual', \n                                 'Functional', 'FireplaceQu', 'GarageFinish', 'GarageQual', 'GarageCond', 'PoolQC', 'Fence', 'LotShape', 'LandSlope', 'PavedDrive']\nprint('Shape all_data: {}'.format(features.shape))\nfeatures.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"4dfce260b8013ca7e997de2f49b3fd8559c3aab5"},"cell_type":"code","source":"# process columns, apply LabelEncoder to categorical features\nfor feat in ordinal_catorgorical_features:\n    lbl = LabelEncoder() \n    lbl.fit(list(features[feat].values)) \n    features[feat] = lbl.transform(list(features[feat].values))","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"7c56f90721d4589d7ebb97c7b7cdbf1414b98f21"},"cell_type":"code","source":"print('Shape all_data: {}'.format(features.shape))\nprint(features[ordinal_catorgorical_features].dtypes)\nfeatures.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2fa51bee967c1fac9225290528880b8d1c858ed5"},"cell_type":"markdown","source":"## 5.6 Column Removal - Incomplete Cases"},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"f81f73fb01a67abbc652d51dd8024f699746b174"},"cell_type":"code","source":"categorical_features = list(features.select_dtypes(include='object'))\ncat_value_counts = [(features[x].value_counts().iloc[0:3] / features.shape[0]) * 100 for x in categorical_features if (features[x].value_counts().iloc[0] / features.shape[0]) * 100 > 98]\n\nfor i,x in enumerate(cat_value_counts):\n    print(x)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c07e371f4c1da53e245f7251d56fa143a25c51d9"},"cell_type":"markdown","source":"From the above we can see that values of Street, Utilities, Condition2, RoofMatl, Heating and PoolQC have are, ay 98% for one value, highly skewed - so we will drop them. We can always rerun, adding them back in, to test the model accuracy. SO we drop the following:\n\n* Street\n* Utilities\n* Condition2\n* RoofMatl\n* Heating\n* PoolQC\n"},{"metadata":{"trusted":false,"_uuid":"86ae84c5550d56a6e64701cc7f429f6d7b8b3678"},"cell_type":"code","source":"cols_dropped = ['Street', 'Utilities', 'Condition2','RoofMatl', 'Heating', 'PoolQC']\nfeatures.drop(cols_dropped, axis=1, inplace=True)\nfeatures.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9c319c36ce96e138606bf2ec5e8fa5307ac15a7a"},"cell_type":"markdown","source":"## 5.7 Creating Features\n\nFull credit to Laurens for all of these.\n\nSize of House - taking into consideration:\n* BsmtFinSF1\n* BsmtFinSF2\n* 1stFlrSF\n* 2ndFlrSF\n\nBathrooms:\n* FullBath\n* HalfBath\n* BsmtFullBath\n* BsmtHalfBath\n\nPorch:\n* OpenPorchSF\n* EnclosedPorch\n* 3SsnPorch\n* Screenporch\n* WoodDeckSF\n\nMisc additional features:\n* haspool\n* has2ndfloor\n* hasgarage\n* hasbsmt\n* hasfireplace"},{"metadata":{"trusted":false,"_uuid":"45266ee5b04c44f4dcbf20eaca6b487966e1a01a"},"cell_type":"code","source":"features['Total_sqr_footage'] = (features['BsmtFinSF1'] + features['BsmtFinSF2'] +\n                                 features['1stFlrSF'] + features['2ndFlrSF'])\n\nfeatures['Total_Bathrooms'] = (features['FullBath'] + (0.5*features['HalfBath']) + \n                               features['BsmtFullBath'] + (0.5*features['BsmtHalfBath']))\n\nfeatures['Total_porch_sf'] = (features['OpenPorchSF'] + features['3SsnPorch'] +\n                              features['EnclosedPorch'] + features['ScreenPorch'] +\n                             features['WoodDeckSF'])\n\n\n#simplified features\nfeatures['haspool'] = features['PoolArea'].apply(lambda x: 1 if x > 0 else 0)\nfeatures['has2ndfloor'] = features['2ndFlrSF'].apply(lambda x: 1 if x > 0 else 0)\nfeatures['hasgarage'] = features['GarageArea'].apply(lambda x: 1 if x > 0 else 0)\nfeatures['hasbsmt'] = features['TotalBsmtSF'].apply(lambda x: 1 if x > 0 else 0)\nfeatures['hasfireplace'] = features['Fireplaces'].apply(lambda x: 1 if x > 0 else 0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2389535ff1b0f8bf97c73e2f8327b1cf2d2e6f55"},"cell_type":"markdown","source":"## 5.8 Creating Dummies\n\nWe have to convert our string features to dummy variables as sklearn.lm.fit() doesn't accept string features."},{"metadata":{"trusted":false,"_uuid":"d9c3c69abbe96bdb9562adea7a4c220f7018bff1"},"cell_type":"code","source":"features.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"694b5b568af026528bf6381a5cb04b8f86f3e611"},"cell_type":"code","source":"final_features = pd.get_dummies(features).reset_index(drop=True)\nfinal_features.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"5766bb509f17566aa76476d935ba0199286ac493"},"cell_type":"code","source":"final_features.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"58f8ab7a1fd3e446284eea31e3f34db86ca7d2a0"},"cell_type":"markdown","source":"Resplit the model data back into train and test"},{"metadata":{"trusted":false,"_uuid":"8c111624d69cfe05969bd1f02795c1ba7c190d04"},"cell_type":"code","source":"y.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"74fe298fb294982f6ab8cdc18a29e086bc5c682f"},"cell_type":"code","source":"X = final_features.iloc[:len(y),:]\ntesting_features = final_features.iloc[len(X):,:]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"1dcfbbaa7ca517b8dabd73a998f0f251ea8e3605"},"cell_type":"code","source":"print(X.shape)\nprint(testing_features.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1da534422ac5a078acdd6b5a3dc97e1faf4cc9c8"},"cell_type":"markdown","source":"## 5.9 Overfitting Prevention\n\n**Outliers**\n\n* Use the outlier test to calculate the Studentized residuals, the unadjusted p-value, and the corrected p-value.\n* Plot some of the values\n* Print the outlier indexes and values\n* drop the outliers from the train and test sets where the corrected p-value < 0.001"},{"metadata":{"trusted":false,"_uuid":"932f45a565686fa29585953df35e86ea778b46d3"},"cell_type":"code","source":"# Imports\nimport statsmodels.api as sm\n\nmodel = sm.OLS(endog = y, exog = X)\nresults = model.fit()\ntest = results.outlier_test()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"8ae03a1cf6dc2bdfdf79f366a4837de2e926e4b6"},"cell_type":"code","source":"# Change the last parameter of the plot to choose another feature\nimport statsmodels.graphics as smgraphics\nfigure = smgraphics.regressionplots.plot_fit(results, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"c48919ec7cef1d3e14a2f1cc1a01fadcabef0d00"},"cell_type":"code","source":"pd.set_option('display.float_format', lambda x: '{:.9f}'.format(x)) #Limiting floats output to 3 decimal points\nprint('Outliers:')\ntest.loc[test['bonf(p)'] < 0.001]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"0ab5a0c83233342aa556a7c6ffd0033e8207f621"},"cell_type":"code","source":"# Check shape before outlier drop\nprint('X shape before: {}'.format(X.shape))\nprint('y shape before: {}'.format(y.shape))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"b4e3c23b22a8bebc52414bd913af655c67c51bc6"},"cell_type":"code","source":"# Drop the outliers\noutlier_indexes = list(test.loc[test['bonf(p)'] < 0.001].index)\nX = X.drop(X.index[outlier_indexes])\ny = y.drop(y.index[outlier_indexes])\nprint('X shape after: {}'.format(X.shape))\nprint('y shape after: {}'.format(y.shape))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"7751168e8c41a1a75811e58cb0e0897f1d015ca2"},"cell_type":"code","source":"# Create the ovefit list\ndf_overfit = pd.DataFrame(columns=['feature', 'feature mode %'])\nfor feat in X.columns:\n    if (X[feat].value_counts().iloc[0] / X.shape[0] * 100) > 97.5:\n        df_overfit = df_overfit.append({'feature': feat, 'feature mode %': (X[feat].value_counts().iloc[0] / X.shape[0] * 100)}, ignore_index=True)\n\n# Print the overfit list and shape of train and test sets\nfor i, row in df_overfit.iterrows():\n    print('Feature: {: <{WIDTH}}   >   % of Values: {: <{WIDTH}}'.format(row[0], row[1], WIDTH=PRINT_WIDTH))\nprint('\\nX shape before: {}'.format(X.shape))\nprint('testing_features shape before: {}'.format(testing_features.shape))\n\n# Create list of overfit features\n\noverfit_features = list(df_overfit['feature'])\nprint('No. of Features: {}'.format(len(overfit_features)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"526f9f3cbfabae2ef5ae24e3379e03f57e602179"},"cell_type":"code","source":"X = X.drop(overfit_features, axis=1)\ntesting_features = testing_features.drop(overfit_features, axis=1)\nprint('X shape after: {}'.format(X.shape))\nprint('testing_features shape after: {}'.format(testing_features.shape))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c40db2dedabcfa97b0628b5bb522a304cb65fedc"},"cell_type":"markdown","source":"## 5.10 Baseline Model\n\nFull Model w/ kfold cross validation."},{"metadata":{"trusted":false,"_uuid":"5fd09390e5bc458298158b4ec0150bea5058110f"},"cell_type":"code","source":"from sklearn.linear_model import LinearRegression\nfrom sklearn.model_selection import KFold, cross_val_score\nfrom sklearn.model_selection import cross_val_predict\nfrom sklearn.preprocessing import RobustScaler\nfrom sklearn.pipeline import make_pipeline\n\n#Build our model method\nlm = LinearRegression()\n\n#Build our cross validation method\nkfolds = KFold(n_splits=10, shuffle=True, random_state=23)\n\n#build our model scoring function\ndef cv_rmse(model):\n    rmse = np.sqrt(-cross_val_score(model, X, y, \n                                   scoring=\"neg_mean_squared_error\", \n                                   cv = kfolds))\n    return(rmse)\n\n\n#second scoring metric\ndef cv_rmsle(model):\n    rmsle = np.sqrt(np.log(-cross_val_score(model, X, y, scoring = 'neg_mean_squared_error', cv=kfolds)))\n    return(rmsle)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"7b4932d3a0dd3d5087fc9389efa043750d8671ae"},"cell_type":"code","source":"benchmark_model = make_pipeline(RobustScaler(),lm).fit(X=X, y=y)\ncv_rmse(benchmark_model).mean()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5fbf962f1013cb4c3ee05aa96ac39aa4f9198cb9"},"cell_type":"markdown","source":"# 6. Feature Selection\n\n    6.1 Filter Methods\n        6.1.1 Baseline Coefficients\n    6.2 Embedded methods\n        6.2.1 L2: Ridge Regression\n        6.2.2 L1: Lasso regression\n            In-depth coefficient analysis\n    6.3 Elasticnet\n    6.4 XGBoost\n    6.5 SVR\n    6.6 LightGBM\n\n7. Ensemble methods\n\n    7.1 Stacked generalizations\n    7.2 Averaging\n    7.3 Standard\n    7.4 Weighted"},{"metadata":{"_uuid":"da4ee75b9a48fc7d82d4253baa8be967b1a5dff0"},"cell_type":"markdown","source":"## 6.1 Filter Methods\n\n### 6.1.1 Baseline Coeficients"},{"metadata":{"scrolled":true,"trusted":false,"_uuid":"e0e2311e007f0a5e04f336835cd4b03309d01e52"},"cell_type":"code","source":"coeffs = pd.DataFrame(list(zip(X.columns, benchmark_model.steps[1][1].coef_)), columns=['Predictors', 'Coefficients'])\n\ncoeffs.sort_values(by='Coefficients', ascending=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eb5c375634669d487cb1e6921cd96300559dcaaf"},"cell_type":"markdown","source":"### 6.2 Embedded Methods\n\n### 6.2.1 L2: Ridge Regression"},{"metadata":{"trusted":false,"_uuid":"a6ffbd85f26d6adf1c2daaf1059601a28a35e770"},"cell_type":"code","source":"from sklearn.linear_model import RidgeCV\n\ndef ridge_selector(k):\n    ridge_model = make_pipeline(RobustScaler(),\n                                RidgeCV(alphas = [k],\n                                        cv=kfolds)).fit(X, y)\n    \n    ridge_rmse = cv_rmse(ridge_model).mean()\n    return(ridge_rmse)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"1778cd92a7b326a285aa8d23d76b29fd55658c1a"},"cell_type":"code","source":"r_alphas = [.0001, .0003, .0005, .0007, .0009, \n          .01, 0.05, 0.1, 0.3, 1, 3, 5, 10, 15, 20, 30, 50, 60, 70, 80]\n\nridge_scores = []\nfor alpha in r_alphas:\n    score = ridge_selector(alpha)\n    ridge_scores.append(score)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"dcb986df4076ed56d115e82f25034a1cadefd5d6"},"cell_type":"code","source":"plt.plot(r_alphas, ridge_scores, label='Ridge')\nplt.legend('center')\nplt.xlabel('alpha')\nplt.ylabel('score')\n\nridge_score_table = pd.DataFrame(ridge_scores, r_alphas, columns=['RMSE'])\nridge_score_table","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"76bacedd51b546c72cfac40dc5a597baa4c6115d"},"cell_type":"code","source":"alphas_alt = [7.5, 7.6, 7.7, 7.8, 7.9, 8, 8.1, 8.2, 8.3, 8.4, 8.5, 8.6, 8.7, 8.8, 8.9, 9, 9.1, 9.2, 9.3, 9.4, 9.5, 9.6, 9.7, 9.8 , \n              9.9, 10]\n# alphas_alt = [14, 14.1, 14.2, 14.3, 14.4, 14.5, 14.6, 14.7, 14.8, 14.9, 15, 15.1, 15.2, 15.3, 15.4, 15.5, 15.6, 15.7, 15.8 , 15.9,\n#              16, 16.1, 16.2, 16.3, 16.4, 16.5]\n\nridge_model2 = make_pipeline(RobustScaler(),\n                            RidgeCV(alphas = alphas_alt,\n                                    cv=kfolds)).fit(X, y)\n\ncv_rmse(ridge_model2).mean()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"754745cf798596770541cff45efbadaec6aa6235"},"cell_type":"code","source":"ridge_model2.steps[1][1].alpha_","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8348eab8f4a49493e87c1c8208bc2d01c97de6a7"},"cell_type":"markdown","source":"### 6.2.2 Lasso Regression (L1 penalty)"},{"metadata":{"trusted":false,"_uuid":"05b70884083682e150fbe6ca5e62150367048068"},"cell_type":"code","source":"from sklearn.linear_model import LassoCV\n\n\nalphas = [0.00005, 0.0001, 0.0003, 0.0005, 0.0007, \n          0.0009, 0.01]\nalphas2 = [0.00005, 0.0001, 0.0002, 0.0003, 0.0004, 0.0005,\n           0.0006, 0.0007, 0.0008]\n\n\nlasso_model2 = make_pipeline(RobustScaler(),\n                             LassoCV(max_iter=1e7,\n                                    alphas = alphas2,\n                                    random_state = 42)).fit(X, y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"31b3e067cc8994dda8419b151cb23a193974221a"},"cell_type":"code","source":"scores = lasso_model2.steps[1][1].mse_path_\n\nplt.plot(alphas2, scores, label='Lasso')\nplt.legend(loc='center')\nplt.xlabel('alpha')\nplt.ylabel('RMSE')\nplt.tight_layout()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"951e5bba5860e33b8837ff86de80d347fb48c8bd"},"cell_type":"code","source":"lasso_model2.steps[1][1].alpha_","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"343473b4cb6188a416cdc8e2ea3283e02a01ed85"},"cell_type":"code","source":"cv_rmse(lasso_model2).mean()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"69bfba95f4978ef8b8a3d481b2a0297a0ec12822"},"cell_type":"code","source":"coeffs = pd.DataFrame(list(zip(X.columns, lasso_model2.steps[1][1].coef_)), columns=['Predictors', 'Coefficients'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"ad545f1328c948f1e4173e1484453b41fb4c3178"},"cell_type":"code","source":"used_coeffs = coeffs[coeffs['Coefficients'] != 0].sort_values(by='Coefficients', ascending=False)\nprint(used_coeffs.shape)\nprint(used_coeffs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"abb15870206b80f60d1775d307bd2a9fad13acc1"},"cell_type":"code","source":"used_coeffs_values = X[used_coeffs['Predictors']]\nused_coeffs_values.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"405d569326c3909e35a660a986c2e1b75036b77e"},"cell_type":"code","source":"overfit_test2 = []\nfor i in used_coeffs_values.columns:\n    counts2 = used_coeffs_values[i].value_counts()\n    zeros2 = counts2.iloc[0]\n    if zeros2 / len(used_coeffs_values) * 100 > 99.5:\n        overfit_test2.append(i)\n        \noverfit_test2","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"37db1f9df3c741c9ce4bf6074d3aa902968d1bdf"},"cell_type":"markdown","source":"## 6.3 Elastic Net (L1 and L2 penalty)"},{"metadata":{"trusted":false,"_uuid":"0ec21ceb404289232e640ffdc43fc391190f9e50"},"cell_type":"code","source":"from sklearn.linear_model import ElasticNetCV\n\ne_alphas = [0.0001, 0.0002, 0.0003, 0.0004, 0.0005, 0.0006, 0.0007]\ne_l1ratio = [0.75, 0.8, 0.85, 0.9, 0.95, 0.99, 1]\n\nelastic_cv = make_pipeline(RobustScaler(), \n                           ElasticNetCV(max_iter=1e7, alphas=e_alphas, \n                                        cv=kfolds, l1_ratio=e_l1ratio))\n\nelastic_model3 = elastic_cv.fit(X, y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"15a0b74e8cbe09011536f27a64cd6846777b9893"},"cell_type":"code","source":"cv_rmse(elastic_model3).mean()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"214b302c23c409f12a81ee6ee85abdb7e9af1a8b"},"cell_type":"code","source":"print(elastic_model3.steps[1][1].l1_ratio_)\nprint(elastic_model3.steps[1][1].alpha_)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a8651a8a415274cf935dc25166e5e602581ec4e4"},"cell_type":"markdown","source":"## 6.4 XGboost"},{"metadata":{"trusted":false,"_uuid":"3c2da7827032bdc4b1c05e38c541dc9ba58d0eab"},"cell_type":"code","source":"from sklearn.model_selection import GridSearchCV\nfrom matplotlib.pylab import rcParams\nrcParams['figure.figsize'] = 12, 4\n%matplotlib inline\nimport xgboost as xgb\nfrom xgboost import XGBRegressor","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"f5bdf294cca26b1143c9f4db9081fc89a9b65341"},"cell_type":"code","source":"from sklearn.metrics import mean_squared_error\n\ndef modelfit(alg, dtrain, target, useTrainCV=True, \n             cv_folds=5, early_stopping_rounds=50):\n    \n    if useTrainCV:\n        xgb_param = alg.get_xgb_params()\n        xgtrain = xgb.DMatrix(dtrain.values, \n                              label=y.values)\n        \n        print(\"\\nGetting Cross-validation result..\")\n        cvresult = xgb.cv(xgb_param, xgtrain, \n                          num_boost_round=alg.get_params()['n_estimators'], \n                          nfold=cv_folds,metrics='rmse', \n                          early_stopping_rounds=early_stopping_rounds,\n                          verbose_eval = True)\n        alg.set_params(n_estimators=cvresult.shape[0])\n    \n    #Fit the algorithm on the data\n    print(\"\\nFitting algorithm to data...\")\n    alg.fit(dtrain, target, eval_metric='rmse')\n        \n    #Predict training set:\n    print(\"\\nPredicting from training data...\")\n    dtrain_predictions = alg.predict(dtrain)\n        \n    #Print model report:\n    print(\"\\nModel Report\")\n    print(\"RMSE : %.4g\" % np.sqrt(mean_squared_error(target.values,\n                                             dtrain_predictions)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"c61b59df0d32b90cd9d1df5580c7f59a6684628e"},"cell_type":"code","source":"xgb3 = XGBRegressor(learning_rate =0.01, n_estimators=3460, max_depth=3,\n                     min_child_weight=0 ,gamma=0, subsample=0.7,\n                     colsample_bytree=0.7,objective= 'reg:linear',\n                     nthread=4,scale_pos_weight=1,seed=27, reg_alpha=0.00006)\n\nxgb_fit = xgb3.fit(X, y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"381fad5f0ce97d455e51ae0478a67f8840ba79ba"},"cell_type":"code","source":"cv_rmse(xgb_fit).mean()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"54891cbc95c7eb26fb6ce4acd136cb6eab737241"},"cell_type":"markdown","source":"## 6.5 Support Vector Regression"},{"metadata":{"trusted":false,"_uuid":"c108f9d1aa710bec3824d96e952d052e24797180"},"cell_type":"code","source":"from sklearn import svm\nsvr_opt = svm.SVR(C = 100000, gamma = 1e-08)\n\nsvr_fit = svr_opt.fit(X, y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"e2f4efd2f1cf5b5c8c9b065e045f11769a259d0f"},"cell_type":"code","source":"cv_rmse(svr_fit).mean()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5e4c5edc967559d9345aafdede383b39972a712a"},"cell_type":"markdown","source":"## 6.6 LightGBM"},{"metadata":{"trusted":false,"_uuid":"cdd1923a9d0b64949243a435b3abdf7598595ad8"},"cell_type":"code","source":"from lightgbm import LGBMRegressor\n\nlgbm_model = LGBMRegressor(objective='regression',num_leaves=5,\n                              learning_rate=0.05, n_estimators=720,\n                              max_bin = 55, bagging_fraction = 0.8,\n                              bagging_freq = 5, feature_fraction = 0.2319,\n                              feature_fraction_seed=9, bagging_seed=9,\n                              min_data_in_leaf =6, min_sum_hessian_in_leaf = 11)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"956b9880d549581c80e4c04fd1802c4bd3acddfb"},"cell_type":"code","source":"cv_rmse(lgbm_model).mean()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"8b65edc53a7e3a783b2e5d98d41ce97112a437dd"},"cell_type":"code","source":"lgbm_fit = lgbm_model.fit(X, y)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"11d6a2461417cbfd76044e124cda2f32198a7e9e"},"cell_type":"markdown","source":"# 7. Ensemble\n\n## 7.1 Ensemble - Stacking"},{"metadata":{"trusted":false,"_uuid":"8e208431182f23961c8eab9f7d4218388a1d2f0c"},"cell_type":"code","source":"from mlxtend.regressor import StackingCVRegressor\nfrom sklearn.pipeline import make_pipeline\n\n#setup models\nridge = make_pipeline(RobustScaler(), \n                      RidgeCV(alphas = alphas_alt, cv=kfolds))\n\nlasso = make_pipeline(RobustScaler(),\n                      LassoCV(max_iter=1e7, alphas = alphas2,\n                              random_state = 42, cv=kfolds))\n\nelasticnet = make_pipeline(RobustScaler(), \n                           ElasticNetCV(max_iter=1e7, alphas=e_alphas, \n                                        cv=kfolds, l1_ratio=e_l1ratio))\n\nlightgbm = make_pipeline(RobustScaler(),\n                        LGBMRegressor(objective='regression',num_leaves=5,\n                                      learning_rate=0.05, n_estimators=720,\n                                      max_bin = 55, bagging_fraction = 0.8,\n                                      bagging_freq = 5, feature_fraction = 0.2319,\n                                      feature_fraction_seed=9, bagging_seed=9,\n                                      min_data_in_leaf =6, \n                                      min_sum_hessian_in_leaf = 11))\n\nxgboost = make_pipeline(RobustScaler(),\n                        XGBRegressor(learning_rate =0.01, n_estimators=3460, \n                                     max_depth=3,min_child_weight=0 ,\n                                     gamma=0, subsample=0.7,\n                                     colsample_bytree=0.7,\n                                     objective= 'reg:linear',nthread=4,\n                                     scale_pos_weight=1,seed=27, \n                                     reg_alpha=0.00006))\n\n\n#stack\nstack_gen = StackingCVRegressor(regressors=(ridge, lasso, elasticnet, \n                                            xgboost, lightgbm), \n                               meta_regressor=xgboost,\n                               use_features_in_secondary=True)\n\n#prepare dataframes\nstackX = np.array(X)\nstacky = np.array(y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"c33bb4224435d44364f4caa2a293bb72b25b20d7"},"cell_type":"code","source":"#scoring \n\nprint(\"cross validated scores\")\n\nfor model, label in zip([ridge, lasso, elasticnet, xgboost, lightgbm, stack_gen],\n                     ['RidgeCV', 'LassoCV', 'ElasticNetCV', 'xgboost', 'lightgbm',\n                      'StackingCVRegressor']):\n    \n    SG_scores = cross_val_score(model, stackX, stacky, cv=kfolds,\n                               scoring='neg_mean_squared_error')\n    print(\"RMSE\", np.sqrt(-SG_scores.mean()), \"SD\", scores.std(), label)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"11e8afae5e204b159fbe55e23355bdff12eccbbb"},"cell_type":"code","source":"stack_gen_model = stack_gen.fit(stackX, stacky)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"55023cc7bdf7376ec34b80a8b407c0f245fc9910"},"cell_type":"markdown","source":"## 7.2 Ensemble - Averaging"},{"metadata":{"trusted":false,"_uuid":"9ce07c27da458b91f7190f6c36f36589b7b44b5a"},"cell_type":"code","source":"em_preds = elastic_model3.predict(testing_features)\nlasso_preds = lasso_model2.predict(testing_features)\nridge_preds = ridge_model2.predict(testing_features)\nstack_gen_preds = stack_gen_model.predict(testing_features)\nxgb_preds = xgb_fit.predict(testing_features)\nsvr_preds = svr_fit.predict(testing_features)\nlgbm_preds = lgbm_fit.predict(testing_features)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"e216dc84e29defc06c945342657b5d7bdf27fd98"},"cell_type":"code","source":"stack_preds = ((0.2*em_preds) + (0.1*lasso_preds) + (0.2*ridge_preds) + \n               (0.2*xgb_preds) + (0.1*lgbm_preds) + (0.2*stack_gen_preds))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"8a2fd67b6c3da9ae87399f909134e2f974606e8c"},"cell_type":"code","source":"submission = pd.read_csv('../input/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"eaa7bbe5b295ea4f0fd7f5b03e7d618e2a114b56"},"cell_type":"code","source":"submission.iloc[:,1] = np.expm1(stack_preds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"415733679388cfab4e70f1d745c5f01cba3ad673"},"cell_type":"code","source":"submission.to_csv(\"final_submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"9b2d2c64c0f16a02ca1f380be098c93a0157edd9"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}