{ "cells": [ { "cell_type": "markdown", "metadata": { "_cell_guid": "b59df2dc-c353-c01c-3da4-60c477b86a5e" }, "source": [ "House Prices Advanced Regression Techniques\n", "----------\n", "\n", "## Summary\n", "You never knew you wanted to go to Ames, Iowa but by the end you will know more than you ever thought possible from a data set compiled by Dean De Cock. There are many cities that have open data sets on housing, like my local town of Burlington Vermont, but in this Kaggle competition we are looking to explore the possible relationship of the sale price to all the other features of a house.\u00a0\n", "\n", "Many but not all of the features of a house share a linear relationship with the Sale Price. By filling missing values, some feature engineering and feature selection determining the Sales Price should not be that far off. In this case I tried applying Random Forest Regression, Lasso and Ridge Regressions to arrive at my final answer \n", "\n", "## Goal\n", "Use the dataset after normalization, filling in missing values and adding new features with a linear regression model to predict sales prices.\n", "\n", "## Method\n", "There are a large number of features in this data set and in order to create a good prediction there will have to be a fair amount of work to do in advance. Combining the train and test set will means changes only need to be done once across both data sets. We will be exploring the following steps:\n", "* Correlation\n", "* Feature Exploration\n", "* Missing Values\n", "* Data Normilization\n", "* Feature Engineering\n", "* Assembling dataset\n", "* Prediction\n", "\n", "There should be several good indicators for sale price, the challenge will be in normalization of the data and feature engineering." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "2d34f629-19d9-c16e-da5b-18766091db5d" }, "outputs": [], "source": [ "# Handle table-like data and matrices\n", "import numpy as np\n", "import pandas as pd\n", "import math\n", "\n", "# Modelling Algorithms\n", "from sklearn.tree import DecisionTreeClassifier\n", "from sklearn.linear_model import LogisticRegression, LassoLarsCV,Ridge\n", "from sklearn.neighbors import KNeighborsClassifier\n", "from sklearn.naive_bayes import GaussianNB\n", "from sklearn.svm import SVC, LinearSVC\n", "from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier, RandomForestRegressor\n", "\n", "# Modelling Helpers\n", "from sklearn.preprocessing import Imputer , Normalizer , scale, OneHotEncoder\n", "from sklearn.cross_validation import train_test_split , StratifiedKFold\n", "from sklearn.feature_selection import RFECV\n", "from sklearn.preprocessing import LabelEncoder,OneHotEncoder\n", "\n", "# Visualisation\n", "import matplotlib \n", "import matplotlib.pyplot as plt\n", "import matplotlib.pylab as pylab\n", "import seaborn as sns\n", "\n", "# Configure visualisations\n", "%matplotlib inline" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "2d11ce1e-9162-11ad-c6ff-0062cb47b27e" }, "outputs": [], "source": [ "traindf = pd.read_csv(\"../input/train.csv\")\n", "testdf = pd.read_csv(\"../input/test.csv\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "bd3173b0-f4f5-16ad-6bb5-703a021debe9" }, "outputs": [], "source": [ "# create a single dataframe of both the training and testing data\n", "wholedf = pd.concat([traindf,testdf])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "79976670-f0fa-95ec-bf26-ef1516d922ee" }, "outputs": [], "source": [ "wholedf.info()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "30de2da9-8a62-5a80-cdc5-44eb1c211bfb" }, "outputs": [], "source": [ "wholedf.head(5)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "c51c0dca-b195-76c8-8c27-ab828bd8106a" }, "source": [ "The Data\n", "-----------\n", "The data has a wide range of integer, float and categorical information. In the end those will end up as numeric, but not quite yet. There are a couple of groupings of different types of measurements and understanding their differences will help in determining what new features can be created.\n", "\n", "Neighborhood\u200a\u2014\u200aInformation about the neighborhood, zoning and lot.\n", "Examples: MSSubClass, LandContour, Neighborhood, BldgType\n", "\n", "Dates\u200a\u2014\u200aTime based data about when it was built, remodeled or sold.\n", "Example: YearBuilt, YearRemodAdd, GarageYrBlt, YrSold\n", "\n", "Quality/Condition\u200a\u2014\u200aThere are categorical assessment of the various features of the houses, most likely from the property assessor.\n", "Example: PoolQC, SaleCondition,GarageQual, HeatingQC\n", "\n", "Property Features\u200a\u2014\u200aCategorical collection of additional features and attributes of the building\n", "Example: Foundation, Exterior1st, BsmtFinType1,Utilities\n", "\n", "Square Footage\u200a\u2014\u200aArea measurement of section of the building and features like porches and lot area(which is in acres)\n", "Example: TotalBsmtSF, GrLivArea, GarageArea, PoolArea, LotArea\n", "\n", "Room/Feature Count\u200a\u2014\u200aQuantitative counts of features (versus categorical) like rooms, prime candidate for feature engineering\n", "Example: FullBath, BedroomAbvGr, Fireplaces,GarageCars\n", "\n", "Pricing\u200a\u2014\u200aMonetary values, one of which is the sales price we are trying to determine\n", "Examples: SalePrice, MiscVal" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "ae5a41e9-72ba-a3da-063e-12890b58a792" }, "outputs": [], "source": [ "sns.lmplot(x=\"GrLivArea\", y=\"SalePrice\", data=wholedf);\n", "plt.title(\"Linear Regression of Above Grade Square Feet and Sale Price\")\n", "plt.ylim(0,)\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "a27e04a3-1794-54bb-b024-41b2c1eb5be7" }, "outputs": [], "source": [ "sns.lmplot(x=\"1stFlrSF\", y=\"SalePrice\", data=wholedf);\n", "plt.title(\"Linear Regression of First Floor Square Feet and Sale Price\")\n", "plt.ylim(0,)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "d0fa7ab1-eead-ceac-f479-dac96a76d051" }, "source": [ "Correlation\n", "--------------\n", "A quick correlation check is the best way to the heart of the data set. There is a far amount of correlation for sales price with a couple of variables:\n", "\n", "* OverallQual - 0.790982\n", "* GrLivArea - 0.708624\n", "* GarageCars - 0.640409\n", "* GarageArea - 0.623431\n", "* TotalBsmtSF - 0.613581\n", "* 1stFlrSF - 0.605852\n", "* FullBath - 0.560664\n", "* TotRmsAbvGrd - 0.533723\n", "* YearBuilt - 0.522897\n", "* YearRemodAdd - 0.507101" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "1e613413-1bcf-11a2-58ea-a3f123646955" }, "outputs": [], "source": [ "traincorr = traindf.corr()['SalePrice']\n", "# convert series to dataframe so it can be sorted\n", "traincorr = pd.DataFrame(traincorr)\n", "# correct column label from SalePrice to correlation\n", "traincorr.columns = [\"Correlation\"]\n", "# sort correlation\n", "traincorr2 = traincorr.sort_values(by=['Correlation'], ascending=False)\n", "traincorr2.head(15)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "5ec42ca1-c8a1-f059-80e9-9308562b64ca" }, "outputs": [], "source": [ "corr = wholedf.corr()\n", "sns.heatmap(corr)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "02842225-8a7b-9356-9f58-b60920ed2263" }, "source": [ "Missing Values\n", "--------------------\n", "There is a wide selection of missing values. First it make sense to hit the low hanging fruit first and deal with those that are missing a value or two, then work through what is left. Some of these look to be missing values not because they don\u2019t have data but rather because the building was missing that feature, like a garage. Using Pandas Get.Dummies will sort that problem out into true/false values" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "8fa6fba0-8b5d-09d6-470b-db8345b1de03" }, "outputs": [], "source": [ "countmissing = wholedf.isnull().sum().sort_values(ascending=False)\n", "percentmissing = (wholedf.isnull().sum()/wholedf.isnull().count()).sort_values(ascending=False)\n", "wholena = pd.concat([countmissing,percentmissing], axis=1)\n", "wholena.head(36)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "28b883d3-7be2-d667-d35c-60363998771e" }, "source": [ "Replacing Missing Data\n", "------------\n", "For the categorical information that is missing a single values a quick check shows which ones are dominant and manually replace the missing values. For most it is a quick process, I commented out the Python as I had in order to keep this shorter but feel free to uncomment and take a look." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "14ded018-d126-6cf9-3c17-e186615c2174" }, "outputs": [], "source": [ "#wholedf[[\"Utilities\", \"Id\"]].groupby(['Utilities'], as_index=False).count()\n", "wholedf['Utilities'] = wholedf['Utilities'].fillna(\"AllPub\")\n", "\n", "# wholedf[[\"Electrical\", \"Id\"]].groupby(['Electrical'], as_index=False).count()\n", "wholedf['Electrical'] = wholedf['Electrical'].fillna(\"SBrkr\")\n", "\n", "# wholedf[[\"Exterior1st\", \"Id\"]].groupby(['Exterior1st'], as_index=False).count()\n", "wholedf['Exterior1st'] = wholedf['Exterior1st'].fillna(\"VinylSd\")\n", "\n", "#wholedf[[\"Exterior2nd\", \"Id\"]].groupby(['Exterior2nd'], as_index=False).count()\n", "wholedf['Exterior2nd'] = wholedf['Exterior2nd'].fillna(\"VinylSd\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "6a5417d7-47fc-3027-2613-b70aa2208cd0" }, "outputs": [], "source": [ "# Missing interger values replace with the median in order to return an integer\n", "wholedf['BsmtFullBath']= wholedf.BsmtFullBath.fillna(wholedf.BsmtFullBath.median())\n", "wholedf['BsmtHalfBath']= wholedf.BsmtHalfBath.fillna(wholedf.BsmtHalfBath.median())\n", "wholedf['GarageCars']= wholedf.GarageCars.fillna(wholedf.GarageCars.median())\n", "\n", "# Missing float values were replaced with the mean for accuracy \n", "wholedf['BsmtUnfSF']= wholedf.BsmtUnfSF.fillna(wholedf.BsmtUnfSF.mean())\n", "wholedf['BsmtFinSF2']= wholedf.BsmtFinSF2.fillna(wholedf.BsmtFinSF2.mean())\n", "wholedf['BsmtFinSF1']= wholedf.BsmtFinSF1.fillna(wholedf.BsmtFinSF1.mean())\n", "wholedf['GarageArea']= wholedf.GarageArea.fillna(wholedf.GarageArea.mean())\n", "wholedf['MasVnrArea']= wholedf.MasVnrArea.fillna(wholedf.MasVnrArea.mean())" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "d92a7940-d4c6-942b-9de8-7c32d9b7c3e3" }, "source": [ "### Infer Missing Values\n", "Some of the missing values can be inferred from other values for that given property. The GarageYearBuilt would have to be at the earliest the year the house was built. Likewise TotalBasementSQFeet would have to be equal to the first floor square footage." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "01d1afea-7010-859c-5695-2b5228f731a3" }, "outputs": [], "source": [ "wholedf.GarageYrBlt.fillna(wholedf.YearBuilt, inplace=True)\n", "wholedf.TotalBsmtSF.fillna(wholedf['1stFlrSF'], inplace=True)\n", "\n", "sns.lmplot(x=\"TotalBsmtSF\", y=\"1stFlrSF\", data=wholedf)\n", "plt.title(\"Linear Regression of Basement SF and 1rst Floor SQ \")\n", "plt.xlim(0,)\n", "plt.ylim(0,)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "c594980f-a87c-bfbe-c071-4c726814c269" }, "source": [ "### Lot Frontage\n", "\n", "Lot Frontage is a bit trickier. There are 486 missing values (17% of total values) but a quick check of correlation shows that there are surprisingly few features that have high correlation outside of Lot Area." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "09befb87-50cc-efce-b48b-55180eb4c212" }, "outputs": [], "source": [ "lot = wholedf[['LotArea','LotConfig','LotFrontage','LotShape']]\n", "lot = pd.get_dummies(lot)\n", "lot.corr()['LotFrontage']" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "1fa53e14-0852-c00e-b636-3d9e8e012796" }, "source": [ "Logic dictates that the Lot Area should have a linear relation to Lot Frontage. A quick check with a linear regression of the relationship between Lot Frontage and the square root of Lot Area (or effectively one side) shows we are in the ballpark." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "60f40eeb-0e2a-5df6-ed79-e70e4667ca84" }, "outputs": [], "source": [ "lot[\"LotAreaUnSq\"] = np.sqrt(lot['LotArea'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "28e2b520-14db-4249-bb3b-e3739ce616d4" }, "outputs": [], "source": [ "sns.regplot(x=\"LotAreaUnSq\", y=\"LotFrontage\", data=lot);\n", "plt.xlim(0,)\n", "plt.ylim(0,)\n", "plt.title(\"Lot Area to Lot Frontage\")\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "16bb6027-716b-c6c2-fdb8-369f002a6ce1" }, "outputs": [], "source": [ "# Remove all lotfrontage is missing values\n", "lot = lot[lot['LotFrontage'].notnull()]\n", "# See the not null values of LotFrontage\n", "lot.describe()['LotFrontage']" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "c8bff8ba-d224-1cb3-31c4-9c3d0dffbca0" }, "outputs": [], "source": [ "wholedf['LotFrontage']= wholedf.LotFrontage.fillna(np.sqrt(wholedf.LotArea))\n", "wholedf['LotFrontage']= wholedf['LotFrontage'].astype(int)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "a772171d-3208-bc02-2761-a38f4b7035f0" }, "source": [ "Once the missing values are filled in, it is good to confirm that the distribution did not go way out of wack. The blue is the original distribution, the green is the new one with missing values inferred and the red is the curve of the square root of lot area. I looks like a majority of these properties unsurprisingly do not have perfectly square lots but for our purposes, everything looks good." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "43003ab1-f936-26ce-aa7b-db0f5caa8e74" }, "outputs": [], "source": [ "# Distribution of values after replacement of missing frontage\n", "sns.kdeplot(wholedf['LotFrontage']);\n", "sns.kdeplot(lot['LotFrontage']);\n", "sns.kdeplot(lot['LotAreaUnSq']);\n", "plt.title(\"Distribution of Lot Frontage\")\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "0745beac-fda8-45f9-ce80-7a90d46af148" }, "outputs": [], "source": [ "countmissing = wholedf.isnull().sum().sort_values(ascending=False)\n", "percentmissing = (wholedf.isnull().sum()/wholedf.isnull().count()).sort_values(ascending=False)\n", "wholena = pd.concat([countmissing,percentmissing], axis=1)\n", "wholena.head(3)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "b60561b3-7712-6cc5-5792-9d5f83888e17" }, "source": [ "## Feaure Engineering\n", "-----------------------\n", "Now it is time to create some new features and see how they can help the model\u2019s accuracy. First are a couple of macro creations, like adding all the internal and external square footage together to get the total living space including both floors, garage and external spaces." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "7bed7d7c-92bd-11fc-cb91-87481649ded2" }, "outputs": [], "source": [ "Livingtotalsq = wholedf['TotalBsmtSF'] + wholedf['1stFlrSF'] + wholedf['2ndFlrSF'] + wholedf['GarageArea'] + wholedf['WoodDeckSF'] + wholedf['OpenPorchSF']\n", "wholedf['LivingTotalSF'] = Livingtotalsq\n", "\n", "# Total Living Area divided by LotArea\n", "wholedf['PercentSQtoLot'] = wholedf['LivingTotalSF'] / wholedf['LotArea']\n", "\n", "# Total count of all bathrooms including full and half through the entire building\n", "wholedf['TotalBaths'] = wholedf['BsmtFullBath'] + wholedf['BsmtHalfBath'] + wholedf['HalfBath'] + wholedf['FullBath']\n", "\n", "# Percentage of total rooms are bedrooms\n", "wholedf['PercentBedrmtoRooms'] = wholedf['BedroomAbvGr'] / wholedf['TotRmsAbvGrd']\n", "\n", "# Number of years since last remodel, if there never was one it would be since it was built\n", "wholedf['YearSinceRemodel'] = 2016 - ((wholedf['YearRemodAdd'] - wholedf['YearBuilt']) + wholedf['YearBuilt'])" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "83cc0da8-0ff6-4152-b78c-a4fa49804c9d" }, "source": [ "## Total Square Footage\n", "There is a minimal increase of the Condition Rating to square foot, which is not a surprise. However when a linear regression with the Sale Price is taken instead a much more obvious pattern emerges." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "42a27185-36e8-5d2e-7341-7eefd374f0f9" }, "outputs": [], "source": [ "sns.barplot(x=\"OverallCond\", y=\"LivingTotalSF\", data=wholedf)\n", "plt.title(\"Total Square Footage by Overall Condition Rating\")\n", "plt.show()\n", "\n", "sns.lmplot(x=\"LivingTotalSF\", y=\"SalePrice\", data=wholedf)\n", "plt.title(\"Relation of Sale Price to Total Square Footage\")\n", "plt.xlim(0,)\n", "plt.ylim(0,)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "a7eef1ae-1c1f-908e-4fc2-7ec1e94c4704" }, "source": [ "## Total Rooms\n", "Conversely if rooms were what interested you, there is a pronounced relationship between rooms and the number of bathrooms. This is the mean of bathrooms and the line is the variance (how much from the average most values deviate)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "7fa8efaf-9e98-cc8b-9db0-7d078bc44067" }, "outputs": [], "source": [ "ax = sns.barplot(x=\"TotRmsAbvGrd\", y=\"TotalBaths\",data=wholedf)\n", "plt.title(\"Total Rooms Versus Total Bathrooms\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "b6abbdf2-5c7c-bf90-97b6-678228ff6d06" }, "source": [ "## Sale Month\n", "Once last colorful graph before wrapping this up. I have wondered this about the Vermont housing market as the weather can have a large impact on many activities, including looking at houses I would imagine." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "a99f975c-01ea-6f1f-c5a4-96223f4f424f" }, "outputs": [], "source": [ "sns.swarmplot(x=\"MoSold\", y=\"SalePrice\", data=wholedf)\n", "plt.title(\"Sale Price by Month\")\n", "plt.show()\n", "\n", "sns.kdeplot(wholedf['MoSold']);\n", "plt.title(\"Distribution of Month Sold\")\n", "plt.xlim(1,12)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "3e023714-3dc9-b961-4ff2-ac266eda81a4" }, "source": [ "## Neighborhoods\n", "All of us know that neighborhoods have a wide variety of characteristics. In some ways there is a relationship to sale price, but it is harder to define and less useful in prediction." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "4b13a7c7-48d2-8e28-02b6-3ecf2818f4c6" }, "outputs": [], "source": [ "plt.figure(figsize = (12, 6))\n", "sns.boxplot(x = 'Neighborhood', y = 'SalePrice', data = wholedf)\n", "plt.xticks(rotation=45)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "154a43fe-3e58-8cd2-d74c-36369c75a204" }, "source": [ "### Features\n", "This is a way to edit what feeatures are included in the final model. I played a bit with which to include, and I have a feeling I may come back to this to re-engineer it. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "cdc45ed7-c2dc-01f1-3a47-f5bef774f5ce" }, "outputs": [], "source": [ "pricing1 = wholedf[['Id','SalePrice','MiscVal']]\n", "\n", "neigh = wholedf[['Neighborhood','MSZoning','MSSubClass','BldgType','HouseStyle']]\n", "\n", "dates = wholedf[['YearBuilt','YearRemodAdd','GarageYrBlt','YearSinceRemodel']]\n", "\n", "quacon = wholedf[['ExterQual','BsmtQual','PoolQC','Condition1','Condition2','SaleCondition',\n", " 'BsmtCond','ExterCond','GarageCond','KitchenQual','GarageQual','HeatingQC','OverallQual','OverallCond']]\n", "\n", "features = wholedf[['Foundation','RoofStyle','RoofMatl','Exterior1st','Exterior2nd',\n", " 'MiscFeature','PavedDrive','Utilities',\n", " 'Heating','CentralAir','Electrical','Fence']]\n", "\n", "sqfoot = wholedf[['LivingTotalSF','TotalBsmtSF','1stFlrSF','2ndFlrSF','GrLivArea',\n", " 'GarageArea','WoodDeckSF','OpenPorchSF','LotArea','PercentSQtoLot','LowQualFinSF']]\n", "\n", "roomfeatcount = wholedf[['PercentBedrmtoRooms','TotalBaths','FullBath','HalfBath',\n", " 'KitchenAbvGr','TotRmsAbvGrd','Fireplaces','GarageCars','GarageType','EnclosedPorch']]\n", "\n", "# Splits out sale price for the training set and only has not null values\n", "pricing = wholedf['SalePrice']\n", "pricing = pricing[pricing.notnull()]\n", "\n", "# Bringing it all together\n", "wholedf = pd.concat([pricing1,neigh,dates,quacon,features,sqfoot,roomfeatcount], axis=1)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "4166522d-944f-b5ee-aeca-aff8ef4da6bc" }, "source": [ "### Categorical Conversion\n", "In order for the model to understand categories, first replace all the categorical data with boolean values through Pandas get_dummies. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "919f27e4-e6a9-2451-2996-22eac93ab151" }, "outputs": [], "source": [ "wholedf = pd.get_dummies(wholedf)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "155c7d9d-1934-3e74-5ec8-28183b38d5cc" }, "outputs": [], "source": [ "traincorr = wholedf.corr()['SalePrice']\n", "# convert series to dataframe so it can be sorted\n", "traincorr = pd.DataFrame(traincorr)\n", "# correct column label from SalePrice to correlation\n", "traincorr.columns = [\"Correlation\"]\n", "# sort correlation\n", "traincorr2 = traincorr.sort_values(by=['Correlation'], ascending=False)\n", "traincorr2.head(15)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "67f0745a-4b35-b9d5-3641-ff78970e7a15" }, "source": [ "## Split Database\n", "--------\n", "Time to split the database back into two parts, one with sales price and one without" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "7763f182-5012-af95-4b9c-32f6678f281e" }, "outputs": [], "source": [ "train_X = wholedf[wholedf['SalePrice'].notnull()]\n", "del train_X['SalePrice']\n", "test_X = wholedf[wholedf['SalePrice'].isnull()]\n", "del test_X['SalePrice']" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "946925de-d038-48d8-b09d-1d0227fdbb71" }, "source": [ "## Training/Test Dataset\n", "\n", "Create training set assembly" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "03926d49-c9f1-32dd-91fb-df5df6dd0e70" }, "outputs": [], "source": [ "# Create all datasets that are necessary to train, validate and test models\n", "train_valid_X = train_X\n", "train_valid_y = pricing\n", "test_X = test_X\n", "train_X , valid_X , train_y , valid_y = train_test_split( train_valid_X , train_valid_y , train_size = .7 )" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "bceca0b8-58a7-313f-a0a5-71b86ec89565" }, "source": [ "## Model\n", "Here are a variety of models you can try, many performed extremely poorly, Lasso was the best but I plan to go back to this and do further work on refining my features, like log regressions and normalizations. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "5b3ec615-4ffd-5cb0-6132-7e9097640459" }, "outputs": [], "source": [ "# model = RandomForestRegressor()\n", "model = Ridge()\n", "# model = LassoLarsCV()\n", "\n", "# Models that performed substantially worse\n", "# model = LinearSVC()\n", "# model = KNeighborsClassifier(n_neighbors = 3)\n", "# model = GaussianNB()\n", "# model = LogisticRegression()\n", "# model = SVC()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "789c0d9d-6f4a-e775-6b43-7fd558378393" }, "source": [ "## Fit/Accurancy" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "0fbedad8-157a-a95b-c609-923377a9bb96" }, "outputs": [], "source": [ "model.fit( train_X , train_y )\n", "\n", "# Print the Training Set Accuracy and the Test Set Accuracy in order to understand overfitting\n", "print (model.score( train_X , train_y ) , model.score( valid_X , valid_y ))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "8555f20d-73ae-cfbc-b2bc-097566613e98" }, "outputs": [], "source": [ "id = test_X.Id\n", "result = model.predict(test_X)\n", "\n", "# output = pd.DataFrame( { 'id': id , 'SalePrice': result}, columns=['id', 'SalePrice'] )\n", "output = pd.DataFrame( { 'id': id , 'SalePrice': result} )\n", "output = output[['id', 'SalePrice']]\n", "\n", "output.to_csv(\"solution.csv\", index = False)\n", "output.head(10)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "f62c79e6-0346-a39b-b09e-433bbdb506b1" }, "source": [ "## Conclusion\n", "\n", "There were a couple of models tried. Random Forest Regression, LassoLarsCV and Ridge were all options and after experimenting I ended up settling on Ridge as the better option. What was fascinating and frustrating was the wide variation in accuracy each time I ran it. The models all seem to have some fluctuation in them. In the end I had about a 82%\u201390% accuracy. In trying to apply this same process to Burlington\u2019s housing data I ran into even stranger inconsistency and huge over fitting problems. I have a feeling this will be a work in progress as I try it on other data sets. \n", "\n", "There were some great kernels out there that helped me work through the problem, definately check the following out:\n", "https://www.kaggle.com/neviadomski/house-prices-advanced-regression-techniques/how-to-get-to-top-25-with-simple-model-sklearn\n", "https://www.kaggle.com/poonaml/house-prices-advanced-regression-techniques/house-prices-data-exploration-and-visualisation\n", "https://www.kaggle.com/apapiu/house-prices-advanced-regression-techniques/regularized-linear-models\n", "https://www.kaggle.com/xchmiao/house-prices-advanced-regression-techniques/detailed-data-exploration-in-python\n" ] } ], "metadata": { "_change_revision": 0, "_is_fork": false, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.0" } }, "nbformat": 4, "nbformat_minor": 0 }