{ "cells": [ { "cell_type": "markdown", "metadata": { "_cell_guid": "65663e28-5b6f-f7bd-9d93-fd32a79227fc" }, "source": [ "## Introduction\n", "\n", "**_Poonam Ligade_**\n", "\n", "*30th Jan 2017*\n", "\n", "I am trying to predict for how much money each house can be sold??\n", "\n", "In this mainly we will look at data exploration and visulisation part\n", "\n", "EDA is often most tedious and boring job.\n", "\n", "But the more time you spend here on understanding, cleaning and preparing data the better fruits your predictive model will bare!!\n", "\n", "Lets start.\n", "\n", "1) **Introduction**\n", "\n", " 1. Import Libraries\n", " 2. Load data\n", " 3. Variable Identification\n", " 4. Run Statistical summaries\n", " 5. Correlation with target variable\n", "\n", " \n", "2) **Missing values imputation**\n", "\n", " 1. Figure out missing value columns\n", " 2. Fill out missing values\n", "\n", "\n", "3) **Visualisations**\n", "\n", " 1. Univariate Analysis\n", " 2. Bivariate Analysis" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "a80c6e1b-d07c-5911-facc-caaa3ce97b1e" }, "source": [ "**Import libraries**\n", "====================" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "82a4b19f-e192-f905-0337-e47d9e497901" }, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "\n", "import matplotlib.pyplot as plt\n", "% matplotlib inline\n", "import seaborn as sns\n", "sns.set(style=\"whitegrid\", color_codes=True)\n", "sns.set(font_scale=1)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "868fd0ec-93e8-2475-68a5-958215551295" }, "source": [ "**Load train & test data**\n", "====================" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "e970882d-841e-d8ec-7350-0e60cb82c3b1" }, "outputs": [], "source": [ "houses=pd.read_csv(\"../input/train.csv\")\n", "houses.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "b4a28437-3e15-72dc-674f-a44fee56b3c4" }, "outputs": [], "source": [ "houses_test = pd.read_csv(\"../input/test.csv\")\n", "#transpose\n", "houses_test.head()\n", "#note their is no \"SalePrice\" column here which is our target varible." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "b4096670-f549-257f-192e-c93abe1eb82a" }, "outputs": [], "source": [ "#shape command will give number of rows/samples/examples and number of columns/features/predictors in dataset\n", "#(rows,columns)\n", "houses.shape" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "fdfc9932-44ce-e09f-d82e-b0e5982ef6f9" }, "source": [ "There are total **1460 samples** which we can use to train model and **80 features** and **1 target variable.**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "9e75e90d-1130-1db5-7c5f-5a85049fce74" }, "outputs": [], "source": [ "houses_test.shape\n", "#1 column less because target variable isn't there in test set!" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "12182a06-901a-cd5a-f986-5661864d546c" }, "source": [ "*Variable Identification*\n", "-----------------------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "17dad815-c249-4b4d-e542-95a082cc2983" }, "outputs": [], "source": [ "#info method provides information about dataset like \n", "#total values in each column, null/not null, datatype, memory occupied etc\n", "houses.info()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "01498707-ed8c-8aa5-4a43-941894415b2d" }, "outputs": [], "source": [ "#How many columns with different datatypes are there?\n", "houses.get_dtype_counts()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "4169c23c-b2a9-0c1a-1513-0be4be3ea9ec" }, "outputs": [], "source": [ "##Describe gives statistical information about numerical columns in the dataset\n", "houses.describe()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "336f8c6d-d698-3379-1bb9-050e9f146017" }, "source": [ "**Correlation in Data**\n", "====================" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "8ee773fc-fe4f-3ee7-d6b4-b4d8ab5762a0" }, "outputs": [], "source": [ "corr=houses.corr()[\"SalePrice\"]\n", "corr[np.argsort(corr, axis=0)[::-1]]" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "c6d6ff08-8ba2-4d4a-9726-3fbba32f1205" }, "source": [ "OverallQual ,GrLivArea ,GarageCars,GarageArea ,TotalBsmtSF, 1stFlrSF ,FullBath,TotRmsAbvGrd,YearBuilt, YearRemodAdd have more than 0.5 correlation with SalePrice. \n", "\n", "EnclosedPorch and KitchenAbvGr have little negative correlation with target variable.\n", "\n", "These can prove to be important features to predict SalePrice.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "21f6f47f-398b-12be-411a-460c1ae2a715" }, "outputs": [], "source": [ "#plotting correlations\n", "num_feat=houses.columns[houses.dtypes!=object]\n", "num_feat=num_feat[1:-1] \n", "labels = []\n", "values = []\n", "for col in num_feat:\n", " labels.append(col)\n", " values.append(np.corrcoef(houses[col].values, houses.SalePrice.values)[0,1])\n", " \n", "ind = np.arange(len(labels))\n", "width = 0.9\n", "fig, ax = plt.subplots(figsize=(12,40))\n", "rects = ax.barh(ind, np.array(values), color='red')\n", "ax.set_yticks(ind+((width)/2.))\n", "ax.set_yticklabels(labels, rotation='horizontal')\n", "ax.set_xlabel(\"Correlation coefficient\")\n", "ax.set_title(\"Correlation Coefficients w.r.t Sale Price\");" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "f6429d98-12d7-46f1-e14d-19024b303cd8" }, "outputs": [], "source": [ "correlations=houses.corr()\n", "attrs = correlations.iloc[:-1,:-1] # all except target\n", "\n", "threshold = 0.5\n", "important_corrs = (attrs[abs(attrs) > threshold][attrs != 1.0]) \\\n", " .unstack().dropna().to_dict()\n", "\n", "unique_important_corrs = pd.DataFrame(\n", " list(set([(tuple(sorted(key)), important_corrs[key]) \\\n", " for key in important_corrs])), \n", " columns=['Attribute Pair', 'Correlation'])\n", "\n", " # sorted by absolute value\n", "unique_important_corrs = unique_important_corrs.ix[\n", " abs(unique_important_corrs['Correlation']).argsort()[::-1]]\n", "\n", "unique_important_corrs" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "9694ee3a-a92e-c946-7fec-05f5a2ec600c" }, "source": [ "This shows multicollinearity.\n", "In regression, \"multicollinearity\" refers to features that are correlated with other features. Multicollinearity occurs when your model includes multiple factors that are correlated not just to your target variable, but also to each other.\n", "\n", "Problem:\n", "\n", "Multicollinearity increases the standard errors of the coefficients.\n", "That means, multicollinearity makes some variables statistically insignificant when they should be significant.\n", "\n", "To avoid this we can do 3 things:\n", "\n", "1. Completely remove those variables\n", "2. Make new feature by adding them or by some other operation.\n", "3. Use PCA, which will reduce feature set to small number of non-collinear features.\n", "\n", "Reference:http://blog.minitab.com/blog/understanding-statistics/handling-multicollinearity-in-regression-analysis" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "599eb385-8d31-7fdb-5ca9-801e5f8f2b47" }, "source": [ "**Heatmap**\n", "-----------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "549ae27a-804c-14f8-b4b6-811f09c576ac" }, "outputs": [], "source": [ "corrMatrix=houses[[\"SalePrice\",\"OverallQual\",\"GrLivArea\",\"GarageCars\",\n", " \"GarageArea\",\"GarageYrBlt\",\"TotalBsmtSF\",\"1stFlrSF\",\"FullBath\",\n", " \"TotRmsAbvGrd\",\"YearBuilt\",\"YearRemodAdd\"]].corr()\n", "\n", "sns.set(font_scale=1.10)\n", "plt.figure(figsize=(10, 10))\n", "\n", "sns.heatmap(corrMatrix, vmax=.8, linewidths=0.01,\n", " square=True,annot=True,cmap='viridis',linecolor=\"white\")\n", "plt.title('Correlation between features');" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "bec44e1a-abee-71c5-79a7-02e98123b12b" }, "source": [ "As we saw above there are few feature which shows high multicollinearity from heatmap.\n", "Lets focus on yellow squares on diagonal line.\n", "\n", "GrLivArea and GarageCars\n", "\n", "TotalBsmtSF and 1stFlrSF\n", "\n", "FullBath and TotRmsAbvGrd\n", "\n", "YearBulit and YearRemodelAdd.\n", "\n", "We have to create a single feature from them before we use them as predictors.\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "1d211341-26d5-67e9-8995-8c9c45fccdec" }, "source": [ "*Pivotal Features*\n", "------------------------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "b82f3410-17f5-412d-79b9-bc97bcb1b40d" }, "outputs": [], "source": [ "houses[['OverallQual','SalePrice']].groupby(['OverallQual'],\n", "as_index=False).mean().sort_values(by='OverallQual', ascending=False)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "c615a4ce-a5c9-fc00-12fa-014f2fa3c720" }, "outputs": [], "source": [ "houses[['GarageCars','SalePrice']].groupby(['GarageCars'],\n", "as_index=False).mean().sort_values(by='GarageCars', ascending=False)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "b820fe61-6bfc-b2ef-dff4-9950bf4d72c5" }, "outputs": [], "source": [ "houses[['Fireplaces','SalePrice']].groupby(['Fireplaces'],\n", "as_index=False).mean().sort_values(by='Fireplaces', ascending=False)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "5744ee64-4a0c-6683-bc18-5bc89c12db94" }, "source": [ "*Visualising Target variable*\n", "-----------------" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "7054ccbd-e345-994a-565d-2f65b82b9dd0" }, "source": [ "*Univariate Analysis*\n", "--------------------\n", "\n", "How 1 single variable is distributed in numeric range.\n", "What is statistical summary of it.\n", "Is it positively skewed or negatively." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "d9adc736-5981-3ec8-7207-273f05bd86e2" }, "outputs": [], "source": [ "sns.distplot(houses['SalePrice'], color=\"r\")\n", "plt.title(\"Distribution of Sale Price\")\n", "plt.ylabel(\"Number of Occurences\")\n", "plt.xlabel(\"Sale Price\");" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "3a6876ef-75a6-70af-c63a-08c300480457" }, "source": [ "Prices are right skewed and graph shows some peakedness.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "599e4e1e-cc6c-ca33-fa6a-46e6000d785b" }, "outputs": [], "source": [ "#skewness \n", "\n", "houses['SalePrice'].skew()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "a51b8dc1-c8f3-ebcc-b530-cdbe8e477951" }, "outputs": [], "source": [ "#kurtosis\n", "\n", "houses['SalePrice'].kurt()\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "68d54928-97f4-d874-b823-ed88a1d9cff8" }, "outputs": [], "source": [ "#there are some outliers.lets remove them.\n", "upperlimit = np.percentile(houses.SalePrice.values, 99.5)\n", "houses['SalePrice'].ix[houses['SalePrice']>upperlimit] = upperlimit\n", "\n", "plt.scatter(range(houses.shape[0]), houses[\"SalePrice\"].values,color='orange')\n", "plt.title(\"Distribution of Sale Price\")\n", "plt.xlabel(\"Number of Occurences\")\n", "plt.ylabel(\"Sale Price\");" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "0b87ed94-2de4-c26e-a3b5-5a69ff38a698" }, "source": [ "**Missing Value Imputation**\n", "====================\n", "\n", "\n", "Missing values in the training data set can affect prediction or classification of a model negatively.\n", "\n", "Also some machine learning algorithms can't accept missing data eg. SVM.\n", "\n", "But filling missing values with mean/median/mode or using another predictive model to predict missing values is also a prediction which may not be 100% accurate, instead you can use models like Decision Trees and Random Forest which handle missing values very well.\n", "\n", "Some of this part is based on this kernel:\n", "https://www.kaggle.com/bisaria/house-prices-advanced-regression-techniques/handling-missing-data" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "526af8fe-b1e3-7b53-5b67-c7cda66e2658" }, "outputs": [], "source": [ "#lets see if there are any columns with missing values \n", "null_columns=houses.columns[houses.isnull().any()]\n", "houses[null_columns].isnull().sum()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "8dc6df99-93b7-9a0d-226b-d829ad168a34" }, "outputs": [], "source": [ "labels = []\n", "values = []\n", "for col in null_columns:\n", " labels.append(col)\n", " values.append(houses[col].isnull().sum())\n", "ind = np.arange(len(labels))\n", "width = 0.9\n", "fig, ax = plt.subplots(figsize=(12,50))\n", "rects = ax.barh(ind, np.array(values), color='violet')\n", "ax.set_yticks(ind+((width)/2.))\n", "ax.set_yticklabels(labels, rotation='horizontal')\n", "ax.set_xlabel(\"Count of missing values\")\n", "ax.set_ylabel(\"Column Names\")\n", "ax.set_title(\"Variables with missing values\");" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "b30def9c-129b-fcc1-887c-bcef8f1f6132" }, "source": [ "*Multivariate Analysis*\n", "--------------------\n", "\n", "When we understand how 3 or more variables behave according to each other." ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "7a053c65-694b-be3d-4bab-f9355c0aa6e2" }, "source": [ "*LotFrontage*\n", "-------------\n", "\n", "We can see if there is some correlation between LotArea and LotFrontage" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "36050698-f36a-9b68-2e1a-244ea0fd7fcd" }, "outputs": [], "source": [ "houses['LotFrontage'].corr(houses['LotArea'])" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "5132e512-24ea-cb37-13b5-fdf5c69103b1" }, "source": [ "This is not great, we will try some polynomial expressions like squareroot " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "5f3ac7bd-c835-3987-e3ec-30e17791eb39" }, "outputs": [], "source": [ "houses['SqrtLotArea']=np.sqrt(houses['LotArea'])\n", "houses['LotFrontage'].corr(houses['SqrtLotArea'])" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "7f1fe1c2-337c-61c6-8ce9-881538fa0d35" }, "source": [ "0.60 looks good to go." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "f9e195cf-bda8-f23f-deab-35674f5445c6" }, "outputs": [], "source": [ "sns.jointplot(houses['LotFrontage'],houses['SqrtLotArea'],color='gold');" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "ef3f5e6c-bc27-007a-08f3-8d723cd3faef" }, "outputs": [], "source": [ "filter = houses['LotFrontage'].isnull()\n", "houses.LotFrontage[filter]=houses.SqrtLotArea[filter]" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "aeb0c00a-5cf2-fd21-d4b2-ecced63eb694" }, "source": [ "*MasVnrType and MasVnrArea*\n", "===========================" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "09611281-eba3-ba8d-1d84-66fd7f26ccad" }, "outputs": [], "source": [ "plt.scatter(houses[\"MasVnrArea\"],houses[\"SalePrice\"])\n", "plt.title(\"MasVnrArea Vs SalePrice \")\n", "plt.ylabel(\"SalePrice\")\n", "plt.xlabel(\"Mas Vnr Area in sq feet\");" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "12005d25-b3c9-4ca3-6f1f-cbc59ca555e3" }, "outputs": [], "source": [ "sns.boxplot(\"MasVnrType\",\"SalePrice\",data=houses);" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "8d583cb6-32cc-a71e-46b1-c4994f1d5060" }, "outputs": [], "source": [ "houses[\"MasVnrType\"] = houses[\"MasVnrType\"].fillna('None')\n", "houses[\"MasVnrArea\"] = houses[\"MasVnrArea\"].fillna(0.0)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "0eb78a8e-b365-14e9-892e-43ebd2ada036" }, "source": [ "*Bivariate Analysis*\n", "--------------------\n", "\n", "When we try to figure out how 2 parameters in dataset are related to each other. in the sense when one decreases, other also decreases or when one increases other also increases i.e Positive Correlation \n", "\n", "And when one increases , other decreases or vice versa i .e Negative correlation." ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "26bf4d66-47da-c362-f55a-92d640965877" }, "source": [ "*Electrical*\n", "------------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "359b8def-62f9-a239-fb8a-0af4c5c18393" }, "outputs": [], "source": [ "sns.boxplot(\"Electrical\",\"SalePrice\",data=houses)\n", "plt.title(\"Electrical Vs SalePrice \")\n", "plt.ylabel(\"SalePrice\")\n", "plt.xlabel(\"Electrical\");" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "0507d351-a7fd-cc9b-88cd-6d04de595a31" }, "outputs": [], "source": [ "#We can replace missing values with most frequent ones.\n", "houses[\"Electrical\"] = houses[\"Electrical\"].fillna('SBrkr')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "fc56e53e-b43c-dd30-78b2-836c85165d56" }, "source": [ "*Alley*\n", "-------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "fc490b5c-3c2c-1124-4314-4ffe74e8d026" }, "outputs": [], "source": [ "sns.stripplot(x=houses[\"Alley\"], y=houses[\"SalePrice\"],jitter=True);" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "4469208f-b4d3-5810-81be-2fc30c992de2" }, "source": [ "All missing value indicate that particular house doesn't have an alley access.we can replace it with 'None'." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "91306208-cfb2-a177-3907-20d3ce5cb9d2" }, "outputs": [], "source": [ "houses[\"Alley\"] = houses[\"Alley\"].fillna('None')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "4f6ff17d-0a7e-e584-7e0a-f7e4a7b7cf21" }, "source": [ "*Basement Features*\n", "-------------------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "ca96c976-d17a-c709-1047-139b3e5e63b4" }, "outputs": [], "source": [ "plt.scatter(houses[\"TotalBsmtSF\"],houses[\"SalePrice\"])\n", "plt.title(\"Total Basement area in Square Feet Vs SalePrice \")\n", "plt.ylabel(\"SalePrice\")\n", "plt.xlabel(\"Total Basement area in Square Feet\");" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "a5bd287c-c2d7-b5a0-dee3-b328f333f8d9" }, "outputs": [], "source": [ "#there are few outliers in total basement area lets remove them\n", "upperlimit = np.percentile(houses.TotalBsmtSF.values, 99.5)\n", "houses['TotalBsmtSF'].ix[houses['TotalBsmtSF']>upperlimit] = upperlimit\n", "\n", "plt.scatter(houses.TotalBsmtSF, houses[\"SalePrice\"].values,color='orange')\n", "plt.title(\"TotalBsmtSF Vs SalePrice \")\n", "plt.ylabel(\"SalePrice\")\n", "plt.xlabel(\"Total Basement in sq feet\");" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "a6b1e38a-9cec-9c63-c78a-df689035d37e" }, "outputs": [], "source": [ "basement_cols=['BsmtQual','BsmtCond','BsmtExposure','BsmtFinType1','BsmtFinType2','BsmtFinSF1','BsmtFinSF2']\n", "houses[basement_cols][houses['BsmtQual'].isnull()==True]" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "e86de3a4-4faf-3e55-ae5c-8f731ed86e2d" }, "source": [ "All categorical variables contains NAN whereas continuous ones have 0.\n", "So that means there is no basement for those houses.\n", "we can replace it with 'None'." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "ae9a930d-a53a-2057-d38a-2ef35aa824ef" }, "outputs": [], "source": [ "for col in basement_cols:\n", " if 'FinSF'not in col:\n", " houses[col] = houses[col].fillna('None')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "76a1b3cf-efee-8953-0587-e7e2772aedb7" }, "source": [ "*Fireplaces*\n", "------------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "352ec337-db2d-81c6-771e-8328b25bbfcb" }, "outputs": [], "source": [ "sns.factorplot(\"Fireplaces\",\"SalePrice\",data=houses,hue=\"FireplaceQu\");" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "6e488451-04bc-9d2c-8612-ea48ee4f2e9f" }, "source": [ "Having 2 fireplaces increases house price and fireplace of Excellent quality is a big plus. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "216ab92b-fb2f-b9e8-7187-51e58f567a7e" }, "outputs": [], "source": [ "#If fireplace quality is missing that means that house doesn't have a fireplace\n", "houses[\"FireplaceQu\"] = houses[\"FireplaceQu\"].fillna('None')\n", "pd.crosstab(houses.Fireplaces, houses.FireplaceQu)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "5ae4c70f-24d3-8840-b0f3-25fdb8bf9eff" }, "source": [ "*Garages*\n", "---------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "dffe90ed-3bdf-901f-f862-67ec8ca3b048" }, "outputs": [], "source": [ "sns.distplot(houses[\"GarageArea\"],color='r');" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "a1fed155-d4d2-3a2d-b804-7ec2a5a834e2" }, "outputs": [], "source": [ "#GarageArea has got some outliers lets remove them.\n", "upperlimit = np.percentile(houses.GarageArea.values, 99.5)\n", "houses['GarageArea'].ix[houses['GarageArea']>upperlimit] = upperlimit\n", "\n", "plt.scatter(houses.GarageArea, houses[\"SalePrice\"].values,color='violet')\n", "plt.title(\"Garage Area Vs SalePrice \")\n", "plt.ylabel(\"SalePrice\")\n", "plt.xlabel(\"Garage Area in sq feet\");" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "6298bc48-64c9-e927-af2b-2b84f000da5b" }, "outputs": [], "source": [ "sns.violinplot(houses[\"GarageCars\"],houses[\"SalePrice\"])\n", "plt.title(\"Garage Cars Vs SalePrice \")\n", "plt.ylabel(\"SalePrice\")\n", "plt.xlabel(\"Number of Garage cars\");" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "2c509364-8dd0-6d34-0215-7c78b6004106" }, "outputs": [], "source": [ "garage_cols=['GarageType','GarageQual','GarageCond','GarageYrBlt','GarageFinish','GarageCars','GarageArea']\n", "houses[garage_cols][houses['GarageType'].isnull()==True]" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "b45b1f51-d50a-3124-ed18-2e1d73b9cda0" }, "source": [ "All garage related features are missing values in same rows.\n", "that means we can replace categorical variables with None and continuous ones with 0." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "81a5478c-af63-5b44-46c2-a30e9c030008" }, "outputs": [], "source": [ "for col in garage_cols:\n", " if houses[col].dtype==np.object:\n", " houses[col] = houses[col].fillna('None')\n", " else:\n", " houses[col] = houses[col].fillna(0)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "11f71806-59eb-7685-0b63-7ef2e9026682" }, "source": [ "*Pool*\n", "-----------------------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "37368aff-a3ff-07cb-df24-0d4de153154a" }, "outputs": [], "source": [ "#If PoolArea is 0, that means that house doesn't have a pool.\n", "#So we can replace PoolQuality with None.\n", "houses[\"PoolQC\"] = houses[\"PoolQC\"].fillna('None')\n", "sns.factorplot(\"PoolArea\",\"SalePrice\",data=houses,hue=\"PoolQC\",kind='bar')\n", "plt.title(\"Pool Area , Pool quality and SalePrice \")\n", "plt.ylabel(\"SalePrice\")\n", "plt.xlabel(\"Pool Area in sq feet\");" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "b9c83afe-5358-9859-37a2-783ed47038eb" }, "source": [ "*Fence*\n", "-----------------------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "f9531d81-18af-c38d-ab8a-5e143b94ab2c" }, "outputs": [], "source": [ "sns.violinplot(houses[\"Fence\"],houses[\"SalePrice\"])\n", "plt.title(\"Fence wrt SalePrice \")\n", "plt.ylabel(\"SalePrice\")\n", "plt.xlabel(\"Type of Fence\");" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "7b0ed129-411a-5e9e-7362-6ec41ee1f177" }, "source": [ "Fence has got 1179 null values.\n", "We can safely assume that those houses doesn't have a Fence and replace those values with None." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "13c663f7-a945-c419-9b3f-7c5e4fe5d7e1" }, "outputs": [], "source": [ "houses[\"Fence\"] = houses[\"Fence\"].fillna('None')" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "270a2e54-f38c-fa41-56d4-fc9d812b9241" }, "source": [ "*MiscFeature*\n", "-----------------------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "128dc1d2-aea1-b9f4-6621-b8a8501fab27" }, "outputs": [], "source": [ "sns.barplot(houses[\"MiscFeature\"],houses[\"SalePrice\"])\n", "plt.title(\"Miscelleneous Features Vs SalePrice \")\n", "plt.ylabel(\"SalePrice\")\n", "plt.xlabel(\"Type of Miscelleneous Features\");" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "5d38c6dc-15b9-6d8f-fa61-bafd373ce171" }, "outputs": [], "source": [ "#Some houses don't have miscellaneous features like shed, Tennis court etc..\n", "houses[\"MiscFeature\"] = houses[\"MiscFeature\"].fillna('None')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "d47a93a5-17dd-7909-96c1-b89926ce6b96" }, "outputs": [], "source": [ "#Let's confirm that we have removed all missing values\n", "houses[null_columns].isnull().sum()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "4ccecd98-0c3b-c9da-c0c5-7605f69dcc50" }, "source": [ "**Visualizations**\n", "==================" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "fc91e0cd-85fa-aac5-e5ca-a897a324bfbf" }, "source": [ "*MSZoning*\n", "-----------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "519ff4b9-37bf-20c8-474f-aaf132a1fdd0" }, "outputs": [], "source": [ "\n", "labels = houses[\"MSZoning\"].unique()\n", "sizes = houses[\"MSZoning\"].value_counts().values\n", "explode=[0.1,0,0,0,0]\n", "parcent = 100.*sizes/sizes.sum()\n", "labels = ['{0} - {1:1.1f} %'.format(i,j) for i,j in zip(labels, parcent)]\n", "\n", "colors = ['yellowgreen', 'gold', 'lightblue', 'lightcoral','blue']\n", "patches, texts= plt.pie(sizes, colors=colors,explode=explode,\n", " shadow=True,startangle=90)\n", "plt.legend(patches, labels, loc=\"best\")\n", "\n", "plt.title(\"Zoning Classification\")\n", "plt.show()\n", "\n", "\n", "\n", "sns.violinplot(houses.MSZoning,houses[\"SalePrice\"])\n", "plt.title(\"MSZoning wrt Sale Price\")\n", "plt.xlabel(\"MSZoning\")\n", "plt.ylabel(\"Sale Price\");" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "2eacf185-0009-88a1-f1df-1aed06eac561" }, "source": [ "*1st Floor in square feet*\n", "--------------------------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "55165960-f0ad-cb30-1617-4a35e0ab0d90" }, "outputs": [], "source": [ "plt.scatter(houses[\"1stFlrSF\"],houses.SalePrice, color='red')\n", "plt.title(\"Sale Price wrt 1st floor\")\n", "plt.ylabel('Sale Price (in dollars)')\n", "plt.xlabel(\"1st Floor in square feet\");" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "e52f461a-eaea-864d-aaaa-7b16b67fe4d2" }, "source": [ "*Ground Living Area w.r.t SalePrice*\n", "--------------------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "66f3a8b6-bde4-bea8-1c41-bb1cb60c4f69" }, "outputs": [], "source": [ "plt.scatter( houses[\"GrLivArea\"],houses[\"SalePrice\"],color='purple')\n", "plt.title(\"Sale Price wrt Ground living area\")\n", "plt.ylabel('Sale Price')\n", "plt.xlabel(\"Ground living area\");" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "2877cc8b-8044-5758-d2e5-db3c7df0edc5" }, "source": [ "*SalePrice per square foot*\n", "--------------------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "9d0cacbb-eb31-1c44-f334-adcefbdcab69" }, "outputs": [], "source": [ "houses['SalePriceSF'] = houses['SalePrice']/houses['GrLivArea']\n", "plt.hist(houses['SalePriceSF'], bins=15,color=\"gold\")\n", "plt.title(\"Sale Price per Square Foot\")\n", "plt.ylabel('Number of Sales')\n", "plt.xlabel('Price per square feet');" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "3d9d7b41-035e-2a98-0f2f-375bc8706c9b" }, "outputs": [], "source": [ "#Average Sale Price per square feet \n", "print(\"$\",houses.SalePriceSF.mean())" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "cdb97a47-019f-710f-a81f-8a5aac04259f" }, "source": [ "*Garage Area*\n", "-------------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "06d82e4b-fcd8-c519-8895-f5aae077672e" }, "outputs": [], "source": [ "plt.scatter(houses[\"GarageArea\"],houses.SalePrice, color='green')\n", "plt.title(\"Sale Price vs Garage Area\")\n", "plt.ylabel('Sale Price(in dollars)')\n", "plt.xlabel(\"Garage Area in sq foot\");" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "3250d58a-3b7d-7205-c742-f4386445505b" }, "source": [ "*Building , remodelling years and age of house*\n", "----------------------------------------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "936722d0-b4a6-b32c-13d8-3763ec9b2ed4" }, "outputs": [], "source": [ "sns.distplot(houses[\"YearBuilt\"],color='seagreen');" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "d2cb72a2-4fdd-bdd4-1927-46ec50cde111" }, "outputs": [], "source": [ "sns.distplot(houses[\"YearRemodAdd\"],color='r');" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "504253bf-d05f-d557-b872-2e9f9e036e62" }, "outputs": [], "source": [ "houses['ConstructionAge'] = houses['YrSold'] - houses['YearBuilt']\n", "plt.scatter(houses['ConstructionAge'], houses['SalePriceSF'])\n", "plt.ylabel('Price per square foot (in dollars)')\n", "plt.xlabel(\"Construction Age of house\");" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "db9da79b-192b-edff-7d17-4a54754fcd4f" }, "source": [ "Price of house goes down with its age." ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "1cc5c20a-72bd-3655-2d28-9a5e85221470" }, "source": [ "*Heating and AC arrangements*\n", "-----------------------------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "cc81b112-0a77-8889-e7ff-a732ceb11b02" }, "outputs": [], "source": [ "sns.stripplot(x=\"HeatingQC\", y=\"SalePrice\",data=houses,hue='CentralAir',jitter=True,split=True)\n", "plt.title(\"Sale Price vs Heating Quality\");" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "a0c844f7-4e68-4024-1917-9adfa1fe1528" }, "source": [ "Having AC definitely escalates price of house." ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "5f33c9a1-b4e5-8e72-b9fe-0253c9be8a95" }, "source": [ "*Bathrooms in house*\n", "--------------------------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "2fd51149-d72b-2533-2bb3-11c46ca4410c" }, "outputs": [], "source": [ "sns.boxplot(houses[\"FullBath\"],houses[\"SalePrice\"])\n", "plt.title(\"Sale Price vs Full Bathrooms\");" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "438c40fd-005a-0ecf-9096-be1d42ac43e8" }, "outputs": [], "source": [ "sns.violinplot( houses[\"HalfBath\"],houses[\"SalePrice\"])\n", "plt.title(\"Sale Price vs Half Bathrooms\");" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "9580fc8c-ab87-7581-4a27-603d84ca24c2" }, "source": [ "*Total rooms above grade*\n", "-------------------------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "28538e2e-c825-a89a-8dcc-ee6badf5c6a8" }, "outputs": [], "source": [ "sns.barplot(houses[\"TotRmsAbvGrd\"],houses[\"SalePrice\"],palette=\"Blues_d\")\n", "plt.title(\"Sale Price vs Number of rooms\");" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "875b12a4-9fcc-623c-a420-167f036fb3ba" }, "source": [ "*Kitchen Quality*\n", "=================" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "a13d53e2-fb99-1721-77f8-0af3b0ed9f32" }, "outputs": [], "source": [ "sns.factorplot(\"KitchenAbvGr\",\"SalePrice\",data=houses,hue=\"KitchenQual\")\n", "plt.title(\"Sale Price vs Kitchen\");" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "de482374-3324-33df-5441-3c4102df5333" }, "source": [ "Having 1 Kitchen of Excellent quality hikes house price like anything." ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "eeb31a0e-499a-8883-f672-56aac64ac40c" }, "source": [ "*Neighbourhood*\n", "--------------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "20e32ecb-7d47-344b-4cbe-af90a9b5a3fb" }, "outputs": [], "source": [ "plt.xticks(rotation=45) \n", "sns.barplot(houses[\"Neighborhood\"],houses[\"SalePrice\"])\n", "plt.title(\"Sale Price vs Neighborhood\");" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "12538606-3ab5-0bc5-132c-244fe12b2082" }, "source": [ "*Overall Quality*\n", "-----------------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "1fedf48b-e200-809d-77d2-951f29994829" }, "outputs": [], "source": [ "plt.barh(houses[\"OverallQual\"],width=houses[\"SalePrice\"],color=\"r\")\n", "plt.title(\"Sale Price vs Overall Quality of house\")\n", "plt.ylabel(\"Overall Quality of house\")\n", "plt.xlabel(\"Sale Price\");" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "5878a644-5ea5-75bd-d57d-a3efb0d7e8c3" }, "source": [ "*2nd Floor with SalePrice*\n", "--------------------------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "846de2d4-6f90-ded9-2265-cf9efae98e37" }, "outputs": [], "source": [ "plt.scatter(houses[\"2ndFlrSF\"],houses[\"SalePrice\"],color=\"gold\")\n", "plt.title(\"Sale Price vs 2nd floor in sq feet\");\n", "plt.xlabel(\"2nd floor in sq feet\")\n", "plt.ylabel(\"Sale Price\");" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "86a40ff4-dc3b-aa72-4d97-4f2605df1eab" }, "source": [ "*Street*\n", "--------" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "d8e8ba7d-6e67-b9e9-1473-1dabb61d67b7" }, "outputs": [], "source": [ "#most streets are paved lets visulalize it\n", "sns.stripplot(x=houses[\"Street\"], y=houses[\"SalePrice\"],jitter=True)\n", "plt.title(\"Sale Price vs Streets\");" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "f155b1db-c9ea-de65-9b1d-34a3c3880fe7" }, "source": [ "More to come .. Watch this space." ] } ], "metadata": { "_change_revision": 0, "_is_fork": false, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.0" } }, "nbformat": 4, "nbformat_minor": 0 }