{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# House Prices - Regression Problem\n\nHello everyone,\n\n\"House Prices - Advanced Regression Techniques\" was the first Kaggle competition that I joined. After participating in different competitions for several months and gaining some experience, I wanted to return to the same notebook and see what I can do better.\n\n---\n\n**Goal**: Predicting the sales price for each house using 79 variables\n\n**Metric**: RMSE between the logarithm of the *predicted value* and the logarithm of the *real value*\n\n---\n\n**Index**\n\n1. [Sale Price Analysis](#1.-Sale-Price-Analysis)\n\n 1.1. [Log Transformation](#1.1.-Log-Transformation)\n\n2. [Exploratory Data Analysis](#2.-Exploratory-Data-Analysis)\n\n 2.1. [Overall Quality](#2.1.-Overall-Quality)\n\n 2.2. [Living Area](#2.2.-Living-Area)\n\n 2.3. [Basement Area](#2.3.-Basement-Area)\n\n 2.4. [Other Basement Variables](#2.4.-Other-Basement-Variables)\n\n 2.5. [Garage](#2.5.-Garage)\n\n 2.6. [Area and Quality](#2.6.-Area-and-Quality)\n\n 2.7. [Date Related Variables](#2.7.-Date-Related-Variables)\n\n 2.8. [Pool](#2.8.-Pool)\n\n 2.9. [Bathroom](#2.9.-Bathroom)\n\n 2.10. [Porch Area](#2.10.-Porch-Area)\n\n 2.11. [MSSubClass](#2.11.-MSSubClass)\n\n 2.12. [Other Features to Drop](#2.12.-Other-Features-to-Drop)\n \n3. [Data Preparation](#3.-Data-Preparation)\n\n 3.1. [Log Transforms and Dropping Columns](#3.1.-Log-Transforms-and-Dropping-Columns)\n \n 3.2. [Missing Values](#3.2.-Missing-Values)\n \n 3.3. [Converting Categorical Variables](#3.3.-Converting-Categorical-Variables)\n\n4. [Model Building](#4.-Model-Building)\n\n 4.1. [Model Selection](#4.1.-Model-Selection)\n \n 4.2. [Random Forest Hyperparameter Tuning](#4.2.-Random-Forest-Hyperparameter-Tuning)\n \n 4.3. [XGBoost](#4.3.-XGBoost)\n \n 4.4. [Prediction and Submission](#4.4.-Prediction-and-Submission)","metadata":{}},{"cell_type":"code","source":"# importing packages\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom scipy.stats import norm\nfrom scipy import stats\n%matplotlib inline\n\ntrain = pd.read_csv(\"//kaggle//input//house-prices-advanced-regression-techniques//train.csv\")\ntest = pd.read_csv(\"//kaggle//input//house-prices-advanced-regression-techniques//test.csv\")\ntrain.head()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Columns:\")\ntrain.columns","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Sale Price Analysis","metadata":{}},{"cell_type":"markdown","source":"Looking at the distribution and the summary, we can say that\n* Sale Price is a **numerical** variable and consists of **positive** values.\n* The **average** house price is **180k**.\n* The **minimum** house price is **34.9k**, while the **maximum** is **755k**.\n* The distribution is **positively skewed**.","metadata":{}},{"cell_type":"code","source":"sns.displot(train[\"SalePrice\"], kde=True, aspect=1.5)\nplt.show()\n\nprint(\"\\nSALE PRICE SUMMARY: \")\ntrain[\"SalePrice\"].describe()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1.1. Log Transformation\n\nFor positively skewed variables, it is a good idea to apply log transformation so that the distribution can converge to a Gaussian distribution. \n\nSale Price after log transform applied:\n* varies between 10.46 and 13.53; with a mean of 12.02\n* The distribution become closer to a normal distribution.\n* Probability plot follows the diagonal line better.","metadata":{}},{"cell_type":"code","source":"# applying log transformation\ntrain[\"SalePrice_Log\"] = np.log(train[\"SalePrice\"])\n\n# plots:\nfig, ax = plt.subplots(2, 2, figsize=(12, 6))\nsns.histplot(ax = ax[0,0], data=train, x=\"SalePrice\", kde=True)\nres = stats.probplot(train[\"SalePrice\"], plot=ax[0,1])\nax[0,0].set_title(\"Sale Price Distribution (Before Log Transform)\")\nax[0,1].set_title(\"Probability Plot (Before Log Transform)\")\n\nsns.histplot(ax = ax[1,0], data=train, x=\"SalePrice_Log\", kde=True)\nres = stats.probplot(train[\"SalePrice_Log\"], plot=ax[1,1])\nax[1,0].set_title(\"Sale Price Distribution (After Log Transform)\")\nax[1,1].set_title(\"Probability Plot (After Log Transform)\")\nfig.tight_layout()\nplt.show()\n\nprint(\"\\nSale Price After Log Transform: \")\ntrain[\"SalePrice_Log\"].describe()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Exploratory Data Analysis","metadata":{}},{"cell_type":"code","source":"# Correlation Matrix\ncorr_matrix = train.corr()\nplt.figure(figsize = (9,8)) # figure size\nsns.heatmap(corr_matrix)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Lets see top 15 variables which have the highest correlation with Sale Price\ntop_cols = corr_matrix.sort_values(by=\"SalePrice_Log\", ascending=False).head(15).index\ntop_corr_matrix = train[top_cols].corr() # correlation matrix\nplt.figure(figsize = (9,8)) # figure size\nsns.heatmap(top_corr_matrix, annot=True, square=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# creating two empty lists to record variables in this section\nvars_to_drop = list()\nvars_log_transform = list()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.1. Overall Quality\n\n* The correlation between overall quality and sale price is high.\n* As the quality increases, the sale price increases as well.\n* I will use OverallQual variable as it is.","metadata":{}},{"cell_type":"code","source":"sns.stripplot(x=\"OverallQual\", y=\"SalePrice_Log\", data=train)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.2. Living Area\n\n* Above grade living area (GrLivArea) is the second most correlated variable with Sale Price.\n* GrLivArea is the sum of:\n    * 1stFlrSF (first floor square feet)\n    * 2ndFlrSF (second floor square feet)\n    * LowQualFinSF (low quality finished square feet)\n* GrLivArea and TotRmsAbvGrd (total rooms above grade) have a strong correlation. ","metadata":{}},{"cell_type":"code","source":"# Scatter plots\ncols = [\"SalePrice\", \"GrLivArea\", \"TotRmsAbvGrd\", \"1stFlrSF\", \"2ndFlrSF\", \"LowQualFinSF\"]\nsns.pairplot(data=train[cols])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I am planning to **use GrLivArea and 1stFlrSF in the model**. They both consist of nonzero values and have a positive relation with Sale Price. Because of the positive skewness, I will apply log transform to these variables.\n\nI will **drop TotRmsAbvGrd, 2ndFlrSF and LowQualFinSF**. The correlation between TotRmsAbvGrd and GrLivArea is high, I'm already using GrLivArea. 2ndFlrSF and LowQualFinSF have zero values which causes nonlinearity.","metadata":{}},{"cell_type":"code","source":"vars_log_transform.append(\"GrLivArea\")\nvars_log_transform.append(\"1stFlrSF\")\n\nvars_to_drop.append(\"TotRmsAbvGrd\")\nvars_to_drop.append(\"2ndFlrSF\")\nvars_to_drop.append(\"LowQualFinSF\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.3. Basement Area\n\nTotalBsmtSF is the total square feet of basement area and it is the sum of:\n* BsmtFinSF1 (type 1 finished square feet)\n* BsmtFinSF2 (type 2 finished square feet)\n* BsmtUnfSF (unfinished square feet of basement area)","metadata":{}},{"cell_type":"code","source":"cols = [\"SalePrice\", \"TotalBsmtSF\", \"1stFlrSF\", \"BsmtFinSF1\", \"BsmtFinSF2\", \"BsmtUnfSF\"]\nsns.pairplot(data=train[cols])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I am already **using 1stFlrSF**, so I will **drop TotalBsmtSF**. \n* 1stFlrSF and TotalBsmtSF have a strong correlation to each other. \n* TotalBsmtSF has zero values, which show houses without a basement.\n* I will create a new column \"HasBsmt\" to see if the house has a basement.","metadata":{}},{"cell_type":"code","source":"# new column as \"Has Basement\"\ntrain[\"HasBsmt\"] = 0\ntrain.loc[train[\"TotalBsmtSF\"]>0, \"HasBsmt\"] = 1\ntest[\"HasBsmt\"] = 0\ntest.loc[test[\"TotalBsmtSF\"]>0, \"HasBsmt\"] = 1\n\n# i will drop TotalBsmtSF\nvars_to_drop.append(\"TotalBsmtSF\")\n\n# HasBsmt vs. SalePrice\nsns.boxplot(x=\"HasBsmt\", y=\"SalePrice_Log\", data=train)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.4. Other Basement Variables\n\nIn the categorical Basement variables, there are NA values. Looking at the data description file, these NA values correspond to the houses without a basement. So, I will fill them with \"NoBsmt\" and see the plots.","metadata":{}},{"cell_type":"code","source":"bsmt_vars = [\"BsmtQual\", \"BsmtCond\", \"BsmtExposure\", \"BsmtFinType1\", \"BsmtFinType2\"]\ntrain[bsmt_vars] = train[bsmt_vars].fillna(value=\"NoBsmt\")\ntest[bsmt_vars] = test[bsmt_vars].fillna(value=\"NoBsmt\")\n\nfig, ax = plt.subplots(2, 3, figsize=(12, 6))\nsns.stripplot(ax = ax[0,0], x=\"BsmtQual\", y=\"SalePrice_Log\", data=train)\nsns.stripplot(ax = ax[0,1], x=\"BsmtCond\", y=\"SalePrice_Log\", data=train)\nsns.stripplot(ax = ax[0,2], x=\"BsmtExposure\", y=\"SalePrice_Log\", data=train)\nsns.stripplot(ax = ax[1,0], x=\"BsmtFinType1\", y=\"SalePrice_Log\", data=train)\nsns.stripplot(ax = ax[1,1], x=\"BsmtFinType2\", y=\"SalePrice_Log\", data=train)\nax[1,2].remove()\nfig.tight_layout()\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.5. Garage\n\nGarage car capacity (GarageCars) and garage area in square feet (GarageArea) are two important **numerical variables**, both have a **strong correlation to Sale Price**, and also to **each other**.","metadata":{}},{"cell_type":"code","source":"cols = [\"SalePrice\", \"GarageCars\", \"GarageArea\"]\nsns.pairplot(data=train[cols])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I am planning to **use GarageCars** and **drop GarageArea**. Correlation between SalePrice_Log and GarageCars is slightly higher than correlation between SalePrice_Log and GarageArea. Also, GarageArea has zero values which causes nonlinearity. These zero values are equivalent to 0 in GarageCars.\n\nIn the categorical garage variables, NA values correspond to the houses without a garage. Lets fill them with \"NoGrg\" and see the plots.","metadata":{}},{"cell_type":"code","source":"vars_to_drop.append(\"GarageArea\")\n\ngarage_vars = [\"GarageType\", \"GarageFinish\", \"GarageQual\", \"GarageCond\"]\ntrain[garage_vars] = train[garage_vars].fillna(value=\"NoGrg\")\ntest[garage_vars] = test[garage_vars].fillna(value=\"NoGrg\")\n\nfig, ax = plt.subplots(2, 2, figsize=(12, 6))\nsns.stripplot(ax = ax[0,0], x=\"GarageType\", y=\"SalePrice_Log\", data=train)\nsns.stripplot(ax = ax[0,1], x=\"GarageFinish\", y=\"SalePrice_Log\", data=train)\nsns.stripplot(ax = ax[1,0], x=\"GarageQual\", y=\"SalePrice_Log\", data=train)\nsns.stripplot(ax = ax[1,1], x=\"GarageCond\", y=\"SalePrice_Log\", data=train)\nfig.tight_layout()\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# For GarageType, I'll replace \"Basment\", \"CarPort\" and \"2Types\" to \"Other\"\nto_replace = [\"Basment\", \"CarPort\", \"2Types\"]\ntrain[\"GarageType\"] = train[\"GarageType\"].replace(to_replace, \"Other\")\ntest[\"GarageType\"] = test[\"GarageType\"].replace(to_replace, \"Other\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.6. Area and Quality\n\nIn the correlation matrix, we have seen that Overall Quality is the most correlated variable with SalePrice. After that, variables related to area is really important for the SalePrice; such as living area, garage area, basement area. In this section, I want to create two new features as:\n1. Total Area, which is the sum of \"Living Area\", \"Basement\" and \"Garage\"\n2. Quality Area, which is the multiplication of \"Total Area\" and \"Overall Quality\"","metadata":{}},{"cell_type":"code","source":"# There are 2 missing values in the test dataset, first filling them with 0\ntest[[\"TotalBsmtSF\", \"GarageArea\"]] = test[[\"TotalBsmtSF\", \"GarageArea\"]].fillna(value=0)\n\n# Total Area\ntrain[\"TotalArea\"] = train[\"GrLivArea\"] + train[\"TotalBsmtSF\"] + train[\"GarageArea\"]\ntest[\"TotalArea\"] = test[\"GrLivArea\"] + test[\"TotalBsmtSF\"] + test[\"GarageArea\"]\n\n# Quality Area\ntrain[\"QualArea\"] = train[\"TotalArea\"] * train[\"OverallQual\"]\ntest[\"QualArea\"] = test[\"TotalArea\"] * test[\"OverallQual\"]\n\n# since area variables are positively skewed, I'll apply log transformation\nvars_log_transform.append(\"TotalArea\")\nvars_log_transform.append(\"QualArea\")\nvars_log_transform.append(\"LotArea\")\n\n# Plots:\nfig, ax = plt.subplots(1, 2, figsize=(12, 3.5))\nsns.scatterplot(ax = ax[0], x=\"TotalArea\", y=\"SalePrice\", data=train)\nsns.scatterplot(ax = ax[1], x=\"QualArea\", y=\"SalePrice\", data=train)\nfig.tight_layout()\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I am very happy with these 2 new features, they both have a positive relationship with the Sale Price. Especially Quality Area and Sale Price have a relation that is really close to a linear.\n\nI have one more thing to fix, there are **two outliers**. Even these two houses have a large space, they are cheap. This can affect the model badly, so I will drop these observations and plot again.","metadata":{}},{"cell_type":"code","source":"row_ind = train[(train[\"TotalArea\"]>8000) & (train[\"SalePrice\"]<300000)].index\ntrain = train.drop(row_ind, axis=0)\n\nfig, ax = plt.subplots(1, 2, figsize=(12, 3.5))\nsns.scatterplot(ax = ax[0], x=\"TotalArea\", y=\"SalePrice\", data=train)\nsns.scatterplot(ax = ax[1], x=\"QualArea\", y=\"SalePrice\", data=train)\nfig.tight_layout()\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.7. Date Related Variables\n\n* YearBuilt and GarageYrBlt have a strong correlation to each other. I will use the YearBuilt column and drop the GarageYrBlt column.\n* YearRemodAdd has also a linear relationship with Sale Price, I will use this variable too.\n* Looking at the sale date, there is no clear relationship between SalePrice and MoSold/YrSold. I will drop MoSold and YrSold.","metadata":{}},{"cell_type":"code","source":"cols = [\"SalePrice_Log\", \"YearBuilt\", \"GarageYrBlt\", \"YearRemodAdd\"]\nsns.pairplot(data=train[cols])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(12, 4))\nsns.regplot(ax = ax[0], x=\"MoSold\", y=\"SalePrice_Log\", data=train)\nsns.regplot(ax = ax[1], x=\"YrSold\", y=\"SalePrice_Log\", data=train)\nfig.tight_layout()\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vars_to_drop.append(\"GarageYrBlt\")\nvars_to_drop.append(\"MoSold\")\nvars_to_drop.append(\"YrSold\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.8. Pool\n\nPool Quality variable has only 7 values, because most of the houses have no pool. Looking at the scatterplot, these 7 houses have a higher price than the average. I will create a new column as \"HasPool\" and drop PoolArea & PoolQC.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(12, 3.5))\nsns.scatterplot(ax = ax[0], x=\"PoolArea\", y=\"SalePrice_Log\", data=train)\nsns.stripplot(ax = ax[1], x=\"PoolQC\", y=\"SalePrice_Log\", data=train)\nfig.tight_layout()\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"HasPool\"] = 0\ntrain.loc[train[\"PoolArea\"]>0, \"HasPool\"] = 1\ntest[\"HasPool\"] = 0\ntest.loc[test[\"PoolArea\"]>0, \"HasPool\"] = 1\n\nvars_to_drop.append(\"PoolArea\")\nvars_to_drop.append(\"PoolQC\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.9. Bathroom\n\nThere are 4 variables related to bathroom: FullBath, BsmtFullBath, HalfBath and BsmtHalfBath. Looking at the plots, BsmtHalfBath has no significant relation with the Sale Price, I will drop this column. Also, I will create a new column for the total number of bathrooms.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(1, 4, figsize=(12, 3))\nsns.regplot(ax = ax[0], x=\"FullBath\", y=\"SalePrice_Log\", data=train)\nsns.regplot(ax = ax[1], x=\"BsmtFullBath\", y=\"SalePrice_Log\", data=train)\nsns.regplot(ax = ax[2], x=\"HalfBath\", y=\"SalePrice_Log\", data=train)\nsns.regplot(ax = ax[3], x=\"BsmtHalfBath\", y=\"SalePrice_Log\", data=train)\nfig.tight_layout()\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# There are 4 missing values in the test dataset, first filling them with 0\ntest[[\"BsmtFullBath\", \"BsmtHalfBath\"]] = test[[\"BsmtFullBath\", \"BsmtHalfBath\"]].fillna(value=0)\n\n# Total Bathrooms:\ntrain[\"TotalBath\"]= train[\"FullBath\"]+ train[\"BsmtFullBath\"]+ 0.5*(train[\"HalfBath\"]+ train[\"BsmtHalfBath\"])\ntest[\"TotalBath\"]= test[\"FullBath\"]+ test[\"BsmtFullBath\"]+ 0.5*(test[\"HalfBath\"]+ test[\"BsmtHalfBath\"])\n\nvars_to_drop.append(\"BsmtHalfBath\")\n\nsns.regplot(x=\"TotalBath\", y=\"SalePrice_Log\", data=train)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.10. Porch Area\n\nThere are several variables related to porch area. I am planning to create a new column for the total porch area.","metadata":{}},{"cell_type":"code","source":"train[\"TotalPorch\"]=train[\"WoodDeckSF\"]+train[\"OpenPorchSF\"]+train[\"EnclosedPorch\"]+train[\"3SsnPorch\"]+train[\"ScreenPorch\"]\ntest[\"TotalPorch\"]=test[\"WoodDeckSF\"]+test[\"OpenPorchSF\"]+test[\"EnclosedPorch\"]+test[\"3SsnPorch\"]+test[\"ScreenPorch\"]\n\nfig, ax = plt.subplots(2, 3, figsize=(12, 6))\nsns.regplot(ax = ax[0,0], x=\"WoodDeckSF\", y=\"SalePrice_Log\", data=train)\nsns.regplot(ax = ax[0,1], x=\"OpenPorchSF\", y=\"SalePrice_Log\", data=train)\nsns.regplot(ax = ax[0,2], x=\"EnclosedPorch\", y=\"SalePrice_Log\", data=train)\nsns.regplot(ax = ax[1,0], x=\"3SsnPorch\", y=\"SalePrice_Log\", data=train)\nsns.regplot(ax = ax[1,1], x=\"ScreenPorch\", y=\"SalePrice_Log\", data=train)\nsns.regplot(ax = ax[1,2], x=\"TotalPorch\", y=\"SalePrice_Log\", data=train)\nfig.tight_layout()\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.11. MSSubClass\n\nAccording to the data description file, MSSubClass variable identifies the type of dwelling involved in the sale. MSSubClass is a categorical variable but the column contains integer values. I will convert this column into string.","metadata":{}},{"cell_type":"code","source":"# saving as string, since it is a categorical variable\ntrain[\"MSSubClass\"] = train[\"MSSubClass\"].astype(str)\ntest[\"MSSubClass\"] = test[\"MSSubClass\"].astype(str)\n\nsns.stripplot(x=\"MSSubClass\", y=\"SalePrice_Log\", data=train)\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.12. Other Features to Drop\n\nIn the following variables, most of the observations are concentrated on a single value. I will drop these variables.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(2, 3, figsize=(12, 6))\nsns.scatterplot(ax = ax[0,0], x=\"MiscVal\", y=\"SalePrice_Log\", data=train)\nsns.stripplot(ax = ax[0,1], x=\"Street\", y=\"SalePrice_Log\", data=train)\nsns.stripplot(ax = ax[0,2], x=\"Utilities\", y=\"SalePrice_Log\", data=train)\nsns.stripplot(ax = ax[1,0], x=\"Condition2\", y=\"SalePrice_Log\", data=train)\nsns.stripplot(ax = ax[1,1], x=\"RoofMatl\", y=\"SalePrice_Log\", data=train)\nsns.stripplot(ax = ax[1,2], x=\"Heating\", y=\"SalePrice_Log\", data=train)\nfig.tight_layout()\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vars_to_drop.append(\"MiscVal\")\nvars_to_drop.append(\"Street\")\nvars_to_drop.append(\"Utilities\")\nvars_to_drop.append(\"Condition2\")\nvars_to_drop.append(\"RoofMatl\")\nvars_to_drop.append(\"Heating\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Data Preparation\n\n## 3.1. Log Transforms and Dropping Columns\n\nIn the EDA section, I recorded some variables to apply log transform or drop. Let's make them.","metadata":{}},{"cell_type":"code","source":"# Log transforms:\nfor col in vars_log_transform:\n    train[col] = np.log(train[col])\n    test[col] = np.log(test[col])\n\nprint(\"Log transform is applied to following variables:\")\nprint(\", \".join(vars_log_transform))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Drops:\ntrain = train.drop(vars_to_drop, axis=1)\ntest = test.drop(vars_to_drop, axis=1)\n\nprint(\"Following variables are dropped:\")\nprint(\", \".join(vars_to_drop))\n\n# since we have saleprice_log, we won't need sale price\ntrain = train.drop(\"SalePrice\", axis=1)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.2. Missing Values","metadata":{}},{"cell_type":"code","source":"# calculating number of missing values in each columns\ntrain_miss = train.isnull().sum().sort_values(ascending=False).to_frame(\"N_MissVal\")\n\n# also i would like to see type of the columns (categorical / numerical)\ncat_cols = train.select_dtypes(include=['object']).columns # categorical columns\ntrain_miss[\"ValType\"] = np.where(train_miss.index.isin(cat_cols), \"Categorical\", \"Numerical\")\n\n# printing columns that have missing values\nprint(\"Training Data:\")\ntrain_miss[train_miss[\"N_MissVal\"]>0]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# For the following 4 features, missing value shows houses without that feature\n# I'll fill them with \"None\"\ncols = [\"FireplaceQu\", \"Fence\", \"Alley\", \"MiscFeature\"]\ntrain[cols] = train[cols].fillna(value=\"None\")\ntest[cols] = test[cols].fillna(value=\"None\")\n\n# For the remaining, I will drop columns with more than 1 missing value\nto_drop = train_miss[train_miss[\"N_MissVal\"]>1].index\nto_drop = [x for x in to_drop if x not in cols] # excluding columns i just filled\ntrain = train.drop(to_drop, axis=1)\ntest = test.drop(to_drop, axis=1)\n\n# Electrical has only 1 missing data, I will drop corresponding row\nrow_ind = train.loc[train[\"Electrical\"].isnull()].index\ntrain = train.drop(row_ind, axis=0)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# now let's see if test data have any other missing data:\ntest_miss = test.isnull().sum().sort_values(ascending=False).to_frame(\"N_MissVal\")\ncat_cols = test.select_dtypes(include=['object']).columns # categorical columns\ntest_miss[\"ValType\"] = np.where(test_miss.index.isin(cat_cols), \"Categorical\", \"Numerical\")\n\nprint(\"Test Data:\")\ntest_miss[test_miss[\"N_MissVal\"]>0]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To fill **MSZoning** in the test data, I want to look at the Neighborhood. There are 4 missing values, 3 of them are in IDOTRR and the other one is in Mitchel. In IDOTRR, most of the houses are in the Residential Medium Density zone (RM), while Residential Low Density (RL) dominates the Mitchel. I will fill missing values according to the most occurent MSZoning value in the same neighborhood.","metadata":{}},{"cell_type":"code","source":"print(\"Missing MSZoning in test data\")\nprint(test.loc[test.MSZoning.isnull(), [\"MSZoning\", \"Neighborhood\"]])\n\nprint(\"\\n\\nMSZoning Value Counts for IDOTRR:\")\nprint(test.loc[test.Neighborhood==\"IDOTRR\", [\"MSZoning\"]].value_counts())\n\nprint(\"\\n\\nMSZoning Value Counts for Mitchel:\")\nprint(test.loc[test.Neighborhood==\"Mitchel\", [\"MSZoning\"]].value_counts())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# filling MSZoning:\ntest.loc[test.MSZoning.isnull() & (test.Neighborhood == \"IDOTRR\"),\n         \"MSZoning\"] = \"RM\"\ntest.loc[test.MSZoning.isnull() & (test.Neighborhood == \"Mitchel\"),\n         \"MSZoning\"] = \"RL\"\n\n# numeric missing values related to garage/basement are houses without garage/basement\n# i will fill them with 0\ncols = [\"GarageCars\", \"BsmtUnfSF\", \"BsmtFinSF1\", \"BsmtFinSF2\"]\ntest[cols] = test[cols].fillna(value=0)\n\n# for remaining categorical variables, I will fill with the most occurent category\ntest = test.apply(lambda x: x.fillna(x.value_counts().index[0]))\n\n#checking if there is any missing value left\nprint(\"Remaining missing values (train): {}\".format(train.isnull().sum().sum()))\nprint(\"Remaining missing values (test): {}\".format(test.isnull().sum().sum()))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.3. Converting Categorical Variables","metadata":{}},{"cell_type":"code","source":"train.columns","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"Train data, shape before conversion: {train.shape}\")\nprint(f\"Test data, shape before conversion:  {test.shape}\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.get_dummies(train)\ntest = pd.get_dummies(test)\n\nprint(f\"Train data, shape after conversion: {train.shape}\")\nprint(f\"Test data, shape after conversion:  {test.shape}\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Before transforming categorical variables, training data have one more column than the test data, which is SalePrice. But after conversion, training data have a lot more columns. That's because there are more categories in the training data. For modeling and prediction, shapes of training and test should be same. I will find missing columns, create them and fill with 0.","metadata":{}},{"cell_type":"code","source":"# finding and creating missing columns\ncols_to_create = list(set(train.columns) - set(test.columns))\ntest[cols_to_create] = 0\n\n# reordering columns of test data, same as train data\ntest = test[train.columns]\ntest = test.drop(\"SalePrice_Log\", axis=1) # of course, don't need SalePrice in test :)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Model Building","metadata":{}},{"cell_type":"code","source":"from sklearn.linear_model import LinearRegression\nfrom sklearn.svm import SVR\nfrom sklearn.tree import DecisionTreeRegressor\nfrom sklearn.ensemble import RandomForestRegressor\nfrom xgboost import XGBRegressor\nfrom sklearn.model_selection import cross_val_score, train_test_split\nfrom sklearn.model_selection import GridSearchCV, RandomizedSearchCV\nfrom sklearn.metrics import mean_squared_error\n\ndef print_rmse(regressor, X, y, cv=5):\n    neg_mse = cross_val_score(estimator=regressor, X=X, y=y, cv=cv,\n                              scoring = \"neg_mean_squared_error\")\n    rmse = np.sqrt(-neg_mse).mean()\n    print(f\"Regressor: {regressor}\")\n    print(f\"RMSE: {rmse:.5f}\\n\")\n    return","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train = train.drop([\"SalePrice_Log\", \"Id\"], axis=1)\nX_test = test.drop([\"Id\"], axis=1)\n\nX_train = X_train.values\nX_test = X_test.values\ny_train = train[\"SalePrice_Log\"].values","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.1. Model Selection","metadata":{}},{"cell_type":"code","source":"# Linear Regression\nregressor = LinearRegression()\nprint_rmse(regressor, X_train, y_train)\n\n# Support Vector Regression\nregressor = SVR()\nprint_rmse(regressor, X_train, y_train)\n\n# Decision Tree Regressor\nregressor = DecisionTreeRegressor(random_state = 42)\nprint_rmse(regressor, X_train, y_train)\n\n# Random Forest Regression\nregressor = RandomForestRegressor(random_state = 42)\nprint_rmse(regressor, X_train, y_train)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.2. Random Forest Hyperparameter Tuning","metadata":{}},{"cell_type":"code","source":"parameters = {\n    \"n_estimators\": [100, 200, 500, 1000],\n    \"max_depth\": [20, 40, 100, None],\n    \"min_samples_split\": [2, 3, 5],\n    \"min_samples_leaf\": [1, 2, 4],\n    \"max_features\": [1, \"sqrt\", \"log2\"]\n    }\n\ngrid = RandomizedSearchCV(RandomForestRegressor(random_state=42),\n                          parameters, n_iter=12, refit=True,\n                          scoring=\"neg_mean_squared_error\",\n                          random_state = 42)\ngrid.fit(X_train, y_train)\n\n# printing results\nprint(\"Best Score: {}\".format(grid.best_score_))\nprint(\"Best Parameters: {}\".format(grid.best_params_))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"regressor = grid.best_estimator_\nprint_rmse(regressor, X_train, y_train)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.3. XGBoost\n\nFor XGBoost, I want to use early stopping to avoid overfitting. To do this, I need to split training data into training & validation. While fitting XGBRegressor, I will set early stopping rounds and use validation data as eval set.","metadata":{}},{"cell_type":"code","source":"# splitting X_train and y_train\nX_train_xgb, X_val, y_train_xgb, y_val = train_test_split(X_train, y_train, \n                                            test_size = 0.2,\n                                            random_state = 0)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"regressor = XGBRegressor(n_estimators=1000,\n                         learning_rate=0.05,\n                         eval_metric=mean_squared_error,\n                         random_state=0)\nregressor.fit(X_train_xgb, y_train_xgb, \n             early_stopping_rounds=10, \n             eval_set=[(X_val, y_val)], \n             verbose=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print_rmse(regressor, X_train_xgb, y_train_xgb)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.4. Prediction and Submission","metadata":{}},{"cell_type":"code","source":"# Prediction\ny_test = regressor.predict(X_test)\nresult = pd.DataFrame({\"Id\": test.Id, \"SalePrice\":y_test})\nresult.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Finally, to obtain real sale values we have to invert log transformation --> exponential\nresult[\"SalePrice\"] = np.exp(result[\"SalePrice\"])\nresult.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result.to_csv('submission.csv', index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}