{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# House Prices Prediction -- EDA & ML Model XGBoost","metadata":{}},{"cell_type":"markdown","source":"**CONTENTS:**\n\n* [Introduction](#1)\n* [Load and Check Data](#2)\n* [Variable Description](#3)\n* [Univariate Analysis](#4)\n* [Missing Value](#5)\n* [Data Analysis and Visualization](#6)\n* [Feature Engineering](#7)\n* [Model Selection](#8)\n* [Generate Prediction Value File](#9)","metadata":{}},{"cell_type":"markdown","source":"<a id = \"1\"></a><br>\n# Introduction","metadata":{}},{"cell_type":"markdown","source":"This project is going to predict the final sale price of houses located in Ames, Iowa. The datasets of this project contain 79 explanatory variables describing (almost) every aspect of residential homes.\\\n**Exploratory Data Analysis (EDA)** method is used to describe the data, viewing the distribution of the data, finding relationships between the data, cleaning the data and so on.\\\nThen Machine Learning methods are used to create a model which can predict sale price of each house listed on test dataset. In this notebook, 2 different machine learning Classifier are considered (**Random Forest** and **XGBoost**). Finally, higher **mean absolute error (MAE)** of which model can be selected to do the prediction.","metadata":{}},{"cell_type":"markdown","source":"<a id = \"2\"></a><br>\n# Load and Check Data","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\ntrain_data = pd.read_csv('../input/home-data-for-ml-course/train.csv')\n\ntest_data = pd.read_csv('../input/home-data-for-ml-course/test.csv')","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:31:34.513119Z","iopub.execute_input":"2022-07-19T07:31:34.513531Z","iopub.status.idle":"2022-07-19T07:31:35.234096Z","shell.execute_reply.started":"2022-07-19T07:31:34.513453Z","shell.execute_reply":"2022-07-19T07:31:35.232989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.head(10)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:31:38.402884Z","iopub.execute_input":"2022-07-19T07:31:38.403194Z","iopub.status.idle":"2022-07-19T07:31:38.439166Z","shell.execute_reply.started":"2022-07-19T07:31:38.403170Z","shell.execute_reply":"2022-07-19T07:31:38.438471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_data.head(5)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T08:23:36.833198Z","iopub.execute_input":"2022-07-19T08:23:36.833519Z","iopub.status.idle":"2022-07-19T08:23:36.858522Z","shell.execute_reply.started":"2022-07-19T08:23:36.833495Z","shell.execute_reply":"2022-07-19T08:23:36.857447Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"3\"></a><br>\n# Variable Description","metadata":{}},{"cell_type":"code","source":"# train data information\ntrain_data.info()","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:31:46.455266Z","iopub.execute_input":"2022-07-19T07:31:46.455686Z","iopub.status.idle":"2022-07-19T07:31:46.497030Z","shell.execute_reply.started":"2022-07-19T07:31:46.455651Z","shell.execute_reply":"2022-07-19T07:31:46.496018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train data description\ntrain_data.describe()","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:31:51.835109Z","iopub.execute_input":"2022-07-19T07:31:51.835535Z","iopub.status.idle":"2022-07-19T07:31:51.937330Z","shell.execute_reply.started":"2022-07-19T07:31:51.835505Z","shell.execute_reply":"2022-07-19T07:31:51.936370Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# type numbers of each variable in training dataset\ntrain_data['Neighborhood'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:31:55.868129Z","iopub.execute_input":"2022-07-19T07:31:55.868549Z","iopub.status.idle":"2022-07-19T07:31:55.876727Z","shell.execute_reply.started":"2022-07-19T07:31:55.868519Z","shell.execute_reply":"2022-07-19T07:31:55.875831Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"4\"></a><br>\n# Univariate Analysis","metadata":{}},{"cell_type":"markdown","source":"* **Categorical Variable:** MSSubClass, MSZoning, Street, Alley, LotShape, LandContour, Utilities, LotConfig, LandSlope, Neighborhood, Condition1, Condition2, BldgType, HouseStyle, OverallQual, OverallCond, YearBuilt, YearRemodAdd, RoofStyle, RoofMatl, Exterior1st, Exterior2nd, MasVnrType, ExterQual, ExterCond, Foundation, BsmtQual, BsmtCond, BsmtExposure, BsmtFinType1, BsmtFinType2, Heating, HeatingQC, CentralAir, Electrical, BsmtFullBath, BsmtHalfBath, FullBath, HalfBath, BedroomAbvGr, KitchenAbvGr, KitchenQual, TotRmsAbvGrd, Functional, Fireplaces, FireplaceQu, GarageType, GarageYrBlt, GarageFinish, GarageCars, GarageQual, GarageCond, PavedDrive, PoolQC, Fence, MiscFeature, MiscVal, MoSold, YrSold, SaleType, SaleCondition\n* **Numerical Variable:** LotFrontage, LotArea, MasVnrArea, BsmtFinSF1, BsmtFinSF2, BsmtUnfSF, TotalBsmtSF, 1stFlrSF, 2ndFlrSF, LowQualFinSF, GrLivArea, GarageArea, WoodDeckSF, OpenPorchSF, EnclosedPorch, 3SsnPorch, ScreenPorch, PoolArea, SalePrice","metadata":{}},{"cell_type":"markdown","source":"**Categorical Variable**\n\nFirstly, we'll find out how many groups each categorical variable can be classified, and the number of items in each group.","metadata":{}},{"cell_type":"code","source":"# A loop to show the groups can be classified for all categorical variables.\ncategorical_vari = ['MSSubClass', 'MSZoning', 'Street', 'Alley', 'LotShape','LandContour',\\\n                    'Utilities', 'LotConfig', 'LandSlope', 'Neighborhood', 'Condition1', \\\n                    'Condition2', 'BldgType', 'HouseStyle', 'OverallQual', 'OverallCond', \\\n                    'YearBuilt', 'YearRemodAdd', 'RoofStyle', 'RoofMatl', 'Exterior1st', \\\n                    'Exterior2nd', 'MasVnrType', 'ExterQual', 'ExterCond', 'Foundation', \\\n                    'BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1','BsmtFinType2',\\\n                    'Heating', 'HeatingQC', 'CentralAir', 'Electrical', 'BsmtFullBath', \\\n                    'BsmtHalfBath', 'FullBath','HalfBath', 'BedroomAbvGr','KitchenAbvGr',\\\n                    'KitchenQual', 'TotRmsAbvGrd','Functional','Fireplaces','FireplaceQu',\\\n                    'GarageType', 'GarageYrBlt', 'GarageFinish','GarageCars','GarageQual',\\\n                    'GarageCond', 'PavedDrive', 'PoolQC', 'Fence','MiscFeature','MiscVal',\\\n                    'MoSold', 'YrSold', 'SaleType', 'SaleCondition']\n\nfor x in categorical_vari:\n    print(\"{}: \\n\".format(x), \"{} \\n\".format(train_data[x].value_counts()))","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:31:59.898627Z","iopub.execute_input":"2022-07-19T07:31:59.899000Z","iopub.status.idle":"2022-07-19T07:31:59.955269Z","shell.execute_reply.started":"2022-07-19T07:31:59.898970Z","shell.execute_reply":"2022-07-19T07:31:59.954318Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the results above, we can define **YearBuilt, YearRemodAdd, GarageYrBlt** as the other kind of categorical variable, since these variables cannot be classified in limited groups (in this project more than 60 groups).","metadata":{}},{"cell_type":"markdown","source":"**Numerical Variable**\n\nSecondly, we try to use numerical variabes to draw some hist plots, and to see the distribution of each of them.","metadata":{}},{"cell_type":"code","source":"# A function to draw the hist plot for all numerical variables.\n\ndef hist_plot(variable):\n    \n    plt.figure(figsize = (9,3))\n    plt.hist(train_data[variable], bins = 50, color = \"#8e82fe\")\n    plt.xlabel(variable)\n    plt.ylabel(\"Frequency\")\n    plt.title(\"{} distribution with hist\".format(variable))\n    \n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:32:09.892549Z","iopub.execute_input":"2022-07-19T07:32:09.892932Z","iopub.status.idle":"2022-07-19T07:32:09.899973Z","shell.execute_reply.started":"2022-07-19T07:32:09.892903Z","shell.execute_reply":"2022-07-19T07:32:09.899033Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"numerical_vari = ['LotFrontage', 'LotArea', 'MasVnrArea', 'BsmtFinSF1', 'BsmtFinSF2', \\\n                  'BsmtUnfSF', 'TotalBsmtSF', '1stFlrSF', '2ndFlrSF', 'LowQualFinSF', \\\n                  'GrLivArea', 'GarageArea', 'WoodDeckSF', 'OpenPorchSF', 'EnclosedPorch',\\\n                  '3SsnPorch', 'ScreenPorch', 'PoolArea', 'SalePrice']\n\nfor y in numerical_vari:\n    hist_plot(y)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:32:12.609933Z","iopub.execute_input":"2022-07-19T07:32:12.610284Z","iopub.status.idle":"2022-07-19T07:32:17.057653Z","shell.execute_reply.started":"2022-07-19T07:32:12.610256Z","shell.execute_reply":"2022-07-19T07:32:17.056642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From all the plots shown above, we can find that the distribution of some numerical variables just concentrated on one specific value, like 0. It seems that those variables maybe have less significant on affecting the house price. we list this kind of numerical variables: **BsmtFinSF2, LowQualFinSF, EnclosedPorch, 3SsnPorch, ScreenPorch, PoolArea**.","metadata":{}},{"cell_type":"markdown","source":"<a id = \"5\"></a><br>\n# Missing Value\n\n* Find Missing Value\n* Fill Missing Value","metadata":{}},{"cell_type":"code","source":"# Combine train data with test data, show in dataframe\n\ntrain_test_combi = pd.concat([train_data, test_data], ignore_index=True)\ntrain_test_combi","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:32:28.746783Z","iopub.execute_input":"2022-07-19T07:32:28.747165Z","iopub.status.idle":"2022-07-19T07:32:28.797993Z","shell.execute_reply.started":"2022-07-19T07:32:28.747135Z","shell.execute_reply":"2022-07-19T07:32:28.797131Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Find Missing Value**","metadata":{}},{"cell_type":"code","source":"train_test_combi.columns[train_test_combi.isnull().any()]","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:32:33.028335Z","iopub.execute_input":"2022-07-19T07:32:33.029699Z","iopub.status.idle":"2022-07-19T07:32:33.056600Z","shell.execute_reply.started":"2022-07-19T07:32:33.029667Z","shell.execute_reply":"2022-07-19T07:32:33.055581Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Show the number of \"null\" in each variable.\n\nnull_num = train_test_combi.isnull().sum()\nnull_num[null_num > 0]","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:32:36.330222Z","iopub.execute_input":"2022-07-19T07:32:36.331493Z","iopub.status.idle":"2022-07-19T07:32:36.359267Z","shell.execute_reply.started":"2022-07-19T07:32:36.331439Z","shell.execute_reply":"2022-07-19T07:32:36.358401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Find the missing rate of variable which is larger than 0.1.\n\nnull_rate = train_test_combi.isnull().sum() / train_test_combi.shape[0]\nnull_rate[null_rate > 0.1]","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:32:40.650130Z","iopub.execute_input":"2022-07-19T07:32:40.650526Z","iopub.status.idle":"2022-07-19T07:32:40.677976Z","shell.execute_reply.started":"2022-07-19T07:32:40.650486Z","shell.execute_reply":"2022-07-19T07:32:40.676905Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since these features have high missing rate, which may have less useful information for price predicting, so we'll drop them from the dataset.","metadata":{}},{"cell_type":"code","source":"# Drop features have high missing rate.\n\ntrain_test_combi1 = train_test_combi.drop(['LotFrontage', 'Alley', 'FireplaceQu','PoolQC',\\\n                                          'Fence', 'MiscFeature'], axis = 1)\ntrain_test_combi1.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:32:43.988250Z","iopub.execute_input":"2022-07-19T07:32:43.988638Z","iopub.status.idle":"2022-07-19T07:32:44.013355Z","shell.execute_reply.started":"2022-07-19T07:32:43.988613Z","shell.execute_reply":"2022-07-19T07:32:44.012733Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Fill Missing Value**\n\nNext, we use SimpleImputer to replace missing values with the **most frequent value** of each feature along each column.","metadata":{}},{"cell_type":"code","source":"# Try to fill the missing value of all the features.\nfrom sklearn.impute import SimpleImputer\n\nmy_imputer = SimpleImputer(strategy = 'most_frequent', fill_value = None)\n\ntrain_test_combi1_X = train_test_combi1.drop(['SalePrice'], axis = 1)\nimputed_train_test_combi1_X = pd.DataFrame(my_imputer.fit_transform(train_test_combi1_X))\n\n# Imputation removed column names, put them back.\nimputed_train_test_combi1_X.columns = train_test_combi1_X.columns\n\nimputed_train_test_combi1_X","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:32:48.176425Z","iopub.execute_input":"2022-07-19T07:32:48.176820Z","iopub.status.idle":"2022-07-19T07:32:48.542923Z","shell.execute_reply.started":"2022-07-19T07:32:48.176787Z","shell.execute_reply":"2022-07-19T07:32:48.542209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# no NaN value in dataset\nimputed_train_test_combi1_X.columns[imputed_train_test_combi1_X.isnull().any()]","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:32:52.594408Z","iopub.execute_input":"2022-07-19T07:32:52.594828Z","iopub.status.idle":"2022-07-19T07:32:52.627802Z","shell.execute_reply.started":"2022-07-19T07:32:52.594792Z","shell.execute_reply":"2022-07-19T07:32:52.626997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"imputed_train_X = imputed_train_test_combi1_X[0:1460]\nimputed_test_X = imputed_train_test_combi1_X[1460:2919]\nimputed_train_X","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:32:55.860739Z","iopub.execute_input":"2022-07-19T07:32:55.861024Z","iopub.status.idle":"2022-07-19T07:32:55.888235Z","shell.execute_reply.started":"2022-07-19T07:32:55.861002Z","shell.execute_reply":"2022-07-19T07:32:55.886885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_fill = imputed_train_X\ntrain_data_fill['SalePrice'] = train_data['SalePrice']\ntrain_data_fill","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:33:00.571380Z","iopub.execute_input":"2022-07-19T07:33:00.571762Z","iopub.status.idle":"2022-07-19T07:33:00.609931Z","shell.execute_reply.started":"2022-07-19T07:33:00.571733Z","shell.execute_reply":"2022-07-19T07:33:00.608475Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_data_fill = imputed_test_X\ntest_data_fill","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:33:05.017023Z","iopub.execute_input":"2022-07-19T07:33:05.017302Z","iopub.status.idle":"2022-07-19T07:33:05.042349Z","shell.execute_reply.started":"2022-07-19T07:33:05.017281Z","shell.execute_reply":"2022-07-19T07:33:05.041258Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"6\"></a><br>\n# Data Analysis and Visualization\n\nWe try to find the **relations** between the variables and the house price.","metadata":{}},{"cell_type":"markdown","source":"**1. Correlation: Numerical Variables for House Features -- Sale Price**","metadata":{}},{"cell_type":"code","source":"list1 = ['LotArea', 'MasVnrArea', 'BsmtFinSF1', 'BsmtFinSF2', 'BsmtUnfSF', 'TotalBsmtSF',\\\n         '1stFlrSF', '2ndFlrSF', 'LowQualFinSF', 'GrLivArea', 'GarageArea', 'WoodDeckSF',\\\n          'OpenPorchSF', 'EnclosedPorch', '3SsnPorch', 'ScreenPorch', 'PoolArea',\\\n         'SalePrice']\n\nplt.figure(figsize=(15, 8))\nsns.heatmap(train_data_fill[list1].astype(int).corr(), cmap = \"coolwarm\", annot = True, \\\n            fmt = \".2f\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:33:08.963701Z","iopub.execute_input":"2022-07-19T07:33:08.964032Z","iopub.status.idle":"2022-07-19T07:33:10.573284Z","shell.execute_reply.started":"2022-07-19T07:33:08.964005Z","shell.execute_reply":"2022-07-19T07:33:10.572436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the heat map above, the correlation between each numerical feature and **SalePrice** is shown clearly. It would seem that **BsmtFinSF2, LowQualFinSF, EnclosedPorch, 3SsnPorch, ScreenPorch** and **PoolArea** have less correlation with **SalePrice** (each absolute value less than 0.2), this result is also the same as that we got from the hist plots of each numerical Variable. So we consider to drop these variables from both of the dataset.","metadata":{}},{"cell_type":"code","source":"# Drop variables with low correlation from both datasets.\n\ntrain_data_fill1 = train_data_fill.drop(['BsmtFinSF2', 'LowQualFinSF', 'EnclosedPorch', \\\n                                         '3SsnPorch', 'ScreenPorch', 'PoolArea'], axis = 1)\ntest_data_fill1 = test_data_fill.drop(['BsmtFinSF2', 'LowQualFinSF', 'EnclosedPorch', \\\n                                         '3SsnPorch', 'ScreenPorch', 'PoolArea'], axis = 1)\ntrain_data_fill1\n#test_data_fill1","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:33:15.023253Z","iopub.execute_input":"2022-07-19T07:33:15.024368Z","iopub.status.idle":"2022-07-19T07:33:15.061205Z","shell.execute_reply.started":"2022-07-19T07:33:15.024315Z","shell.execute_reply":"2022-07-19T07:33:15.060339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**2. Relation: Categorical Variables for House Features -- Sale Price**\n\nNext, let's find out the relation between those categorical house features with the sale price of the house.","metadata":{}},{"cell_type":"code","source":"list2 = ['MSSubClass', 'MSZoning', 'Street', 'LotShape', 'LandContour', 'Utilities', \\\n         'LotConfig', 'LandSlope', 'Neighborhood', 'Condition1','Condition2','BldgType', \\\n         'HouseStyle', 'OverallQual', 'OverallCond', 'YearBuilt', 'YearRemodAdd', \\\n         'RoofStyle', 'RoofMatl', 'Exterior1st', 'Exterior2nd', 'MasVnrType', 'ExterQual',\\\n         'ExterCond','Foundation','BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1', \\\n         'BsmtFinType2', 'Heating','HeatingQC', 'CentralAir', 'Electrical','BsmtFullBath',\\\n         'BsmtHalfBath', 'FullBath','HalfBath', 'BedroomAbvGr', 'KitchenAbvGr', \\\n         'KitchenQual', 'TotRmsAbvGrd', 'Functional', 'Fireplaces', 'GarageType', \\\n         'GarageYrBlt', 'GarageFinish', 'GarageCars', 'GarageQual', 'GarageCond', \\\n         'PavedDrive', 'MiscVal', 'MoSold', 'YrSold', 'SaleType', 'SaleCondition', \\\n         'SalePrice']\n\nimport math\n\nn_cols = 4\nn_rows = math.ceil((len(train_data_fill1[list2].columns) - 1) / n_cols)\n\nfig, axes = plt.subplots(n_rows, n_cols, figsize = (30,60), constrained_layout = True)\nfor col, ax in zip(train_data_fill1[list2].columns, axes.ravel()):\n    sns.boxplot(ax = ax, x = col, y = 'SalePrice', data = train_data_fill1[list2])\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:33:19.032929Z","iopub.execute_input":"2022-07-19T07:33:19.033290Z","iopub.status.idle":"2022-07-19T07:33:38.688081Z","shell.execute_reply.started":"2022-07-19T07:33:19.033262Z","shell.execute_reply":"2022-07-19T07:33:38.687055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the 56 box plots shown above, we can find that some of the categorical variables have less or even no relationships with **SalePrice** feature.\n1. **Utilities, LotConfig, LandSlope, BsmtFinType1, BsmtFinType2, HeatingQC, BsmtFullBath, BsmtHalfBath, KitchenAbvGr, Functional, GarageCond, PavedDrive, MoSold** and **YrSold** seem to have no relationship with **SalePrice** feature;\n2. **LotShape, LandContour, BldgType, HouseStyle, RoofStyle, ExterCond, Foundation, BsmtCond, BsmtExposure, Heating, Electrical, HalfBath** and **Fireplaces** seem to have less relationship with **SalePrice** feature.\\\n\\\nSo we consider to drop these categorical variables from both of the dataset.","metadata":{}},{"cell_type":"code","source":"# Drop categorical variables with less and no relationship with SalePrice from datasets.\n\ntrain_data_fill2 = train_data_fill1.drop(['Utilities', 'LotConfig', 'LandSlope', \\\n                                        'BsmtFinType1', 'BsmtFinType2', 'HeatingQC', \\\n                                        'BsmtFullBath', 'BsmtHalfBath', 'KitchenAbvGr', \\\n                                        'Functional', 'GarageCond', 'PavedDrive','MoSold',\\\n                                        'YrSold', 'LotShape', 'LandContour', 'BldgType', \\\n                                        'HouseStyle','RoofStyle','ExterCond','Foundation',\\\n                                        'BsmtCond', 'BsmtExposure','Heating','Electrical',\\\n                                        'HalfBath', 'Fireplaces'], axis = 1)\ntest_data_fill2 = test_data_fill1.drop(['Utilities', 'LotConfig', 'LandSlope', \\\n                                        'BsmtFinType1', 'BsmtFinType2', 'HeatingQC', \\\n                                        'BsmtFullBath', 'BsmtHalfBath', 'KitchenAbvGr', \\\n                                        'Functional', 'GarageCond','PavedDrive','MoSold',\\\n                                        'YrSold', 'LotShape', 'LandContour', 'BldgType', \\\n                                        'HouseStyle','RoofStyle','ExterCond','Foundation',\\\n                                        'BsmtCond', 'BsmtExposure','Heating','Electrical',\\\n                                        'HalfBath', 'Fireplaces'], axis = 1)\ntrain_data_fill2\n#test_data_fill2","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:33:46.580125Z","iopub.execute_input":"2022-07-19T07:33:46.580456Z","iopub.status.idle":"2022-07-19T07:33:46.617836Z","shell.execute_reply.started":"2022-07-19T07:33:46.580431Z","shell.execute_reply":"2022-07-19T07:33:46.616680Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"7\"></a><br>\n# Feature Engineering\n\nThe train and test datasets still contain categorical variables, while modeling needs something numerical, so we need to do some conversion to get numerical values of these data.","metadata":{}},{"cell_type":"code","source":"num_list = ['LotArea', 'MasVnrArea', 'BsmtFinSF1', 'BsmtUnfSF', 'TotalBsmtSF', '1stFlrSF',\\\n            '2ndFlrSF', 'GrLivArea', 'GarageArea', 'WoodDeckSF', 'OpenPorchSF', 'Id']\n\nyear_list = ['YearBuilt', 'YearRemodAdd', 'GarageYrBlt']\n\nCate_list = ['MSSubClass', 'MSZoning', 'Street', 'Neighborhood','Condition1','Condition2',\\\n             'OverallQual', 'OverallCond', 'RoofMatl', 'Exterior1st', 'Exterior2nd', \\\n             'MasVnrType', 'ExterQual', 'BsmtQual','CentralAir','FullBath','BedroomAbvGr',\\\n             'KitchenQual', 'TotRmsAbvGrd', 'GarageType', 'GarageFinish', 'GarageCars', \\\n             'GarageQual', 'MiscVal', 'SaleType', 'SaleCondition']","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:33:58.325661Z","iopub.execute_input":"2022-07-19T07:33:58.326235Z","iopub.status.idle":"2022-07-19T07:33:58.334753Z","shell.execute_reply.started":"2022-07-19T07:33:58.326205Z","shell.execute_reply":"2022-07-19T07:33:58.333657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_fill3 = train_data_fill2\ntrain_data_fill3[num_list] = train_data_fill2[num_list].astype(int)\ntrain_data_fill3[year_list] = train_data_fill2[year_list].astype(int)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:34:02.790793Z","iopub.execute_input":"2022-07-19T07:34:02.791170Z","iopub.status.idle":"2022-07-19T07:34:02.810796Z","shell.execute_reply.started":"2022-07-19T07:34:02.791140Z","shell.execute_reply":"2022-07-19T07:34:02.809864Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_data_fill3 = test_data_fill2\ntest_data_fill3[num_list] = test_data_fill2[num_list].astype(int)\ntest_data_fill3[year_list] = test_data_fill2[year_list].astype(int)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:34:15.168627Z","iopub.execute_input":"2022-07-19T07:34:15.169023Z","iopub.status.idle":"2022-07-19T07:34:15.179244Z","shell.execute_reply.started":"2022-07-19T07:34:15.168993Z","shell.execute_reply":"2022-07-19T07:34:15.178168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# show the cardinality of each categorical variable\nfor x in Cate_list:\n    print(\"{}: \\n\".format(x), \"{} \\n\".format(train_data_fill2[x].value_counts()))","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:34:20.222620Z","iopub.execute_input":"2022-07-19T07:34:20.222979Z","iopub.status.idle":"2022-07-19T07:34:20.258369Z","shell.execute_reply.started":"2022-07-19T07:34:20.222951Z","shell.execute_reply":"2022-07-19T07:34:20.257105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# cardinality in these categorical variables are defined by numerical number.\nCate_list1 = ['MSSubClass', 'OverallQual', 'OverallCond', 'FullBath', 'BedroomAbvGr', \\\n              'TotRmsAbvGrd', 'GarageCars', 'MiscVal']\n\n# cardinality in these categorical variables are defined by letters.\nCate_list2 = ['MSZoning', 'Street', 'Neighborhood', 'Condition1', 'Condition2','RoofMatl',\\\n              'Exterior1st', 'Exterior2nd', 'MasVnrType', 'ExterQual', 'BsmtQual', \\\n              'CentralAir', 'KitchenQual', 'GarageType', 'GarageFinish', 'GarageQual', \\\n              'SaleType', 'SaleCondition']","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:34:38.209465Z","iopub.execute_input":"2022-07-19T07:34:38.210528Z","iopub.status.idle":"2022-07-19T07:34:38.216322Z","shell.execute_reply.started":"2022-07-19T07:34:38.210462Z","shell.execute_reply":"2022-07-19T07:34:38.215592Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_fill3[Cate_list1] = train_data_fill2[Cate_list1].astype(int)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:34:41.666155Z","iopub.execute_input":"2022-07-19T07:34:41.666542Z","iopub.status.idle":"2022-07-19T07:34:41.680330Z","shell.execute_reply.started":"2022-07-19T07:34:41.666512Z","shell.execute_reply":"2022-07-19T07:34:41.678442Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_data_fill3[Cate_list1] = test_data_fill2[Cate_list1].astype(int)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:34:43.953625Z","iopub.execute_input":"2022-07-19T07:34:43.953993Z","iopub.status.idle":"2022-07-19T07:34:43.966993Z","shell.execute_reply.started":"2022-07-19T07:34:43.953963Z","shell.execute_reply":"2022-07-19T07:34:43.966009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The number of groups in each categorical variables\nnum_group = train_data_fill2[Cate_list2].nunique().sort_values(ascending=False)\nprint(num_group)\n\n# which categorical variable has more different groups, more than 10 groups\nhigh_cardinality_cols = list(num_group[num_group >= 10].index)\nprint('high_cardinality_cols =', high_cardinality_cols)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:34:46.778618Z","iopub.execute_input":"2022-07-19T07:34:46.779037Z","iopub.status.idle":"2022-07-19T07:34:46.796086Z","shell.execute_reply.started":"2022-07-19T07:34:46.779007Z","shell.execute_reply":"2022-07-19T07:34:46.795081Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**one-hot encoding**\n\nNext, we'll create a one-hot encoding for categorical variables shown above.","metadata":{}},{"cell_type":"code","source":"train_test_fill_combi = pd.concat([train_data_fill2[Cate_list2], \\\n                                   test_data_fill2[Cate_list2]], ignore_index = True)\ntrain_test_fill_combi","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:34:49.903141Z","iopub.execute_input":"2022-07-19T07:34:49.903540Z","iopub.status.idle":"2022-07-19T07:34:49.934506Z","shell.execute_reply.started":"2022-07-19T07:34:49.903510Z","shell.execute_reply":"2022-07-19T07:34:49.933780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.preprocessing import OneHotEncoder\n\n# Apply one-hot encoder to each column with categorical data\nOH_encoder = OneHotEncoder(handle_unknown = 'ignore', sparse = False)\n\nOH_train_test_fill_combi = pd.DataFrame(OH_encoder.fit_transform(train_test_fill_combi))\nOH_train_test_fill_combi.index = train_test_fill_combi.index\nOH_train_test_fill_combi","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:34:53.347986Z","iopub.execute_input":"2022-07-19T07:34:53.348358Z","iopub.status.idle":"2022-07-19T07:34:53.406329Z","shell.execute_reply.started":"2022-07-19T07:34:53.348304Z","shell.execute_reply":"2022-07-19T07:34:53.405391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"OH_train_data_fill3 = OH_train_test_fill_combi[0:1460]\nOH_test_data_fill3 = OH_train_test_fill_combi[1460:2919]","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:34:57.080890Z","iopub.execute_input":"2022-07-19T07:34:57.081292Z","iopub.status.idle":"2022-07-19T07:34:57.087037Z","shell.execute_reply.started":"2022-07-19T07:34:57.081262Z","shell.execute_reply":"2022-07-19T07:34:57.086142Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Remove categorical columns (will replace with one-hot encoding)\nDrop_train_data_fill3 = train_data_fill3.drop(Cate_list2, axis=1)\n\n# Add one-hot encoded columns to other features\nPre_train_data = pd.concat([Drop_train_data_fill3,OH_train_data_fill3.astype(int)], axis=1)\nPre_train_data","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:34:59.930447Z","iopub.execute_input":"2022-07-19T07:34:59.930765Z","iopub.status.idle":"2022-07-19T07:34:59.954734Z","shell.execute_reply.started":"2022-07-19T07:34:59.930742Z","shell.execute_reply":"2022-07-19T07:34:59.953737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Remove categorical columns (will replace with one-hot encoding)\nDrop_test_data_fill3 = test_data_fill3.drop(Cate_list2, axis=1)\n\n# Add one-hot encoded columns to other features\nPre_test_data = pd.concat([Drop_test_data_fill3, OH_test_data_fill3.astype(int)], axis=1)\nPre_test_data","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:35:23.765174Z","iopub.execute_input":"2022-07-19T07:35:23.765552Z","iopub.status.idle":"2022-07-19T07:35:23.796151Z","shell.execute_reply.started":"2022-07-19T07:35:23.765523Z","shell.execute_reply":"2022-07-19T07:35:23.795257Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Pre_train_data.info()","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:35:28.421841Z","iopub.execute_input":"2022-07-19T07:35:28.422135Z","iopub.status.idle":"2022-07-19T07:35:28.439944Z","shell.execute_reply.started":"2022-07-19T07:35:28.422111Z","shell.execute_reply":"2022-07-19T07:35:28.438870Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"8\"></a><br>\n# Model Selection\n\n**Testing Different Models**\n\nThe **mean absolute error (MAE)** of these two models below will be tested using training data:\n\n* Random Forest\n* XGBoost","metadata":{}},{"cell_type":"markdown","source":"**Splitting the Training Data**\n\nWe will use part of the training data (at about 30%) to test the MAE of the models.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\nX_Variable = Pre_train_data.drop(['Id', 'SalePrice'], axis = 1)\ny_Target = Pre_train_data['SalePrice']\n\nX_train, X_valid, y_train, y_valid = train_test_split(X_Variable, y_Target, \\\n                                                      test_size = 0.3, random_state = 0)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:35:34.878059Z","iopub.execute_input":"2022-07-19T07:35:34.878457Z","iopub.status.idle":"2022-07-19T07:35:34.890380Z","shell.execute_reply.started":"2022-07-19T07:35:34.878422Z","shell.execute_reply":"2022-07-19T07:35:34.889433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Calculate the MAE of each model**\n\nFor each model, we set the model classifier, and then fit it with 70% of our training data, predict for 30% of the validation data, test and check the MAE.","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import mean_absolute_error","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:35:41.848430Z","iopub.execute_input":"2022-07-19T07:35:41.848809Z","iopub.status.idle":"2022-07-19T07:35:41.853406Z","shell.execute_reply.started":"2022-07-19T07:35:41.848782Z","shell.execute_reply":"2022-07-19T07:35:41.852740Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Random Forest Classifier\nfrom sklearn.ensemble import RandomForestClassifier\n\nmodel_RF = RandomForestClassifier()\nmodel_RF.fit(X_train, y_train)\ny_pred_RF = model_RF.predict(X_valid)\nMAE_RF = mean_absolute_error(y_pred_RF, y_valid)\nprint('The MAE of Random Forest model is:', MAE_RF)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:35:47.395634Z","iopub.execute_input":"2022-07-19T07:35:47.395975Z","iopub.status.idle":"2022-07-19T07:35:49.112422Z","shell.execute_reply.started":"2022-07-19T07:35:47.395953Z","shell.execute_reply":"2022-07-19T07:35:49.111118Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# XGBoost\nfrom xgboost import XGBRegressor\n\nmodel_XGB = XGBRegressor(random_state = 0)\nmodel_XGB.fit(X_train, y_train)\ny_pred_XGB = model_XGB.predict(X_valid)\nMAE_XGB = mean_absolute_error(y_pred_XGB, y_valid)\nprint('The MAE of XGBoost model is:', MAE_XGB)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T07:36:05.604671Z","iopub.execute_input":"2022-07-19T07:36:05.605014Z","iopub.status.idle":"2022-07-19T07:36:06.318169Z","shell.execute_reply.started":"2022-07-19T07:36:05.604987Z","shell.execute_reply":"2022-07-19T07:36:06.317330Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The results of two models are shown above, the model using the **XGBRegressor** have lower MAE than the model using Random Forest Classifier, so we select XGBoost to make the prediction.","metadata":{}},{"cell_type":"markdown","source":"**Improve the model -- Parameter Tuning**\n\nNow that we've trained a default model, it's time to tinker with the parameters, to see if we can get better performance.","metadata":{}},{"cell_type":"code","source":"# XGBoost\nfrom xgboost import XGBRegressor\n\nmodel_XGB = XGBRegressor(random_state = 0, n_estimators=1000, learning_rate=0.01)\nmodel_XGB.fit(X_train, y_train)\ny_pred_XGB = model_XGB.predict(X_valid)\nMAE_XGB = mean_absolute_error(y_pred_XGB, y_valid)\nprint('The MAE of XGBoost model is:', MAE_XGB)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T08:12:24.465295Z","iopub.execute_input":"2022-07-19T08:12:24.465662Z","iopub.status.idle":"2022-07-19T08:12:30.291052Z","shell.execute_reply.started":"2022-07-19T08:12:24.465633Z","shell.execute_reply":"2022-07-19T08:12:30.289932Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id = \"9\"></a><br>\n# Generate Prediction Value File","metadata":{}},{"cell_type":"code","source":"# XGBoost\nfrom xgboost import XGBRegressor\n\nmodel_XGB = XGBRegressor(random_state = 0, n_estimators=1000, learning_rate=0.01)\nmodel_XGB.fit(X_Variable, y_Target)\n\ny_predictions_XGB = model_XGB.predict(Pre_test_data.drop(['Id'], axis = 1)).astype(int)\nprint(y_predictions_XGB)","metadata":{"execution":{"iopub.status.busy":"2022-07-19T08:19:48.725732Z","iopub.execute_input":"2022-07-19T08:19:48.726097Z","iopub.status.idle":"2022-07-19T08:19:55.745093Z","shell.execute_reply.started":"2022-07-19T08:19:48.726072Z","shell.execute_reply":"2022-07-19T08:19:55.743073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#set the output as a dataframe and convert to csv file named submission.csv\n\noutput = pd.DataFrame({'Id': test_data['Id'], 'SalePrice': y_predictions_XGB})\n\noutput.to_csv('submission.csv', index = False)\noutput","metadata":{"execution":{"iopub.status.busy":"2022-07-19T08:22:48.634341Z","iopub.execute_input":"2022-07-19T08:22:48.634717Z","iopub.status.idle":"2022-07-19T08:22:48.654149Z","shell.execute_reply.started":"2022-07-19T08:22:48.634689Z","shell.execute_reply":"2022-07-19T08:22:48.653448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Successfully, we predicted one version of the Sale Price value of houses listed in the testing dataset. A part of the result is shown above.","metadata":{}},{"cell_type":"markdown","source":"<font size=\"+2\" color=blue ><b><u> Upvote If you like it !!! 😄🔝</u></b></font>","metadata":{}}]}