Housing Prices Prediction with Random Forest
\n","metadata":{}},{"cell_type":"markdown","source":"Hi guys 😀\n\nIn this notebook, I'm going to show you how to perform random forest using housing prices dataset.\n\nTable of contents:\nLoading the Dataset
","metadata":{}},{"cell_type":"markdown","source":"The dataset I'm going to load is the housing prices dataset. The dataset includes the train and test set. Let's read these datasets with the `read_csv` method and then look at the first five rows with the `head` method.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\ndf_train = pd.read_csv(\"../input/home-data-for-ml-course/train.csv\")\ndf_test = pd.read_csv(\"../input/home-data-for-ml-course/test.csv\")\ndf_train.head()","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:47.093319Z","iopub.status.busy":"2022-07-22T12:22:47.092911Z","iopub.status.idle":"2022-07-22T12:22:47.153393Z","shell.execute_reply":"2022-07-22T12:22:47.152487Z","shell.execute_reply.started":"2022-07-22T12:22:47.093271Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#Understanding the Dataset
","metadata":{}},{"cell_type":"markdown","source":"Let's take a look at the shape of train and test set with the `shape` attribute.","metadata":{}},{"cell_type":"code","source":"print(\"The shape of train set: \", df_train.shape)\nprint(\"The shape of test set: \", df_test.shape)","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:47.155353Z","iopub.status.busy":"2022-07-22T12:22:47.154526Z","iopub.status.idle":"2022-07-22T12:22:47.159693Z","shell.execute_reply":"2022-07-22T12:22:47.158713Z","shell.execute_reply.started":"2022-07-22T12:22:47.155314Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's have a look at the column types with the `dtypes` attribute.","metadata":{}},{"cell_type":"code","source":"df_train.dtypes","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:47.213090Z","iopub.status.busy":"2022-07-22T12:22:47.212316Z","iopub.status.idle":"2022-07-22T12:22:47.220136Z","shell.execute_reply":"2022-07-22T12:22:47.219392Z","shell.execute_reply.started":"2022-07-22T12:22:47.213055Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can also use the `info` method to see information such as the index dtype and columns, non-null values and memory usage.","metadata":{}},{"cell_type":"code","source":"df_train.info()","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:47.271520Z","iopub.status.busy":"2022-07-22T12:22:47.270630Z","iopub.status.idle":"2022-07-22T12:22:47.300224Z","shell.execute_reply":"2022-07-22T12:22:47.299241Z","shell.execute_reply.started":"2022-07-22T12:22:47.271474Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's see the summary statistics of numerical columns with the `describe` method.","metadata":{}},{"cell_type":"code","source":"df_train.describe().T","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:47.325783Z","iopub.status.busy":"2022-07-22T12:22:47.325487Z","iopub.status.idle":"2022-07-22T12:22:47.432039Z","shell.execute_reply":"2022-07-22T12:22:47.431054Z","shell.execute_reply.started":"2022-07-22T12:22:47.325757Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#Data Preprocessing
","metadata":{}},{"cell_type":"markdown","source":"The first column is the `Id`. Let's convert this column into the index.","metadata":{}},{"cell_type":"code","source":"df_train.set_index(\"Id\", inplace=True)\ndf_test.set_index(\"Id\", inplace=True)\ndf_train.head()","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:47.434151Z","iopub.status.busy":"2022-07-22T12:22:47.433773Z","iopub.status.idle":"2022-07-22T12:22:47.456575Z","shell.execute_reply":"2022-07-22T12:22:47.455775Z","shell.execute_reply.started":"2022-07-22T12:22:47.434111Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Handling Missing Data\n","metadata":{}},{"cell_type":"markdown","source":"Let's take a look at missing data in each column with the `isnull` method.","metadata":{}},{"cell_type":"code","source":"df_train.isnull().sum()","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:47.457924Z","iopub.status.busy":"2022-07-22T12:22:47.457452Z","iopub.status.idle":"2022-07-22T12:22:47.472540Z","shell.execute_reply":"2022-07-22T12:22:47.471536Z","shell.execute_reply.started":"2022-07-22T12:22:47.457897Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since there are many columns in the dataset, we can't see the number of missing data in all columns. Let's sort the columns with the most missing data using the `sort_values` method and look at the first twenty rows.","metadata":{}},{"cell_type":"code","source":"cols_with_null = df_train.isnull().sum().sort_values(ascending=False)\ncols_with_null.head(20)","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:47.503621Z","iopub.status.busy":"2022-07-22T12:22:47.503089Z","iopub.status.idle":"2022-07-22T12:22:47.518362Z","shell.execute_reply":"2022-07-22T12:22:47.517457Z","shell.execute_reply.started":"2022-07-22T12:22:47.503588Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To see the count of all missing data in the dataset, let me use the `sum` method one more time.","metadata":{}},{"cell_type":"code","source":"print(\"Total number of missing data in the dataset: \", df_train.isnull().sum().sum())","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:47.534003Z","iopub.status.busy":"2022-07-22T12:22:47.533713Z","iopub.status.idle":"2022-07-22T12:22:47.547568Z","shell.execute_reply":"2022-07-22T12:22:47.546592Z","shell.execute_reply.started":"2022-07-22T12:22:47.533977Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's look at the number of missing data in the Sales Price target variable.","metadata":{}},{"cell_type":"code","source":"df_train[\"SalePrice\"].isnull().sum()","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:47.582078Z","iopub.status.busy":"2022-07-22T12:22:47.581544Z","iopub.status.idle":"2022-07-22T12:22:47.588605Z","shell.execute_reply":"2022-07-22T12:22:47.587570Z","shell.execute_reply.started":"2022-07-22T12:22:47.582045Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's remove the first six columns with the most missing data with the `drop` method.","metadata":{}},{"cell_type":"code","source":"cols_to_drop = (cols_with_null.head(6).index).tolist()\ndf_train.drop(cols_to_drop, axis=1, inplace=True)\ndf_test.drop(cols_to_drop, axis=1, inplace=True)","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:47.630458Z","iopub.status.busy":"2022-07-22T12:22:47.629736Z","iopub.status.idle":"2022-07-22T12:22:47.641718Z","shell.execute_reply":"2022-07-22T12:22:47.640985Z","shell.execute_reply.started":"2022-07-22T12:22:47.630423Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Creating the target and feature variables","metadata":{}},{"cell_type":"markdown","source":"The SalePrice column is the target variable and the other columns is features. Let's assign y and X variables to these columns, respectively.","metadata":{}},{"cell_type":"code","source":"y = df_train.SalePrice\nX = df_train.drop([\"SalePrice\"], axis=1)","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:47.669586Z","iopub.status.busy":"2022-07-22T12:22:47.668737Z","iopub.status.idle":"2022-07-22T12:22:47.675197Z","shell.execute_reply":"2022-07-22T12:22:47.674374Z","shell.execute_reply.started":"2022-07-22T12:22:47.669550Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Splitting the dataset","metadata":{}},{"cell_type":"markdown","source":"Let's split the dataset into the train and test set with the `train_test_split` method.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nX_train, X_val, y_train, y_val = train_test_split(X,y,train_size=0.8, random_state=0)","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:47.728479Z","iopub.status.busy":"2022-07-22T12:22:47.727929Z","iopub.status.idle":"2022-07-22T12:22:47.734615Z","shell.execute_reply":"2022-07-22T12:22:47.733812Z","shell.execute_reply.started":"2022-07-22T12:22:47.728449Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Handling the categorical and numerical columns","metadata":{}},{"cell_type":"markdown","source":"Data preprocessing is different for categorical and numeric columns. Let's select categorical and numeric columns. I'm going to remove columns with more than ten subcategories.","metadata":{}},{"cell_type":"code","source":"categorical_cols=[cname for cname in X_train.columns \n if X_train[cname].nunique()<10 and X_train[cname].dtype == \"object\"]","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:47.759405Z","iopub.status.busy":"2022-07-22T12:22:47.758922Z","iopub.status.idle":"2022-07-22T12:22:47.777718Z","shell.execute_reply":"2022-07-22T12:22:47.777030Z","shell.execute_reply.started":"2022-07-22T12:22:47.759373Z"}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"numerical_cols=[cname for cname in X_train.columns \n if X_train[cname].dtype in [\"int64\", \"float64\"]]","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:47.819461Z","iopub.status.busy":"2022-07-22T12:22:47.818936Z","iopub.status.idle":"2022-07-22T12:22:47.823951Z","shell.execute_reply":"2022-07-22T12:22:47.822894Z","shell.execute_reply.started":"2022-07-22T12:22:47.819423Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's have a look at the number of categorical and numerical columns.","metadata":{}},{"cell_type":"code","source":"print(\"The number of categorical columns: \", len(categorical_cols))\nprint(\"The number of numerical columns: \", len(numerical_cols))","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:47.859511Z","iopub.status.busy":"2022-07-22T12:22:47.858963Z","iopub.status.idle":"2022-07-22T12:22:47.864390Z","shell.execute_reply":"2022-07-22T12:22:47.863339Z","shell.execute_reply.started":"2022-07-22T12:22:47.859480Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We've selected `70(=35+35)` columns. Let's remove any other columns we didn't select from the datasets.","metadata":{}},{"cell_type":"code","source":"my_cols=categorical_cols+numerical_cols\nX_train = X_train[my_cols]\nX_val = X_val[my_cols]\nX_test = df_test[my_cols]","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:47.899907Z","iopub.status.busy":"2022-07-22T12:22:47.899156Z","iopub.status.idle":"2022-07-22T12:22:47.909164Z","shell.execute_reply":"2022-07-22T12:22:47.908381Z","shell.execute_reply.started":"2022-07-22T12:22:47.899872Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Pipelines for data preprocessing","metadata":{}},{"cell_type":"markdown","source":"A machine learning pipeline allows us to combine a series of steps involved in training a model. Let's import the necessary libraries to build the pipelines.","metadata":{}},{"cell_type":"code","source":"from sklearn.pipeline import Pipeline\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.preprocessing import OneHotEncoder\nfrom sklearn.compose import ColumnTransformer","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:47.920915Z","iopub.status.busy":"2022-07-22T12:22:47.920478Z","iopub.status.idle":"2022-07-22T12:22:47.924526Z","shell.execute_reply":"2022-07-22T12:22:47.923800Z","shell.execute_reply.started":"2022-07-22T12:22:47.920887Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's build a pipeline for numerical columns to handle missing data and scale data.","metadata":{}},{"cell_type":"code","source":"numerical_transformer = Pipeline(steps=[\n (\"imputer_num\", SimpleImputer(strategy=\"median\")),\n (\"scaler\", StandardScaler())\n])","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:47.969383Z","iopub.status.busy":"2022-07-22T12:22:47.968828Z","iopub.status.idle":"2022-07-22T12:22:47.973483Z","shell.execute_reply":"2022-07-22T12:22:47.972635Z","shell.execute_reply.started":"2022-07-22T12:22:47.969347Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's build an other pipeline for categorical columns to handle missing data and perform one-hot encoding.","metadata":{}},{"cell_type":"code","source":"categorical_transformer = Pipeline(steps = [\n (\"imputer_cal\", SimpleImputer(strategy=\"most_frequent\")),\n (\"onehot\", OneHotEncoder(handle_unknown=\"ignore\"))\n])","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:47.975248Z","iopub.status.busy":"2022-07-22T12:22:47.974747Z","iopub.status.idle":"2022-07-22T12:22:47.983325Z","shell.execute_reply":"2022-07-22T12:22:47.982434Z","shell.execute_reply.started":"2022-07-22T12:22:47.975220Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's apply these transformers to categorical and numerical columns.","metadata":{}},{"cell_type":"code","source":"preprocessor = ColumnTransformer(transformers=[\n (\"num\", numerical_transformer, numerical_cols),\n (\"cat\", categorical_transformer, categorical_cols)\n])","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:47.989787Z","iopub.status.busy":"2022-07-22T12:22:47.989236Z","iopub.status.idle":"2022-07-22T12:22:47.994059Z","shell.execute_reply":"2022-07-22T12:22:47.992983Z","shell.execute_reply.started":"2022-07-22T12:22:47.989760Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#Model Building
","metadata":{}},{"cell_type":"markdown","source":"Random forests is an ensemble learning method used for classification and regression. Let's build a simple random forest model.","metadata":{}},{"cell_type":"code","source":"from sklearn.ensemble import RandomForestRegressor\nrf = RandomForestRegressor(random_state=0)","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:48.010732Z","iopub.status.busy":"2022-07-22T12:22:48.010283Z","iopub.status.idle":"2022-07-22T12:22:48.085082Z","shell.execute_reply":"2022-07-22T12:22:48.084245Z","shell.execute_reply.started":"2022-07-22T12:22:48.010705Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's create a pipeline for data preprocessing and model building steps.","metadata":{}},{"cell_type":"code","source":"my_pipeline = Pipeline(steps=[ (\"preprocessor\", preprocessor),(\"model\", rf)])","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:48.090222Z","iopub.status.busy":"2022-07-22T12:22:48.089414Z","iopub.status.idle":"2022-07-22T12:22:48.094349Z","shell.execute_reply":"2022-07-22T12:22:48.093241Z","shell.execute_reply.started":"2022-07-22T12:22:48.090188Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's train the model with the `train` method.","metadata":{}},{"cell_type":"code","source":"my_pipeline.fit(X_train, y_train)","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:48.095970Z","iopub.status.busy":"2022-07-22T12:22:48.095584Z","iopub.status.idle":"2022-07-22T12:22:50.222177Z","shell.execute_reply":"2022-07-22T12:22:50.221048Z","shell.execute_reply.started":"2022-07-22T12:22:48.095852Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model evaluation","metadata":{}},{"cell_type":"markdown","source":"Let's predict the validation data with the `predict` method.","metadata":{}},{"cell_type":"code","source":"val_preds = my_pipeline.predict(X_val)","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:50.225107Z","iopub.status.busy":"2022-07-22T12:22:50.224512Z","iopub.status.idle":"2022-07-22T12:22:50.256924Z","shell.execute_reply":"2022-07-22T12:22:50.256191Z","shell.execute_reply.started":"2022-07-22T12:22:50.225073Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's see the performance of model on validation data with the `mean_absulate_error` function.","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import mean_absolute_error\nprint(\"Validation MAE: \", mean_absolute_error(y_val, val_preds))","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:50.258475Z","iopub.status.busy":"2022-07-22T12:22:50.258173Z","iopub.status.idle":"2022-07-22T12:22:50.264011Z","shell.execute_reply":"2022-07-22T12:22:50.262934Z","shell.execute_reply.started":"2022-07-22T12:22:50.258447Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#Cross-Validation
","metadata":{}},{"cell_type":"markdown","source":"Cross-validation is a resampling method that allows us to use different portions of the data to test and train a model on different iterations. Let's find the cross validation score with the `cross_val_score` function and calculate the mean of cross validation scores with the `mean` method.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import cross_val_score\nscores = -1 * cross_val_score(my_pipeline, X,y, cv = 5, scoring=\"neg_mean_absolute_error\")\nprint(\"Mean Cross Validation Score: \", scores.mean())","metadata":{"execution":{"iopub.execute_input":"2022-07-22T12:22:59.000056Z","iopub.status.busy":"2022-07-22T12:22:58.999636Z","iopub.status.idle":"2022-07-22T12:23:09.669617Z","shell.execute_reply":"2022-07-22T12:23:09.668394Z","shell.execute_reply.started":"2022-07-22T12:22:59.000020Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#Grid Search
","metadata":{}},{"cell_type":"markdown","source":"The grid search allows us to generate candidates from the grid of parameter values specified by the param_grid parameter. Let's find the best hyperparameters of random forest model with the `GridSearchCV` class.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import GridSearchCV\nparam_grid = { \n 'model__n_estimators': [500, 600, 700],\n 'model__max_features': ['auto','sqrt','log2'],\n 'model__max_depth' : [5,6,7],\n 'model__criterion' :['squared_error','absolute_error','poisson']}\nGridCV = GridSearchCV(my_pipeline, param_grid, n_jobs= -1)\nGridCV.fit(X_train,y_train) \nprint(GridCV.best_params_) \nprint(GridCV.best_score_)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's predict the test data with the `predict` method.","metadata":{}},{"cell_type":"code","source":"preds_test = GridCV.predict(X_test)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's convert these predictions into a dataframe.","metadata":{}},{"cell_type":"code","source":"output = pd.DataFrame({'Id': X_test.index, 'SalePrice': preds_test})\noutput.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's write this dataframe to an `csv` file.","metadata":{}},{"cell_type":"code","source":"output.to_csv('submission.csv', index=False)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can now submit this file to the competition!","metadata":{}},{"cell_type":"markdown","source":"\n#Conclusion
","metadata":{}},{"cell_type":"markdown","source":"### That's it. In this notebook, I first performed EDA and then built a random forest model to predict house prices. I also used the grid search technique to find the combination of best hyperparameters.\n\n### Thanks for reading 😀 If you like this notebook, please upvote it 😊\n\n### Don't forget to follow us on [YouTube](http://youtube.com/tirendazacademy) | [Medium](http://tirendazacademy.medium.com) | [Twitter](http://twitter.com/tirendazacademy) | [GitHub](http://github.com/tirendazacademy) | [Linkedin](https://www.linkedin.com/in/tirendaz-academy) | [Kaggle](https://www.kaggle.com/tirendazacademy)","metadata":{}}]}