{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "30a9b757-a497-5506-bb5f-79d4e7fcc5ea"
      },
      "source": [
        "As the name says, all I am doing here is cleaning the data - more specifically, replacing missing values in the training data. The result should be the starting point for further exploration and/or feature engineering. In order to build the actual model, further steps will have to be taken."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "cd93b979-f815-c010-d618-92562bcb2ade"
      },
      "outputs": [],
      "source": [
        "# This Python 3 environment comes with many helpful analytics libraries installed\n",
        "# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n",
        "# For example, here's several helpful packages to load in \n",
        "\n",
        "import numpy as np # linear algebra\n",
        "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
        "\n",
        "# Input data files are available in the \"../input/\" directory.\n",
        "# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n",
        "\n",
        "from subprocess import check_output\n",
        "print(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n",
        "\n",
        "# Any results you write to the current directory are saved as output."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "6ded24b4-d6ae-9ef2-0fe7-a57bb0971e7a"
      },
      "outputs": [],
      "source": [
        "houseprice=pd.read_csv('../input/train.csv')\n",
        "houseprice.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "3ff5aa7d-297b-44f5-2793-f810dc88052d"
      },
      "source": [
        "You can already see that there are NaNs in some columns. So let's see where exactly and how many"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "28440547-4ae8-9c38-2be0-d29324b0b07f"
      },
      "outputs": [],
      "source": [
        "# To check how many columns have missing values - this can be repeated to see the progress made\n",
        "def show_missing():\n",
        "    missing = houseprice.columns[houseprice.isnull().any()].tolist()\n",
        "    return missing"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "aeb5081d-42d3-02a9-c3ae-e5fc27e92349"
      },
      "outputs": [],
      "source": [
        "houseprice[show_missing()].isnull().sum()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "26e7a3f0-9215-0356-a298-75591541383a"
      },
      "source": [
        "## Data Cleaning Plan\n",
        "Let's look at these variables in the data dictionary:\n",
        "\n",
        "* LotFrontage: Linear feet of street connected to property. I can't imagine that this would be 0 (as this would be a property without access), so either impute mean, or maybe see if there's a correlation with LotArea (like square root?).\n",
        "\n",
        "* Alley: Type of alley access to property -> Many missing values, I would presume that these properties just don't have an alley access.\n",
        "\n",
        "* MasVnrType/MasVnrArea -> both have 8 values missing, I presume they are the same ones. Either set as \"None\"/0 or use most frequent value/median.\n",
        "\n",
        "* Bsmt... Variables: A  number of variables in connection with the basement. About the same number of missing values. However, there are two basement-related variables without missing values \"BsmtFinSF1\" and \"BsmtFinSF2\" - look at those and then decide what to do with the missing values.\n",
        "\n",
        "* Electrical: Just one missing value - here just impute most frequent one.\n",
        "\n",
        "* FireplaceQu: I assume the properties with missing values just don't have a fireplace. There's also the variable Fireplaces (without missing values) - check this and then decide.\n",
        "\n",
        "* Garage ... Variables: 81 missing in these columns. However, there are some Garage-related variables without missing values: GarageCars, GarageArea - check these and then decide.\n",
        "\n",
        "* PoolQC - probably no pool - but check against PoolArea (which has no missing values).\n",
        "\n",
        "* Fence: Many missing values - probably no fence, just impute 'None'\n",
        "\n",
        "* MiscFeature: Assuming none - probably no special features, just impute 'None'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "2e298917-834c-1665-3f88-4001b9d61eb7"
      },
      "outputs": [],
      "source": [
        "# Looking at categorical values\n",
        "def cat_exploration(column):\n",
        "    return houseprice[column].value_counts()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "eec06cc7-7099-6baa-8a32-3d92476b5649"
      },
      "outputs": [],
      "source": [
        "# Imputing the missing values\n",
        "def cat_imputation(column, value):\n",
        "    houseprice.loc[houseprice[column].isnull(),column] = value"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "66738f4e-9f56-03b9-74ec-c14d042411fc"
      },
      "source": [
        "### LotFrontage/LotArea\n",
        "A number of values are missing and one possibility would be to just impute the mean. However, there should actually be a correlation with LotArea, which has no missing values."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1148996c-6020-7e96-5f1e-24fd573a50b4"
      },
      "outputs": [],
      "source": [
        "# check correlation with LotArea\n",
        "houseprice['LotFrontage'].corr(houseprice['LotArea'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "d44a4217-8f3f-c0d8-c019-7481091e0a3d"
      },
      "source": [
        "Ok, that's not great. I we assume that most lots are rectangular, using the square root might be an improvement. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "95161499-7203-1be9-89d7-482933e8236e"
      },
      "outputs": [],
      "source": [
        "# improvement - and good enough for now\n",
        "houseprice['SqrtLotArea']=np.sqrt(houseprice['LotArea'])\n",
        "houseprice['LotFrontage'].corr(houseprice['SqrtLotArea'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "92199733-1584-f189-f81b-f5f6aa9fd145"
      },
      "outputs": [],
      "source": [
        "import seaborn as sns\n",
        "%pylab inline\n",
        "sns.pairplot(houseprice[['LotFrontage','SqrtLotArea']].dropna())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "cdd8ff86-2a33-e6b5-d115-0ba368c5807a"
      },
      "outputs": [],
      "source": [
        "cond = houseprice['LotFrontage'].isnull()\n",
        "houseprice.LotFrontage[cond]=houseprice.SqrtLotArea[cond]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "07412b09-2ae9-d3ab-488f-89144e3c2df2"
      },
      "outputs": [],
      "source": [
        "# This column is not needed anymore\n",
        "del houseprice['SqrtLotArea']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "0b085e40-d883-19dd-52f8-3969c465d978"
      },
      "source": [
        "### Alley"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "90db0f3c-ccd6-82cf-e39f-ba0ee2b8d92c"
      },
      "outputs": [],
      "source": [
        "cat_exploration('Alley')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b7184787-74a3-5408-986a-a0a3758499a1"
      },
      "outputs": [],
      "source": [
        "# I assume empty fields here mean no alley access\n",
        "cat_imputation('Alley','None')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "a8eb4bf4-cf56-a8a2-99f8-8c0d72298104"
      },
      "source": [
        "### MasVnr"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "45161bbf-3043-052c-bfcf-a3ba6d015a9a"
      },
      "outputs": [],
      "source": [
        "houseprice[['MasVnrType','MasVnrArea']][houseprice['MasVnrType'].isnull()==True]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "c877f46d-2fd5-86d3-262c-4ef863f04e19"
      },
      "source": [
        "So the missing values for the \"MasVnr...\" Variables are in the same rows."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5b32b85a-c2cd-089b-7b6d-8e348d295d3b"
      },
      "outputs": [],
      "source": [
        "cat_exploration('MasVnrType')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "a276d727-e75b-c6ea-1580-df41767f12f9"
      },
      "source": [
        "Since \"None\" is the most frequent value, I will impute \"None\" for the Type, and 0.0 for the area."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3d710c38-23de-61a7-b747-34fa0b32237e"
      },
      "outputs": [],
      "source": [
        "cat_imputation('MasVnrType', 'None')\n",
        "cat_imputation('MasVnrArea', 0.0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "3038e37f-6fc7-edf0-2d2b-fb9037de22c7"
      },
      "source": [
        "### Basement"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3ce366bd-4acc-8417-79c2-2c7ece816991"
      },
      "outputs": [],
      "source": [
        "basement_cols=['BsmtQual','BsmtCond','BsmtExposure','BsmtFinType1','BsmtFinType2','BsmtFinSF1','BsmtFinSF2']\n",
        "houseprice[basement_cols][houseprice['BsmtQual'].isnull()==True]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "f8b8241d-dc61-0e22-eac7-8079f5ad6170"
      },
      "source": [
        "So in the cases where the categorical variables are NaN, the numerical ones are 0. Which means there's no basement, so the categorical ones should also be set to \"None\"."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3bc01ccc-9f44-ad42-f477-b94d3e1b8b24"
      },
      "outputs": [],
      "source": [
        "for cols in basement_cols:\n",
        "    if 'FinSF'not in cols:\n",
        "        cat_imputation(cols,'None')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "04c34658-b8e0-f074-8eff-8df9f579c40c"
      },
      "source": [
        "### Electrical"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1653b12f-468e-6a5a-ed81-37e054cf0e67"
      },
      "outputs": [],
      "source": [
        "cat_exploration('Electrical')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "58e57cd2-fc2b-9569-d4ac-1497a12dadaa"
      },
      "outputs": [],
      "source": [
        "# Impute most frequent value\n",
        "cat_imputation('Electrical','SBrkr')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "ec8cbe6f-03b6-bf2d-8dd2-115a880ed3d2"
      },
      "source": [
        "### Fireplace"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3b34fa0f-755a-b4be-6628-f954c2d12d38"
      },
      "outputs": [],
      "source": [
        "cat_exploration('FireplaceQu')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "f82cfb52-6b27-a13d-ffa3-8ad95405bff3"
      },
      "source": [
        "I would assume that the 690 just don't have a fireplace. Let's check:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "fb40fd33-6bea-35d8-9067-efaea61e05d3"
      },
      "outputs": [],
      "source": [
        "houseprice['Fireplaces'][houseprice['FireplaceQu'].isnull()==True].describe()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "01b55cd2-1ea2-3864-1c92-81e52d06a67f"
      },
      "outputs": [],
      "source": [
        "cat_imputation('FireplaceQu','None')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "17b589a0-c4c2-ebfa-dfbc-5cf7cf6f1467"
      },
      "outputs": [],
      "source": [
        "pd.crosstab(houseprice.Fireplaces, houseprice.FireplaceQu)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "928da1fd-66df-8abc-ad1d-b1f657eb470f"
      },
      "source": [
        "### Garages"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "bb6c15a7-ae7f-3b2c-43ef-e716bdc639ce"
      },
      "outputs": [],
      "source": [
        "garage_cols=['GarageType','GarageQual','GarageCond','GarageYrBlt','GarageFinish','GarageCars','GarageArea']\n",
        "houseprice[garage_cols][houseprice['GarageType'].isnull()==True]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "8b62b636-ec5a-3486-6d98-065e242449a5"
      },
      "outputs": [],
      "source": [
        "#Garage Imputation\n",
        "for cols in garage_cols:\n",
        "    if houseprice[cols].dtype==np.object:\n",
        "        cat_imputation(cols,'None')\n",
        "    else:\n",
        "        cat_imputation(cols, 0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "2be17b6b-446b-4238-2615-e5c3c6a4f71c"
      },
      "source": [
        "### Pool"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "0ccd42be-c564-5d82-2a09-a973b699eaf8"
      },
      "outputs": [],
      "source": [
        "cat_exploration('PoolQC')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "c6686eb7-b630-4543-e828-a485cee5efae"
      },
      "source": [
        "Many missing values - are they all without a pool?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "92dd4099-172a-5426-a603-28fd69816397"
      },
      "outputs": [],
      "source": [
        "houseprice['PoolArea'][houseprice['PoolQC'].isnull()==True].describe()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "b8884e33-c4ee-2d3d-4623-ac9cf2772649"
      },
      "source": [
        "Yes, seems like it - if PoolQC is empty, PoolArea is 0"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "21cbf4e8-cabe-1aa6-f4a2-b3cfd70a7f88"
      },
      "outputs": [],
      "source": [
        "cat_imputation('PoolQC', 'None')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "e8f295b1-a535-7b28-e997-2dca1bf77d73"
      },
      "source": [
        "### Fence"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b2da9717-0c8e-f336-4f83-47729284b9f9"
      },
      "outputs": [],
      "source": [
        "cat_imputation('Fence', 'None')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "3e124599-cc4a-a14d-7321-e42042be06f2"
      },
      "source": [
        "### MiscFeature"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "2b4e17d0-23ee-4550-8e72-a1470d5b2656"
      },
      "outputs": [],
      "source": [
        "cat_imputation('MiscFeature', 'None')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "3a79cc2b-4999-4daf-5a84-2b5e4f232a6b"
      },
      "source": [
        "### Are we done?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "58682e3a-1282-d633-4ae3-9f36d6905ac2"
      },
      "outputs": [],
      "source": [
        "houseprice[show_missing()].isnull().sum()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "20416e0d-6261-2681-4052-323466555b84"
      },
      "source": [
        "Yes, all missing values are gone!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a87a680a-ccb8-4f41-163a-ada39cb70636"
      },
      "outputs": [],
      "source": [
        "houseprice"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "65e69793-ee4f-b193-c11d-a52b6773afdc"
      },
      "outputs": [],
      "source": [
        "from xgboost.sklearn import XGBRegressor"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "4acc694d-3bf6-c8fb-a041-d3d1fa0b2d72"
      },
      "outputs": [],
      "source": [
        "clf = XGBRegressor()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "77984bc8-04cd-b82d-07ae-182d8cd496af"
      },
      "outputs": [],
      "source": [
        "from sklearn.model_selection import cross_val_score"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "6610bac7-571f-166e-adfd-e03838457097"
      },
      "outputs": [],
      "source": [
        "x_train = pd.get_dummies(houseprice.drop(['Id', 'SalePrice'], axis=1))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "34b93a11-2199-7b5a-862c-a35f90887d01"
      },
      "outputs": [],
      "source": [
        "y_train = houseprice['SalePrice']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "2ebf58ae-ff8c-1de1-4b5a-647c14f49ab4"
      },
      "outputs": [],
      "source": [
        "cross_val_score(clf, x_train, y_train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1539d10b-5e8b-fc76-532f-1252a0b3143b"
      },
      "outputs": [],
      "source": [
        "clf.fit(x_train, y_train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "16a63abd-5593-29ab-929b-6a789fbc85f9"
      },
      "outputs": [],
      "source": [
        "houseprice=pd.read_csv('../input/test.csv')"
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}