{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "30f89f90-298a-1ddc-517b-fd58b0ee67ef"
      },
      "source": [
        "Reading CSV of training data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d701aa0d-c00a-891b-6722-bf1d216299ee"
      },
      "outputs": [],
      "source": [
        "house.train = read.csv('../input/train.csv',header=TRUE)\n",
        "head(house.train)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "9db0c66f-149b-a6d5-5039-64619ed91a2e"
      },
      "source": [
        "### Summary of Data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "70cd53c9-3ad3-7d78-9c5a-9991a0047e6a"
      },
      "outputs": [],
      "source": [
        "dim(house.train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "dd7e31a5-4293-e0f7-ddcc-8efa24d69c36"
      },
      "outputs": [],
      "source": [
        "summary(house.train)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "f00d51ff-96e2-f847-e8c0-222bcf68c870"
      },
      "source": [
        "There are some variables which have null values. It is possible that these variables might not be valid for a house.(Like, Basement features are null as the house may not have any basement).\n",
        "\n",
        "**Names of variables which have null values**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "876995dc-00f6-ae57-7456-304a40b7e392"
      },
      "outputs": [],
      "source": [
        "null.cols <- sapply(names(house.train), function(x)any(is.na(house.train[,x])))\n",
        "null.cols <- null.cols[null.cols == TRUE]\n",
        "names(null.cols)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "cb97edb2-bcdf-5348-bdff-9b2244c52526"
      },
      "source": [
        "\n",
        "###Let's check all the variables which have null values one by one\n",
        "\n",
        "####Lot Frontage  : \n",
        "Certain Houses may not have Lot frontage. So, there are NA. We can set them to zero and add it to LotArea and see if there is a relationship between TotalLot and SalePrice"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d6deca2f-c0ab-21b5-904d-4868efc4aae8"
      },
      "outputs": [],
      "source": [
        "house.train[is.na(house.train[,'LotFrontage']),'LotFrontage'] = 0\n",
        "house.train[,'TotalLot'] = house.train[,\"LotFrontage\"] + house.train[,\"LotArea\"]\n",
        "house.train <- house.train[,!(names(house.train) %in% c('LotArea', 'LotFrontage', 'Id'))]\n",
        "with(house.train, plot(TotalLot, SalePrice))\n",
        "model <- lm(SalePrice~TotalLot, data = house.train)\n",
        "abline(model, lwd = 3)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "162ada74-c8b2-f5ac-f195-01a1bf903f65"
      },
      "source": [
        "This is not giving any solid relationship because SalePrice and TotalLot on different scale. Taking log might help."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "8f5d3e6f-35c1-49d2-514f-1cb31ca77cfe"
      },
      "outputs": [],
      "source": [
        "LogSalePrice <- log(house.train[,'SalePrice'])\n",
        "LogTotalLot <- log(house.train[,'TotalLot'])\n",
        "plot(LogTotalLot, LogSalePrice)\n",
        "model <- lm(LogSalePrice~LogTotalLot)\n",
        "abline(model, lwd = 3)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "75718187-a40e-f5d8-3c23-e79f85388ea3"
      },
      "source": [
        "There seems to be a linear relationship between SalePrice and TotalLot\n",
        "\n",
        "####Alley : \n",
        "There are houses which dont have alley access so in that case we can set Alley to 'None'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5a4ac3ba-4528-8e77-9166-350bd9c85b3a"
      },
      "outputs": [],
      "source": [
        "house.train$Alley <- as.character(house.train$Alley)\n",
        "house.train[is.na(house.train[,'Alley']),'Alley'] <- 'None'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "131c90a8-2598-724e-e7dd-2962ee64cdaa"
      },
      "source": [
        "####MasVnrType and MasVnrArea :\n",
        "From the summary data there are equal number of NAs in both so it means Masonry Veneer is not present for these homes. So, we can set MasVnrType to None and MasVnrArea to 0."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "2db1a02b-4e3d-dbec-2f8b-4546bd9075c7"
      },
      "outputs": [],
      "source": [
        "house.train$MasVnrType <- as.character(house.train$MasVnrType)\n",
        "house.train[is.na(house.train$MasVnrType),'MasVnrType'] <- 'None'\n",
        "house.train[is.na(house.train$MasVnrArea),'MasVnrArea'] <- 0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "7f2f948b-19ad-1c1f-0b5f-e6e7c40fa97d"
      },
      "source": [
        "####GarageType, GarageYrBlt, GarageQual, GarageCond, GarageFinish :"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d72cd5d4-b533-13da-b143-bcf7d8b24b40"
      },
      "outputs": [],
      "source": [
        "house.train[is.na(house.train$GarageType),c(\"GarageType\", \"GarageYrBlt\",\"GarageQual\", \"GarageCond\", \"GarageFinish\", \"GarageCars\", \"GarageArea\")]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "caf29f8e-1502-eea0-4680-6c5f608c35ea"
      },
      "source": [
        "For these garage cars and garage area are also 0 so we can set them to None or 0 acccording to their type."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "8f2ec71a-f161-c188-928d-e2754b72b4a6"
      },
      "outputs": [],
      "source": [
        "for (name in c(\"GarageType\", \"GarageQual\", \"GarageCond\", \"GarageFinish\")){\n",
        "  house.train[,name]<- as.character(house.train[,name])\n",
        "  house.train[is.na(house.train[,name]),name] <- 'None'\n",
        "}\n",
        "house.train[is.na(house.train$GarageYrBlt),'GarageYrBlt'] <- 0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "04db5a71-dc16-5127-676d-971f7c472593"
      },
      "source": [
        "####PoolArea and PoolQC :"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b26c0505-c531-0299-a70b-ab73193b50c0"
      },
      "outputs": [],
      "source": [
        "house.train[is.na(house.train$PoolQC),c(\"PoolArea\", \"PoolQC\")] "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ca274bb6-64e7-079e-4492-072c99b08658"
      },
      "outputs": [],
      "source": [
        "house.train$PoolQC <- as.character(house.train$PoolQC)\n",
        "house.train[is.na(house.train$PoolQC),'PoolQC'] <- 'None'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "a3dbb0cd-edfa-5241-8567-88e52d84b502"
      },
      "source": [
        "####Basement Variables :"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "56edb714-2779-8469-35fc-63230a613c76"
      },
      "outputs": [],
      "source": [
        "house.train[is.na(house.train$BsmtQual), c(\"BsmtQual\", \"BsmtCond\", \"BsmtExposure\", \"BsmtFinType1\", \"BsmtFinType2\",\n",
        "                                           \"BsmtFinSF1\", \"BsmtFinSF2\", \"BsmtUnfSF\", \"TotalBsmtSF\")]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "8cafec21-a337-2a8a-6e8a-420b8e97b5dc"
      },
      "outputs": [],
      "source": [
        "for (name in c(\"BsmtQual\", \"BsmtCond\", \"BsmtExposure\", \"BsmtFinType1\", \"BsmtFinType2\")){\n",
        "  house.train[,name]<- as.character(house.train[,name])\n",
        "  house.train[is.na(house.train[,name]),name] <- 'None'\n",
        "}\n",
        "for (name in c(\"BsmtFinSF1\", \"BsmtFinSF2\", \"BsmtUnfSF\", \"TotalBsmtSF\")){\n",
        "  house.train[is.na(house.train[,name]),name] <- 0\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "4110467e-0eb0-8c97-161b-728d112b7429"
      },
      "source": [
        "####FireplaceQu and Fireplaces :"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "4b0ac186-5ca2-61cb-9269-b94a47b19ce4"
      },
      "outputs": [],
      "source": [
        "house.train[is.na(house.train$FireplaceQu),c(\"FireplaceQu\",\"Fireplaces\")]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "08534a1b-1868-9ed3-cfd7-e83c35a04d93"
      },
      "outputs": [],
      "source": [
        "house.train$FireplaceQu <- as.character(house.train$FireplaceQu)\n",
        "house.train[is.na(house.train$FireplaceQu),'FireplaceQu'] <- 'None'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "46fac2dc-ec0b-b5ab-4693-485e9486d20c"
      },
      "source": [
        "####Electrical, Fence and MiscFeature fill all with None :"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d974f5c7-8b85-4ea5-7270-4adbf5f79ad9"
      },
      "outputs": [],
      "source": [
        "for (name in c(\"Electrical\", \"Fence\", \"MiscFeature\")){\n",
        "  house.train[,name]<- as.character(house.train[,name])\n",
        "  house.train[is.na(house.train[,name]),name] <- 'None'\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "fb52f6fa-4f86-60dd-6313-f317ead066af"
      },
      "source": [
        "Checking if there are still NA present"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "2696109a-2fdc-d297-d5fe-029aab18bfe1"
      },
      "outputs": [],
      "source": [
        "sapply(null.cols,function(name)any(is.na(house.train[,name])))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "88aa5f54-e307-5ae9-7007-5369377c6ed6"
      },
      "source": [
        "-------------------------------------------------------------------------------------"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "3d3ccad9-d3de-bf00-c6ec-49b6bbcc08a3"
      },
      "source": [
        "###Now dealing with test data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "2dd690b1-9ed4-476b-2dc0-44039bb7f58a"
      },
      "outputs": [],
      "source": [
        "house.test <- read.csv('../input/test.csv',header=TRUE)\n",
        "dim(house.test)\n",
        "summary(house.test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "199465a0-3096-2081-1573-2b87624745fc"
      },
      "source": [
        "### Dealing with missing values in test data\n",
        "\n",
        "Names of columns which have null values."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "9c266d5d-75da-6d5a-83a0-a7152ba9455a"
      },
      "outputs": [],
      "source": [
        "null.cols.test <- sapply(names(house.test), function(col.name)any(is.na(house.test[, col.name])))\n",
        "null.cols.test <- null.cols.test[null.cols.test == TRUE]\n",
        "null.cols.test <- names(null.cols.test)\n",
        "null.cols.test"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "0d55a7ce-d58c-04b3-a6e1-772c939906bf"
      },
      "source": [
        "#### MSZoning :\n",
        "null values of MSZoning can be predicted by neighbourhood."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f0d89db8-005c-07d2-4aef-8dd90c043d33"
      },
      "outputs": [],
      "source": [
        "house.test[is.na(house.test$MSZoning),c(\"MSZoning\", \"Neighborhood\")]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a202a010-98b3-2c4e-ef10-1518e268f383"
      },
      "outputs": [],
      "source": [
        "temp.data <- house.test[house.test$Neighborhood =='IDOTRR'|house.test$Neighborhood =='Mitchel', c(\"MSZoning\", \"Neighborhood\")]\n",
        "elements <- c('IDOTRR', 'Mitchel')\n",
        "for (name in elements){\n",
        "  print(name)\n",
        "  print(summary(temp.data[temp.data$Neighborhood==name,'MSZoning']))\n",
        "}\n",
        "library(ggplot2)\n",
        "qplot(MSZoning, data = temp.data, facets = .~Neighborhood)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "62e61fb4-b3a5-b010-7e49-bbdd753acdff"
      },
      "source": [
        "So, for IDOTRR it is most likely to be RM, for Mitchell it is most likely to be RL."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d184e999-fe2e-6972-7142-08fdfdfd459e"
      },
      "outputs": [],
      "source": [
        "house.test[,'MSZoning'] <- as.character(house.test$MSZoning)\n",
        "house.test[is.na(house.test$MSZoning) & house.test$Neighborhood == 'IDOTRR', 'MSZoning'] <- 'RM'\n",
        "house.test[is.na(house.test$MSZoning) & house.test$Neighborhood == 'Mitchel', 'MSZoning'] <- 'RL'\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "d1ebe2d2-0feb-e12d-b4a8-4f47066beea2"
      },
      "source": [
        "#### LotFrontage :"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f17c8e55-8f4c-0e4e-8863-debba17e3374"
      },
      "outputs": [],
      "source": [
        "house.test[is.na(house.test$LotFrontage), c(\"LotFrontage\", \"LotArea\")]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "bd4eee83-3c9a-16c8-86cb-d899cce1bc36"
      },
      "source": [
        "We can add LotFrontage and LotArea together"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ccff8321-ab33-04ea-381b-31b81b7b707d"
      },
      "outputs": [],
      "source": [
        "house.test[is.na(house.test$LotFrontage), 'LotFrontage'] <- 0\n",
        "house.test[,'TotalLot'] <- house.test$LotFrontage + house.test$LotArea\n",
        "house.test <- house.test[,!names(house.test) %in% c(\"Id\", \"LotArea\", \"LotFrontage\")]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "300648b7-cdcb-ef72-e232-088fc0c51b84"
      },
      "source": [
        "#### Alley : "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "29784ccf-cc1b-5243-3698-b31e94d0d612"
      },
      "outputs": [],
      "source": [
        "house.test[,'Alley'] <- as.character(house.test$Alley)\n",
        "house.test[is.na(house.test$Alley), 'Alley'] <- 'None'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "ab116993-9236-ec96-21f5-a1d0558881f7"
      },
      "source": [
        "#### Utilities : \n",
        "We can assume where utilities is NA, there are no utilities available."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "bbc9086b-67ec-91ac-4988-e80e698344b2"
      },
      "outputs": [],
      "source": [
        "house.test[,'Utilities'] <- as.character(house.test$Utilities)\n",
        "house.test[is.na(house.test$Utilities), 'Utilities'] <- 'None'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "291a749f-b59b-edf0-43e6-ff761c0333bd"
      },
      "source": [
        "#### Exterior1st & Exterior2nd :\n",
        "Exterior 1 (Nominal): Exterior covering on house\n",
        "\n",
        "       AsbShng\tAsbestos Shingles\n",
        "       AsphShn\tAsphalt Shingles\n",
        "       BrkComm\tBrick Common\n",
        "       BrkFace\tBrick Face\n",
        "       CBlock\tCinder Block\n",
        "       CemntBd\tCement Board\n",
        "       HdBoard\tHard Board\n",
        "       ImStucc\tImitation Stucco\n",
        "       MetalSd\tMetal Siding\n",
        "       Other\tOther\n",
        "       Plywood\tPlywood\n",
        "       PreCast\tPreCast\t\n",
        "       Stone\tStone\n",
        "       Stucco\tStucco\n",
        "       VinylSd\tVinyl Siding\n",
        "       Wd Sdng\tWood Siding\n",
        "       WdShing\tWood Shingles\n",
        "Exterior 2 (Nominal): Exterior covering on house (if more than one material)\n",
        "\n",
        "       AsbShng\tAsbestos Shingles\n",
        "       AsphShn\tAsphalt Shingles\n",
        "       BrkComm\tBrick Common\n",
        "       BrkFace\tBrick Face\n",
        "       CBlock\tCinder Block\n",
        "       CemntBd\tCement Board\n",
        "       HdBoard\tHard Board\n",
        "       ImStucc\tImitation Stucco\n",
        "       MetalSd\tMetal Siding\n",
        "       Other\tOther\n",
        "       Plywood\tPlywood\n",
        "       PreCast\tPreCast\n",
        "       Stone\tStone\n",
        "       Stucco\tStucco\n",
        "       VinylSd\tVinyl Siding\n",
        "       Wd Sdng\tWood Siding\n",
        "       WdShing\tWood Shingles\n",
        "If Exterior1st and Exterior2nd are NA for a given house, it is safe to assume that they are not present for the house "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "bcd452f2-c061-36a1-5feb-b6a8342b76bc"
      },
      "outputs": [],
      "source": [
        "for (name in c(\"Exterior1st\", \"Exterior2nd\")){\n",
        "  house.test[,name] <- as.character(house.test[,name])\n",
        "  house.test[is.na(house.test[,name]), name] <- 'None'\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "d7435036-89f9-6529-1a49-209d4129aad8"
      },
      "source": [
        "#### MasVnrType and MasVnrArea :\n",
        "Check if what is MasVnrArea for null MasVnrType, so that we can make decision whether we can assume MasVnrType None or not."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ee864aa3-92ec-6e4e-0910-9e02c288cf03"
      },
      "outputs": [],
      "source": [
        "house.test[is.na(house.test$MasVnrType), c(\"MasVnrType\", \"MasVnrArea\")]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "8dfa2713-099f-01a5-14bb-17fac99973fd"
      },
      "source": [
        "From the summary above None is already a category for which area is :"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b07e9a0a-64b1-28b6-8e2b-069af4dbd301"
      },
      "outputs": [],
      "source": [
        "house.test[house.test$MasVnrType=='None', c(\"MasVnrType\", \"MasVnrArea\")]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "7abc90ea-5b9c-eab8-00da-8f7b7bcd5c39"
      },
      "source": [
        "We can predict MasVnrType from MasVnrArea "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e9995480-ae39-45ca-c7b4-04ba9fdb4f77"
      },
      "outputs": [],
      "source": [
        "boxplot(MasVnrArea ~ MasVnrType, data = house.test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "51a32fc0-ad77-ae1c-f04c-530e3019cefb"
      },
      "source": [
        "So, for one NA MasVnrType which has MasVnrArea as 198, it is highly probable that it belongs to BrkCmn."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "cc67efd0-55c0-d972-9b14-cf3b271b1b05"
      },
      "outputs": [],
      "source": [
        "house.test[,'MasVnrType'] <- as.character(house.test$MasVnrType)\n",
        "house.test[is.na(house.test$MasVnrType) & !is.na(house.test$MasVnrArea), 'MasVnrType'] <- 'BrkCmn'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "ee076788-925c-04b9-3c34-b3fcd6856b30"
      },
      "source": [
        "for others NA MasVnrType and MasVnrArea, we can set them to None & 0, respectively. \n",
        "Right now, we dont have any way to predict their values from other variables."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "7031493f-11c9-e75a-d0f4-9139d6cafcc3"
      },
      "outputs": [],
      "source": [
        "house.test[,'MasVnrType'] <- as.character(house.test$MasVnrType)\n",
        "house.test[is.na(house.test$MasVnrType), 'MasVnrType'] <- 'None'\n",
        "house.test[is.na(house.test$MasVnrArea), 'MasVnrArea'] <- 0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "df8f8a57-006a-399d-fa42-04c01472cc12"
      },
      "source": [
        "#### Basement Variables :"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ecdcdbd9-f045-91a8-9eec-d35cca27ce4d"
      },
      "outputs": [],
      "source": [
        "house.test[is.na(house.test$BsmtCond), c(\"BsmtQual\", \"BsmtCond\", \"BsmtExposure\", \"BsmtFinType1\", \"BsmtFinType2\",\n",
        "                                           \"BsmtFinSF1\", \"BsmtFinSF2\", \"BsmtUnfSF\", \"TotalBsmtSF\")]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "bbab3a70-aeab-1115-385a-a999a086c9d5"
      },
      "source": [
        "#####BsmtFin Type 1\t: Rating of basement finished area"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f53c6812-2fb3-0b6b-5912-5b3895ae519d"
      },
      "outputs": [],
      "source": [
        "house.test[is.na(house.test$BsmtFinType2), c(\"BsmtQual\", \"BsmtCond\", \"BsmtExposure\", \"BsmtFinType1\", \"BsmtFinType2\",\n",
        "                                           \"BsmtFinSF1\", \"BsmtFinSF2\", \"BsmtUnfSF\", \"TotalBsmtSF\")]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "a3183ca9-a886-ea0e-1f38-7b5aebb68e8b"
      },
      "source": [
        "As all the Bsmt Variables for null BsmntFinType1 or 2 are nulls or zero, thus we can set them to None or zero by assuming that the basement is not be present for these houses."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "16dc016b-c76a-35a3-3e30-e75f31ebb8a4"
      },
      "outputs": [],
      "source": [
        "for (name in c(\"BsmtQual\", \"BsmtCond\", \"BsmtExposure\", \"BsmtFinType2\")){\n",
        "  house.test[, name]<- as.character(house.test[, name])\n",
        "  house.test[is.na(house.test$BsmtFinType1),name] <- 'None'\n",
        "}\n",
        "for (name in c(\"BsmtFinSF1\", \"BsmtFinSF2\", \"BsmtUnfSF\", \"TotalBsmtSF\")){\n",
        "  house.test[is.na(house.test$BsmtFinType1),name] <- 0\n",
        "}\n",
        "house.test[, \"BsmtFinType1\"]<- as.character(house.test[, \"BsmtFinType1\"])\n",
        "house.test[is.na(house.test$BsmtFinType1),\"BsmtFinType1\"] <- 'None'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "0cd0e193-143d-59e3-a664-699dd455fa86"
      },
      "source": [
        "\n",
        "#####Bsmt Exposure : Refers to walkout or garden level walls\n",
        "It is independent variable thus cannot be predicted from other Bsmt values. Thus, we can safely assume that either exposure is not available or Bsmt itself is not available."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "024c03f6-10e8-0d8e-c636-abbe530397eb"
      },
      "outputs": [],
      "source": [
        "house.test[,'BsmtExposure'] <- as.character(house.test$BsmtExposure)\n",
        "house.test[is.na(house.test$BsmtExposure), 'BsmtExposure'] <- 'None'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "1bd61f25-20a0-b33f-6ca3-59ed6ad7a179"
      },
      "source": [
        "After all this assigmnet let's see if we can predict values of Bsmt Qual or Bsmt Cond.\n",
        "\n",
        "#####Bsmt Qual : Evaluates the height of the basement"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "0d56d643-1613-bcc3-8b70-1aac91c9603b"
      },
      "outputs": [],
      "source": [
        "house.test[is.na(house.test$BsmtQual), c(\"BsmtQual\", \"BsmtCond\", \"BsmtExposure\", \"BsmtFinType1\", \"BsmtFinType2\",\n",
        "                                           \"BsmtFinSF1\", \"BsmtFinSF2\", \"BsmtUnfSF\", \"TotalBsmtSF\")]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "fb52a551-db2b-afae-4d78-62e66df3a5ee"
      },
      "source": [
        "What is the general Bsmt Qual of unfinished basement with no exposure and Bsmt Condition is Fa(Fair - dampness or some cracking or settling)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "228b0182-3004-287b-4747-70f4d4673ee3"
      },
      "outputs": [],
      "source": [
        "summary(as.factor(house.test[house.test$BsmtFinType1 == 'Unf' & house.test$BsmtExposure == 'No' & house.test$BsmtCond == 'Fa', c('BsmtQual')]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "f94d3e88-2c62-b83e-e4ab-3662d5f1cc40"
      },
      "source": [
        "So it is TA(typical)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "0b8065bc-bcfe-c63a-594b-e36286ef5bfd"
      },
      "outputs": [],
      "source": [
        "house.test[,'BsmtQual'] <- as.character(house.test$BsmtQual)\n",
        "house.test[is.na(house.test$BsmtQual) & house.test$BsmtFinType1 == 'Unf' & house.test$BsmtExposure == 'No' & house.test$BsmtCond == 'Fa', 'BsmtQual'] <- 'TA'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "fb5bb33b-b05b-3b98-f78b-350112979c4c"
      },
      "source": [
        "What is the general Bsmt Qual of unfinished basement with no exposure and Bsmt Condition is TA(Typical - slight dampness allowed)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b6132fe3-37f9-0f74-332c-160b72a82680"
      },
      "outputs": [],
      "source": [
        "summary(as.factor(house.test[house.test$BsmtFinType1 == 'Unf' & house.test$BsmtExposure == 'No' & house.test$BsmtCond == 'TA', c('BsmtQual')]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "a38d830a-3aa1-4bd4-b69f-2235344fd5c9"
      },
      "source": [
        "It also seems to be typical."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3dfb6766-9efb-2606-e3c0-31ba09e6e674"
      },
      "outputs": [],
      "source": [
        "house.test[,'BsmtQual'] <- as.character(house.test$BsmtQual)\n",
        "house.test[is.na(house.test$BsmtQual) & house.test$BsmtFinType1 == 'Unf' & house.test$BsmtExposure == 'No' & house.test$BsmtCond == 'TA', 'BsmtQual'] <- 'TA'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "890054ac-4a63-315d-64d6-a0605f2bace7"
      },
      "source": [
        "##### BsmtCond : Evaluates the general condition of the basement"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "21359292-30c0-d4a6-736a-041571074c48"
      },
      "outputs": [],
      "source": [
        "house.test[is.na(house.test$BsmtCond), c(\"BsmtQual\", \"BsmtCond\", \"BsmtExposure\", \"BsmtFinType1\", \"BsmtFinType2\",\n",
        "                                           \"BsmtFinSF1\", \"BsmtFinSF2\", \"BsmtUnfSF\", \"TotalBsmtSF\")]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5085fd60-8580-e58e-74be-174462354c33"
      },
      "outputs": [],
      "source": [
        "summary(as.factor(house.test[house.test$BsmtFinType1 == 'BLQ' & house.test$BsmtExposure == 'No' & house.test$BsmtQual == 'TA', c('BsmtCond')]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "b565273a-a7b7-5154-3a9b-cbb2a38a2fba"
      },
      "source": [
        "It is most probably TA"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "590d9019-3b78-99ae-0370-54634999399d"
      },
      "outputs": [],
      "source": [
        "house.test[,'BsmtCond'] <- as.character(house.test$BsmtCond)\n",
        "house.test[is.na(house.test$BsmtCond) & house.test$BsmtFinType1 == 'BLQ' & house.test$BsmtExposure == 'No' & house.test$BsmtQual == 'TA', 'BsmtCond'] <- 'TA'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "56d0b98b-100b-ee05-54ce-494661428f76"
      },
      "outputs": [],
      "source": [
        "summary(as.factor(house.test[house.test$BsmtFinType1 == 'ALQ' & house.test$BsmtExposure == 'Av' & house.test$BsmtQual == 'TA', c('BsmtCond')]))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "4f823239-aa52-4cf9-e93e-f01e531780cc"
      },
      "outputs": [],
      "source": [
        "house.test[,'BsmtCond'] <- as.character(house.test$BsmtCond)\n",
        "house.test[is.na(house.test$BsmtCond) & house.test$BsmtFinType1 == 'ALQ' & house.test$BsmtExposure == 'Av' & house.test$BsmtQual == 'TA', 'BsmtCond'] <- 'TA'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "1490f5e2-c210-f1a1-30eb-ad65c888cbb8"
      },
      "source": [
        "With basement quality good, basement exposure as minimum exposure"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1e1538ab-533b-341e-3b82-d757115303b8"
      },
      "outputs": [],
      "source": [
        "summary(as.factor(house.test[house.test$BsmtFinType1 == 'GLQ' & house.test$BsmtExposure == 'Mn' & house.test$BsmtQual == 'Gd', c('BsmtCond')]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "261c4c92-a9ac-a40a-6034-351a4a35426f"
      },
      "source": [
        "Including BsmtFinType2 with Rec(Average Rec Room)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "41c392b4-fe57-4caf-f3dc-69e6345a7f81"
      },
      "outputs": [],
      "source": [
        "summary(as.factor(house.test[house.test$BsmtFinType1 == 'GLQ' & house.test$BsmtFinType2 == 'Rec' & house.test$BsmtExposure == 'Mn' & house.test$BsmtQual == 'Gd', c('BsmtCond')]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "f3407486-0ccd-0930-32ba-4134329bfe72"
      },
      "source": [
        "It is not really helping since there is only one record with these conditions. So, if we ignore finType1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "72ad0d1e-c1d9-977f-31e1-0f3440e6b13d"
      },
      "outputs": [],
      "source": [
        "summary(as.factor(house.test[house.test$BsmtFinType2 == 'Rec' & house.test$BsmtExposure == 'Mn' & house.test$BsmtQual == 'Gd', c('BsmtCond')]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "7b759836-ce2d-0d39-f0f8-ad3ca0885c1a"
      },
      "source": [
        "Basement Fine type 2 is not really helping to make our assumption.\n",
        "We cannot ignore Basement exposure and basement qual as both are factors to determine the condition of basement like dampness etc."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ba07e938-fc9e-dea8-c2b9-5e21bd5e0c71"
      },
      "outputs": [],
      "source": [
        "house.test[,'BsmtCond'] <- as.character(house.test$BsmtCond)\n",
        "house.test[is.na(house.test$BsmtCond) & house.test$BsmtFinType1 == 'GLQ' & house.test$BsmtExposure == 'Mn' & house.test$BsmtQual == 'Gd', 'BsmtCond'] <- 'TA'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "176960cb-5652-8299-fb70-3bcfcbd3c81b"
      },
      "source": [
        "Are we left with any other null Basement variables?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3d7efcd5-2707-4c70-bdbe-653c746da928"
      },
      "outputs": [],
      "source": [
        "sapply(c(\"BsmtQual\", \"BsmtCond\", \"BsmtExposure\", \"BsmtFinType1\", \"BsmtFinType2\",\n",
        "                                           \"BsmtFinSF1\", \"BsmtFinSF2\", \"BsmtUnfSF\", \"TotalBsmtSF\"), function(name)any(is.na(house.test[,name])))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "0b32d8c8-ff99-74d8-730e-788de1371c3c"
      },
      "source": [
        "####Bsmt Full Bath and Bsmt Half Bath :"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5e5bf1f8-ad52-e09c-4584-9e106b26683b"
      },
      "outputs": [],
      "source": [
        "house.test[is.na(house.test$BsmtFullBath), c(\"BsmtFullBath\", \"BsmtHalfBath\",\"BsmtCond\")]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "b0345874-eb4d-8a80-2c95-c77d994dfc66"
      },
      "source": [
        "It is safe to assume theat Basement full and half bathrooms doea not exist as there is no basement."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "2b7d62c7-5270-1cec-5960-09e1c97cb134"
      },
      "outputs": [],
      "source": [
        "for (name in c(\"BsmtHalfBath\", \"BsmtFullBath\")){\n",
        "  house.test[, name]<- as.character(house.test[, name])\n",
        "  house.test[is.na(house.test[,name]),name] <- 'None'\n",
        "  \n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "3233ea8f-2183-dc7f-d2e6-6ab1be77d7cb"
      },
      "source": [
        "####KitchenQual :"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5918e17a-6f82-ad55-7b1d-2d4b70d7e2c4"
      },
      "outputs": [],
      "source": [
        "house.test[is.na(house.test$KitchenQual),]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "130a3685-7be1-f395-9020-3adc17b7219b"
      },
      "source": [
        "Thus, kitchen quality value is missing for this record. How can we guess its value?\n",
        "the quality of house roughly depends on its condition1, condition2, YearBuilt, Building Type and house style."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "9e3cde55-d04a-1e0a-c3d0-5081a3e6ec6c"
      },
      "outputs": [],
      "source": [
        "summary(as.factor(house.test[house.test$Condition1 == 'Norm' & house.test$Condition2 == 'Norm' &  house.test$BldgType == '1Fam' & house.test$HouseStyle == '1.5Fin' & house.test$KitchenAbvGr == 1 & house.test$YearRemodAdd == 1950,'KitchenQual']))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "6f588ed8-67d9-6484-4273-3ec7baee5ae5"
      },
      "source": [
        "Again, quality is coming out to be TA"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ccabdeaa-571c-621d-d75d-a812d5db4bf0"
      },
      "outputs": [],
      "source": [
        "house.test[,'KitchenQual'] <- as.character(house.test$KitchenQual)\n",
        "house.test[is.na(house.test$KitchenQual) & house.test$Condition1 == 'Norm' & house.test$Condition2 == 'Norm' &  house.test$BldgType == '1Fam' & house.test$HouseStyle == '1.5Fin' & house.test$KitchenAbvGr == 1 & house.test$YearRemodAdd == 1950, 'KitchenQual'] <- 'TA'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "799f4885-3d23-e8bf-f776-da76cdb544bb"
      },
      "source": [
        "####Functional : \n",
        "We are assuming it to be typical as there is no way to deduce it on the basis of other variables."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5921822a-dba0-982d-dfc9-393edde28255"
      },
      "outputs": [],
      "source": [
        "house.test[,'Functional'] <- as.character(house.test$Functional)\n",
        "house.test[is.na(house.test$Functional), 'Functional'] <- 'Typ'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "0c3034ae-2332-b4be-ea2d-b6f4717a45ee"
      },
      "source": [
        "####Fireplaces & FireplaceQu :"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d2b293b3-efed-74e7-2b48-f6206e629eef"
      },
      "outputs": [],
      "source": [
        "house.test[is.na(house.test$FireplaceQu), c(\"Fireplaces\", \"FireplaceQu\")]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "ad50701f-15b0-3532-e4e8-0d09533c1d7a"
      },
      "source": [
        "So, it is safe to assume that FireplaceQu is none as there are no fireplaces."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "741b3537-5d30-8856-0abe-99cbf7dd3096"
      },
      "outputs": [],
      "source": [
        "house.test[,'FireplaceQu'] <- as.character(house.test$FireplaceQu)\n",
        "house.test[is.na(house.test$FireplaceQu), 'FireplaceQu'] <- 'None'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "ff0b6529-60ef-26c6-fe4b-249cf4492b5b"
      },
      "source": [
        "####Garage Variables :"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "2dc152d3-2a30-19c1-6fe7-8dc03698c913"
      },
      "outputs": [],
      "source": [
        "house.test[is.na(house.test$GarageYrBlt),c(\"GarageType\", \"GarageYrBlt\", \"GarageFinish\", \"GarageCars\", \"GarageArea\", \"GarageQual\", \"GarageCond\")]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "b3f1916a-32bb-f7db-10f5-953a6d475776"
      },
      "source": [
        "There is one entry which has detached garage for 1 car with area of 360. We cannot guess its other values so let just assume they are None and 0."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "650060c0-85cf-49e9-3f12-308a26645e92"
      },
      "outputs": [],
      "source": [
        "for (name in c(\"GarageType\", \"GarageQual\", \"GarageCond\", \"GarageFinish\")){\n",
        "  house.test[, name]<- as.character(house.test[, name])\n",
        "  house.test[is.na(house.test[,name]),name] <- 'None'\n",
        "}\n",
        "for (name in c(\"GarageYrBlt\", \"GarageCars\", \"GarageArea\")){\n",
        "  house.test[is.na(house.test[,name]),name] <- 0\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "1e4d5d84-3e66-b944-6c87-abbb4d406522"
      },
      "source": [
        "####PoolArea and PoolQc :"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "34877ceb-e191-6afe-4304-7ea9b4451e34"
      },
      "outputs": [],
      "source": [
        "house.test[is.na(house.test$PoolQC) & house.test$PoolArea != 0,c(\"PoolArea\", \"PoolQC\")]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "9fc37f8e-5c85-9737-c88d-1389a5e28f58"
      },
      "source": [
        "There are entries which has pool but pool quality is miissing.\n",
        "We can calculate mean pool area for each quality and then assign quality to the pool areas which lie btw minimum and maximum in each quality.\n",
        "Here we will use training data and test data as well.\n",
        "\n",
        "Merging two tables train and test"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "950b0d77-64aa-1ef4-2926-a64583f0b5d2"
      },
      "outputs": [],
      "source": [
        "temp.data <- rbind(house.test[,c(\"PoolArea\", \"PoolQC\")], house.train[,c(\"PoolArea\", \"PoolQC\")])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "6bd8e032-5952-37d9-de10-fb2e75706d36"
      },
      "source": [
        "**Minimum**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "72f92c13-2703-c219-07d0-c09a8e998324"
      },
      "outputs": [],
      "source": [
        "aggregate(PoolArea ~ PoolQC, temp.data, min)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "b5c4e9bd-8cc5-de73-6acb-f4109517aa74"
      },
      "source": [
        "**Maximum**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "596add53-f9b9-9a85-a500-86bb8e59a9d1"
      },
      "outputs": [],
      "source": [
        "aggregate(PoolArea ~ PoolQC, temp.data, max)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "79314632-8efc-8803-0e43-316aa888ad9c"
      },
      "source": [
        "There is a pool with poolArea 800. This is an oulier. Removing pool which has this area and taking max again."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "4d1a88ab-fea5-ab8e-8ef2-3201896ddb73"
      },
      "outputs": [],
      "source": [
        "temp.data <- temp.data[temp.data$PoolArea < 700,]\n",
        "aggregate(PoolArea ~ PoolQC, temp.data, max)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "b0219cf6-408b-70ef-460f-3a90b1a64490"
      },
      "source": [
        "So, the pool which has a area of 368 will be Ex, pool which has a area of 444 will be Ex, pool which has a area of 561 could either be in Gd or Fa.\n",
        "\n",
        "**Mean**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "2d3ca5cb-0d64-b234-c836-b6cab2fbd006"
      },
      "outputs": [],
      "source": [
        "aggregate(PoolArea ~ PoolQC, temp.data, mean)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "077f773a-3765-dc78-6be8-fcf0deb0fa2a"
      },
      "source": [
        "We above data since 561 is close to 583, we can make a rough guess that it is Fa."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b1d7e539-8b9f-0b5b-46c4-dc4a6095c99f"
      },
      "outputs": [],
      "source": [
        "house.test[,'PoolQC'] <- as.character(house.test$PoolQC)\n",
        "house.test[is.na(house.test$PoolQC) & (house.test$PoolArea == 368 | house.test$PoolArea == 444), 'PoolQC'] <- 'Ex'\n",
        "house.test[is.na(house.test$PoolQC) & (house.test$PoolArea == 561), 'PoolQC'] <- 'Fa'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "fc26674e-0dfa-8ed0-8c68-0bee98f227b0"
      },
      "source": [
        " All those for which pool area is 0 and Pool Qc is null we can assume that pool does not exist."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "23a303d9-2f62-a6d2-5f5d-62fdcc97c7ef"
      },
      "outputs": [],
      "source": [
        "house.test[,'PoolQC'] <- as.character(house.test$PoolQC)\n",
        "house.test[is.na(house.test$PoolQC), 'PoolQC'] <- 'None'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "e2244f04-9647-fe50-c7e7-2199a9165684"
      },
      "source": [
        "####Fence\n",
        "\n",
        "Total 169 entries which can be set to none as there may no fences available"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ea3d72cb-6f85-55a8-d3ca-7c58b48d9bf5"
      },
      "outputs": [],
      "source": [
        "house.test[,'Fence'] <- as.character(house.test$Fence)\n",
        "house.test[is.na(house.test$Fence), 'Fence'] <- 'None'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "923254f8-6795-7b4f-63b4-90728ef9b704"
      },
      "source": [
        "####MiscFeature"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "4ef8714a-2728-4d1a-93bb-d58587ba789f"
      },
      "outputs": [],
      "source": [
        "house.test[is.na(house.test$MiscFeature) & (house.test$MiscVal!=0) , c(\"MiscFeature\", \"MiscVal\")]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "9bdecf41-fe62-b38a-5af8-074fa0249b24"
      },
      "source": [
        "Combining train and test data to look at Misc Features,"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "4d4c0950-d998-da87-3d24-857afee552d1"
      },
      "outputs": [],
      "source": [
        "temp.data <- rbind(house.train[,c(\"MiscFeature\", \"MiscVal\")], house.test[,c(\"MiscFeature\", \"MiscVal\")])\n",
        "temp.data[(temp.data$MiscVal!=0) , c(\"MiscFeature\", \"MiscVal\")]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "cf9b8800-cb3d-a9d0-e1a4-81772b39e899"
      },
      "source": [
        "It is evident that Gar2 feature is costly so it is most likely to be Gar2\n",
        "Looking at train data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f290e0c3-e3d2-89fb-837e-6e14feb8ea68"
      },
      "outputs": [],
      "source": [
        "house.test[,'MiscFeature'] <- as.character(house.test$MiscFeature)\n",
        "house.test[is.na(house.test$MiscFeature) & (house.test$MiscVal==0) , 'MiscFeature'] <- 'None'\n",
        "house.test[is.na(house.test$MiscFeature) & (house.test$MiscVal!=0) , 'MiscFeature'] <- 'Gar2'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "64821fab-e455-fe4c-be4c-7e01005cc857"
      },
      "source": [
        "####SaleType"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3d8871b8-bc68-0625-72b2-2dafb269ad55"
      },
      "outputs": [],
      "source": [
        "elements <- unique(house.test[,'SaleCondition'])\n",
        "for (each.element in elements){\n",
        "  print(each.element)\n",
        "  print(summary(house.test[house.test$SaleCondition == each.element, c(\"SaleType\")]))\n",
        "\n",
        "}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5ed4ab53-8215-bfe1-6e5f-4450de26d34e"
      },
      "outputs": [],
      "source": [
        "g <- ggplot(data = house.test, aes(x = SaleType))\n",
        "g <- g+geom_bar() + facet_wrap(~SaleCondition, nrow = 3)\n",
        "g"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "45eecacc-0372-8908-abbc-b63adffe21c8"
      },
      "source": [
        "So, in case of normal sale condition sale type is more likely to be WD."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1668ea28-5e89-af81-0a50-a1ff15c8a51a"
      },
      "outputs": [],
      "source": [
        "house.test[,'SaleType'] <- as.character(house.test$SaleType)\n",
        "house.test[is.na(house.test$SaleType), 'SaleType'] <- 'WD'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "3fd16a33-12cd-1371-27a0-0f4839479ee8"
      },
      "source": [
        "Check if any null values left in house.test"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "0db54a59-1e3d-cff9-61c6-1735c89bd787"
      },
      "outputs": [],
      "source": [
        "null.cols.test <- sapply(names(house.test), function(col)any(is.na(house.test[,col])))\n",
        "null.cols.test <- null.cols.test[null.cols.test == TRUE]\n",
        "null.cols.test"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "ad5883d6-4250-3b5e-ea16-2d33d28ff869"
      },
      "source": [
        "=============================================================================="
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "e7097fd8-86a5-d390-f66b-cda16ae10dc5"
      },
      "source": [
        "### For Training Model :\n",
        "Test and Train data are combined and all the character variables are converted to factor variables. This is done so that any training model should know that what are all the factors possible for a variable even if some factors are available in train and missing in test for the variable or vice versa.\n",
        "\n",
        "Now, combining train and test data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "6ae9c02f-be6d-bef8-28ba-f77b7013d511"
      },
      "outputs": [],
      "source": [
        "house.test[,'SalePrice'] <- 0\n",
        "#set a flag to differentiate between test and train\n",
        "house.train[,'Flag'] <- 0\n",
        "house.test[,'Flag'] <- 1\n",
        "temp.data <- rbind(house.train, house.test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "b618038f-95a6-2486-d881-19f9708e136e"
      },
      "source": [
        "Getting all the character variables and converting it to factor variables"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "35075d0b-3a72-5c82-15da-76591c61b7d4"
      },
      "outputs": [],
      "source": [
        "factor.var <- sapply(names(temp.data), function(name)!is.numeric(temp.data[,name]))\n",
        "factor.var <- names(factor.var[factor.var==TRUE])\n",
        "factor.var"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "3c9d96b3-a73c-7e0e-2db4-9c5da1ca2a9d"
      },
      "source": [
        "Converting all to factors"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a4acd7c4-2e2f-2aba-4220-4509b2c4bea0"
      },
      "outputs": [],
      "source": [
        "for (name in factor.var){\n",
        "  temp.data[,name] <- as.factor(temp.data[,name])\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "725645cf-afbd-e06d-8855-72a58177e513"
      },
      "source": [
        "Splitting temp.data to train and test again"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f6d11f87-51c3-d836-fd2e-397ccf684c9d"
      },
      "outputs": [],
      "source": [
        "house.train <- temp.data[temp.data$Flag==0,]\n",
        "house.test <- temp.data[temp.data$Flag==1,]\n",
        "house.train <- house.train[,!(names(house.train) %in% c(\"Flag\"))]\n",
        "house.test <- house.test[,!(names(house.test) %in% c(\"Flag\",\"SalePrice\"))]"
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "R",
      "language": "R",
      "name": "ir"
    },
    "language_info": {
      "codemirror_mode": "r",
      "file_extension": ".r",
      "mimetype": "text/x-r-source",
      "name": "R",
      "pygments_lexer": "r",
      "version": "3.3.3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}