{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "0239809d-07b4-1948-098c-5cf93f194542"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "fd126e11-c50a-872a-d60d-3d0f5437f463"
      },
      "outputs": [],
      "source": [
        "# This Python 3 environment comes with many helpful analytics libraries installed\n",
        "# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n",
        "\n",
        "import numpy as np \n",
        "import pandas as pd\n",
        "\n",
        "# Input data files are available in the \"../input/\" directory.\n",
        "# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n",
        "import kagglegym\n",
        "train_df = pd.read_json(\"../input/train.json\")\n",
        "test_df = pd.read_json(\"../input/test.json\")\n",
        "train_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "8f73332e-d932-8056-2972-1ce3da647cc0"
      },
      "source": [
        "We see a number of different categorical and numerical data, features are shown as an array as well as photos.\n",
        "\n",
        "Lets look into the distribution of the target variable Interest\n",
        "\n",
        "**Interest Distribtuion**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "02e8f953-c5e1-efb3-bb1d-8fc171696187"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "color = sns.color_palette()\n",
        "int_level = train_df['interest_level'].value_counts()\n",
        "\n",
        "plt.figure(figsize=(6,3))\n",
        "sns.barplot(int_level.index, int_level.values, alpha=0.6, color=color[2])\n",
        "plt.ylabel('Number of Occurrences', fontsize=12)\n",
        "plt.xlabel('Interest level', fontsize=12)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "a08fc6e9-cf3f-0c5d-deee-e1d2cee84bd3"
      },
      "source": [
        "We can see that the majority of the houses have a low interest level associated with the listing.\n",
        "Lets explore the other features of our data set"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "edca9de8-5c20-5fef-ad3f-501aba2017a3"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(10,4))\n",
        "bathroom = train_df['bathrooms'].value_counts()\n",
        "\n",
        "sns.barplot(bathroom.index,bathroom.values,alpha=.8,color=color[1])\n",
        "plt.ylabel('Number of Occurances',fontsize=12)\n",
        "plt.xlabel('Number of Bathrooms',fontsize=12)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "2efa3ab3-9de1-79b6-81c3-d27eb757af71"
      },
      "source": [
        "We can see that the majority of apartments have only 1 bathroom, lets see if interest level is at all affected by the number of bathrooms"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "43165302-65ed-57f9-d32b-d2d007b8ef0e"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(8,4))\n",
        "sns.countplot(x='bathrooms', hue='interest_level', data=train_df)\n",
        "plt.ylabel('Number of Occurances',fontsize=12)\n",
        "plt.xlabel('Number of Bathrooms',fontsize=12)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "7235e65b-0a3c-2716-6b1a-636bad8dc5ef"
      },
      "source": [
        "There is no apparent indication of differences in interest among varying number of bathrooms.\n",
        "\n",
        "Lets explore the number of bedrooms"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "2f13c43d-dc11-60fd-f6d2-4795067f7849"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(6,4))\n",
        "\n",
        "sns.countplot(x='bedrooms',data=train_df)\n",
        "plt.ylabel('Number of Occurances',fontsize=12)\n",
        "plt.xlabel('Number of Bedrooms',fontsize=12)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "cd73738d-b784-4ab8-fb75-8d9fe7cf7c91"
      },
      "source": [
        "Lets see if there is a relationship between interest and bedrooms."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e393874d-47a8-5091-1b12-ec3a747dd57e"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(6,4))\n",
        "\n",
        "sns.countplot(x='bedrooms',hue='interest_level',data=train_df)\n",
        "plt.ylabel('Number of Occurances',fontsize=12)\n",
        "plt.xlabel('Number of Bedrooms',fontsize=12)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "26184d86-a4ff-0ba6-57b2-e352278a2e9e"
      },
      "source": [
        "There is nothing apparent about the relationship. Between all numbers of bedrooms there seems to be the same proportion of interest levels.\n",
        "\n",
        "Let's look at the distribution of prices"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "24d03702-e960-810d-9e01-1f68225296b8"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(8,4))\n",
        "\n",
        "plt.scatter(range(train_df.shape[0]), np.sort(train_df.price.values))\n",
        "plt.ylabel('Price',fontsize=12)\n",
        "plt.xlabel('Individual Listing')\n",
        "plt.title('Distribution of House prices')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "93142bd5-6bec-7806-c677-a9f480da8562"
      },
      "source": [
        "We can see that there are a couple of outliers in our data set, let see if a log transform can fix this issue"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "a5eb4667-1adc-365a-d222-a025647740e6"
      },
      "source": [
        "Our distribution of prices is still not normal, but better when dealing with outliers, will a log transform make this distribution more normal?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "66d68c94-5036-1d8c-ddcc-fa0f77a49da2"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(8,4))\n",
        "\n",
        "sns.distplot(np.log(train_df.price.values))\n",
        "plt.ylabel('Price',fontsize=12)\n",
        "plt.xlabel('Individual Listing')\n",
        "plt.title('Distribution of House prices')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "a3fe1fb9-8e8d-7346-a221-5ea48f5cbb98"
      },
      "source": [
        "This looks much better, we will transform all the prices to now be the logarithm of usd."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e8a93692-4f4a-2224-7f1d-b073081f8887"
      },
      "outputs": [],
      "source": [
        "#transforms price to be logarithm\n",
        "train_df.price = train_df.price.apply(lambda x: np.log(x))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "211f7cfa-9271-1307-7536-b84806c664e5"
      },
      "source": [
        "**Features**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "c4c689dd-1d57-4ca4-a593-0bb08306bc4c"
      },
      "outputs": [],
      "source": [
        "#get number of features for each listing\n",
        "train_df['num_features']  = train_df.features.apply(lambda x: len(x))\n",
        "#plot distribution\n",
        "f, (ax1, ax2) = plt.subplots(1, 2, sharey=True,figsize=(14,4))\n",
        "\n",
        "sns.countplot(x='num_features',data=train_df,ax=ax1)\n",
        "ax1.set_title('Distribution of number of features')\n",
        "\n",
        "sns.countplot(x='num_features',hue='interest_level',data=train_df,ax=ax2)\n",
        "ax2.set_title('Distribution of number of features by interest')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "eaf524ea-797e-68a3-2263-53bccb9894e3"
      },
      "source": [
        "We can see the majority of features per listing is around 3, there is no evident association between interest level and number of features. Getting the actual features of a listing should be explored.\n",
        "Let's explore the most common features"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f1b926b0-3d69-bf1a-e96a-140b4116669c"
      },
      "outputs": [],
      "source": [
        "import itertools\n",
        "from collections import Counter\n",
        "\n",
        "#puts all features into one list and gets word count\n",
        "featureCount =Counter(list(itertools.chain.from_iterable(train_df.features)))\n",
        "featureCount = dict((k.replace(\" \", \"_\"), v) for k, v in featureCount.items() if v >= 5)\n",
        "sortedFeatures = sorted(featureCount, key=featureCount.get,reverse=True)\n",
        "print('The top ten features are ' + str(sortedFeatures[:10]))\n",
        "\n",
        "#transform feature column to be put in vectorizer\n",
        "train_df.features = [','.join(x) for x in train_df.features]\n",
        "train_df.features = train_df.features.apply(lambda x: x.replace(' ','_'))\n",
        "train_df.features = train_df.features.apply(lambda x: x.replace(',',' '))\n",
        "\n",
        "#transform features into a sparse matrix\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "#fit count vectorizer\n",
        "vectorizer = CountVectorizer().fit(sortedFeatures)#fit vectorizer with top features\n",
        "featVec = vectorizer.fit_transform(train_df.features)#transform training data into sparse matrix"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b1f34b9d-58f6-0714-e0dd-cf9acd4eb607"
      },
      "outputs": [],
      "source": [
        ""
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}