{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "a2edf4fb-3410-2137-e3f3-17b900038daf"
      },
      "source": [
        "# Data Exploration\n",
        "\n",
        "Let's take a look at the data, visualize some relationships, and see if there is anything unexpected that will help model building and validation later."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "93444aa9-31d4-7f0f-a30e-01abf9c82f90"
      },
      "outputs": [],
      "source": [
        "# Load some useful packages and configure plotting options\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "\n",
        "plt.style.use('fivethirtyeight')\n",
        "# plt.rcParams['font.family'] = 'serif'\n",
        "plt.rcParams['font.serif'] = 'Helvetica'\n",
        "plt.rcParams['font.monospace'] = 'Consolas'\n",
        "plt.rcParams['font.size'] = 14\n",
        "plt.rcParams['axes.labelsize'] = 14\n",
        "plt.rcParams['axes.labelweight'] = 'bold'\n",
        "plt.rcParams['xtick.labelsize'] = 12\n",
        "plt.rcParams['ytick.labelsize'] = 12\n",
        "plt.rcParams['legend.fontsize'] = 14\n",
        "plt.rcParams['figure.titlesize'] = 16\n",
        "plt.rcParams['lines.linewidth'] = 2\n",
        "\n",
        "%matplotlib inline\n",
        "\n",
        "# for auto-reloading external modules\n",
        "%load_ext autoreload\n",
        "%autoreload 2"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "0ef1351f-8487-4db0-a731-9719e8f70d28"
      },
      "source": [
        "# Loading data\n",
        "\n",
        "Let's load the data, and print out information on how many rows there are, if any are NAs, and see how the summary statistics look. We'll combine the ``train_df`` and `test_df` into a single dataframe `all_df`. This allows us to check for any distributional shift between training and test sets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "c8c47750-c02e-21a5-a831-cb6a070e96bb"
      },
      "outputs": [],
      "source": [
        "INPUT_DIR = '../input/'\n",
        "TRAIN_FILE = 'train.json'\n",
        "TEST_FILE = 'test.json'\n",
        "\n",
        "\n",
        "train_df = pd.read_json(INPUT_DIR + TRAIN_FILE)\n",
        "test_df = pd.read_json(INPUT_DIR + TEST_FILE)\n",
        "all_df = pd.concat((train_df, test_df), axis=0) # Can be useful to facet on train vs test\n",
        "# Add a bool to facet train vs test sets\n",
        "all_df['train'] = all_df['interest_level'].notnull()\n",
        "all_df['test'] = all_df['interest_level'].isnull()\n",
        "\n",
        "n_train = train_df.shape[0]\n",
        "n_test = test_df.shape[0]\n",
        "n_total = n_train + n_test\n",
        "n_train_pct = (n_train / n_total) * 100.0\n",
        "n_test_pct = (n_test / n_total) * 100.0\n",
        "\n",
        "print('Train DF Shape: {}, %age: {:.1f}'.format(train_df.shape, n_train_pct))\n",
        "print('Test  DF Shape: {}, %age: {:.1f}'.format(test_df.shape, n_test_pct))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "7b3259be-b13d-737a-2ea5-78cf5a3b8ed7"
      },
      "outputs": [],
      "source": [
        "def print_df_info(df, name):\n",
        "    \"\"\"\n",
        "    Prints out more detailed DF info\n",
        "    \"\"\"\n",
        "    print('\\n{} Info:\\n'.format(name))\n",
        "    print(df.info())\n",
        "    print('\\n{} Null info by column:\\n'.format(name))\n",
        "    print(df.isnull().sum(axis=0))\n",
        "    print('\\n{} Statistical Description:\\n'.format(name))\n",
        "    print(df.describe())\n",
        "\n",
        "print_df_info(train_df, 'Train')\n",
        "print_df_info(test_df, 'Test')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "8fda0b42-6e63-2ad9-81ad-9788e3009e85"
      },
      "outputs": [],
      "source": ""
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "3e5253c0-13c0-c82c-6617-056333858d69"
      },
      "source": [
        "Here are a few brief observations from the statistics above:\n",
        "\n",
        "### Train\n",
        "\n",
        "No null values in any of the columns. The bedrooms and bathrooms look sensible, there are some zero values for latitude and longitude which might be a null value (maybe the location wasn't entered for the apartment?). The price looks a bit strange ($43 dollars per week?)\n",
        "\n",
        "### Test\n",
        "\n",
        "No null values in these columns. There are some bathroom outliers (112?). Longitude and Latitude of 0 seems to be a reserved value again."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "927134e6-3589-3fe9-7563-75422b17ecfc"
      },
      "source": [
        "# Numeric column exploration\n",
        "\n",
        "Let's start off by looking at the numeric columns, and their distributions across training and test datasets. How many bathrooms do the listings have?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "14c9b83c-48a6-4b0d-969c-741b9f8720ab"
      },
      "outputs": [],
      "source": [
        "g = sns.FacetGrid(all_df, col=\"test\", sharex=True, sharey=False, size=5)\n",
        "g = g.map(sns.distplot, \"bathrooms\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "fc7d57a7-2034-4b9f-9423-3c4ad9d67d46"
      },
      "source": [
        "# Bathrooms and Bedrooms"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "33e4ced2-12df-40eb-c4ea-432961bc2949"
      },
      "source": [
        "There are some crazy outliers in the test set (112 bathrooms ?!). Let's drop listings with more than 10 bathrooms and replot."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "eb0df4f5-9fcb-4fbc-2c0d-dbf881c16b6f"
      },
      "outputs": [],
      "source": [
        "g = sns.FacetGrid(all_df[all_df['bathrooms'] < 10], col=\"test\", sharex=True, sharey=False, size=5)\n",
        "g = g.map(sns.countplot, \"bathrooms\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "63ac53c6-9d1a-40ad-bc66-c2e35ba13a86"
      },
      "outputs": [],
      "source": [
        "g = sns.FacetGrid(all_df, col=\"test\", sharex=True, sharey=False, size=5)\n",
        "g = g.map(sns.countplot, \"bedrooms\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "36ebc832-b4f3-4534-8638-d260d92b7f66"
      },
      "outputs": [],
      "source": [
        "# Check for correlation between bedrooms and bathrooms\n",
        "sns.jointplot(data=all_df[all_df['train']], x=\"bedrooms\", y=\"bathrooms\", kind='reg');"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "2797904f-a8db-22a6-8f6a-a75aa2caf98b"
      },
      "source": [
        "## Building ID\n",
        "\n",
        "* The building ID of 0 seems to mean a missing value. Maybe we need to impute building values based on longitude / latitude?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d4ae9b4a-4ad5-b5fb-5936-3e1b19e9bec7"
      },
      "outputs": [],
      "source": [
        "# How many unique building IDs are there compared to the listings?\n",
        "building_df = all_df.groupby('building_id').size().reset_index(name=\"count\")\n",
        "building_df = building_df.sort_values(\"count\", ascending=False).reset_index(drop=True)\n",
        "building_df.head(10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "9eabc4b0-d2f7-6de4-c2ed-0671981434c6"
      },
      "source": [
        "## `Created` field data exploration\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "37bc97d3-c517-3803-c17c-377596a78949"
      },
      "outputs": [],
      "source": [
        "# First of all we need to convert the date and time - \"2016-06-24 07:54:24\"\n",
        "DATE_FORMAT = \"%Y-%m-%d %H:%M:%S\"\n",
        "train_df['created'] = pd.to_datetime(train_df['created'], format=DATE_FORMAT)\n",
        "test_df['created'] = pd.to_datetime(test_df['created'], format=DATE_FORMAT)\n",
        "all_df['created'] = pd.to_datetime(all_df['created'], format=DATE_FORMAT)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b5927426-957f-f38c-6ae5-9acde715b0af"
      },
      "outputs": [],
      "source": [
        "g = sns.FacetGrid(all_df, col=\"train\", sharex=True, sharey=True, size=5)\n",
        "g = g.map(sns.distplot, \"bathrooms\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1cd9ba0f-8600-1f46-64e2-ce5d44039b95"
      },
      "outputs": [],
      "source": [
        "# How does the train / test split depend on the creation date?\n",
        "all_agg_df = all_df.copy()\n",
        "all_agg_df = all_agg_df.set_index('created', drop=True)\n",
        "all_agg_df = all_agg_df.groupby('train').resample('1D').size().transpose()\n",
        "\n",
        "fig, ax = plt.subplots(1,1, figsize=(18,10))\n",
        "ax = all_agg_df.plot.bar(ax=ax, stacked=True)\n",
        "ax.set_xticklabels(all_agg_df.index.strftime('%a %b %d'))\n",
        "ax.set_title('All listings creation date and train/test split');"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "56499e0b-1ffb-c859-5df0-70eab1b3bb23"
      },
      "outputs": [],
      "source": [
        "# How does creation vary by Day of Week?\n",
        "all_agg_df = all_df.copy()\n",
        "all_agg_df['dayofweek'] = all_df['created'].dt.dayofweek\n",
        "all_agg_df = all_agg_df.groupby('dayofweek').size()\n",
        "\n",
        "all_agg_df.plot.bar(title=\"All row creation by day of week\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f3eddbde-6376-0350-296c-cda124a8fa6e"
      },
      "outputs": [],
      "source": [
        "# How does creation vary by Hour?\n",
        "all_agg_df = all_df.copy()\n",
        "all_agg_df['hour'] = all_df['created'].dt.hour\n",
        "all_agg_df = all_agg_df.groupby('hour').size()\n",
        "\n",
        "all_agg_df.plot.bar(title=\"All row creation by hour of day\", figsize=(10,6));"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e82235bc-77cd-5fc6-c832-051e29b7abf3"
      },
      "outputs": [],
      "source": [
        "# Separate line plot for day vs hour of day creation\n",
        "all_agg_df = all_df.copy()\n",
        "\n",
        "all_agg_df = all_agg_df.reset_index()\n",
        "all_agg_df['dayofweek'] = all_agg_df['created'].dt.weekday_name\n",
        "all_agg_df['hour'] = all_agg_df['created'].dt.hour\n",
        "all_agg_df = all_agg_df.groupby(['dayofweek', 'hour']).size().reset_index(name=\"count\")\n",
        "# all_agg_df = all_agg_df[['dayofweek', 'hour', 'checkouts']]\n",
        "all_agg_df = all_agg_df.pivot_table(values='count', index='hour', columns='dayofweek')\n",
        "\n",
        "all_agg_df = all_agg_df[['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday']]\n",
        "day_palette = sns.color_palette(\"hls\", 7) # Need to have 7 distinct colours\n",
        "\n",
        "fig, ax = plt.subplots(1,1, figsize=(10,8))\n",
        "all_agg_df.plot.line(ax=ax, linewidth=3, color=day_palette, title=\"Created by hour and day\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "68547ecb-d250-ce18-ca20-71b84a3a4eee"
      },
      "outputs": [],
      "source": [
        "# Proportion of interest level by day-of-week\n",
        "INT_ORDER = ['low', 'medium', 'high']\n",
        "\n",
        "plot_df = train_df.copy()\n",
        "plot_df['day'] = plot_df['created'].dt.dayofweek\n",
        "plot_df = plot_df.groupby([\"day\", \"interest_level\"]).size().reset_index(name='count')\n",
        "day_total = plot_df.groupby('day').sum()   \n",
        "plot_df = pd.merge(plot_df, day_total, left_on=\"day\", right_index=True, suffixes=[\"_value\", \"_day_total\"])\n",
        "plot_df['fraction'] = plot_df['count_value'] / plot_df['count_day_total']\n",
        "plot_df = plot_df.pivot_table(index=\"day\", columns=\"interest_level\", values=\"fraction\")\n",
        "\n",
        "plot_df = plot_df[INT_ORDER]\n",
        "plot_df.plot.bar(stacked=True, title=\"Proportion of interest level by day-of-week\")\n",
        "plt.legend(bbox_to_anchor=(1.1, 0.7, 1., .102), loc=3);"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ff96ec69-3622-5f8a-1ab7-b6907b02ba3f"
      },
      "outputs": [],
      "source": [
        "# Proportion of interest level by hour-of-day\n",
        "INT_ORDER = ['low', 'medium', 'high']\n",
        "\n",
        "plot_df = train_df.copy()\n",
        "plot_df['hour'] = plot_df['created'].dt.hour\n",
        "plot_df = plot_df.groupby([\"hour\", \"interest_level\"]).size().reset_index(name='count')\n",
        "day_total = plot_df.groupby('hour').sum()   \n",
        "plot_df = pd.merge(plot_df, day_total, left_on=\"hour\", right_index=True, suffixes=[\"_value\", \"_day_total\"])\n",
        "plot_df['fraction'] = plot_df['count_value'] / plot_df['count_day_total']\n",
        "plot_df = plot_df.pivot_table(index=\"hour\", columns=\"interest_level\", values=\"fraction\")\n",
        "\n",
        "plot_df = plot_df[INT_ORDER]\n",
        "plot_df.plot.bar(stacked=True, figsize=(8,6),\n",
        "                 title=\"Proportion of interest level by day-of-week\")\n",
        "plt.legend(bbox_to_anchor=(1.05, 0.8, 1., .102), loc=3);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "f5ff7e7d-7e60-bb79-d31c-8d12a32ef10f"
      },
      "source": [
        "## Description"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "39233904-6cf4-36e9-1f01-3761bcd93651"
      },
      "outputs": [],
      "source": [
        "plot_df = all_df.copy()\n",
        "plot_df['len_description'] = plot_df['description'].apply(len)\n",
        "# plot_df['len_description'].plot.hist(bins=40, figsize=(8,6), title=\"Description length\");\n",
        "g = sns.FacetGrid(plot_df, col=\"test\", sharex=True, sharey=True, size=5)\n",
        "g = g.map(sns.distplot, \"len_description\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "6e966fce-70e7-2f80-bd37-aa534b1b583d"
      },
      "outputs": [],
      "source": [
        "plot_df = plot_df[plot_df['len_description'] < 2500]\n",
        "#plot_df['len_description'].plot.hist(bins=40, figsize=(8,6), title=\"Description length (outliers removed)\");\n",
        "g = sns.FacetGrid(plot_df, col=\"test\", sharex=True, sharey=True, size=5)\n",
        "g = g.map(sns.distplot, \"len_description\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "fcc59d4f-0da7-adb3-98cd-a5195af19bf3"
      },
      "outputs": [],
      "source": [
        "plot_df = all_df.copy()\n",
        "plot_df['words'] = plot_df['description'].apply(str.split).apply(len)\n",
        "#plot_df['words'].plot.hist(bins=40, figsize=(8,6), title=\"Description words\");\n",
        "g = sns.FacetGrid(plot_df, col=\"test\", sharex=True, sharey=True, size=5)\n",
        "g = g.map(sns.distplot, \"words\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "0e9025d4-792c-a103-fbb8-26f8b9728f17"
      },
      "outputs": [],
      "source": [
        "plot_df = plot_df[plot_df['words'] < 500]\n",
        "#plot_df['words'].plot.hist(bins=40, figsize=(8,6), title=\"Description words\");\n",
        "g = sns.FacetGrid(plot_df, col=\"test\", sharex=True, sharey=True, size=5)\n",
        "g = g.map(sns.distplot, \"words\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d3994daa-0926-7419-a376-91f3300bbbb7"
      },
      "outputs": [],
      "source": [
        "# Interest level by length of description\n",
        "plot_df = train_df.copy()\n",
        "plot_df['len_description'] = plot_df['description'].apply(len)\n",
        "fix, ax = plt.subplots(1,1,figsize=(8,6))\n",
        "ax.set_title(\"Interest level by number of characters in description\")\n",
        "sns.violinplot(data=plot_df, x=\"interest_level\", y=\"len_description\", order=INT_ORDER, ax=ax);"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d1098e47-a78e-4a50-6ad5-77dc0dd3635e"
      },
      "outputs": [],
      "source": [
        "# Interest level by length of description\n",
        "plot_df = train_df.copy()\n",
        "plot_df['words'] = plot_df['description'].apply(str.split).apply(len)\n",
        "fix, ax = plt.subplots(1,1,figsize=(8,6))\n",
        "ax.set_title(\"Interest level by number of words in description\")\n",
        "sns.violinplot(data=plot_df, x=\"interest_level\", y=\"words\", order=INT_ORDER, ax=ax);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "762db2de-429a-58fc-81a0-d78e6be80d11"
      },
      "source": [
        "## Display Address\n",
        "\n",
        "On the renthop listing it has a summary like:\n",
        "\n",
        "```\n",
        "1BR, 1BA at 50 Murray Street, Apt 1604\n",
        "Tribeca, Downtown Manhattan, Manhattan\n",
        "```\n",
        "\n",
        "The display address is the rough area. People sometimes include details like 'No Fee'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "9d9fe316-c657-e2ab-24c5-defeaeb9a02f"
      },
      "outputs": [],
      "source": [
        "# Plot Histograms of display address words for train and test sites\n",
        "plot_df = all_df.copy()\n",
        "plot_df['words'] = plot_df['display_address'].apply(str.split).apply(len)\n",
        "#plot_df['words'].plot.hist(bins=40, figsize=(8,6), title=\"Description words\");\n",
        "g = sns.FacetGrid(plot_df, col=\"test\", sharex=True, sharey=True, size=5)\n",
        "g = g.map(sns.countplot, \"words\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e4483225-a23e-da78-b2d4-32681d6ca4b8"
      },
      "outputs": [],
      "source": [
        "# Remove outliers and re-plot\n",
        "plot_df = plot_df[plot_df['words'] < 12]\n",
        "g = sns.FacetGrid(plot_df, col=\"test\", sharex=True, sharey=True, size=5)\n",
        "g = g.map(sns.countplot, \"words\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "84ae14e9-f43d-b7f0-e21e-9adb3c070ca1"
      },
      "outputs": [],
      "source": [
        "# Interest level by length of display_address\n",
        "plot_df = train_df.copy()\n",
        "plot_df['words'] = plot_df['display_address'].apply(str.split).apply(len)\n",
        "plot_df = plot_df[plot_df['words'] < 12]\n",
        "plot_df.head()\n",
        "\n",
        "fix, ax = plt.subplots(1,1,figsize=(8,6))\n",
        "sns.violinplot(data=plot_df, x=\"interest_level\", y=\"words\", order=INT_ORDER)\n",
        "ax.set_title(\"Interest level by number of words in display address\");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "31e72077-82a2-9fc3-0842-27ffd5c2de15"
      },
      "source": [
        "## Latitude and Longitude"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "45f81597-a816-8358-211c-bd0cd8996b84"
      },
      "outputs": [],
      "source": [
        "# Plot Histograms of display address words for train and test sites\n",
        "plot_df = all_df.copy()\n",
        "#plot_df['words'].plot.hist(bins=40, figsize=(8,6), title=\"Description words\");\n",
        "g = sns.FacetGrid(plot_df, col=\"test\", sharex=True, sharey=True, size=5)\n",
        "g = g.map(sns.countplot, \"latitude\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "059bed25-12bc-fa16-8c1d-61168b168441"
      },
      "outputs": [],
      "source": ""
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "862ead5d-dd05-5a54-960b-877970774bb8"
      },
      "outputs": [],
      "source": ""
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1f0f4373-f6e0-2482-df92-7e5927e690e4"
      },
      "outputs": [],
      "source": ""
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5a29c79a-67a2-73e0-2221-a85c916a2d83"
      },
      "outputs": [],
      "source": ""
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "cf1f7137-1604-701b-9e29-0c03cdfae6d7"
      },
      "source": [
        "## Listing ID"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1b8f4651-4790-45dc-b0f3-ab375572a481"
      },
      "outputs": [],
      "source": ""
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "36d4cf10-c00f-d70b-0e5f-4ac89c30c005"
      },
      "source": [
        "# Display address"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "c435f916-8f5e-2391-b45c-333cd79526f3"
      },
      "outputs": [],
      "source": [
        "# Interest level by length of description\n",
        "plot_df = train_df.copy()\n",
        "plot_df['len_disp_address'] = plot_df['display_address'].apply(len)\n",
        "fix, ax = plt.subplots(1,1,figsize=(8,6))\n",
        "ax.set_title(\"Interest level by length of display address\")\n",
        "sns.violinplot(data=plot_df, x=\"interest_level\", y=\"len_disp_address\", order=INT_ORDER, ax=ax);"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1b8cadb0-ee21-36e9-2a03-e3fd78c7dfd5"
      },
      "outputs": [],
      "source": [
        "# Interest level by length of description\n",
        "plot_df = plot_df[plot_df['len_disp_address'] < 20]\n",
        "fix, ax = plt.subplots(1,1,figsize=(8,6))\n",
        "ax.set_title(\"Interest level by length of display address\")\n",
        "sns.violinplot(data=plot_df, x=\"interest_level\", y=\"len_disp_address\", order=INT_ORDER, ax=ax);"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a3245067-976d-c156-57b0-8da7a71aef42"
      },
      "outputs": [],
      "source": [
        "# Interest level by length of description words\n",
        "plot_df = train_df.copy()\n",
        "plot_df['disp_addess_words'] = plot_df['display_address'].apply(str.split).apply(len)\n",
        "fix, ax = plt.subplots(1,1,figsize=(8,6))\n",
        "ax.set_title(\"Interest level by words in display address\")\n",
        "sns.violinplot(data=plot_df, x=\"interest_level\", y=\"disp_addess_words\", order=INT_ORDER, ax=ax);"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "facb093f-88ef-2b5a-dde0-854859f3abf5"
      },
      "outputs": [],
      "source": [
        "# Interest level by length of description words\n",
        "plot_df = plot_df[plot_df['disp_addess_words'] < 8]\n",
        "plot_df['disp_addess_words'] = plot_df['display_address'].apply(str.split).apply(len)\n",
        "fix, ax = plt.subplots(1,1,figsize=(8,6))\n",
        "ax.set_title(\"Interest level by words in display address\")\n",
        "sns.violinplot(data=plot_df, x=\"interest_level\", y=\"disp_addess_words\", order=INT_ORDER, ax=ax);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "4265ac5d-38c6-a8a7-e662-c344d8f63841"
      },
      "source": [
        "## Latitude and Longitude"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "0a323884-0a71-214f-dbdd-b47da2f6807f"
      },
      "outputs": [],
      "source": [
        "plot_df = all_df.copy()\n",
        "g = sns.FacetGrid(plot_df, col=\"test\", sharex=True, sharey=True, size=5)\n",
        "g = g.map(sns.countplot, \"latitude\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3cd7f47b-5f4e-e5a9-882b-f3b639da40b8"
      },
      "outputs": [],
      "source": [
        "g = sns.FacetGrid(plot_df, col=\"test\", sharex=True, sharey=True, size=5)\n",
        "g = g.map(sns.countplot, \"longitude\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "91da1a46-ddcc-819a-5e54-b70bf6649fc5"
      },
      "outputs": [],
      "source": [
        "plot_df.plot.scatter(x=\"latitude\", y=\"longitude\", title=\"Scatter plot of latitude/longitude\");\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "8f2a63cf-f651-f0b9-cb30-32cda0d7a231"
      },
      "outputs": [],
      "source": ""
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "6cda1da5-c87e-5974-828a-7507dd9015fb"
      },
      "source": [
        "## Interest Level"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "9d608b05-881d-0fa5-0426-15b0ede39542"
      },
      "outputs": [],
      "source": [
        "# Class balance on interest level?\n",
        "plot_df = train_df.groupby('interest_level').size()\n",
        "plot_df = plot_df[INT_ORDER]\n",
        "plot_df.plot.bar(title=\"Interest level counts in training data\")\n",
        "\n",
        "total = plot_df.sum()\n",
        "print('Percentage of data:\\n{}'.format((plot_df / total).round(3) * 100.0))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "278a2e3a-03b1-d1fd-cfff-bb53e7d0c4db"
      },
      "outputs": [],
      "source": ""
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "06481dd6-7bcb-f3b2-50b4-9981da88243f"
      },
      "outputs": [],
      "source": ""
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "33a52354-c3ce-84b5-4989-ab2305507082"
      },
      "outputs": [],
      "source": ""
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}