{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "99f9bb90-4688-14ea-2592-51f7ef027512"
      },
      "source": [
        "In this notebook I show two methods to encode location features (longitude, latitude) to get a \"neighborhood\" proxy feature."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "301ac311-6e4b-2c27-e748-292511ce5654"
      },
      "outputs": [],
      "source": [
        "%matplotlib inline\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import matplotlib as mpl\n",
        "import seaborn as sns\n",
        "from sklearn.cluster import KMeans"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "80e619b4-5134-5add-2088-ef628041dbb7"
      },
      "source": [
        "### Load dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "c64d65de-40b9-e70a-4778-428ccdbc3d42"
      },
      "outputs": [],
      "source": [
        "df = pd.read_json('../input/train.json')[['longitude','latitude','interest_level','price']]\n",
        "df['response'] = 0.\n",
        "df.loc[df.interest_level=='medium', 'response'] = 0.5\n",
        "df.loc[df.interest_level=='high', 'response'] = 1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "b9a091d4-2b4b-f4b6-076f-34521abfdb8a"
      },
      "source": [
        "### Outlier removal\n",
        "\n",
        "If you take a look to the location coordinates you'll notice a few outliers far away from NY city center. I'll start by excluding them using a recursive approach. At each iteration all the listings with |z-score|>3 are excluded. The removal algorithm stops once all the listings have |z-score|<3."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "67ada435-0acf-1fe0-8d21-0e106558d3a5"
      },
      "outputs": [],
      "source": [
        "# Show location coordinates before oulier removal\n",
        "fig, ax = plt.subplots(1, 2, figsize=(9,6))\n",
        "print('Length before removing ouliers', len(df))\n",
        "ax[0].plot(df.longitude, df.latitude, '.');\n",
        "ax[0].set_title('Before outlier removal');\n",
        "ax[0].set_xlabel('Longitude');\n",
        "ax[0].set_ylabel('Latitude');\n",
        "# Outlier removal\n",
        "for i in ['latitude', 'longitude']:\n",
        "    while(1):\n",
        "        x = df[i].median()\n",
        "        ix = abs(df[i] - x) > 3*df[i].std()\n",
        "        if ix.sum()==0: # no more outliers -> stop\n",
        "            break\n",
        "        df.loc[ix, i] = np.nan # exclude outliers\n",
        "# Keep only non-outlier listings\n",
        "df = df.loc[df[['latitude', 'longitude']].isnull().sum(1) == 0, :]\n",
        "print('Length after removing ouliers', len(df))\n",
        "# Show location coordinates after outlier removal\n",
        "ax[1].plot(df.longitude, df.latitude, '.');\n",
        "ax[1].set_title('After outlier removal');\n",
        "ax[1].set_xlabel('Longitude');\n",
        "ax[1].set_ylabel('Latitude');"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "0561845b-72bc-3496-0fe7-dce025220a1d"
      },
      "source": [
        "### Neighborhoods\n",
        "Here, I'll use Kmeans to cluster listings based on their coordinates. I'll show the results of using 5, 10, 20 and 40 \"neighborhoods\"."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5d40b35d-c77e-9cdb-b038-2146139abd2d"
      },
      "outputs": [],
      "source": [
        "fig, ax = plt.subplots(4,1,figsize=(9,30))\n",
        "for ix, ncomp in enumerate([5, 10, 20, 40]):\n",
        "    r = KMeans(ncomp, random_state=1)\n",
        "    # Normalize (longitude, latitude) before K-means\n",
        "    temp = df[['longitude', 'latitude']].copy()\n",
        "    temp['longitude'] = (temp['longitude']-temp['longitude'].mean())/temp['longitude'].std()\n",
        "    temp['latitude'] = (temp['latitude']-temp['latitude'].mean())/temp['latitude'].std()\n",
        "    # Fit k-means and get labels\n",
        "    r.fit(temp[['longitude', 'latitude']])\n",
        "    df['labels'] = r.labels_\n",
        "    # Plot results\n",
        "    cols = sns.color_palette(\"Set2\", n_colors=ncomp, desat=.5)\n",
        "    cl = [cols[i] for i in r.labels_]\n",
        "    area = 12\n",
        "    ax[ix].scatter(df.longitude, df.latitude, s=area, c=cl, alpha=0.5);\n",
        "    ax[ix].set_title('Number of components: ' + str(ncomp))\n",
        "    ax[ix].set_xlabel('Longitude')\n",
        "    ax[ix].set_ylabel('Latitude')\n",
        "    # Show aggregated volume and interest at each neighborhood\n",
        "    x = df.groupby('labels')[['longitude','latitude','response']].mean().sort_values(['response'])\n",
        "    x = pd.concat([x, df['labels'].value_counts()], axis=1).sort_values(['response'])\n",
        "    cols = sns.color_palette(\"RdBu_r\", ncomp)[::-1]\n",
        "    for i in range(ncomp):\n",
        "        props = dict(boxstyle='round', facecolor=cols[i], alpha=0.8)\n",
        "        ax[ix].text(x.longitude.values[i], x.latitude.values[i], \n",
        "                str(np.array(np.round(x.response.values,2), '|S8')[i])+'\\n'+str(np.array(x['labels'].values, '|S8')[i]), \n",
        "                fontsize=9, verticalalignment='center', horizontalalignment='center', bbox=props);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "d6cf0b1a-866c-9c3c-6738-aa9e6f9b1fe6"
      },
      "source": [
        "### Supervised encoding of location coordinates\n",
        "The aim of this analysis is to provide a way to numericaly encode listing locations based on the mean location interest. I'll proceed as follows:\n",
        "\n",
        "- Define a fixed grid of 40 points both for 'latitude' and 'longitude' values\n",
        "- At each square defined by two consecutive longitude and latitude breaks compute the mean interest of the listings within the square"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "2c98517b-16ad-8311-29d5-e36c0fe2954d"
      },
      "outputs": [],
      "source": [
        "res = 40 # grid size\n",
        "min_n = 30 # minimum size to perform inference\n",
        "# Define grids\n",
        "nx = np.linspace(df.longitude.min(), df.longitude.max(), res)\n",
        "ny = np.linspace(df.latitude.min(), df.latitude.max(), res)\n",
        "# Encode\n",
        "Y = pd.DataFrame()\n",
        "for i in range(res-1):\n",
        "    for j in range(res-1):\n",
        "        # Identify listings within the square\n",
        "        ix = (df.longitude >= nx[i])&(df.longitude < nx[i+1])&(df.latitude >= ny[j])&(df.latitude < ny[j+1])\n",
        "        # Compute mean interest if the number of listings is greated than 'min_n'\n",
        "        if ix.sum() > min_n:\n",
        "            y = df.loc[ix, :].mean() # mean interest\n",
        "            y['n'] = ix.sum() # volume\n",
        "            Y = pd.concat([Y, y], axis=1)\n",
        "Y = Y.transpose()\n",
        "# Show results\n",
        "cols = sns.color_palette(\"RdBu_r\", 5)\n",
        "cl = [cols[i] for i in np.digitize(Y.response.values, Y.response.quantile(np.arange(1/5., 1, 1/5.)))]\n",
        "area = 12 + 5*np.log1p(Y.n - min_n)\n",
        "fig, ax = plt.subplots(1, 1, figsize=(9,9))\n",
        "ax.scatter(Y.longitude, Y.latitude, s=area, c=cl, alpha=0.5);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "d8b7a2e2-394f-d85a-6271-3b970cad15cc"
      },
      "source": [
        "Any feedback or comment will be appreciated! Upvote if you found it interesting :)\n",
        "\n",
        "Thanks!"
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}