{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "c892e370-ff2b-25ba-be34-755194f59ca5"
      },
      "source": [
        "## Cleansing addresses - am I doing it wrong? ##"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "98d2429d-a631-ff13-f720-3692ee751fb5"
      },
      "outputs": [],
      "source": [
        "# This Python 3 environment comes with many helpful analytics libraries installed\n",
        "# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n",
        "# For example, here's several helpful packages to load in \n",
        "import numpy as np # linear algebra\n",
        "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
        "from sklearn.preprocessing import LabelEncoder\n",
        "\n",
        "X = pd.read_json(\"../input/train.json\")\n",
        "X_test = pd.read_json(\"../input/test.json\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "209ef1b2-584b-1ce0-a694-9d3eaf447546"
      },
      "source": [
        "We have 2 features: 'display_address' and 'street_address'. We are using these columns in the following way:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "c5dedb2b-9fe0-dfbd-b5a0-0273e47ebf5e"
      },
      "outputs": [],
      "source": [
        "street_encoder = LabelEncoder()\n",
        "street_encoder.fit(list(X['display_address']) + list(X_test['display_address']))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "2c3b98dc-32ba-a6d5-0683-75654890945b"
      },
      "source": [
        "Let's take a closer look to feature called 'display_address'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a9a57585-2db8-ba6b-00a4-d4feec1cfa24"
      },
      "outputs": [],
      "source": [
        "X['display_address'].head(15)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "170352da-c773-8ba2-1d28-e14ca7761b6d"
      },
      "source": [
        "First let's deal with the **Street** addresses. There are several main representations:\n",
        "\n",
        " 1. It can be **West/East** or simply **W/E**\n",
        " 2. It can be **11th** or **11**\n",
        " 3. It can be **Street**, **Str**, **St**, **St.**\n",
        "\n",
        "So we are going to convert all these addresses to the form: **n 11 st** or **name st**.\n",
        "\n",
        "About **Avenue**:\n",
        "\n",
        " 1. If can be **1st**, **1** or even **First**\n",
        " 2. It can be **Avenue**, **Ave**, **Ave.** or **Av**\n",
        "\n",
        "Convert all these addresses to the form **Name/Number av**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "46919531-1b2d-75d4-22f0-678794c6a5cd"
      },
      "source": [
        "To measure how well we normalize addresses, we will check the unique number of addresses before normalization and after. Also as a last step of cleansing we will remove all characters like **.** and **,** and then strip all space symbols."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "9a70a788-3f1f-68e8-ca58-61574a980056"
      },
      "outputs": [],
      "source": [
        "def normalize_address(X, column):\n",
        "    print(\"Before: {0}\".format(len(X[column].unique())))\n",
        "    substitution = [('west', 'w'), ('east', 'e'), ('south', 's'), ('north', 'n'),\n",
        "                    ('1st', '1'), ('1th', '1'), ('2nd', '2'), ('2th', '2'),\n",
        "                    ('3rd', '3'), ('3th', '3'), ('4th', '4'), ('5th', '5'),\n",
        "                    ('6th', '6'), ('7th', '7'), ('8th', '8'), ('9th', '9'),\n",
        "                    ('0th', '0'),\n",
        "                    ('street', 'st'), ('str', 'st'),\n",
        "                    ('avenue', 'av'), ('ave', 'av'),\n",
        "                    ('place', 'pl'), ('boulevard', 'blvd'), ('road', 'rd'),\n",
        "                    ('first', '1'), ('second', '2'), ('third', '3'),\n",
        "                    ('fourth', '4'), ('fifth', '5'), ('sixth', '6'),\n",
        "                    ('seventh', '7'), ('eighth', '8'), ('nineth', '9'),\n",
        "                    ('tenth', '10'),                    \n",
        "                    (',', ''), ('.', '')]\n",
        "    \n",
        "    def apply_normalization(s):\n",
        "        for subst in substitution:\n",
        "            s = s.lower().replace(subst[0], subst[1])\n",
        "        s = s.strip()\n",
        "        \n",
        "        return s\n",
        "        \n",
        "    X[column] = X[column].apply(apply_normalization)\n",
        "    print(\"After: {0}\".format(len(X[column].unique())))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "fb5b4f77-37ee-b1dd-7b75-44c5851b202b"
      },
      "outputs": [],
      "source": [
        "normalize_address(X, 'display_address')   \n",
        "normalize_address(X_test, 'display_address')\n",
        "normalize_address(X, 'street_address')   \n",
        "normalize_address(X_test, 'street_address')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "a8dda585-ae33-034e-5271-c458ab62970c"
      },
      "source": [
        "As you can see we decreased number of unique addresses by nearly 30%. Let's check top 15 addresses from the training dataframe:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "9827f1ec-f104-aa89-078b-8a79eccf030e"
      },
      "outputs": [],
      "source": [
        "X['display_address'].head(15)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "ce6bb466-6f4a-5fa2-2c57-0d90615b3fcf"
      },
      "source": [
        "Now I can see the following steps for further improvement:\n",
        "\n",
        " 1. Think about house numbers, do we need them or not (e.g. \"521 E 11\" and \"456 E 11\" wil be treated as different addresses)\n",
        " 2. Think about outliers: \"williamsburg - NO FEE\", \"W 10 and Waverly Place \""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "9879ef1d-9e8b-d2f6-909b-1f9fdf2a58c3"
      },
      "source": [
        "And now the really disappointing bit of all these normalizations: **the score has been increased**, both on CV and public! Whit it can be? Any help will be greatly appreciated! The only thought: I have removed some information and now addresses are more correlated with the longitude/latitude thus not having value for algorithms."
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}