{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "e8acc802-80ba-e4b0-403c-df40ce20cf20"
      },
      "source": [
        "# Visualizing Word Vectors with t-SNE\n",
        "\n",
        "TSNE is pretty useful when it comes to visualizing similarity between objects. It works by taking a group of high-dimensional (100 dimensions via Word2Vec) vocabulary word feature vectors, then compresses them down to 2-dimensional x,y coordinate pairs. The idea is to keep similar words close together on the plane, while maximizing the distance between dissimilar words. \n",
        "\n",
        "### Steps\n",
        "\n",
        "1. Clean the data\n",
        "2. Build a corpus\n",
        "3. Train a Word2Vec Model\n",
        "4. Visualize t-SNE representations of the most common words \n",
        "\n",
        "Credit: Some of the code was inspired by this awesome [NLP repo][1]. \n",
        "\n",
        "\n",
        "\n",
        "\n",
        "  [1]: https://github.com/rouseguy/DeepLearningNLP_Py"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "327a2a48-c101-959c-af2d-cabd82276e65"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "pd.options.mode.chained_assignment = None \n",
        "import numpy as np\n",
        "import re\n",
        "import nltk\n",
        "\n",
        "from gensim.models import word2vec\n",
        "\n",
        "from sklearn.manifold import TSNE\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "\n",
        "data = pd.read_csv('../input/train.csv').sample(50000, random_state=23)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "c5d7458b-d380-8af7-13cf-5ed65fb42a83"
      },
      "outputs": [],
      "source": [
        "STOP_WORDS = nltk.corpus.stopwords.words()\n",
        "\n",
        "def clean_sentence(val):\n",
        "    \"remove chars that are not letters or numbers, downcase, then remove stop words\"\n",
        "    regex = re.compile('([^\\s\\w]|_)+')\n",
        "    sentence = regex.sub('', val).lower()\n",
        "    sentence = sentence.split(\" \")\n",
        "    \n",
        "    for word in list(sentence):\n",
        "        if word in STOP_WORDS:\n",
        "            sentence.remove(word)  \n",
        "            \n",
        "    sentence = \" \".join(sentence)\n",
        "    return sentence\n",
        "\n",
        "def clean_dataframe(data):\n",
        "    \"drop nans, then apply 'clean_sentence' function to question1 and 2\"\n",
        "    data = data.dropna(how=\"any\")\n",
        "    \n",
        "    for col in ['question1', 'question2']:\n",
        "        data[col] = data[col].apply(clean_sentence)\n",
        "    \n",
        "    return data\n",
        "\n",
        "data = clean_dataframe(data)\n",
        "data.head(5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e72326d7-e707-d4e9-928a-519a9193bfc5"
      },
      "outputs": [],
      "source": [
        "def build_corpus(data):\n",
        "    \"Creates a list of lists containing words from each sentence\"\n",
        "    corpus = []\n",
        "    for col in ['question1', 'question2']:\n",
        "        for sentence in data[col].iteritems():\n",
        "            word_list = sentence[1].split(\" \")\n",
        "            corpus.append(word_list)\n",
        "            \n",
        "    return corpus\n",
        "\n",
        "corpus = build_corpus(data)        \n",
        "corpus[0:2]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "c652ad03-be65-f4e6-0afd-02c237449b43"
      },
      "source": [
        "# Word 2 Vec\n",
        "\n",
        "The Word to Vec model produces a vocabulary, with each word being represented by an n-dimensional numpy array (100 values in this example)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ee9f9d57-5b3a-16c0-916f-169ef6d7b920"
      },
      "outputs": [],
      "source": [
        "model = word2vec.Word2Vec(corpus, size=100, window=20, min_count=200, workers=4)\n",
        "model.wv['trump']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "4512c76e-f4da-c793-be73-5b18b5bb70e9"
      },
      "outputs": [],
      "source": [
        "def tsne_plot(model):\n",
        "    \"Creates and TSNE model and plots it\"\n",
        "    labels = []\n",
        "    tokens = []\n",
        "\n",
        "    for word in model.wv.vocab:\n",
        "        tokens.append(model[word])\n",
        "        labels.append(word)\n",
        "    \n",
        "    tsne_model = TSNE(perplexity=40, n_components=2, init='pca', n_iter=2500, random_state=23)\n",
        "    new_values = tsne_model.fit_transform(tokens)\n",
        "\n",
        "    x = []\n",
        "    y = []\n",
        "    for value in new_values:\n",
        "        x.append(value[0])\n",
        "        y.append(value[1])\n",
        "        \n",
        "    plt.figure(figsize=(16, 16)) \n",
        "    for i in range(len(x)):\n",
        "        plt.scatter(x[i],y[i])\n",
        "        plt.annotate(labels[i],\n",
        "                     xy=(x[i], y[i]),\n",
        "                     xytext=(5, 2),\n",
        "                     textcoords='offset points',\n",
        "                     ha='right',\n",
        "                     va='bottom')\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "19ec33d2-5160-6556-c8da-a5a53316619a"
      },
      "outputs": [],
      "source": [
        "tsne_plot(model)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b5ffb880-585d-6ea8-51a5-a6351ea2ff20"
      },
      "outputs": [],
      "source": [
        "# A more selective model\n",
        "model = word2vec.Word2Vec(corpus, size=100, window=20, min_count=500, workers=4)\n",
        "tsne_plot(model)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "9f64e341-1967-617f-4004-ef7c6d109277"
      },
      "outputs": [],
      "source": [
        "# A less selective model\n",
        "model = word2vec.Word2Vec(corpus, size=100, window=20, min_count=100, workers=4)\n",
        "tsne_plot(model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "2ba89a55-30b7-15ea-c571-9e402e1c03d2"
      },
      "source": [
        "# It's Becoming Hard to Read\n",
        "\n",
        "With a dataset this large, its difficult to make an easy-to-read TSNE visualization. What you can do is use the model to look up the most similar words from any given point. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "109ae353-5679-6f7a-74f6-ae13d7042639"
      },
      "outputs": [],
      "source": [
        "model.most_similar('trump')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "67a0844e-83d6-22ab-a89b-ae15c19860a8"
      },
      "outputs": [],
      "source": [
        "model.most_similar('universe')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "a88070c7-87cd-0daa-61f7-b2d5ab1ca6ad"
      },
      "source": [
        "# The End\n",
        "\n",
        "Good luck!"
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}