{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "dc2a1f77-c06f-6a04-ad44-d67555691ef8"
      },
      "source": [
        "# Initial Analysis of Two Sigma Competition"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "5398e512-0f92-23e3-dca0-d3b086450d65"
      },
      "source": [
        "## ML Society\n",
        "### Courtesy of Jason C."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "8de1ae0e-b944-627b-8235-0d93c7819d90"
      },
      "source": [
        "### Import libraries and read data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "0e126bc6-a859-6cd1-1f60-c9eff754011c"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "from sklearn.linear_model import SGDClassifier\n",
        "from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer, TfidfVectorizer\n",
        "from sklearn.pipeline import Pipeline\n",
        "from sklearn.cross_validation import train_test_split\n",
        "from sklearn import metrics"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "9e53a233-811c-fb52-9298-80634ce2275d"
      },
      "outputs": [],
      "source": [
        "train = pd.read_json('../input/train.json')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "a491ceb0-5527-053e-d397-1c4af452a5e5"
      },
      "source": [
        "### What is the majority class?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "43a04603-7a2d-5a4f-94ad-c8ae0c6dbd6a"
      },
      "outputs": [],
      "source": [
        "train.interest_level.value_counts(normalize=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "7672ce6b-48a8-6c4e-163a-1870cc330109"
      },
      "source": [
        "Okay so 0.695 prediction accuracy is the baseline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "57a1f3fc-e7cd-02b6-c3ae-5ff6c270273d"
      },
      "source": [
        "### How predictive is the property description alone?\n",
        "Note: Going forward, it will make sense to first apply some text cleaning."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "abc53026-9f65-1f4b-e85b-114826efb76c"
      },
      "outputs": [],
      "source": [
        "pipe = Pipeline([('tfidf', TfidfVectorizer()), ('clf', SGDClassifier())])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "bcda8935-877c-9c61-3d5b-d89f8ca5de9e"
      },
      "outputs": [],
      "source": [
        "pipe.fit(train.description, train.interest_level)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f3f1edff-2e59-9465-29e7-f388be9f8ccf"
      },
      "outputs": [],
      "source": [
        "pipe.score(train.description, train.interest_level)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "9a6c355a-1e69-823d-c4a5-6f421fdb1d8c"
      },
      "source": [
        "### How predictive are the numeric features alone?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "c9205047-ddaf-092a-b09c-83fb2702b478"
      },
      "outputs": [],
      "source": [
        "train_numer_df = train.select_dtypes(include=['float64', 'int64'])\n",
        "train_target = train.interest_level"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "05ee9f99-c7a2-3f5d-fcea-b0e3b9ac6f30"
      },
      "outputs": [],
      "source": [
        "model = SGDClassifier()\n",
        "model.fit(train_numer_df, train_target)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d5e1a5dc-0dc0-0327-67d6-09c6fffa797d"
      },
      "outputs": [],
      "source": [
        "model.score(train_numer_df, train_target)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "cebc3509-133e-15fd-76f0-b30de11a5a77"
      },
      "source": [
        "### Build list of unique features included in property descriptions\n",
        "Note: Other competition participants have interesting takes on building this feature list, such as deduplicating features or excluding ones that occur less than 5 times in the dataset: https://www.kaggle.com/jxnlco/two-sigma-connect-rental-listing-inquiries/deduplicating-features"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a46a1df3-8ffb-abf9-3147-3afc04b44506"
      },
      "outputs": [],
      "source": [
        "features = []\n",
        "for i in train.features:\n",
        "    for j in i:\n",
        "        if j not in features:\n",
        "            features.append(j)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "3a467182-bfbf-f223-7e7e-f216e0daae30"
      },
      "source": [
        "### Initialize feature ndarray, iterate over properties dataframe, updating feature ndarray appropriately."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "0905ee0f-72bd-ac68-c162-e83f648c9950"
      },
      "outputs": [],
      "source": [
        "feat_array = np.ndarray((len(train),len(features)))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b2b3b040-dd89-f183-b4a5-6c43744c6d8d"
      },
      "outputs": [],
      "source": [
        "for i in range(len(train)):\n",
        "    for word in train.features.iloc[i]:\n",
        "        if word in features:\n",
        "            feat_array[i,features.index(word)] = 1\n",
        "            #print features.index(word)\n",
        "    "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ecd837cb-3b3d-8789-5402-c954db1373f1"
      },
      "outputs": [],
      "source": [
        "target_array = np.array(train_target)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "ac43d7ad-f979-cbc9-cc88-618d9ed39433"
      },
      "source": [
        "### Train linear regression model on feature array and test performance."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f00aa4b5-ab6f-65a0-d274-71058a4d6d0e"
      },
      "outputs": [],
      "source": [
        "svm_model = SGDClassifier()\n",
        "svm_model.fit(feat_array, train_target)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "da57e6ec-00cd-f9b9-86a0-bc89c26a8dee"
      },
      "outputs": [],
      "source": [
        "svm_model.score(feat_array, train_target)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "8a1e43d0-b981-d114-37e6-a5687d283d54"
      },
      "outputs": [],
      "source": [
        "predictions = svm_model.predict(feat_array)\n",
        "predictions = pd.Series(predictions)\n",
        "predictions.value_counts()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "2eb70810-9d85-314c-1f32-8350d076197a"
      },
      "outputs": [],
      "source": [
        "train_target.value_counts()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "c5c8363d-921a-d8c4-1cdf-d8d75aa8315f"
      },
      "source": [
        "### Thoughts going forward:\n",
        "The class imbalance of the training data is obvious. It may be worthwhile to undersample the 'low' and 'medium' interest groups and see if better performance is achieved. In the end, however, I suspect training a deep learning model on the provided image data will be necessary to be competitive."
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}