{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "832eebdc-9bd3-7439-eac3-548f4730dc3e"
      },
      "source": [
        "In this simple exploration notebook, let us try and explore the dataset given for this competition.\n",
        "\n",
        "**Objective:**\n",
        "\n",
        "To classify whether question pairs are duplicate or not. \n",
        "\n",
        "Let us start with importing the necessary modules for exploring the data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "7f540a2b-39a5-b72c-7cbe-c24456547a89"
      },
      "outputs": [],
      "source": [
        "import numpy as np # linear algebra\n",
        "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "from nltk.corpus import stopwords\n",
        "from nltk import word_tokenize, ngrams\n",
        "from sklearn import ensemble\n",
        "from sklearn.model_selection import KFold\n",
        "from sklearn.metrics import log_loss\n",
        "import xgboost as xgb\n",
        "\n",
        "eng_stopwords = set(stopwords.words('english'))\n",
        "color = sns.color_palette()\n",
        "\n",
        "%matplotlib inline\n",
        "\n",
        "pd.options.mode.chained_assignment = None  # default='warn'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "8caac5ad-e5ff-0f1b-88e9-4f4b5095210f"
      },
      "source": [
        "Let us read both the train and test dataset and check the number of rows."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "cd8b05b9-db86-1544-5ab9-00ce8b925bb0"
      },
      "outputs": [],
      "source": [
        "train_df = pd.read_csv(\"../input/train.csv\")\n",
        "test_df = pd.read_csv(\"../input/test.csv\")\n",
        "print(train_df.shape)\n",
        "print(test_df.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "3c313511-0a14-0055-0257-e1461cc183df"
      },
      "source": [
        "Okay. So there are about 400K rows in train set and about 2.35M rows in test set.\n",
        "\n",
        "Also there are 6 columns in train set but only 3 of them are in test set. So we shall first look at the top few lines to understand the columns that are missing in the test set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ead95ec8-f4af-4936-c87f-6d764f2fbddf"
      },
      "outputs": [],
      "source": [
        "train_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "dd3b36a1-744d-c1e1-7adf-8d9d72aa23e4"
      },
      "source": [
        "**Data fields**\n",
        "\n",
        "id - the id of a training set question pair\n",
        "\n",
        "qid1, qid2 - unique ids of each question (only available in train.csv)\n",
        "\n",
        "question1, question2 - the full text of each question\n",
        "\n",
        "is_duplicate - the target variable, set to 1 if question1 and question2 have essentially the same meaning, and 0 otherwise."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e6d497c3-8ad5-c2c7-fd36-318335574910"
      },
      "outputs": [],
      "source": [
        "test_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "48238b39-6e93-8419-cd5d-ce3451e9b5cc"
      },
      "source": [
        "So we do not have question ids for the test set. I hope the reason is as follows:\n",
        "\n",
        "*As an anti-cheating measure, Kaggle has supplemented the test set with computer-generated question pairs. Those rows do not come from Quora, and are not counted in the scoring. All of the questions in the training set are genuine examples from Quora.*\n",
        "\n",
        "Since some questions are not from Quora, question ids are not present I think."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "1454ab55-ba46-843c-e3cd-6d838c573521"
      },
      "source": [
        "**Target Variable Exploration:**\n",
        "\n",
        "First let us look at the target variable distribution."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a3206317-0202-af3a-6bd2-589c2341bb3d"
      },
      "outputs": [],
      "source": [
        "is_dup = train_df['is_duplicate'].value_counts()\n",
        "\n",
        "plt.figure(figsize=(8,4))\n",
        "sns.barplot(is_dup.index, is_dup.values, alpha=0.8, color=color[1])\n",
        "plt.ylabel('Number of Occurrences', fontsize=12)\n",
        "plt.xlabel('Is Duplicate', fontsize=12)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f77614e0-9691-d277-3f93-c9845bfaac4b"
      },
      "outputs": [],
      "source": [
        "is_dup / is_dup.sum()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "531edf4a-103f-6e8e-31d3-d86681ade14d"
      },
      "source": [
        "So we have about 63% non-duplicate questions and 37% duplicate questions in the training data set."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "8d7fee43-3474-d5bd-40d5-e22a70915f45"
      },
      "source": [
        "**Questions Exploration:**\n",
        "\n",
        "Now let us explore the question fields present in the train data. First let us check the number of words distribution in the questions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b8f19318-a7f5-70c3-7d5c-909d88c0fe5d"
      },
      "outputs": [],
      "source": [
        "all_ques_df = pd.DataFrame(pd.concat([train_df['question1'], train_df['question2']]))\n",
        "all_ques_df.columns = [\"questions\"]\n",
        "\n",
        "all_ques_df[\"num_of_words\"] = all_ques_df[\"questions\"].apply(lambda x : len(str(x).split()))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ee9ebab8-74ca-eb46-6425-0a8cf2895194"
      },
      "outputs": [],
      "source": [
        "cnt_srs = all_ques_df['num_of_words'].value_counts()\n",
        "\n",
        "plt.figure(figsize=(12,6))\n",
        "sns.barplot(cnt_srs.index, cnt_srs.values, alpha=0.8, color=color[0])\n",
        "plt.ylabel('Number of Occurrences', fontsize=12)\n",
        "plt.xlabel('Number of words in the question', fontsize=12)\n",
        "plt.xticks(rotation='vertical')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "8978d303-eec5-4623-d617-c4f265abd2b0"
      },
      "source": [
        "So the distribution is right skewed with upto 237 words in a question. There are also few questions with 1 or 2 words as well.\n",
        "\n",
        "Now let us explore the number of characters distribution as well."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "7316a117-f34a-913f-6620-4a420649e958"
      },
      "outputs": [],
      "source": [
        "all_ques_df[\"num_of_chars\"] = all_ques_df[\"questions\"].apply(lambda x : len(str(x)))\n",
        "cnt_srs = all_ques_df['num_of_chars'].value_counts()\n",
        "\n",
        "plt.figure(figsize=(50,8))\n",
        "sns.barplot(cnt_srs.index, cnt_srs.values, alpha=0.8, color=color[3])\n",
        "plt.ylabel('Number of Occurrences', fontsize=12)\n",
        "plt.xlabel('Number of characters in the question', fontsize=12)\n",
        "plt.xticks(rotation='vertical')\n",
        "plt.show()      \n",
        "\n",
        "del all_ques_df"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "07ae62e4-f063-a5c7-ccec-973b08c20297"
      },
      "source": [
        "Number of characters distribution as well is right skewed.\n",
        "\n",
        "One interesting point is the sudden dip at the 150 character mark. Not sure why is that so.!\n",
        "\n",
        "Now let us look at the distribution of common unigrams between the given question pairs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "912aedd0-224f-3bf1-9d39-bf9cbfc78d67"
      },
      "outputs": [],
      "source": [
        "def get_unigrams(que):\n",
        "    return [word for word in word_tokenize(que.lower()) if word not in eng_stopwords]\n",
        "\n",
        "def get_common_unigrams(row):\n",
        "    return len( set(row[\"unigrams_ques1\"]).intersection(set(row[\"unigrams_ques2\"])) )\n",
        "\n",
        "def get_common_unigram_ratio(row):\n",
        "    return float(row[\"unigrams_common_count\"]) / max(len( set(row[\"unigrams_ques1\"]).union(set(row[\"unigrams_ques2\"])) ),1)\n",
        "\n",
        "train_df[\"unigrams_ques1\"] = train_df['question1'].apply(lambda x: get_unigrams(str(x)))\n",
        "train_df[\"unigrams_ques2\"] = train_df['question2'].apply(lambda x: get_unigrams(str(x)))\n",
        "train_df[\"unigrams_common_count\"] = train_df.apply(lambda row: get_common_unigrams(row),axis=1)\n",
        "train_df[\"unigrams_common_ratio\"] = train_df.apply(lambda row: get_common_unigram_ratio(row), axis=1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a95823cc-6356-59b2-7dc9-07cd3c653072"
      },
      "outputs": [],
      "source": [
        "cnt_srs = train_df['unigrams_common_count'].value_counts()\n",
        "\n",
        "plt.figure(figsize=(12,6))\n",
        "sns.barplot(cnt_srs.index, cnt_srs.values, alpha=0.8)\n",
        "plt.ylabel('Number of Occurrences', fontsize=12)\n",
        "plt.xlabel('Common unigrams count', fontsize=12)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "82f3c3a9-d3f9-8b1f-cba8-200aeff0ee5f"
      },
      "source": [
        "It is interesting to see that there are very few question pairs with no common words. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "94a5f33f-ca3e-8f82-b404-bf26d43ad587"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(12,6))\n",
        "sns.boxplot(x=\"is_duplicate\", y=\"unigrams_common_count\", data=train_df)\n",
        "plt.xlabel('Is duplicate', fontsize=12)\n",
        "plt.ylabel('Common unigrams count', fontsize=12)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "d4560efc-0de5-e118-d1cd-34f095227abb"
      },
      "source": [
        "There is some good difference between 0 and 1 class using the common unigram count variable. Let us look at the same graph using common unigrams ratio."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "576de972-bf27-0f20-9a5e-ef49210a84c5"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(12,6))\n",
        "sns.boxplot(x=\"is_duplicate\", y=\"unigrams_common_ratio\", data=train_df)\n",
        "plt.xlabel('Is duplicate', fontsize=12)\n",
        "plt.ylabel('Common unigrams ratio', fontsize=12)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "73f9b99b-3221-990a-08c4-2313f1a17180"
      },
      "source": [
        "Now let us do the same analysis using bigrams.\n",
        "\n",
        "**BIgrams:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3e728431-54b0-904a-e7a3-ffadeb168095"
      },
      "outputs": [],
      "source": [
        "def get_bigrams(que):\n",
        "    return [i for i in ngrams(que, 2)]\n",
        "\n",
        "def get_common_bigrams(row):\n",
        "    return len( set(row[\"bigrams_ques1\"]).intersection(set(row[\"bigrams_ques2\"])) )\n",
        "\n",
        "def get_common_bigram_ratio(row):\n",
        "    return float(row[\"bigrams_common_count\"]) / max(len( set(row[\"bigrams_ques1\"]).union(set(row[\"bigrams_ques2\"])) ),1)\n",
        "\n",
        "train_df[\"bigrams_ques1\"] = train_df[\"unigrams_ques1\"].apply(lambda x: get_bigrams(x))\n",
        "train_df[\"bigrams_ques2\"] = train_df[\"unigrams_ques2\"].apply(lambda x: get_bigrams(x)) \n",
        "train_df[\"bigrams_common_count\"] = train_df.apply(lambda row: get_common_bigrams(row),axis=1)\n",
        "train_df[\"bigrams_common_ratio\"] = train_df.apply(lambda row: get_common_bigram_ratio(row), axis=1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f21737f8-cef4-6c80-2ced-454a7ecc6825"
      },
      "outputs": [],
      "source": [
        "cnt_srs = train_df['bigrams_common_count'].value_counts()\n",
        "\n",
        "plt.figure(figsize=(12,6))\n",
        "sns.barplot(cnt_srs.index, cnt_srs.values, alpha=0.8)\n",
        "plt.ylabel('Number of Occurrences', fontsize=12)\n",
        "plt.xlabel('Common bigrams count', fontsize=12)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e8f18a77-10b0-e233-235f-ac60b1ff4d68"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(12,6))\n",
        "sns.boxplot(x=\"is_duplicate\", y=\"bigrams_common_count\", data=train_df)\n",
        "plt.xlabel('Is duplicate', fontsize=12)\n",
        "plt.ylabel('Common bigrams count', fontsize=12)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "185d3f85-8dc0-7736-cabd-7a3cac9924ba"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(12,6))\n",
        "sns.boxplot(x=\"is_duplicate\", y=\"bigrams_common_ratio\", data=train_df)\n",
        "plt.xlabel('Is duplicate', fontsize=12)\n",
        "plt.ylabel('Common bigrams ratio', fontsize=12)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "f97650db-f43d-e7c3-7058-b1630f89b0e4"
      },
      "source": [
        "We could see a good class difference. So this ratio could be a good predictor between both classes.  \n",
        "\n",
        "**Basic Model:**\n",
        "\n",
        "We see that common unigrams and bigrams are good at differentiating the two classes. So we shall also include trigrams and build a XGB model on top of it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "438cf69f-de0c-d289-4945-5ed83fa88761"
      },
      "outputs": [],
      "source": [
        "def feature_extraction(row):\n",
        "    que1 = str(row['question1'])\n",
        "    que2 = str(row['question2'])\n",
        "    out_list = []\n",
        "    # get unigram features #\n",
        "    unigrams_que1 = [word for word in que1.lower().split() if word not in eng_stopwords]\n",
        "    unigrams_que2 = [word for word in que2.lower().split() if word not in eng_stopwords]\n",
        "    common_unigrams_len = len(set(unigrams_que1).intersection(set(unigrams_que2)))\n",
        "    common_unigrams_ratio = float(common_unigrams_len) / max(len(set(unigrams_que1).union(set(unigrams_que2))),1)\n",
        "    out_list.extend([common_unigrams_len, common_unigrams_ratio])\n",
        "\n",
        "    # get bigram features #\n",
        "    bigrams_que1 = [i for i in ngrams(unigrams_que1, 2)]\n",
        "    bigrams_que2 = [i for i in ngrams(unigrams_que2, 2)]\n",
        "    common_bigrams_len = len(set(bigrams_que1).intersection(set(bigrams_que2)))\n",
        "    common_bigrams_ratio = float(common_bigrams_len) / max(len(set(bigrams_que1).union(set(bigrams_que2))),1)\n",
        "    out_list.extend([common_bigrams_len, common_bigrams_ratio])\n",
        "\n",
        "    # get trigram features #\n",
        "    trigrams_que1 = [i for i in ngrams(unigrams_que1, 3)]\n",
        "    trigrams_que2 = [i for i in ngrams(unigrams_que2, 3)]\n",
        "    common_trigrams_len = len(set(trigrams_que1).intersection(set(trigrams_que2)))\n",
        "    common_trigrams_ratio = float(common_trigrams_len) / max(len(set(trigrams_que1).union(set(trigrams_que2))),1)\n",
        "    out_list.extend([common_trigrams_len, common_trigrams_ratio])\n",
        "    return out_list"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a0bc5f3f-7877-7ddc-8341-6686b5b03c5d"
      },
      "outputs": [],
      "source": [
        "def runXGB(train_X, train_y, test_X, test_y=None, feature_names=None, seed_val=0):\n",
        "        params = {}\n",
        "        params[\"objective\"] = \"binary:logistic\"\n",
        "        params['eval_metric'] = 'logloss'\n",
        "        params[\"eta\"] = 0.02\n",
        "        params[\"subsample\"] = 0.7\n",
        "        params[\"min_child_weight\"] = 1\n",
        "        params[\"colsample_bytree\"] = 0.7\n",
        "        params[\"max_depth\"] = 4\n",
        "        params[\"silent\"] = 1\n",
        "        params[\"seed\"] = seed_val\n",
        "        num_rounds = 300 \n",
        "        plst = list(params.items())\n",
        "        xgtrain = xgb.DMatrix(train_X, label=train_y)\n",
        "\n",
        "        if test_y is not None:\n",
        "                xgtest = xgb.DMatrix(test_X, label=test_y)\n",
        "                watchlist = [ (xgtrain,'train'), (xgtest, 'test') ]\n",
        "                model = xgb.train(plst, xgtrain, num_rounds, watchlist, early_stopping_rounds=100, verbose_eval=10)\n",
        "        else:\n",
        "                xgtest = xgb.DMatrix(test_X)\n",
        "                model = xgb.train(plst, xgtrain, num_rounds)\n",
        "                \n",
        "        pred_test_y = model.predict(xgtest)\n",
        "\n",
        "        loss = 1\n",
        "        if test_y is not None:\n",
        "                loss = log_loss(test_y, pred_test_y)\n",
        "                return pred_test_y, loss, model\n",
        "        else:\n",
        "            return pred_test_y, loss, model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3535dd0e-6df3-04b0-3ae7-727826ca879d"
      },
      "outputs": [],
      "source": [
        "train_X = np.vstack( np.array(train_df.apply(lambda row: feature_extraction(row), axis=1)) ) \n",
        "test_X = np.vstack( np.array(test_df.apply(lambda row: feature_extraction(row), axis=1)) )\n",
        "train_y = np.array(train_df[\"is_duplicate\"])\n",
        "test_id = np.array(test_df[\"test_id\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "d37507c7-768f-e141-1200-d660d7b6ae99"
      },
      "source": [
        "From this [excellent notebook from David Thaler][1], we can see that the duplicate question ratio is just 0.165 as opposed to 0.37 in the given training dataset. \n",
        "\n",
        "Since our metric is log loss, resampling the data to represent the same distribution (of 0.165) will give us a much better score in Public LB (Thanks to [anokas for this great script][2] as well)\n",
        "\n",
        "**Disclaimer : Please do this resampling at own risk since there is a potential of overfitting to the public LB.**\n",
        "\n",
        "  [1]: https://www.kaggle.com/davidthaler/quora-question-pairs/how-many-1-s-are-in-the-public-lb\n",
        "  [2]: https://www.kaggle.com/anokas/quora-question-pairs/data-analysis-xgboost-starter-0-35460-lb"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d6ba0556-ec62-9feb-79c7-62f47aad9cb6"
      },
      "outputs": [],
      "source": [
        "train_X_dup = train_X[train_y==1]\n",
        "train_X_non_dup = train_X[train_y==0]\n",
        "\n",
        "train_X = np.vstack([train_X_non_dup, train_X_dup, train_X_non_dup, train_X_non_dup])\n",
        "train_y = np.array([0]*train_X_non_dup.shape[0] + [1]*train_X_dup.shape[0] + [0]*train_X_non_dup.shape[0] + [0]*train_X_non_dup.shape[0])\n",
        "del train_X_dup\n",
        "del train_X_non_dup\n",
        "print(\"Mean target rate : \",train_y.mean())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f3964a44-efd2-3630-4e98-90db2e929a0c"
      },
      "outputs": [],
      "source": [
        "kf = KFold(n_splits=5, shuffle=True, random_state=2016)\n",
        "for dev_index, val_index in kf.split(range(train_X.shape[0])):\n",
        "    dev_X, val_X = train_X[dev_index,:], train_X[val_index,:]\n",
        "    dev_y, val_y = train_y[dev_index], train_y[val_index]\n",
        "    preds, lloss, model = runXGB(dev_X, dev_y, val_X, val_y)\n",
        "    break"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "989caaaf-da40-6830-becc-66d10b711c19"
      },
      "outputs": [],
      "source": [
        "xgtest = xgb.DMatrix(test_X)\n",
        "preds = model.predict(xgtest)\n",
        "\n",
        "out_df = pd.DataFrame({\"test_id\":test_id, \"is_duplicate\":preds})\n",
        "out_df.to_csv(\"xgb_starter.csv\", index=False)"
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}