{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "14baa9d5-565a-0b39-2e8b-135b72845d81"
      },
      "source": [
        "Exploring the data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e5115d83-b4aa-efff-4618-8ea7dc847e52"
      },
      "outputs": [],
      "source": [
        "import numpy as np # linear algebra\n",
        "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
        "import os\n",
        "import gc\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "%matplotlib inline\n",
        "\n",
        "p = sns.color_palette()\n",
        "\n",
        "print('# File sizes')\n",
        "for f in os.listdir('../input'):\n",
        "    if 'zip' not in f:\n",
        "        print(f.ljust(30) + str(round(os.path.getsize('../input/' + f) / 1000000, 2)) + 'MB')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d0a16a30-ebd4-06ac-f116-aec6cf6446d9"
      },
      "outputs": [],
      "source": [
        "df_train = pd.read_csv('../input/train.csv')\n",
        "df_train[1:10]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "0e5880fd-ce7f-9a33-3542-7047c93571e1"
      },
      "source": [
        "We are given a minimal number of data fields here, consisting of:\n",
        "\n",
        "**`id`:** Looks like a simple rowID    \n",
        "**`qid{1, 2}`:** The unique ID of each question in the pair    \n",
        "**`question{1, 2}`:** The actual textual contents of the questions.    \n",
        "**`is_duplicate`:** The **label** that we are trying to predict - whether the two questions are duplicates of each other."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "67c0cca7-2e41-2340-eacc-7e7a40001f08"
      },
      "outputs": [],
      "source": [
        "print('Total number of question pairs for training: {}'.format(len(df_train)))\n",
        "print('Duplicate pairs: {}%'.format(round(df_train['is_duplicate'].mean()*100, 2)))\n",
        "qids = pd.Series(df_train['qid1'].tolist() + df_train['qid2'].tolist())\n",
        "print('Total number of questions in the training data: {}'.format(len(np.unique(qids))))\n",
        "print('Number of questions that appear multiple times: {}'.format(np.sum(qids.value_counts() > 1)))\n",
        "\n",
        "plt.hist(qids.value_counts(), bins=50)\n",
        "plt.yscale('log', nonposy='clip')\n",
        "plt.title('Log-Histogram of question appearance counts')\n",
        "plt.xlabel('Number of occurences of question')\n",
        "plt.ylabel('Number of questions')\n",
        "print()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "2089517f-6757-8727-81a2-17f2329ee38b"
      },
      "outputs": [],
      "source": [
        "df_train[df_train[\"is_duplicate\"] == 1].dtypes"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "f84ed854-c070-d946-c6c0-4a6d39eceb7b"
      },
      "source": [
        "In terms of questions, everything looks as I would expect here. Most questions only appear a few times, with very few questions appearing several times (and a few questions appearing many times). One question appears more than 160 times, but this is an outlier.\n",
        "\n",
        "We can see that we have a 37% positive class in this dataset. Since we are using the [LogLoss](https://www.kaggle.com/wiki/LogarithmicLoss) metric, and LogLoss looks at the actual predicts as opposed to the order of predictions, we should be able to get a decent score by creating a submission predicting the mean value of the label.\n",
        "\n",
        "## Test Submission"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "9743a85f-1e46-9615-32b7-51b254f4567d"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import log_loss\n",
        "\n",
        "p = df_train['is_duplicate'].mean() # Our predicted probability\n",
        "print('Predicted score:', log_loss(df_train['is_duplicate'], np.zeros_like(df_train['is_duplicate']) + p))\n",
        "\n",
        "df_test = pd.read_csv('../input/test.csv')\n",
        "sub = pd.DataFrame({'test_id': df_test['test_id'], 'is_duplicate': p})\n",
        "sub.to_csv('naive_submission.csv', index=False)\n",
        "sub.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "3807679d-4e93-69bf-63e2-761c383f51da"
      },
      "source": [
        "**0.55 on the leaderboard! Score!**\n",
        "\n",
        "However, not all is well. The discrepancy between our local score and the LB one indicates that the distribution of values on the leaderboard is very different to what we have here, which could cause problems with validation later on in the competition"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "507bfa42-9d0c-3f7d-dec0-191ae4280650"
      },
      "source": ""
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}