{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "be1fb74a-ea76-5d12-af9c-21651032ac37"
      },
      "source": [
        "**It was modified from *\"Titanic: Machine Learning from Disaster\"* for learning.**\n",
        "\n",
        "# Titanic: Machine Learning from Disaster"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "f35f9107-7b1a-184a-9dbf-51a5b2b3ee67"
      },
      "source": [
        "* Description:\n",
        "    The sinking of the RMS Titanic is one of the most infamous shipwrecks in history.  On April 15, 1912, during her maiden voyage, the Titanic sank after colliding with an iceberg, killing 1502 out of 2224 passengers and crew. This sensational tragedy shocked the international community and led to better safety regulations for ships.\n",
        "* Problem definition: In this challenge, we ask you to complete the analysis of what sorts of people were likely to survive. In particular, we ask you to apply the tools of machine learning to predict which passengers survived the tragedy."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "73c6ae96-3a21-d2ea-3728-9d33adee18ba"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import sklearn.linear_model as lm\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "907c91be-7361-fe03-6cea-5a9ee12c4ea9"
      },
      "source": [
        "## Data Exploration"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5bb5d0e3-39a8-edfd-3776-8b639e2d784d"
      },
      "outputs": [],
      "source": [
        "train = pd.read_csv('../input/train.csv')\n",
        "test = pd.read_csv('../input/test.csv')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "889004f2-e10d-0687-4ccc-3e4b5a825d57"
      },
      "outputs": [],
      "source": [
        "print (\"Dimension of train data {}\".format(train.shape))\n",
        "print (\"Dimension of test data {}\".format(test.shape))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ed8dd3e6-915a-41fd-7308-ed7e29b7287c"
      },
      "outputs": [],
      "source": [
        "print (\"Basic statistical description:\")\n",
        "train.describe()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "838ff612-6fc8-1baf-d86e-e6fb94a5c8d2"
      },
      "source": [
        "### Comparison between test and train data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "a13737ae-a781-1e23-0f3d-ecfe31b87239"
      },
      "source": [
        "#### From following cells, we could know that train and test data are split by PassengerId."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "84b58fb2-c8c6-cfe0-6bcf-a867ec2d57dd"
      },
      "outputs": [],
      "source": [
        "train.tail()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "01d0bd1a-f464-fa3e-0551-314b5260745e"
      },
      "outputs": [],
      "source": [
        "test.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "3fbfd867-85bc-db51-bbfd-38b0aeb8c8c5"
      },
      "source": [
        "#### Let's look at data graphically. We could see that all the distribution of features are similar."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "99fd046f-3756-b85b-2c11-d4505d503c35"
      },
      "outputs": [],
      "source": [
        "plt.rc('font', size=13)\n",
        "fig = plt.figure(figsize=(18, 8))\n",
        "alpha = 0.6\n",
        "\n",
        "ax1 = plt.subplot2grid((2,3), (0,0))\n",
        "train.Age.value_counts().plot(kind='kde', color='#FA2379', label='train', alpha=alpha)\n",
        "test.Age.value_counts().plot(kind='kde', label='test', alpha=alpha)\n",
        "ax1.set_xlabel('Age')\n",
        "ax1.set_title(\"What's the distribution of age?\" )\n",
        "plt.legend(loc='best')\n",
        "\n",
        "ax2 = plt.subplot2grid((2,3), (0,1))\n",
        "train.Pclass.value_counts().plot(kind='barh', color='#FA2379', label='train', alpha=alpha)\n",
        "test.Pclass.value_counts().plot(kind='barh', label='test', alpha=alpha)\n",
        "ax2.set_ylabel('Pclass')\n",
        "ax2.set_xlabel('Frequency')\n",
        "ax2.set_title(\"What's the distribution of Pclass?\" )\n",
        "plt.legend(loc='best')\n",
        "\n",
        "ax3 = plt.subplot2grid((2,3), (0,2))\n",
        "train.Sex.value_counts().plot(kind='barh', color='#FA2379', label='train', alpha=alpha)\n",
        "test.Sex.value_counts().plot(kind='barh', label='test', alpha=alpha)\n",
        "ax3.set_ylabel('Sex')\n",
        "ax3.set_xlabel('Frequency')\n",
        "ax3.set_title(\"What's the distribution of Sex?\" )\n",
        "plt.legend(loc='best')\n",
        "\n",
        "ax4 = plt.subplot2grid((2,3), (1,0), colspan=2)\n",
        "train.Fare.value_counts().plot(kind='kde', color='#FA2379', label='train', alpha=alpha)\n",
        "test.Fare.value_counts().plot(kind='kde', label='test', alpha=alpha)\n",
        "ax4.set_xlabel('Fare')\n",
        "ax4.set_title(\"What's the distribution of Fare?\" )\n",
        "plt.legend(loc='best')\n",
        "\n",
        "ax5 = plt.subplot2grid((2,3), (1,2))\n",
        "train.Embarked.value_counts().plot(kind='barh', color='#FA2379', label='train', alpha=alpha)\n",
        "test.Embarked.value_counts().plot(kind='barh', label='test', alpha=alpha)\n",
        "ax5.set_ylabel('Embarked')\n",
        "ax5.set_xlabel('Frequency')\n",
        "ax5.set_title(\"What's the distribution of Embarked?\" )\n",
        "plt.legend(loc='best')\n",
        "plt.tight_layout()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "e4e31ee2-b800-cd4a-3c29-5a064b252298"
      },
      "source": [
        "#### We could know that the numbers of survived and died people are close to balanced."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f6027b27-d006-095f-c43d-ae11691ba2bc"
      },
      "outputs": [],
      "source": [
        "print (train.Survived.value_counts())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "ca7cfd29-5fa2-8c78-e1dc-ca1d4a64cf44"
      },
      "source": [
        "### Look closely to the train data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5c25cfea-9ffa-7a84-6d20-01364fb478f5"
      },
      "outputs": [],
      "source": [
        "\n",
        "d=len(train[train.Survived==0][train.Sex=='female'])\n",
        "s=len(train[train.Survived==1][train.Sex=='female'])\n",
        "print (\"%d %d %f\" % (s, d, s/(s+d)))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "c5aabd1e-0038-c8cd-b40e-bf1bafa670aa"
      },
      "source": [
        "#### We could see that the density of survived is higher than not survived under age 10."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "4e814ab0-feb0-b8e3-73c1-cdfe9324bc0e"
      },
      "outputs": [],
      "source": [
        "fig = plt.figure(figsize=(15, 6))\n",
        "\n",
        "train[train.Survived==0].Age.value_counts().plot(kind='density', color='#FA2379', label='Not Survived', alpha=alpha)\n",
        "train[train.Survived==1].Age.value_counts().plot(kind='density', label='Survived', alpha=alpha)\n",
        "plt.xlabel('Age')\n",
        "plt.title(\"What's the distribution of Age?\" )\n",
        "plt.legend(loc='best')\n",
        "plt.grid()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "d7b4901d-ee4a-796f-6e93-952aa7ccc637"
      },
      "source": [
        "#### We could observe that the survived rate of female is higher than male about 50%"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "54aefba0-a90a-981e-227e-24e7f19683ff"
      },
      "outputs": [],
      "source": [
        "df_male = train[train.Sex=='male'].Survived.value_counts().sort_index()\n",
        "df_female = train[train.Sex=='female'].Survived.value_counts().sort_index()\n",
        "fig = plt.figure(figsize=(18, 6))\n",
        "\n",
        "ax1 = plt.subplot2grid((1,2), (0,0))\n",
        "df_female.plot(kind='barh', color='#FA2379', label='Female', alpha=alpha)\n",
        "df_male.plot(kind='barh', label='Male', alpha=alpha-0.1)\n",
        "ax1.set_xlabel('Frequrncy')\n",
        "ax1.set_yticklabels(['Died', 'Survived'])\n",
        "ax1.set_title(\"Who will survived with respect to sex?\" )\n",
        "plt.legend(loc='best')\n",
        "plt.grid()\n",
        "\n",
        "ax2 = plt.subplot2grid((1,2), (0,1))\n",
        "(df_female/train[train.Sex=='female'].shape[0]).plot(kind='barh', color='#FA2379', label='Female', alpha=alpha)\n",
        "(df_male/train[train.Sex=='male'].shape[0]).plot(kind='barh', label='Male', alpha=alpha-0.1)\n",
        "ax2.set_xlabel('Rate')\n",
        "ax2.set_yticklabels(['Died', 'Survived'])\n",
        "ax2.set_title(\"What's the survived rate with respect to sex?\" )\n",
        "plt.legend(loc='best')\n",
        "plt.grid()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "9febe872-5fd7-1f7c-80bb-e3e54194a266"
      },
      "source": [
        "#### We could observed that the class of people affected the posibility of being survived."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "c3944cf0-f252-fea3-bd6e-c6b30ab83d0a"
      },
      "outputs": [],
      "source": [
        "df_male = train[train.Sex=='male']\n",
        "df_female = train[train.Sex=='female']\n",
        "fig = plt.figure(figsize=(18, 6))\n",
        "\n",
        "ax1 = plt.subplot2grid((1,4), (0,0))\n",
        "df_female[df_female.Pclass<3].Survived.value_counts().sort_index().plot(kind='bar', color='#FA2379', alpha=alpha)\n",
        "ax1.set_ylabel('Frequrncy')\n",
        "ax1.set_ylim((0,350))\n",
        "ax1.set_xticklabels(['Died', 'Survived'])\n",
        "ax1.set_title(\"How will high-class female survived?\", y=1.05)\n",
        "plt.grid()\n",
        "\n",
        "ax2 = plt.subplot2grid((1,4), (0,1))\n",
        "df_female[df_female.Pclass==3].Survived.value_counts().sort_index().plot(kind='bar', color='#23FA79', alpha=alpha)\n",
        "ax2.set_ylabel('Frequrncy')\n",
        "ax2.set_ylim((0,350))\n",
        "ax2.set_xticklabels(['Died', 'Survived'])\n",
        "ax2.set_title(\"How will low-class female survived?\", y=1.05)\n",
        "plt.grid()\n",
        "\n",
        "ax3 = plt.subplot2grid((1,4), (0,2))\n",
        "df_male[df_male.Pclass<3].Survived.value_counts().sort_index().plot(kind='bar', color='#00FA23', alpha=alpha)\n",
        "ax3.set_ylabel('Frequrncy')\n",
        "ax3.set_ylim((0,350))\n",
        "ax3.set_xticklabels(['Died', 'Survived'])\n",
        "ax3.set_title(\"How will high-class male survived?\", y=1.05)\n",
        "plt.grid()\n",
        "\n",
        "ax4 = plt.subplot2grid((1,4), (0,3))\n",
        "df_male[df_male.Pclass==3].Survived.value_counts().sort_index().plot(kind='bar', color='#2379FA', alpha=alpha)\n",
        "ax4.set_ylabel('Frequrncy')\n",
        "ax4.set_ylim((0,350))\n",
        "ax4.set_xticklabels(['Died', 'Survived'])\n",
        "ax4.set_title(\"How will low-class male survived?\", y=1.05)\n",
        "plt.grid()\n",
        "plt.tight_layout()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "226c37a6-7857-9427-5506-ef6da0e1e778"
      },
      "source": [
        "#### We could see that if the value of Ticket is the same, passenger would be close, like friends or familes. But sometimes it will not be the case, so we need to carefully handle it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "4a0675a2-82a1-d765-0471-0e2209830b77"
      },
      "outputs": [],
      "source": [
        "train[train.Ticket=='1601']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a09181a1-e6d0-bc56-2eea-337b6062af79"
      },
      "outputs": [],
      "source": [
        "train[train.Ticket=='CA 2144']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "b47e7d97-26c4-9a89-66ac-3811055b21ed"
      },
      "source": [
        "## Data Cleaning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "00b4c390-b98e-9197-d464-475927359f72"
      },
      "source": [
        "#### Let's see how many missing values we have on each column. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "4cc0fe90-15a1-8b98-3e3b-4cabce0742b2"
      },
      "outputs": [],
      "source": [
        "train.isnull().sum()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "c214124b-48d7-2d8b-2dff-886e2eb18a8d"
      },
      "outputs": [],
      "source": [
        "test.isnull().sum()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "89a371d1-aa19-8bb8-de5b-f219a36f3b46"
      },
      "source": [
        "#### Thanks for [this amazing sharing](https://www.kaggle.com/mrisdal/titanic/exploring-survival-on-the-titanic/notebook), now we have a good way to replace missing values with sensible values. Let's assume the embarked is related with fare and pclass."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "d5f7cdd2-6198-933b-19ad-b8e9d9af3ff7"
      },
      "source": [
        "### Missing values on Embarked"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b8285d1b-025c-f8b2-c89f-31b5b44d7287"
      },
      "outputs": [],
      "source": [
        "fig = plt.figure(figsize=(8, 5))\n",
        "ax = fig.add_subplot(111)\n",
        "ax = train.boxplot(column='Fare', by=['Embarked','Pclass'], ax=ax)\n",
        "plt.axhline(y=80, color='green')\n",
        "ax.set_title('', y=1.1)\n",
        "\n",
        "train[train.Embarked.isnull()][['Fare', 'Pclass', 'Embarked']]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "58318d75-03aa-426d-2be9-ba01fbe7bf2f"
      },
      "source": [
        "#### From the above boxplot, we should replace NA with C because most people who had Pclass 1 and Fare 80 would be Embarked C"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5cc45aba-b1ec-0501-32cb-5667b2915524"
      },
      "outputs": [],
      "source": [
        "_ = train.set_value(train.Embarked.isnull(), 'Embarked', 'C')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "253de3db-c58a-929f-f53f-ab9a42d7ac4c"
      },
      "source": [
        "### Missing values on Fare"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "ea434ea8-acb8-2b72-9213-443486a6ca21"
      },
      "source": [
        "#### By fixing the values of Embarked and Pclass, we could plot histogram of Fare. And we should use the most common value to replace the NA value of Fare."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1ccf3356-42e5-6d93-f143-837851b85fa7"
      },
      "outputs": [],
      "source": [
        "fig = plt.figure(figsize=(8, 5))\n",
        "ax = fig.add_subplot(111)\n",
        "test[(test.Pclass==3)&(test.Embarked=='S')].Fare.hist(bins=100, ax=ax)\n",
        "test[test.Fare.isnull()][['Pclass', 'Fare', 'Embarked']]\n",
        "plt.xlabel('Fare')\n",
        "plt.ylabel('Frequency')\n",
        "plt.title('Histogram of Fare, Plcass 3 and Embarked S')\n",
        "\n",
        "test[test.Fare.isnull()][['Pclass', 'Fare', 'Embarked']]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "befe5367-7929-02e7-1e06-25e8eed92ae3"
      },
      "outputs": [],
      "source": [
        "print (\"The top 5 most common value of Fare\")\n",
        "test[(test.Pclass==3)&(test.Embarked=='S')].Fare.value_counts().head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "db2e070f-5456-437a-867d-a69d14214482"
      },
      "outputs": [],
      "source": [
        "_ = test.set_value(test.Fare.isnull(), 'Fare', 8.05)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "46fd84db-79de-f5a4-b181-3e35c8c83a79"
      },
      "source": [
        "#### Replace the missing value of Cabin with U0"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1963b770-8e48-3eac-f5db-704e553ca968"
      },
      "outputs": [],
      "source": [
        "full = pd.concat([train, test], ignore_index=True)\n",
        "_ = full.set_value(full.Cabin.isnull(), 'Cabin', 'U0')\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "fc2ba8d9-5d13-9285-8ca7-6409b6466416"
      },
      "source": [
        "## Feature Engineering"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "c522ba7d-aa15-a05a-dbc7-33e9e622be26"
      },
      "source": [
        "#### Create a feature, Names, to store the length of words in name."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "dae555ca-cb1e-944e-9bed-bf2f81445024"
      },
      "outputs": [],
      "source": [
        "import re\n",
        "names = full.Name.map(lambda x: len(re.split(' ', x)))\n",
        "_ = full.set_value(full.index, 'Names', names)\n",
        "del names"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "d12425f3-4791-c906-59c5-7af97236b587"
      },
      "source": [
        "#### Create a feature, Title."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5aa8b416-de06-b017-bb46-3b27c9de9b1c"
      },
      "outputs": [],
      "source": [
        "title = full.Name.map(lambda x: re.compile(', (.*?)\\.').findall(x)[0])\n",
        "title[title=='Mme'] = 'Mrs'\n",
        "title[title.isin(['Ms','Mlle'])] = 'Miss'\n",
        "title[title.isin(['Don', 'Jonkheer'])] = 'Sir'\n",
        "title[title.isin(['Dona', 'Lady', 'the Countess'])] = 'Lady'\n",
        "title[title.isin(['Capt', 'Col', 'Major', 'Dr', 'Officer', 'Rev'])] = 'Officer'\n",
        "_ = full.set_value(full.index, 'Title', title)\n",
        "del title"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "8f445a7f-c5fd-3284-3b47-4bdbf6efd2c7"
      },
      "source": [
        "#### Create a feature, Deck. It may represents the socioeconomic status."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "60ae10cd-f38b-14e7-e4db-6efb3f75a6e4"
      },
      "outputs": [],
      "source": [
        "deck = full[~full.Cabin.isnull()].Cabin.map( lambda x : re.compile(\"([a-zA-Z]+)\").search(x).group())\n",
        "deck = pd.factorize(deck)[0]\n",
        "_ = full.set_value(full.index, 'Deck', deck)\n",
        "del deck"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "56ace2c8-aa46-3e32-963d-c6763947d8bf"
      },
      "source": [
        "#### Create a feature, Room. It may represents the geo lacation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "7638a500-b3b3-44f5-48cf-66fcf79d7853"
      },
      "outputs": [],
      "source": [
        "checker = re.compile(\"([0-9]+)\")\n",
        "def roomNum(x):\n",
        "    nums = checker.search(x)\n",
        "    if nums:\n",
        "        return int(nums.group())+1\n",
        "    else:\n",
        "        return 1\n",
        "rooms = full.Cabin.map(roomNum)\n",
        "_ = full.set_value(full.index, 'Room', rooms)\n",
        "del checker, roomNum\n",
        "full['Room'] = full.Room/full.Room.sum()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "88d889ef-7a1a-165d-10de-5e04251caf07"
      },
      "source": [
        "#### Create a feature, Group_num. It may represents the size of family."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ef041721-f437-87d6-46ec-25310e7974a5"
      },
      "outputs": [],
      "source": [
        "full['Group_num'] = full.Parch + full.SibSp + 1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "70ffc73b-c1a9-0a9a-90cf-0a732016a955"
      },
      "source": [
        "#### Create a feature, Group_size. When the size is between 2 and 4, more people are survived."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "21798f34-9c01-bde7-e2de-b788d1573e3c"
      },
      "outputs": [],
      "source": [
        "full['Group_size'] = pd.Series('M', index=full.index)\n",
        "_ = full.set_value(full.Group_num>4, 'Group_size', 'L')\n",
        "_ = full.set_value(full.Group_num==1, 'Group_size', 'S')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "d6ad6202-ef7f-650c-32d0-7fb5ae1963d5"
      },
      "source": [
        "#### Normalized the fare."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "59e3ba95-f15b-7651-8d83-ef8608fc3720"
      },
      "outputs": [],
      "source": [
        "from sklearn.preprocessing import StandardScaler\n",
        "scaler = StandardScaler()\n",
        "full['NorFare'] = pd.Series(scaler.fit_transform(full.Fare.values.reshape(-1,1)).reshape(-1), index=full.index)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "02efb2bd-54e8-c785-5edf-40768abfb36f"
      },
      "outputs": [],
      "source": [
        "def setValue(col):\n",
        "    _ = train.set_value(train.index, col, full[:891][col].values)\n",
        "    _ = test.set_value(test.index, col, full[891:][col].values)\n",
        "\n",
        "for col in ['Deck', 'Room', 'Group_size', 'Group_num', 'Names', 'Title']:\n",
        "    setValue(col)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "69b8bf53-adbb-b467-e3c4-8e171c01552c"
      },
      "source": [
        "#### Predict Age"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "baaa46a5-370b-8536-73bd-74331862dabe"
      },
      "outputs": [],
      "source": [
        "full.drop(labels=['PassengerId', 'Name', 'Cabin', 'Survived', 'Ticket', 'Fare'], axis=1, inplace=True)\n",
        "full = pd.get_dummies(full, columns=['Embarked', 'Sex', 'Title', 'Group_size'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "c8a0dc2b-1059-a811-0ff0-61227d1476fa"
      },
      "outputs": [],
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "X = full[~full.Age.isnull()].drop('Age', axis=1)\n",
        "y = full[~full.Age.isnull()].Age\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "205a752f-1a55-1f98-07a0-9ebb6b3bbb9f"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import mean_absolute_error\n",
        "from sklearn.metrics import make_scorer\n",
        "\n",
        "def get_model(estimator, parameters, X_train, y_train, scoring):  \n",
        "    model = GridSearchCV(estimator, param_grid=parameters, scoring=scoring)\n",
        "    model.fit(X_train, y_train)\n",
        "    return model.best_estimator_"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "40dc56c0-cfde-2cc2-7e22-2ecd3029a3e4"
      },
      "outputs": [],
      "source": [
        "'''\n",
        "import xgboost as xgb\n",
        "\n",
        "XGB = xgb.XGBRegressor(max_depth=4, seed= 42)\n",
        "scoring = make_scorer(mean_absolute_error, greater_is_better=False)\n",
        "parameters = {'reg_alpha':np.linspace(0.1,1.0,5), 'reg_lambda': np.linspace(1.0,3.0,5)}\n",
        "reg_xgb = get_model(XGB, parameters, X_train, y_train, scoring)\n",
        "print (reg_xgb)\n",
        "'''"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e890a1bc-aa25-b2f0-e54f-dbb296b79bc3"
      },
      "outputs": [],
      "source": [
        "#print (\"Mean absolute error of test data: {}\".format(mean_absolute_error(y_test, reg_xgb.predict(X_test))))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "8866a054-8210-da69-b424-3ef5bcb67335"
      },
      "outputs": [],
      "source": [
        "'''\n",
        "fig = plt.figure(figsize=(15, 6))\n",
        "alpha = 0.5\n",
        "full.Age.value_counts().plot(kind='density', color='#FA2379', label='Before', alpha=alpha)\n",
        "\n",
        "pred = reg_xgb.predict(full[full.Age.isnull()].drop('Age', axis=1))\n",
        "full.set_value(full.Age.isnull(), 'Age', pred)\n",
        "\n",
        "full.Age.value_counts().plot(kind='density', label='After', alpha=alpha)\n",
        "plt.xlabel('Age')\n",
        "plt.title(\"What's the distribution of Age after predicting?\" )\n",
        "plt.legend(loc='best')\n",
        "plt.grid()\n",
        "'''"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "792625f3-3a39-36d6-4083-39f3cb5027bf"
      },
      "source": [
        "**Learn by keras**\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "8b0f5f5f-e308-18b9-57f2-2236a3e720f0"
      },
      "outputs": [],
      "source": [
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "\n",
        "# build a neural network from the 1st layer to the last layer\n",
        "model = Sequential()\n",
        "model.add(Dense(output_dim=1, input_dim=X_train.shape[1]))\n",
        "\n",
        "# choose loss function and optimizing method\n",
        "model.compile(loss='mae', optimizer='sgd')\n",
        "\n",
        "# training\n",
        "print('Training -----------')\n",
        "for step in range(1001):\n",
        "    cost = model.train_on_batch(X_train.as_matrix(), y_train.as_matrix())\n",
        "    if step % 100 == 0:\n",
        "        print('train cost: ', cost)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "923d8863-a45e-3bad-96d7-9422584e0403"
      },
      "outputs": [],
      "source": [
        "# test\n",
        "print('\\nTesting ------------')\n",
        "cost = model.evaluate(X_test.as_matrix(), y_test.as_matrix())\n",
        "print('test cost:', cost)\n",
        "W, b = model.layers[0].get_weights()\n",
        "print('Weights=', W, '\\nbiases=', b)\n",
        "\n",
        "y_pred = model.predict(X_test.as_matrix())\n",
        "full.set_value(full.Age.isnull(), 'Age', y_pred)\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3a80c205-ed3b-ba19-20c2-0c661f38ea0d"
      },
      "outputs": [],
      "source": [
        "full['NorAge'] = pd.Series(scaler.fit_transform(full.Age.values.reshape(-1,1)).reshape(-1), index=full.index)\n",
        "full['NorNames'] = pd.Series(scaler.fit_transform(full.Names.values.reshape(-1,1)).reshape(-1), index=full.index)\n",
        "full['Group_num'] = pd.Series(scaler.fit_transform(full.Group_num.values.reshape(-1,1)).reshape(-1), index=full.index)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "639acba3-012c-6d42-4ce4-cbd87d056824"
      },
      "outputs": [],
      "source": [
        "for col in ['NorAge', 'NorFare', 'NorNames', 'Group_num']:\n",
        "    setValue(col)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "9e0174fa-466d-5fee-e2ec-96b828d050cd"
      },
      "source": [
        "#### Encoding sex, feamle: 0 and male: 1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5d63fafb-951f-8fa8-79f5-f34aefff5c32"
      },
      "outputs": [],
      "source": [
        "train.Sex = np.where(train.Sex=='female', 0, 1)\n",
        "test.Sex = np.where(test.Sex=='female', 0, 1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "def377d2-3a58-69cd-3e9f-d4b7971c89d4"
      },
      "source": [
        "#### Convert values of Embarked and Ticket into dummy variables"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "066399f8-cfe0-cb46-dc08-496951eacd10"
      },
      "outputs": [],
      "source": [
        "train.drop(labels=['PassengerId', 'Name', 'Names', 'Cabin', 'Ticket', 'Age', 'Fare'], axis=1, inplace=True)\n",
        "test.drop(labels=['Name', 'Names', 'Cabin', 'Ticket', 'Age', 'Fare'], axis=1, inplace=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "44a6703c-1bd2-4657-09a2-e3f922a21b21"
      },
      "outputs": [],
      "source": [
        "train = pd.get_dummies(train, columns=['Embarked', 'Pclass', 'Title', 'Group_size'])\n",
        "test = pd.get_dummies(test, columns=['Embarked', 'Pclass', 'Title', 'Group_size'])\n",
        "test['Title_Sir'] = pd.Series(0, index=test.index)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "da4a819f-0cbc-77f7-58f3-73c1286636e6"
      },
      "source": [
        "## Build Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "0bf4caea-a8b1-6e32-84d7-725b4919cbb0"
      },
      "outputs": [],
      "source": [
        "from sklearn.model_selection import learning_curve\n",
        "def plot_learning_curve(estimator, title, X, y, ylim=None, cv=None,\n",
        "                        n_jobs=1, train_sizes=np.linspace(.1, 1.0, 5), scoring='accuracy'):\n",
        "    plt.figure(figsize=(10,6))\n",
        "    plt.title(title)\n",
        "    if ylim is not None:\n",
        "        plt.ylim(*ylim)\n",
        "    plt.xlabel(\"Training examples\")\n",
        "    plt.ylabel(scoring)\n",
        "    train_sizes, train_scores, test_scores = learning_curve(estimator, X, y, cv=cv, scoring=scoring,\n",
        "                                                            n_jobs=n_jobs, train_sizes=train_sizes)\n",
        "    train_scores_mean = np.mean(train_scores, axis=1)\n",
        "    train_scores_std = np.std(train_scores, axis=1)\n",
        "    test_scores_mean = np.mean(test_scores, axis=1)\n",
        "    test_scores_std = np.std(test_scores, axis=1)\n",
        "    plt.grid()\n",
        "\n",
        "    plt.fill_between(train_sizes, train_scores_mean - train_scores_std,\n",
        "                     train_scores_mean + train_scores_std, alpha=0.1,\n",
        "                     color=\"r\")\n",
        "    plt.fill_between(train_sizes, test_scores_mean - test_scores_std,\n",
        "                     test_scores_mean + test_scores_std, alpha=0.1, color=\"g\")\n",
        "    plt.plot(train_sizes, train_scores_mean, 'o-', color=\"r\",\n",
        "             label=\"Training score\")\n",
        "    plt.plot(train_sizes, test_scores_mean, 'o-', color=\"g\",\n",
        "             label=\"Cross-validation score\")\n",
        "\n",
        "    plt.legend(loc=\"best\")\n",
        "    return plt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1eb8b7de-cc0e-e47c-76c3-ff408bad439e"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import roc_curve, auc\n",
        "def plot_roc_curve(estimator, X, y, title):\n",
        "    # Determine the false positive and true positive rates\n",
        "    fpr, tpr, _ = roc_curve(y, estimator.predict_proba(X)[:,1])\n",
        "\n",
        "    # Calculate the AUC\n",
        "    roc_auc = auc(fpr, tpr)\n",
        "    print ('ROC AUC: %0.2f' % roc_auc)\n",
        "\n",
        "    # Plot of a ROC curve for a specific class\n",
        "    plt.figure(figsize=(10,6))\n",
        "    plt.plot(fpr, tpr, label='ROC curve (area = %0.2f)' % roc_auc)\n",
        "    plt.plot([0, 1], [0, 1], 'k--')\n",
        "    plt.xlim([0.0, 1.0])\n",
        "    plt.ylim([0.0, 1.05])\n",
        "    plt.xlabel('False Positive Rate')\n",
        "    plt.ylabel('True Positive Rate')\n",
        "    plt.title('ROC Curve - {}'.format(title))\n",
        "    plt.legend(loc=\"lower right\")\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d3899afc-aebf-01e4-566f-321a876da31b"
      },
      "outputs": [],
      "source": [
        "X = train.drop(['Survived'], axis=1)\n",
        "y = train.Survived\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "93a21cdf-9ea6-b277-200a-39acc42d1fa2"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import accuracy_score\n",
        "scoring = make_scorer(accuracy_score, greater_is_better=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "4e74ca8a-658c-97f2-34a2-a4b8ff21d6b2"
      },
      "source": [
        "### KNN"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "951643dd-6806-6d3d-7449-d11d7ee67e93"
      },
      "outputs": [],
      "source": [
        "'''\n",
        "from sklearn.neighbors import KNeighborsClassifier\n",
        "KNN = KNeighborsClassifier(weights='uniform')\n",
        "parameters = {'n_neighbors':[3,4,5], 'p':[1,2]}\n",
        "clf_knn = get_model(KNN, parameters, X_train, y_train, scoring)\n",
        "'''"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "87debc07-5d61-dd22-86d3-7dab7e26a57b"
      },
      "outputs": [],
      "source": [
        "'''\n",
        "print (accuracy_score(y_test, clf_knn.predict(X_test)))\n",
        "print (clf_knn)\n",
        "plot_learning_curve(clf_knn, 'KNN', X, y, cv=4);\n",
        "'''"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "d3988022-15a6-4c66-6de8-8f716c732091"
      },
      "source": [
        "### Random Forest"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "fbc5c39a-9002-9173-bbde-0a0a76d59c52"
      },
      "outputs": [],
      "source": [
        "'''\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "rfc = RandomForestClassifier(random_state=42, criterion='entropy', min_samples_split=5, oob_score=True)\n",
        "parameters = {'n_estimators':[500], 'min_samples_leaf':[12]}\n",
        "clf_rfc1 = get_model(rfc, parameters, X_train, y_train, scoring)\n",
        "'''"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "caf76752-35b8-cc36-2b33-257d1aece91e"
      },
      "outputs": [],
      "source": [
        "'''\n",
        "print (accuracy_score(y_test, clf_rfc1.predict(X_test)))\n",
        "print (clf_rfc1)\n",
        "plot_learning_curve(clf_rfc1, 'Random Forest', X, y, cv=4);\n",
        "'''"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "029c300d-b703-07fe-a91d-252861c7dcb1"
      },
      "outputs": [],
      "source": [
        "'''\n",
        "plt.figure(figsize=(10,6))\n",
        "plt.barh(np.arange(X_train.columns.shape[0]), clf_rfc1.feature_importances_, 0.5)\n",
        "plt.yticks(np.arange(X_train.columns.shape[0]), X_train.columns)\n",
        "plt.grid()\n",
        "plt.xticks(np.arange(0,0.2,0.02));\n",
        "'''"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "cc2b4b34-7465-39b0-caab-a7546ed5c5a7"
      },
      "outputs": [],
      "source": [
        "#cols = X_train.columns[clf_rfc1.feature_importances_>=0.016]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d01ba707-e275-3c3d-adde-2394b6856720"
      },
      "outputs": [],
      "source": [
        "'''\n",
        "rfc = RandomForestClassifier(random_state=42, criterion='entropy', min_samples_split=5, oob_score=True)\n",
        "parameters = {'n_estimators':[500], 'min_samples_leaf':[12]}\n",
        "clf_rfc2 = get_model(rfc, parameters, X_train[cols], y_train, scoring)\n",
        "'''"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ca9184ce-abdb-fe6f-7317-4a081af161a4"
      },
      "outputs": [],
      "source": [
        "'''\n",
        "print (clf_rfc2)\n",
        "print (accuracy_score(y_test, clf_rfc2.predict(X_test[cols])))\n",
        "plot_learning_curve(clf_rfc2, 'Random Forest', X[cols], y, cv=4);\n",
        "'''"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "80180e1b-fb99-6094-2c25-8c6121fbdff2"
      },
      "source": [
        "### Logistic Regression"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ad3fdeca-b04e-426a-bcef-1c8112195636"
      },
      "outputs": [],
      "source": [
        "'''\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "lg = LogisticRegression(random_state=42, penalty='l1')\n",
        "parameters = {'C':[0.5]}\n",
        "clf_lg1 = get_model(lg, parameters, X_train, y_train, scoring)\n",
        "'''"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3af20be0-fbd2-6823-bcf1-f81d3630fde3"
      },
      "outputs": [],
      "source": [
        "'''\n",
        "print (clf_lg1)\n",
        "print (accuracy_score(y_test, clf_lg1.predict(X_test)))\n",
        "plot_learning_curve(clf_lg1, 'Logistic Regression', X, y, cv=4);\n",
        "'''"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "02ab3ecb-4389-fd39-8199-52875342598d"
      },
      "source": [
        "### SVC"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f629f983-b96b-3de3-8c23-323952c3efdd"
      },
      "outputs": [],
      "source": [
        "from sklearn.svm import SVC\n",
        "svc = SVC(random_state=42, kernel='poly', probability=True)\n",
        "parameters = {'C': [35], 'gamma': [0.0055], 'coef0': [0.1],\n",
        "              'degree':[2]}\n",
        "clf_svc = get_model(svc, parameters, X_train, y_train, scoring)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "4203eb35-58ba-dccf-6b53-e40885b57dfe"
      },
      "outputs": [],
      "source": [
        "print (clf_svc)\n",
        "print (accuracy_score(y_test, clf_svc.predict(X_test)))\n",
        "plot_learning_curve(clf_svc, 'SVC', X, y, cv=4);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "86580051-dc5f-32ee-1573-0fd94ff88ab1"
      },
      "source": [
        "### XGBoost"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "480be50b-5dd4-1d7d-6399-df33f9764d7d"
      },
      "outputs": [],
      "source": [
        "'''\n",
        "import xgboost as XGB\n",
        "xgb = XGB.XGBClassifier(seed=42, max_depth=3, objective='binary:logistic', n_estimators=400)\n",
        "parameters = {'learning_rate':[0.1],\n",
        "              'reg_alpha':[3.0], 'reg_lambda': [4.0]}\n",
        "clf_xgb1 = get_model(xgb, parameters, X_train, y_train, scoring)\n",
        "'''"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3917b123-5364-ba90-5dce-b560b5b0045f"
      },
      "outputs": [],
      "source": [
        "'''\n",
        "print (accuracy_score(y_test, clf_xgb1.predict(X_test)))\n",
        "print (clf_xgb1)\n",
        "plot_learning_curve(clf_xgb1, 'XGB', X, y, cv=4);\n",
        "'''"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "98c7ffeb-1ec6-7102-bdd5-90dd47fd5702"
      },
      "source": [
        "## Ensemble"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "749466e3-cc30-6c95-23fd-d7fdb4c33e12"
      },
      "outputs": [],
      "source": [
        "'''\n",
        "from sklearn.ensemble import VotingClassifier\n",
        "clf_vc = VotingClassifier(estimators=[('xgb1', clf_xgb1), ('lg1', clf_lg1), ('svc', clf_svc), \n",
        "                                      ('rfc1', clf_rfc1),('rfc2', clf_rfc2), ('knn', clf_knn)], \n",
        "                          voting='hard', weights=[4,1,1,1,1,2])\n",
        "clf_vc = clf_vc.fit(X_train, y_train)\n",
        "'''"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "47b071ad-a128-37dc-604a-8c6c38e9fc85"
      },
      "outputs": [],
      "source": [
        "'''\n",
        "print (accuracy_score(y_test, clf_vc.predict(X_test)))\n",
        "print (clf_vc)\n",
        "plot_learning_curve(clf_vc, 'Ensemble', X, y, cv=4);\n",
        "'''"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "399f2b4d-22ad-9c75-1923-302740f6ca2b"
      },
      "outputs": [],
      "source": [
        "X = train.drop(['Survived',\n",
        "              'SibSp', 'Room', 'Group_num', 'NorFare', 'Embarked_C',\n",
        "                'Sex', 'Parch', 'NorAge', 'Embarked_Q', 'Embarked_S',\n",
        "                ], axis=1)\n",
        "y = train.Survived\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)\n",
        "\n",
        "#print(X_train.head())\n",
        "\n",
        "from keras.layers import Activation\n",
        "from keras.optimizers import RMSprop\n",
        "import keras.optimizers as opt\n",
        "\n",
        "# build a neural network from the 1st layer to the last layer\n",
        "model = Sequential([\n",
        "    Dense(2, input_dim=X_train.shape[1]),\n",
        "    Activation('softmax')\n",
        "])\n",
        "\n",
        "# choose loss function and optimizing method\n",
        "#model.compile(loss='mae', optimizer='rmsprop')\n",
        "\n",
        "# Another way to define your optimizer\n",
        "optobj = RMSprop(lr=0.1)#, rho=0.9)#, epsilon=1e-08, decay=0.0)\n",
        "#optobj = opt.SGD(lr=0.1, momentum=0.0, decay=0.0, nesterov=False)\n",
        "# We add metrics to get more results you want to see\n",
        "model.compile(optimizer=optobj,\n",
        "              loss='categorical_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "#model.compile(loss='categorical_crossentropy', metrics=['accuracy'], optimizer='rmsprop')\n",
        "\n",
        "from keras.utils.np_utils import to_categorical\n",
        "y_binary = to_categorical(y_train.as_matrix())\n",
        "#y_binary = y_train.astype(int).as_matrix()\n",
        "\n",
        "# training\n",
        "model.fit(X_train.as_matrix(), y_binary, nb_epoch=3)\n",
        "\n",
        "y_binary = to_categorical(y_test.as_matrix())\n",
        "#y_binary = y_test.astype(int).as_matrix()\n",
        "loss, accuracy = model.evaluate(X_test.as_matrix(), y_binary)\n",
        "print ('%f %f' % (loss, accuracy))\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "19e6f36e-2a91-566f-ba74-b5a84e5644d5"
      },
      "outputs": [],
      "source": [
        "def test(droplist):\n",
        "  X = train.drop(['Survived'], axis=1).drop(droplist, axis=1)\n",
        "  y = train.Survived\n",
        "  X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)\n",
        "\n",
        "  print(X_train.head())\n",
        "\n",
        "  from keras.layers import Activation\n",
        "  from keras.optimizers import RMSprop\n",
        "  import keras.optimizers as opt\n",
        "\n",
        "  # build a neural network from the 1st layer to the last layer\n",
        "  model = Sequential([\n",
        "      Dense(2, input_dim=X_train.shape[1]),\n",
        "      Activation('softmax')\n",
        "  ])\n",
        "\n",
        "  # choose loss function and optimizing method\n",
        "  #model.compile(loss='mae', optimizer='rmsprop')\n",
        "\n",
        "  # Another way to define your optimizer\n",
        "  optobj = RMSprop(lr=0.1)#, rho=0.9)#, epsilon=1e-08, decay=0.0)\n",
        "  #optobj = opt.SGD(lr=0.1, momentum=0.0, decay=0.0, nesterov=False)\n",
        "  # We add metrics to get more results you want to see\n",
        "  model.compile(optimizer=optobj,\n",
        "                loss='categorical_crossentropy',\n",
        "                metrics=['accuracy'])\n",
        "  #model.compile(loss='categorical_crossentropy', metrics=['accuracy'], optimizer='rmsprop')\n",
        "\n",
        "  from keras.utils.np_utils import to_categorical\n",
        "  y_binary = to_categorical(y_train.as_matrix())\n",
        "  #y_binary = y_train.astype(int).as_matrix()\n",
        "\n",
        "  # training\n",
        "  model.fit(X_train.as_matrix(), y_binary, nb_epoch=3)\n",
        "\n",
        "  y_binary = to_categorical(y_test.as_matrix())\n",
        "  #y_binary = y_test.astype(int).as_matrix()\n",
        "  loss, accuracy = model.evaluate(X_test.as_matrix(), y_binary)\n",
        "  print ('%f %f' % (loss, accuracy))\n",
        "    \n",
        "  return accuracy\n",
        "  \n",
        "droplist=['SibSp', 'Deck', 'Room', 'Group_num', 'NorFare', 'Embarked_C',\n",
        "                  'Title_Lady', 'Title_Master', 'Title_Miss', 'Title_Mr', 'Title_Mrs', 'Title_Officer',\n",
        "                  'Title_Sir', 'Group_size_L', 'Group_size_M', 'Group_size_S',\n",
        "                  'Sex', 'Parch', 'NorAge', 'NorNames', 'Embarked_Q', 'Embarked_S',\n",
        "                  'Pclass_1', 'Pclass_2', 'Pclass_3']\n",
        "\n",
        "keeps=['SibSp', 'Deck', 'Room', 'Group_num', 'NorFare', \n",
        "       ['Embarked_C', 'Embarked_Q', 'Embarked_S'],\n",
        "       ['Title_Lady', 'Title_Master', 'Title_Miss', 'Title_Mr', 'Title_Mrs', 'Title_Officer', 'Title_Sir'], \n",
        "       ['Group_size_L', 'Group_size_M', 'Group_size_S'],\n",
        "       'Sex', 'Parch', 'NorAge', 'NorNames',\n",
        "       ['Pclass_1', 'Pclass_2', 'Pclass_3']]\n",
        "\n",
        "records={}\n",
        "for keep in keeps:\n",
        "    tmpdroplist = droplist.copy()\n",
        "    if type(keep) == list:\n",
        "        for i in keep:\n",
        "            tmpdroplist.remove(i)\n",
        "    else:\n",
        "        tmpdroplist.remove(keep)\n",
        "    acc = test(tmpdroplist)\n",
        "    if type(keep) == list:\n",
        "        records[keep[0]] = acc\n",
        "    else:\n",
        "        records[keep] = acc\n",
        "\n",
        "for k in records.keys():\n",
        "  print ('%s = %f' % (k, records[k]))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "8ef8a237-0dc5-02db-333f-27518bdd3f7b"
      },
      "outputs": [],
      "source": [
        "\n",
        "print('\\nTesting ------------')\n",
        "y_binary = to_categorical(y_test.as_matrix())\n",
        "cost = model.evaluate(X_test.as_matrix(), y_binary, batch_size=len(X_test))\n",
        "print('test cost:', cost)\n",
        "\n",
        "rst = model.predict(X_test.as_matrix())\n",
        "#print('rst: ' + str(rst))\n",
        "r = []\n",
        "for i in rst:\n",
        "    if i[0] >= i[1]:\n",
        "        r.append(0)\n",
        "    else:\n",
        "        r.append(1)\n",
        "print (accuracy_score(y_test.as_matrix(), r))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "06af75c1-ade4-037d-3cee-7bc7fb6064df"
      },
      "source": [
        "## Make submission"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "05f9534a-2a5b-1f44-9e99-0007bcd34531"
      },
      "outputs": [],
      "source": [
        "test = pd.read_csv('../input/test.csv')\n",
        "\n",
        "PassengerId = test.PassengerId\n",
        "test.drop('PassengerId', axis=1, inplace=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1318d03c-352c-0b5b-9eb4-cb0571caed33"
      },
      "outputs": [],
      "source": [
        "def submission(model, fname, X):\n",
        "    ans = pd.DataFrame(columns=['PassengerId', 'Survived'])\n",
        "    ans.PassengerId = PassengerId\n",
        "    ans.Survived = pd.Series(model.predict(X), index=ans.index)\n",
        "    ans.to_csv(fname, index=False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5644e2c0-f072-98ba-fb4f-b5afab32f690"
      },
      "outputs": [],
      "source": [
        ""
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}