{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "25b1e1db-8bc5-7029-f719-91da523bd121"
      },
      "source": [
        "## Introduction ##\n",
        "\n",
        "This is my first work of machine learning. the notebook is written in python and has inspired from [\"Exploring Survival on Titanic\" by Megan Risdal, a Kernel in R on Kaggle][1].\n",
        "\n",
        "\n",
        "  [1]: https://www.kaggle.com/mrisdal/titanic/exploring-survival-on-the-titanic"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "2ce68358-02ec-556d-ba88-e773a50bc18b"
      },
      "outputs": [],
      "source": [
        "%matplotlib inline\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import re as re\n",
        "\n",
        "train = pd.read_csv('../input/train.csv', header = 0, dtype={'Age': np.float64})\n",
        "test  = pd.read_csv('../input/test.csv' , header = 0, dtype={'Age': np.float64})\n",
        "full_data = [train, test]\n",
        "\n",
        "print (train.info())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "f9595646-65c9-6fc4-395f-0befc4d122ce"
      },
      "source": [
        "# Feature Engineering #"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "9b4c278b-aaca-e92c-ba77-b9b48379d1f1"
      },
      "source": [
        "## 1. Pclass ##\n",
        "there is no missing value on this feature and already a numerical value. so let's check it's impact on our train set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "4680d950-cf7d-a6ae-e813-535e2247d88e"
      },
      "outputs": [],
      "source": [
        "print (train[['Pclass', 'Survived']].groupby(['Pclass'], as_index=False).mean())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "5e70f81c-d4e2-1823-f0ba-a7c9b46984ff"
      },
      "source": [
        "## 2. Sex ##"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "6729681d-7915-1631-78d2-ddf3c35a424c"
      },
      "outputs": [],
      "source": [
        "print (train[[\"Sex\", \"Survived\"]].groupby(['Sex'], as_index=False).mean())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "7c58b7ee-d6a1-0cc9-2346-81c47846a54a"
      },
      "source": [
        "## 3. SibSp and Parch ##\n",
        "With the number of siblings/spouse and the number of children/parents we can create new feature called Family Size."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1a537f10-7cec-d0b7-8a34-fa9975655190"
      },
      "outputs": [],
      "source": [
        "for dataset in full_data:\n",
        "    dataset['FamilySize'] = dataset['SibSp'] + dataset['Parch'] + 1\n",
        "print (train[['FamilySize', 'Survived']].groupby(['FamilySize'], as_index=False).mean())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "e4861d3e-10db-1a23-8728-44e4d5251844"
      },
      "source": [
        "it seems has a good effect on our prediction but let's go further and categorize people to check whether they are alone in this ship or not."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "8c35e945-c928-e3bc-bd9c-d6ddb287e4c9"
      },
      "outputs": [],
      "source": [
        "for dataset in full_data:\n",
        "    dataset['IsAlone'] = 0\n",
        "    dataset.loc[dataset['FamilySize'] == 1, 'IsAlone'] = 1\n",
        "print (train[['IsAlone', 'Survived']].groupby(['IsAlone'], as_index=False).mean())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "2780ca4e-7923-b845-0b6b-5f68a45f6b93"
      },
      "source": [
        "good! the impact is considerable."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "8aa419c0-6614-7efc-7797-97f4a5158b19"
      },
      "source": [
        "## 4. Embarked ##\n",
        "the embarked feature has some missing value. and we try to fill those with the most occurred value ( 'S' )."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "0e70e9af-d7cc-8c40-b7d4-2643889c376d"
      },
      "outputs": [],
      "source": [
        "for dataset in full_data:\n",
        "    dataset['Embarked'] = dataset['Embarked'].fillna('S')\n",
        "print (train[['Embarked', 'Survived']].groupby(['Embarked'], as_index=False).mean())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "e08c9ee8-d6d1-99b7-38bd-f0042c18a5d9"
      },
      "source": [
        "## 5. Fare ##\n",
        "Fare also has some missing value and we will replace it with the median. then we categorize it into 4 ranges."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a21335bd-4e8d-66e8-e6a5-5d2173b72d3b"
      },
      "outputs": [],
      "source": [
        "for dataset in full_data:\n",
        "    dataset['Fare'] = dataset['Fare'].fillna(train['Fare'].median())\n",
        "train['CategoricalFare'] = pd.qcut(train['Fare'], 4)\n",
        "print (train[['CategoricalFare', 'Survived']].groupby(['CategoricalFare'], as_index=False).mean())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "ec8d1b22-a95f-9f16-77ab-7b60d2103852"
      },
      "source": [
        "## 6. Age ##\n",
        "we have plenty of missing values in this feature. # generate random numbers between (mean - std) and (mean + std).\n",
        "then we categorize age into 5 range."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b90c2870-ce5d-ae0e-a33d-59e35445500e"
      },
      "outputs": [],
      "source": [
        "for dataset in full_data:\n",
        "    age_avg \t   = dataset['Age'].mean()\n",
        "    age_std \t   = dataset['Age'].std()\n",
        "    age_null_count = dataset['Age'].isnull().sum()\n",
        "    \n",
        "    age_null_random_list = np.random.randint(age_avg - age_std, age_avg + age_std, size=age_null_count)\n",
        "    dataset['Age'][np.isnan(dataset['Age'])] = age_null_random_list\n",
        "    dataset['Age'] = dataset['Age'].astype(int)\n",
        "    \n",
        "train['CategoricalAge'] = pd.cut(train['Age'], 5)\n",
        "\n",
        "print (train[['CategoricalAge', 'Survived']].groupby(['CategoricalAge'], as_index=False).mean())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "bd25ec3f-b601-c1cc-d701-991fac1621f9"
      },
      "source": [
        "## 7. Name ##\n",
        "inside this feature we can find the title of people."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ad042f43-bfe0-ded0-4171-379d8caaa749"
      },
      "outputs": [],
      "source": [
        "def get_title(name):\n",
        "\ttitle_search = re.search(' ([A-Za-z]+)\\.', name)\n",
        "\t# If the title exists, extract and return it.\n",
        "\tif title_search:\n",
        "\t\treturn title_search.group(1)\n",
        "\treturn \"\"\n",
        "\n",
        "for dataset in full_data:\n",
        "    dataset['Title'] = dataset['Name'].apply(get_title)\n",
        "\n",
        "print(pd.crosstab(train['Title'], train['Sex']))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "ca5fff8c-7a0d-6c18-2173-b8df6293c50a"
      },
      "source": [
        " so we have titles. let's categorize it and check the title impact on survival rate."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "8357238b-98fe-632a-acd5-33674a6132ce"
      },
      "outputs": [],
      "source": [
        "for dataset in full_data:\n",
        "    dataset['Title'] = dataset['Title'].replace(['Lady', 'Countess','Capt', 'Col',\\\n",
        " \t'Don', 'Dr', 'Major', 'Rev', 'Sir', 'Jonkheer', 'Dona'], 'Rare')\n",
        "\n",
        "    dataset['Title'] = dataset['Title'].replace('Mlle', 'Miss')\n",
        "    dataset['Title'] = dataset['Title'].replace('Ms', 'Miss')\n",
        "    dataset['Title'] = dataset['Title'].replace('Mme', 'Mrs')\n",
        "\n",
        "print (train[['Title', 'Survived']].groupby(['Title'], as_index=False).mean())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "68fa2057-e27a-e252-0d1b-869c00a303ba"
      },
      "source": [
        "# Data Cleaning #\n",
        "great! now let's clean our data and map our features into numerical values."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "2502bb70-ce6f-2497-7331-7d1f80521470"
      },
      "outputs": [],
      "source": [
        "for dataset in full_data:\n",
        "    # Mapping Sex\n",
        "    dataset['Sex'] = dataset['Sex'].map( {'female': 0, 'male': 1} ).astype(int)\n",
        "    \n",
        "    # Mapping titles\n",
        "    title_mapping = {\"Mr\": 1, \"Miss\": 2, \"Mrs\": 3, \"Master\": 4, \"Rare\": 5}\n",
        "    dataset['Title'] = dataset['Title'].map(title_mapping)\n",
        "    dataset['Title'] = dataset['Title'].fillna(0)\n",
        "    \n",
        "    # Mapping Embarked\n",
        "    dataset['Embarked'] = dataset['Embarked'].map( {'S': 0, 'C': 1, 'Q': 2} ).astype(int)\n",
        "    \n",
        "    # Mapping Fare\n",
        "    dataset.loc[ dataset['Fare'] <= 7.91, 'Fare'] \t\t\t\t\t\t        = 0\n",
        "    dataset.loc[(dataset['Fare'] > 7.91) & (dataset['Fare'] <= 14.454), 'Fare'] = 1\n",
        "    dataset.loc[(dataset['Fare'] > 14.454) & (dataset['Fare'] <= 31), 'Fare']   = 2\n",
        "    dataset.loc[ dataset['Fare'] > 31, 'Fare'] \t\t\t\t\t\t\t        = 3\n",
        "    dataset['Fare'] = dataset['Fare'].astype(int)\n",
        "    \n",
        "    # Mapping Age\n",
        "    dataset.loc[ dataset['Age'] <= 16, 'Age'] \t\t\t\t\t       = 0\n",
        "    dataset.loc[(dataset['Age'] > 16) & (dataset['Age'] <= 32), 'Age'] = 1\n",
        "    dataset.loc[(dataset['Age'] > 32) & (dataset['Age'] <= 48), 'Age'] = 2\n",
        "    dataset.loc[(dataset['Age'] > 48) & (dataset['Age'] <= 64), 'Age'] = 3\n",
        "    dataset.loc[ dataset['Age'] > 64, 'Age']                           = 4\n",
        "\n",
        "# Feature Selection\n",
        "drop_elements = ['PassengerId', 'Name', 'Ticket', 'Cabin', 'SibSp',\\\n",
        "                 'Parch', 'FamilySize']\n",
        "train = train.drop(drop_elements, axis = 1)\n",
        "train = train.drop(['CategoricalAge', 'CategoricalFare'], axis = 1)\n",
        "\n",
        "test  = test.drop(drop_elements, axis = 1)\n",
        "\n",
        "print (train.head(10))\n",
        "\n",
        "train = train.values\n",
        "test  = test.values"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "8aaaf2bc-e282-79cc-008a-e2e801b51b07"
      },
      "source": [
        "good! now we have a clean dataset and ready to predict. let's find which classifier works better on this dataset. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "23b55b45-572b-7276-32e7-8f7a0dcfd25e"
      },
      "source": [
        "# Classifier Comparison #"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "31ded30a-8de4-6507-e7f7-5805a0f1eaf1"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "\n",
        "from sklearn.model_selection import StratifiedShuffleSplit\n",
        "from sklearn.metrics import accuracy_score, log_loss\n",
        "from sklearn.neighbors import KNeighborsClassifier\n",
        "from sklearn.svm import SVC\n",
        "from sklearn.tree import DecisionTreeClassifier\n",
        "from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier, GradientBoostingClassifier\n",
        "from sklearn.naive_bayes import GaussianNB\n",
        "from sklearn.discriminant_analysis import LinearDiscriminantAnalysis, QuadraticDiscriminantAnalysis\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "\n",
        "classifiers = [\n",
        "    KNeighborsClassifier(3),\n",
        "    SVC(probability=True),\n",
        "    DecisionTreeClassifier(),\n",
        "    RandomForestClassifier(),\n",
        "\tAdaBoostClassifier(),\n",
        "    GradientBoostingClassifier(),\n",
        "    GaussianNB(),\n",
        "    LinearDiscriminantAnalysis(),\n",
        "    QuadraticDiscriminantAnalysis(),\n",
        "    LogisticRegression()]\n",
        "\n",
        "log_cols = [\"Classifier\", \"Accuracy\"]\n",
        "log \t = pd.DataFrame(columns=log_cols)\n",
        "\n",
        "sss = StratifiedShuffleSplit(n_splits=10, test_size=0.1, random_state=0)\n",
        "\n",
        "X = train[0::, 1::]\n",
        "y = train[0::, 0]\n",
        "\n",
        "acc_dict = {}\n",
        "\n",
        "for train_index, test_index in sss.split(X, y):\n",
        "\tX_train, X_test = X[train_index], X[test_index]\n",
        "\ty_train, y_test = y[train_index], y[test_index]\n",
        "\t\n",
        "\tfor clf in classifiers:\n",
        "\t\tname = clf.__class__.__name__\n",
        "\t\tclf.fit(X_train, y_train)\n",
        "\t\ttrain_predictions = clf.predict(X_test)\n",
        "\t\tacc = accuracy_score(y_test, train_predictions)\n",
        "\t\tif name in acc_dict:\n",
        "\t\t\tacc_dict[name] += acc\n",
        "\t\telse:\n",
        "\t\t\tacc_dict[name] = acc\n",
        "\n",
        "for clf in acc_dict:\n",
        "\tacc_dict[clf] = acc_dict[clf] / 10.0\n",
        "\tlog_entry = pd.DataFrame([[clf, acc_dict[clf]]], columns=log_cols)\n",
        "\tlog = log.append(log_entry)\n",
        "\n",
        "plt.xlabel('Accuracy')\n",
        "plt.title('Classifier Accuracy')\n",
        "\n",
        "sns.set_color_codes(\"muted\")\n",
        "sns.barplot(x='Accuracy', y='Classifier', data=log, color=\"b\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "438585cf-b7ad-73ba-49aa-87688ff21233"
      },
      "source": [
        "# Prediction #\n",
        "now we can use SVC classifier to predict our data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "24967b57-732b-7180-bfd5-005beff75974"
      },
      "outputs": [],
      "source": [
        "candidate_classifier = SVC()\n",
        "candidate_classifier.fit(train[0::, 1::], train[0::, 0])\n",
        "result = candidate_classifier.predict(test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b464aec6-c9d0-922e-dd8f-558922f9f4f4"
      },
      "outputs": [],
      "source": [
        "test  = pd.read_csv('../input/test.csv' , header = 0, dtype={'Age': np.float64})\n",
        "submission = pd.DataFrame({\n",
        "        \"PassengerId\": test[\"PassengerId\"],\n",
        "        \"Survived\": result\n",
        "    })\n",
        "\n",
        "submission.to_csv(\"kaggle.csv\", index=False)"
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}