{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "0de78501-a4da-8a25-3a9e-c2d89aabfbf2"
      },
      "source": [
        "This is a simple example and starting point for neural networks with TensorFlow.\n",
        "We create a feed-forward neural network with two hidden layers (128 and 256 nodes)\n",
        "and ReLU units.\n",
        "The test accuracy is around 78.5 % - which is not too bad for such a simple model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "72f90850-768f-0429-9896-11e233dbf7cd"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import pandas as pd        # For loading and processing the dataset\n",
        "import tensorflow as tf    # Of course, we need TensorFlow.\n",
        "from sklearn.model_selection import train_test_split"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "597549eb-6cdf-c65c-955b-8ac84f5ee820"
      },
      "source": [
        "## Reading and cleaning the input data\n",
        "\n",
        "We first read the CSV input file using Pandas.\n",
        "Next, we remove irrelevant entries, and prepare the data for our neural network."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "bb29fa7a-32f7-872a-1635-8560069b8995"
      },
      "outputs": [],
      "source": [
        "# Read the CSV input file and show first 5 rows\n",
        "df_train = pd.read_csv('../input/train.csv')\n",
        "df_train.head(5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "24392b10-cee1-6acf-520b-c44f5fc9f378"
      },
      "outputs": [],
      "source": [
        "# We can't do anything with the Name, Ticket number, and Cabin, so we drop them.\n",
        "df_train = df_train.drop(['PassengerId','Name','Ticket', 'Cabin'], axis=1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "cae98c23-bc27-4a52-b5fa-b34ec1446ed4"
      },
      "outputs": [],
      "source": [
        "# To make 'Sex' numeric, we replace 'female' by 0 and 'male' by 1\n",
        "df_train['Sex'] = df_train['Sex'].map({'female':0, 'male':1}).astype(int) "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "25ea9c4d-6e08-de6b-5f80-a456d1234bc7"
      },
      "outputs": [],
      "source": [
        "# We replace 'Embarked' by three dummy variables 'Embarked_S', 'Embarked_C', and 'Embarked Q',\n",
        "# which are 1 if the person embarked there, and 0 otherwise.\n",
        "df_train = pd.concat([df_train, pd.get_dummies(df_train['Embarked'], prefix='Embarked')], axis=1)\n",
        "df_train = df_train.drop('Embarked', axis=1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "0b884328-f167-931e-537d-86430be4a29d"
      },
      "outputs": [],
      "source": [
        "# We normalize the age and the fare by subtracting their mean and dividing by the standard deviation\n",
        "age_mean = df_train['Age'].mean()\n",
        "age_std = df_train['Age'].std()\n",
        "df_train['Age'] = (df_train['Age'] - age_mean) / age_std\n",
        "\n",
        "fare_mean = df_train['Fare'].mean()\n",
        "fare_std = df_train['Fare'].std()\n",
        "df_train['Fare'] = (df_train['Fare'] - fare_mean) / fare_std"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a727b3f3-d2f1-d6b3-0e60-06a28a83c144"
      },
      "outputs": [],
      "source": [
        "# In many cases, the 'Age' is missing - which can cause problems. Let's look how bad it is:\n",
        "print(\"Number of missing 'Age' values: {:d}\".format(df_train['Age'].isnull().sum()))\n",
        "\n",
        "# A simple method to handle these missing values is to replace them by the mean age.\n",
        "df_train['Age'] = df_train['Age'].fillna(df_train['Age'].mean())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5a4fb0e5-77ac-4f07-a8ad-9d548fe8ee99"
      },
      "outputs": [],
      "source": [
        "# With that, we're almost ready for training\n",
        "df_train.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "91488012-44cc-c5e2-fbf3-cd116b28d6e8"
      },
      "outputs": [],
      "source": [
        "# Finally, we convert the Pandas dataframe to a NumPy array, and split it into a training and test set\n",
        "X_train = df_train.drop('Survived', axis=1).as_matrix()\n",
        "y_train = df_train['Survived'].as_matrix()\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(X_train, y_train, test_size=0.2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e0069a9b-8d1f-3792-0f4f-feeaa6d2d42d"
      },
      "outputs": [],
      "source": [
        "# We'll build a classifier with two classes: \"survived\" and \"didn't survive\",\n",
        "# so we create the according labels\n",
        "# This is taken from https://www.kaggle.com/klepacz/titanic/tensor-flow\n",
        "labels_train = (np.arange(2) == y_train[:,None]).astype(np.float32)\n",
        "labels_test = (np.arange(2) == y_test[:,None]).astype(np.float32)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "c2f5a0a4-07ae-481c-8c7b-3904931c12b5"
      },
      "source": [
        "## Define TensorFlow model\n",
        "In a first step, we define how our neural network will look.\n",
        "We create a network with 2 hidden layers with ReLU activations, and an output layer with softmax.\n",
        "We use dropout for regularization. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d21f9d00-4877-aa18-d887-bfd31d89e6fc"
      },
      "outputs": [],
      "source": [
        "inputs = tf.placeholder(tf.float32, shape=(None, X_train.shape[1]), name='inputs')\n",
        "label = tf.placeholder(tf.float32, shape=(None, 2), name='labels')\n",
        "\n",
        "# First layer\n",
        "hid1_size = 128\n",
        "w1 = tf.Variable(tf.random_normal([hid1_size, X_train.shape[1]], stddev=0.01), name='w1')\n",
        "b1 = tf.Variable(tf.constant(0.1, shape=(hid1_size, 1)), name='b1')\n",
        "y1 = tf.nn.dropout(tf.nn.relu(tf.add(tf.matmul(w1, tf.transpose(inputs)), b1)), keep_prob=0.5)\n",
        "\n",
        "# Second layer\n",
        "hid2_size = 256\n",
        "w2 = tf.Variable(tf.random_normal([hid2_size, hid1_size], stddev=0.01), name='w2')\n",
        "b2 = tf.Variable(tf.constant(0.1, shape=(hid2_size, 1)), name='b2')\n",
        "y2 = tf.nn.dropout(tf.nn.relu(tf.add(tf.matmul(w2, y1), b2)), keep_prob=0.5)\n",
        "\n",
        "# Output layer\n",
        "wo = tf.Variable(tf.random_normal([2, hid2_size], stddev=0.01), name='wo')\n",
        "bo = tf.Variable(tf.random_normal([2, 1]), name='bo')\n",
        "yo = tf.transpose(tf.add(tf.matmul(wo, y2), bo))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "7f68fee1-da60-b69f-f08e-211688027ad3"
      },
      "source": [
        "The output is a softmax output, and we train it with the cross entropy loss.\n",
        "We further define functions which calculate the predicted label, and the accuracy of the network."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "b386f438-326d-05ee-bbd9-5254e50f57bc"
      },
      "outputs": [],
      "source": [
        "# Loss function and optimizer\n",
        "lr = tf.placeholder(tf.float32, shape=(), name='learning_rate')\n",
        "loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=yo, labels=label))\n",
        "optimizer = tf.train.GradientDescentOptimizer(lr).minimize(loss)\n",
        "\n",
        "# Prediction\n",
        "pred = tf.nn.softmax(yo)\n",
        "pred_label = tf.argmax(pred, 1)\n",
        "correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(label, 1))\n",
        "accuracy = tf.reduce_mean(tf.cast(correct_prediction, \"float\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "5c572dc5-a17d-c8cf-430e-0500384fce1d"
      },
      "source": [
        "## Train the network!\n",
        "\n",
        "Finally, we are ready to train our network. Let's initialize TensorFlow and start training."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "c73094e9-a637-fc90-4e38-a1fb86bc3835"
      },
      "outputs": [],
      "source": [
        "# Create operation which will initialize all variables\n",
        "init = tf.global_variables_initializer()\n",
        "\n",
        "# Configure GPU not to use all memory\n",
        "config = tf.ConfigProto()\n",
        "config.gpu_options.allow_growth = True\n",
        "\n",
        "# Start a new tensorflow session and initialize variables\n",
        "sess = tf.InteractiveSession(config=config)\n",
        "sess.run(init)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "7b50693a-afe3-0791-6193-e48a485daa7a"
      },
      "outputs": [],
      "source": [
        "# This is the main training loop: we train for 50 epochs with a learning rate of 0.05 and another \n",
        "# 50 epochs with a smaller learning rate of 0.01\n",
        "for learning_rate in [0.05, 0.01]:\n",
        "    for epoch in range(50):\n",
        "        avg_cost = 0.0\n",
        "\n",
        "        # For each epoch, we go through all the samples we have.\n",
        "        for i in range(X_train.shape[0]):\n",
        "            # Finally, this is where the magic happens: run our optimizer, feed the current example into X and the current target into Y\n",
        "            _, c = sess.run([optimizer, loss], feed_dict={lr:learning_rate, \n",
        "                                                          inputs: X_train[i, None],\n",
        "                                                          label: labels_train[i, None]})\n",
        "            avg_cost += c\n",
        "        avg_cost /= X_train.shape[0]    \n",
        "\n",
        "        # Print the cost in this epcho to the console.\n",
        "        if epoch % 10 == 0:\n",
        "            print(\"Epoch: {:3d}    Train Cost: {:.4f}\".format(epoch, avg_cost))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "ac530578-edcf-a4db-ff74-7e1434c8e65b"
      },
      "source": [
        "We calculate the accuracy on our training set, and (more importantly) our test set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "cb4f3289-c154-b8e9-d419-f414cf362bd5"
      },
      "outputs": [],
      "source": [
        "acc_train = accuracy.eval(feed_dict={inputs: X_train, label: labels_train})\n",
        "print(\"Train accuracy: {:3.2f}%\".format(acc_train*100.0))\n",
        "\n",
        "acc_test = accuracy.eval(feed_dict={inputs: X_test, label: labels_test})\n",
        "print(\"Test accuracy:  {:3.2f}%\".format(acc_test*100.0))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "feaf7214-3e8b-764f-ada7-3b45ce1912d2"
      },
      "source": [
        "## Predict new passengers\n",
        "\n",
        "If we're happy with these results, we load the test dataset, and do all pre-processing steps we also did for the training set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "dfec4549-edca-e4e1-2b70-86978d5125f8"
      },
      "outputs": [],
      "source": [
        "df_test = pd.read_csv('../input/test.csv')\n",
        "df_test.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "247d3afc-f32d-7b9f-d299-4abef5d7d94b"
      },
      "outputs": [],
      "source": [
        "# Do all pre-processing steps as above\n",
        "df_test = df_test.drop(['Name', 'Ticket', 'Cabin'], axis=1)\n",
        "df_test['Sex'] = df_test['Sex'].map({'female':0, 'male':1}).astype(int)\n",
        "df_test = pd.concat([df_test, pd.get_dummies(df_test['Embarked'], prefix='Embarked')], axis=1)\n",
        "df_test = df_test.drop('Embarked', axis=1)\n",
        "df_test['Age'] = (df_test['Age'] - age_mean) / age_std\n",
        "df_test['Fare'] = (df_test['Fare'] - fare_mean) / fare_std\n",
        "df_test.head()\n",
        "X_test = df_test.drop('PassengerId', axis=1).as_matrix()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "7978d492-a9a7-16a8-4a13-cf985f929a44"
      },
      "source": [
        "Then we predict the label of all our test data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "67850d56-e154-1266-d159-8358e6e7e545"
      },
      "outputs": [],
      "source": [
        "# Predict\n",
        "for i in range(X_test.shape[0]):\n",
        "    df_test.loc[i, 'Survived'] = sess.run(pred_label, feed_dict={inputs: X_test[i, None]}).squeeze()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "58769a62-44bb-ddd7-e2fc-8a6f0a442cde"
      },
      "outputs": [],
      "source": [
        "# Important: close the TensorFlow session, now that we're finished.\n",
        "sess.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "0725ef27-405e-2d8c-1484-d42a423408ba"
      },
      "source": [
        "Finally, we can create an output to upload to Kaggle."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "2e7a6185-b79c-db36-36ac-5f2f75e7cb4d"
      },
      "outputs": [],
      "source": [
        "output = pd.DataFrame()\n",
        "output['PassengerId'] = df_test['PassengerId']\n",
        "output['Survived'] = df_test['Survived'].astype(int)\n",
        "output.to_csv('./prediction.csv', index=False)\n",
        "output.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "80769c08-a1e4-bf7f-13cb-bc4268e387cf"
      },
      "outputs": [],
      "source": [
        ""
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}