{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "ebbe6bba-c88c-0d49-502a-573d19476310"
      },
      "source": [
        "A Convolutional Neural Network for MNIST Classification. This solution got me a score of 0.98929 on the leaderboard.\n",
        "\n",
        "Note: this solution is loosely based on the official [tensorflow tutorial](https://www.tensorflow.org/tutorials/mnist/pros/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "6f552ccc-f472-107c-e20a-1c03a9362532"
      },
      "source": [
        "## Packages and Imports"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "15754c6a-4ee3-0ac0-3d01-9ec4831370e3"
      },
      "outputs": [],
      "source": [
        "%matplotlib inline\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import tensorflow as tf\n",
        "import matplotlib.pyplot as plt\n",
        "from sklearn.model_selection import ShuffleSplit\n",
        "from sklearn.preprocessing import StandardScaler, LabelEncoder, OneHotEncoder"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "d3088dc8-906f-32e9-db54-59a4c075bc31"
      },
      "source": [
        "## Simulation Constants ##\n",
        "\n",
        "Download notebook and use commented out values for better performance."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "50083ab2-7752-56fb-54e8-249a4017bd29"
      },
      "outputs": [],
      "source": [
        "LABELS = 10 # Number of different types of labels (1-10)\n",
        "WIDTH = 28 # width / height of the image\n",
        "CHANNELS = 1 # Number of colors in the image (greyscale)\n",
        "\n",
        "VALID = 10000 # Validation data size\n",
        "\n",
        "STEPS = 3500 #20000   # Number of steps to run\n",
        "BATCH = 100 # Stochastic Gradient Descent batch size\n",
        "PATCH = 5 # Convolutional Kernel size\n",
        "DEPTH = 8 #32 # Convolutional Kernel depth size == Number of Convolutional Kernels\n",
        "HIDDEN = 100 #1024 # Number of hidden neurons in the fully connected layer\n",
        "\n",
        "LR = 0.001 # Learning rate"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "4572603b-cc72-031c-f710-6196172efea4"
      },
      "source": [
        "## Import Data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "b734dc0f-50d6-37c7-7906-409d5b49f3cb"
      },
      "source": [
        "prepare data by \n",
        "\n",
        "- applying 1-hot encoding: `1 = [1,0,0...0], 2 = [0,1,0...0] ...`\n",
        "- reshaping into image shape: `(# images, # vertical height, # horizontal width, # colors)`\n",
        "- splitting data into train and validation set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "4cd92946-fe19-5590-7946-1f43e2a68e48"
      },
      "outputs": [],
      "source": [
        "data = pd.read_csv('../input/train.csv') # Read csv file in pandas dataframe\n",
        "labels = np.array(data.pop('label')) # Remove the labels as a numpy array from the dataframe\n",
        "labels = LabelEncoder().fit_transform(labels)[:, None]\n",
        "labels = OneHotEncoder().fit_transform(labels).todense()\n",
        "data = StandardScaler().fit_transform(np.float32(data.values)) # Convert the dataframe to a numpy array\n",
        "data = data.reshape(-1, WIDTH, WIDTH, CHANNELS) # Reshape the data into 42000 2d images\n",
        "train_data, valid_data = data[:-VALID], data[-VALID:]\n",
        "train_labels, valid_labels = labels[:-VALID], labels[-VALID:]\n",
        "\n",
        "print('train data shape = ' + str(train_data.shape) + ' = (TRAIN, WIDTH, WIDTH, CHANNELS)')\n",
        "print('labels shape = ' + str(labels.shape) + ' = (TRAIN, LABELS)')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "616030e9-9c6e-c369-c6f2-c19cd76a542b"
      },
      "source": [
        "## Model\n",
        "\n",
        "Let's now build a network with two convolutional layers, followed by one fully connected layer. Since this is computationally pretty expensive, we'll limit the depth and number of fully connected nodes for this online notebook.\n",
        "\n",
        "We initialize the input data with placeholders"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f7b2abd6-d89d-1149-751f-3697f9afb52c"
      },
      "outputs": [],
      "source": [
        "tf_data = tf.placeholder(tf.float32, shape=(None, WIDTH, WIDTH, CHANNELS))\n",
        "tf_labels = tf.placeholder(tf.float32, shape=(None, LABELS))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "fa565617-eaf9-af98-d4f6-b7a513f019f8"
      },
      "source": [
        "We choose a 4 layered network consisting of 2 convolutional layers with weights and biases `(w1, b1)` and `(w2,b2)`, followed by a fully connected hidden layer `(w3,b3)` with #`HIDDEN` hidden neurons and an output layer `(w4, b4)` with `10` output nodes (one-hot encoding).\n",
        "\n",
        "We initialize the weights and biases such that the kernel has a patch size of `PATCH` and the depth of the second convolutional layer is twice the depth of the first convolutional layer `(DEPTH)`. For the rest, the fully connected hidden layer has `HIDDEN` hidden neurons."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "8c23848b-e5df-548c-698a-d30313129b23"
      },
      "outputs": [],
      "source": [
        "w1 = tf.Variable(tf.truncated_normal([PATCH, PATCH, CHANNELS, DEPTH], stddev=0.1))\n",
        "b1 = tf.Variable(tf.zeros([DEPTH]))\n",
        "w2 = tf.Variable(tf.truncated_normal([PATCH, PATCH, DEPTH, 2*DEPTH], stddev=0.1))\n",
        "b2 = tf.Variable(tf.constant(1.0, shape=[2*DEPTH]))\n",
        "w3 = tf.Variable(tf.truncated_normal([WIDTH // 4 * WIDTH // 4 * 2*DEPTH, HIDDEN], stddev=0.1))\n",
        "b3 = tf.Variable(tf.constant(1.0, shape=[HIDDEN]))\n",
        "w4 = tf.Variable(tf.truncated_normal([HIDDEN, LABELS], stddev=0.1))\n",
        "b4 = tf.Variable(tf.constant(1.0, shape=[LABELS]))\n",
        "\n",
        "def logits(data):\n",
        "    # Convolutional layer 1\n",
        "    x = tf.nn.conv2d(data, w1, [1, 1, 1, 1], padding='SAME')\n",
        "    x = tf.nn.max_pool(x, [1, 2, 2, 1], [1, 2, 2, 1], padding='SAME')\n",
        "    x = tf.nn.relu(x + b1)\n",
        "    # Convolutional layer 2\n",
        "    x = tf.nn.conv2d(x, w2, [1, 1, 1, 1], padding='SAME')\n",
        "    x = tf.nn.max_pool(x, [1, 2, 2, 1], [1, 2, 2, 1], padding='SAME')\n",
        "    x = tf.nn.relu(x + b2)\n",
        "    # Fully connected layer\n",
        "    x = tf.reshape(x, (-1, WIDTH // 4 * WIDTH // 4 * 2*DEPTH))\n",
        "    x = tf.nn.relu(tf.matmul(x, w3) + b3)\n",
        "    return tf.matmul(x, w4) + b4\n",
        "\n",
        "# Prediction:\n",
        "tf_pred = tf.nn.softmax(logits(tf_data))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "bc95200a-61a2-5f5a-0dc0-71009c5601dc"
      },
      "source": [
        "We use the categorical cross entropy loss for training the model.\n",
        "\n",
        "As optimizer we could use a Gradient Descent optimizer [with or without decaying learning rate] or one of the more sophisticated (and easier to optimize) optimizers like `Adam` or `RMSProp`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "429d6634-ba23-9644-84f0-9173627d1f96"
      },
      "outputs": [],
      "source": [
        "tf_loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits(tf_data), \n",
        "                                                                 labels=tf_labels))\n",
        "tf_acc = 100*tf.reduce_mean(tf.to_float(tf.equal(tf.argmax(tf_pred, 1), tf.argmax(tf_labels, 1))))\n",
        "\n",
        "#tf_opt = tf.train.GradientDescentOptimizer(LR)\n",
        "#tf_opt = tf.train.AdamOptimizer(LR)\n",
        "tf_opt = tf.train.RMSPropOptimizer(LR)\n",
        "tf_step = tf_opt.minimize(tf_loss)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "6dde9903-2b67-6d57-0200-3a2041a786b6"
      },
      "source": [
        "## Train"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "3a183269-2293-7a6b-0cc9-ba51d3fc9428"
      },
      "source": [
        "open the session"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "555fd8d0-7716-7cc8-17c3-7427cb9276b6"
      },
      "outputs": [],
      "source": [
        "init = tf.global_variables_initializer()\n",
        "session = tf.Session()\n",
        "session.run(init)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "9f83f5cc-dd67-1e27-d46f-02a0d6b08fce"
      },
      "source": [
        "Run the session (Run this cell again if the desired accuracy is not yet reached)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "bf9c6a09-58e8-ff57-9002-ebfa8ca41175"
      },
      "outputs": [],
      "source": [
        "ss = ShuffleSplit(n_splits=STEPS, train_size=BATCH)\n",
        "ss.get_n_splits(train_data, train_labels)\n",
        "history = [(0, np.nan, 10)] # Initial Error Measures\n",
        "for step, (idx, _) in enumerate(ss.split(train_data,train_labels), start=1):\n",
        "    fd = {tf_data:train_data[idx], tf_labels:train_labels[idx]}\n",
        "    session.run(tf_step, feed_dict=fd)\n",
        "    if step%500 == 0:\n",
        "        fd = {tf_data:valid_data, tf_labels:valid_labels}\n",
        "        valid_loss, valid_accuracy = session.run([tf_loss, tf_acc], feed_dict=fd)\n",
        "        history.append((step, valid_loss, valid_accuracy))\n",
        "        print('Step %i \\t Valid. Acc. = %f'%(step, valid_accuracy), end='\\n')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "ec8208b4-6a7d-f6f1-d23b-7164a2b455ed"
      },
      "source": [
        "Visualize the training history:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "31a957e0-5883-9f44-53c4-255777b16ebe"
      },
      "outputs": [],
      "source": [
        "steps, loss, acc = zip(*history)\n",
        "\n",
        "fig = plt.figure()\n",
        "plt.title('Validation Loss / Accuracy')\n",
        "ax_loss = fig.add_subplot(111)\n",
        "ax_acc = ax_loss.twinx()\n",
        "plt.xlabel('Training Steps')\n",
        "plt.xlim(0, max(steps))\n",
        "\n",
        "ax_loss.plot(steps, loss, '-o', color='C0')\n",
        "ax_loss.set_ylabel('Log Loss', color='C0');\n",
        "ax_loss.tick_params('y', colors='C0')\n",
        "ax_loss.set_ylim(0.01, 0.5)\n",
        "\n",
        "ax_acc.plot(steps, acc, '-o', color='C1')\n",
        "ax_acc.set_ylabel('Accuracy [%]', color='C1');\n",
        "ax_acc.tick_params('y', colors='C1')\n",
        "ax_acc.set_ylim(1,100)\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "769077b0-eff8-ceee-723a-0cbfe1fa87e0"
      },
      "source": [
        "## Results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "35244d22-1ada-54e5-2a83-92debc3ede2f"
      },
      "outputs": [],
      "source": [
        "test = pd.read_csv('../input/test.csv') # Read csv file in pandas dataframe\n",
        "test_data = StandardScaler().fit_transform(np.float32(test.values)) # Convert the dataframe to a numpy array\n",
        "test_data = test_data.reshape(-1, WIDTH, WIDTH, CHANNELS) # Reshape the data into 42000 2d images"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "f3d8bc95-1efe-8429-8eda-b7fcc78052be"
      },
      "source": [
        "Make a prediction about the test labels"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "0faca4c6-f71e-60be-ffca-66441fed07ec"
      },
      "outputs": [],
      "source": [
        "test_pred = session.run(tf_pred, feed_dict={tf_data:test_data})\n",
        "test_labels = np.argmax(test_pred, axis=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "bed7ce11-5a18-56d7-f526-8ebc103b0ce7"
      },
      "source": [
        "Plot an example:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3d9691dd-1bd9-b6d5-9920-c619fd7443fc"
      },
      "outputs": [],
      "source": [
        "k = 0 # Try different image indices k\n",
        "print(\"Label Prediction: %i\"%test_labels[k])\n",
        "fig = plt.figure(figsize=(2,2)); plt.axis('off')\n",
        "plt.imshow(test_data[k,:,:,0]); plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "83654d47-11bc-acc7-4761-d443efd44474"
      },
      "source": [
        "## Submission"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3db12c10-bb63-22b8-5658-9f6bd740cc6c"
      },
      "outputs": [],
      "source": [
        "submission = pd.DataFrame(data={'ImageId':(np.arange(test_labels.shape[0])+1), 'Label':test_labels})\n",
        "submission.to_csv('submission.csv', index=False)\n",
        "submission.tail()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "982a581a-ef18-fd33-848c-e53afcdde40d"
      },
      "source": [
        "## Close Session\n",
        "\n",
        "(note: once the session is closed, the training cell cannot be run again...)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "53d600e1-cae8-a8fb-29e7-dcafae3d7979"
      },
      "outputs": [],
      "source": [
        "#session.close()"
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}