{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "ccca4adf-8d83-3b87-337d-045753b1d3bf"
      },
      "source": [
        "Simple Neural Network notebook using:\n",
        "-------------------------------------\n",
        "\n",
        "- TFLearn\n",
        "- Feature Standardization\n",
        "- Labels Hot Encoding\n",
        "- Softmax\n",
        "- Stochastic Gradient Descent\n",
        "- Cross Entropy"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "49c0b922-b17a-ab3d-5a51-260066ec4ca1"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import tflearn\n",
        "import tensorflow as tf\n",
        "# Remove regular python warnings\n",
        "import warnings\n",
        "warnings.filterwarnings('ignore')\n",
        "# Remove TensorFlow warnings\n",
        "tf.logging.set_verbosity(tf.logging.ERROR)\n",
        "# Visualizations\n",
        "from IPython.display import display, Math, Latex\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "e01d64e9-ec76-091f-cc80-1cb2b41869d5"
      },
      "source": [
        "# Data Load"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "b55ee027-bc7e-f508-ba63-3f4ca139ca87"
      },
      "source": [
        "I do the following:\n",
        "\n",
        "- Load train.csv into data\n",
        "- Load test.csv into test"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "dacdb0ed-0340-b5a6-868e-d45de2cbbb9d"
      },
      "outputs": [],
      "source": [
        "data=pd.read_csv(\"../input/train.csv\")\n",
        "test=pd.read_csv(\"../input/test.csv\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "8fe6e4da-3b46-3c64-5887-4cdb61186ffd"
      },
      "source": [
        "- Since labels are the first column of train, I separate it in **train** and **labels**.\n",
        "- Concatenate **train** and **test** so data standardization is done just once."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3c4318a1-5182-e9cc-9535-9d30e023dfb9"
      },
      "outputs": [],
      "source": [
        "train=data.ix[:,1:]\n",
        "labels=data.ix[:,0:1]\n",
        "data = pd.concat([train,test],ignore_index=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "89c28c00-5900-a45c-2ab3-9fe9c4e35663"
      },
      "source": [
        "- Print shapes of each dataframe so we don't make mistakes when fitting the Neural Network."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "2cdcb9fc-2cca-763f-6fae-9b7731f90f3c"
      },
      "outputs": [],
      "source": [
        "print(train.shape)\n",
        "print(test.shape)\n",
        "print(data.shape)\n",
        "print(labels.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "1a92cd74-4bee-7508-b85f-bef32de0fe6a"
      },
      "source": [
        "# Data Standardization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "345f1a30-5081-f337-dada-928dbe5bc22b"
      },
      "source": [
        "Pixel values goes from 255 to 0. To speedup the training process we can standardizate the data. The changes to apply:\n",
        "\n",
        " - Zero mean.\n",
        " - Low variance. \n",
        "\n",
        "Some have stdev 0, so the result will be NaN. We change those to 0."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "0fe42d7b-ee9b-ff70-0f8f-18fa7a8bc5c0"
      },
      "source": [
        "$$x'=\\frac{x-\\bar{x}}{\\sigma}$$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "786acd71-92c5-8392-376e-ad9829eb1405"
      },
      "outputs": [],
      "source": [
        "norm_data = (data - data.mean()) / (data.std())\n",
        "norm_data = norm_data.fillna(0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "c114eb1c-9b12-ead7-43d5-187a99b81b58"
      },
      "source": [
        "Labels Hot-Encoding\n",
        "----------------------\n",
        "\n",
        "Labels have the following values:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "7b951518-6242-2a90-22c3-133553d66257"
      },
      "outputs": [],
      "source": [
        "labels[0:5]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "e64916ab-c91d-3ff1-3214-8f91ba4f12ca"
      },
      "source": [
        "We need to convert it to a binary array. So, if 0 is the label the array would be:\n",
        "\n",
        "[1,0,0,0,0,0,0,0,0,0]\n",
        "\n",
        "For 1:\n",
        "\n",
        "[0,1,0,0,0,0,0,0,0,0]\n",
        "\n",
        "and so on...\n",
        "\n",
        "To do that, I am going to **hot-encode** the dataframe and store the result in a numpy array."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "063ea0c3-cb79-cfbb-76ee-afa5eaa7a338"
      },
      "outputs": [],
      "source": [
        "norm_labels=[]\n",
        "for value in labels.iterrows():\n",
        "    new_label=np.zeros(10)\n",
        "    new_label[value[1]]=1\n",
        "    norm_labels.append(new_label)\n",
        "norm_labels=np.array(norm_labels)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "366e0eea-43a1-65f8-35b7-535af0357fdb"
      },
      "outputs": [],
      "source": [
        "print(labels.ix[12:12,0:1])\n",
        "print(norm_labels[12])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "01be96fd-e8e6-837a-7d04-52e203a632e5"
      },
      "source": [
        "#\u00a0Preparing the data for TensorFlow"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "54f5ff31-f6ac-f231-181c-887645b60f60"
      },
      "source": [
        "The data is separated again in two variables, train and test."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3a278f33-6a1f-2512-5515-be58efa1bc82"
      },
      "outputs": [],
      "source": [
        "train = norm_data.as_matrix()[0:42000]\n",
        "test = norm_data.as_matrix()[42000:]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "4b023dba-2e8e-5238-542b-959a1b08209c"
      },
      "source": [
        "Tensorflow doesn't read Panda's Dataframes, so it is necessary to convert them to numpy arrays. This will avoid the following error:\n",
        "\n",
        "**IndexError: indices are out-of-bounds**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "90900031-180d-a6eb-4915-7355db95eca7"
      },
      "source": [
        "# Neural Network\n",
        "\n",
        "It is going to have the following characteristics:\n",
        "\n",
        "- An input layer.\n",
        "- A hidden layer with ReLu activation.\n",
        "- An output layer using softmax.\n",
        "- Backpropagation using Stochastic Gradient Descent.\n",
        "- Cross-entropy with labels.\n",
        "\n",
        "First it is needed to clean Tensorflow's graph, so it doesn't show erros when we try to create the model again."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "4369e3e8-40fa-a620-37b0-07d8bb8461ab"
      },
      "outputs": [],
      "source": [
        "tf.reset_default_graph()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "a3b7b545-7acc-ed4f-9773-8ee9d0b9f418"
      },
      "source": [
        "Input Layer\n",
        "-----------\n",
        "\n",
        "With 784 neurons, the number of features we have. The images are 28x28 pixels, so 784."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a312076c-7ad1-2a92-dbd5-3667141261cf"
      },
      "outputs": [],
      "source": [
        "net = tflearn.input_data(shape=[None, 784])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "065a24d9-857a-ad44-014b-a6fbc0079b4f"
      },
      "source": [
        "Hidden layer\n",
        "------------\n",
        "\n",
        "Using ReLu (Rectifier Neural Network).\n",
        "\n",
        "$$f(x)=max(0,x)$$\n",
        "\n",
        "It is a very simple and fast function. If X value is greater than 0, it returns that value. If it is smaller or equal to 0 it returns 0. It is important to configure the correct learning_rate (usually a low value). If a neuron gets to 0, it will die and will be useless during the training process."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "0eb108dc-7cb8-7af3-464c-5ff8a2fadef8"
      },
      "outputs": [],
      "source": [
        "x=np.arange(-10,10,1)\n",
        "y=np.maximum(x, 0)\n",
        "plt.plot(x,y)\n",
        "plt.xlim(-10,10)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "02f1425b-c237-d9ff-3a22-0e9986c30a8e"
      },
      "outputs": [],
      "source": [
        "net = tflearn.fully_connected(net, 128, activation='ReLu')\n",
        "# add a second hidden layer\n",
        "net = tflearn.fully_connected(net, 64, activation='ReLu')\n",
        "#\u00a0third layer, better going deeper than wider\n",
        "net = tflearn.fully_connected(net, 32, activation='ReLu')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "39243cdb-963c-a282-a219-1a4b894e43cf"
      },
      "source": [
        "Output Layer\n",
        "------------\n",
        "\n",
        "It has 10 neurons, one for each possible number. It uses softmax as the activation function. Softmax is a probability distribution function. It highlights the largest value and suppress values which are significantly below the maximum one.\n",
        "\n",
        "$$f(v_i) = \\displaystyle\\frac{e^{v_i}}{\\displaystyle\\sum_{j} e^{v_j}}$$\n",
        "\n",
        "So, with the following input:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1d56876c-ace4-8be2-6732-7634ab8a8217"
      },
      "outputs": [],
      "source": [
        "i=np.array([1,2,3,4,1,2,3,7])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "009e6e23-7595-6d3f-40a6-e43339267ac9"
      },
      "source": [
        "We get the following output after applying softmax:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "e797d4f2-5a23-1a24-b2dd-5ac62c4bec35"
      },
      "outputs": [],
      "source": [
        "o=np.exp(i)/np.sum(np.exp(i))\n",
        "o"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "b1d6deca-d0c7-0448-e2f5-f5797e14c5b6"
      },
      "source": [
        "And since softmax is a probability distribution, it sums up to 1."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "bcfce08e-d11e-8387-8936-198a260ee764"
      },
      "outputs": [],
      "source": [
        "int(np.sum(o))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d5f6e943-6e08-c701-4d3f-76b0a7191671"
      },
      "outputs": [],
      "source": [
        "net = tflearn.fully_connected(net, 10, activation='softmax')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "90916382-a448-fb99-b3c7-2a1682059761"
      },
      "source": [
        "Regression with Gradient Descent\n",
        "--------------------------------"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "0e3ca629-4025-4053-cd1b-cc3f4cb05a38"
      },
      "outputs": [],
      "source": [
        "net = tflearn.regression(net, optimizer='sgd', learning_rate=0.01, loss='categorical_crossentropy')\n",
        "model = tflearn.DNN(net)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5abfd2e1-f3b6-367f-07e1-3b0292a10a07"
      },
      "outputs": [],
      "source": [
        "model.fit(train, norm_labels,show_metric=True,validation_set=0.1,batch_size=100, n_epoch=50)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "273f7f41-f0bf-4fa6-87f3-6236fbf95b38"
      },
      "source": [
        "Predictions\n",
        "-------------------------------------\n",
        "\n",
        "Let's predict ten first numbers and show the actual image:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "eddc7c55-e902-7ad9-4b61-3938f45f3e1b"
      },
      "outputs": [],
      "source": [
        "for i in range(3):\n",
        "    ran=np.random.randint(0,test.shape[0])\n",
        "    pred=model.predict(test)[ran]\n",
        "    pred_digit=pred.index(max(pred))\n",
        "    digit=test[ran].reshape(28,28)\n",
        "    plt.imshow(digit, cmap='gray_r')\n",
        "    plt.text(1, -1,\"PREDICTION: {}\".format(pred_digit),fontsize=20) \n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "971f628a-8cee-f715-4fec-da39b00d8932"
      },
      "source": [
        "#\u00a0Predict test's labels"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "fcee2bbb-65b1-9d72-0dd8-559fe911a0c8"
      },
      "outputs": [],
      "source": [
        "ids=[]\n",
        "predictions=[]\n",
        "pred=model.predict(test)\n",
        "for i, values in enumerate(pred):\n",
        "    pred_digit=values.index(max(values))\n",
        "    ids.append(i+1)\n",
        "    predictions.append(pred_digit)\n",
        "    \n",
        "# Make predictions\n",
        "\n",
        "sub = pd.DataFrame({\n",
        "        \"ImageId\": ids,\n",
        "        \"Label\": predictions\n",
        "    })\n",
        "\n",
        "sub.to_csv(\"digit_submission.csv\", index=False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "be393ac0-33a6-ae31-5930-b39bb999c028"
      },
      "outputs": [],
      "source": [
        ""
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}