{ "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.0" } }, "nbformat": 4, "nbformat_minor": 0, "cells": [ { "cell_type": "markdown", "metadata": { "_cell_guid": "f210dc7b-1ed4-4370-86ed-e4c3e40b5108", "_active": false }, "source": "# TensorFlow deep NN\n#### A high-level tutorial into Deep Learning using MNIST data and TensorFlow library.\nby [@kakauandme](https://twitter.com/KaKaUandME) and [@thekoshkina](https://twitter.com/thekoshkina)\n\nAccuracy: 0.99\n\n**Prerequisites:** fundamental coding skills, a bit of linear algebra, especially matrix operations and perhaps understanding how images are stored in computer memory. To start with machine learning, we suggest [coursera course](https://www.coursera.org/learn/machine-learning) by Andrew Ng.\n\n\nNote: \n\n*Feel free to fork and adjust* CONSTANTS *to tweak network behaviour and explore how it changes algorithm performance and accuracy. Besides **TensorFlow graph** section can also be modified for learning purposes.*\n\n*It is highly recommended printing every variable that isn’t 100% clear for you. Also, [tensorboard](https://www.tensorflow.org/versions/master/how_tos/summaries_and_tensorboard/index.html) can be used on a local environment for visualisation and debugging.*\n## Libraries and settings", "outputs": [] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false, "_cell_guid": "f4c4db96-6d01-480c-b86c-9a74b8092548", "_active": false }, "outputs": [], "source": "import numpy as np\nimport pandas as pd\n\n%matplotlib inline\nimport matplotlib.pyplot as plt\nimport matplotlib.cm as cm\n\nimport tensorflow as tf\n\n# settings\nLEARNING_RATE = 1e-4\n# set to 20000 on local environment to get 0.99 accuracy\nTRAINING_ITERATIONS = 2500 \n \nDROPOUT = 0.5\nBATCH_SIZE = 50\n\n# set to 0 to train on all available data\nVALIDATION_SIZE = 2000\n\n# image number to output\nIMAGE_TO_DISPLAY = 10", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "cb76a020-d2c8-4ddd-a8cd-6e5086fe2984", "_active": false }, "source": "## Data preparation\nTo start, we read provided data. The *train.csv* file contains 42000 rows and 785 columns. Each row represents an image of a handwritten digit and a label with the value of this digit.", "outputs": [] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false, "_cell_guid": "644f11e0-a25b-479c-9f9d-620c6765994e", "_active": false }, "outputs": [], "source": "# read training data from CSV file \ndata = pd.read_csv('../input/train.csv')\n\nprint('data({0[0]},{0[1]})'.format(data.shape))\nprint (data.head())", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "a2450a7f-15b8-4fdf-9e9c-4ab253eee86b", "_active": false }, "source": "Every image is a \"stretched\" array of pixel values.", "outputs": [] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false, "_cell_guid": "5b286468-0900-4f09-8990-372c4caf7b3c", "_active": false }, "outputs": [], "source": "images = data.iloc[:,1:].values\nimages = images.astype(np.float)\n\n# convert from [0:255] => [0.0:1.0]\nimages = np.multiply(images, 1.0 / 255.0)\n\nprint('images({0[0]},{0[1]})'.format(images.shape))", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "0a646648-ca8e-4322-b1eb-a24baf2a4665", "_active": false }, "source": "In this case it's 784 pixels => 28 * 28px", "outputs": [] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false, "_cell_guid": "021f4400-2b1e-454f-9fda-bbc68b279cfe", "_active": false }, "outputs": [], "source": "image_size = images.shape[1]\nprint ('image_size => {0}'.format(image_size))\n\n# in this case all images are square\nimage_width = image_height = np.ceil(np.sqrt(image_size)).astype(np.uint8)\n\nprint ('image_width => {0}\\nimage_height => {1}'.format(image_width,image_height))", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "03fdbf45-432b-4028-a376-5cf557dc6e1b", "_active": false }, "source": "To output one of the images, we reshape this long string of pixels into a 2-dimensional array, which is basically a grayscale image.", "outputs": [] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false, "_cell_guid": "89475ddd-08ca-47f2-b0e5-444cd9e41fba", "_active": false }, "outputs": [], "source": "# display image\ndef display(img):\n \n # (784) => (28,28)\n one_image = img.reshape(image_width,image_height)\n \n plt.axis('off')\n plt.imshow(one_image, cmap=cm.binary)\n\n# output image \ndisplay(images[IMAGE_TO_DISPLAY])", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "d8853a1c-bdc1-488b-8924-f9f4d565fd13", "_active": false }, "source": "The corresponding labels are numbers between 0 and 9, describing which digit a given image is of.", "outputs": [] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": false, "_cell_guid": "18b09276-b2db-484c-be7b-ac55aa572182", "_active": false }, "outputs": [], "source": "labels_flat = data[[0]].values.ravel()\n\nprint('labels_flat({0})'.format(len(labels_flat)))\nprint ('labels_flat[{0}] => {1}'.format(IMAGE_TO_DISPLAY,labels_flat[IMAGE_TO_DISPLAY]))", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "10e01d44-e5df-404a-9081-cf020410de3b", "_active": false }, "source": "In this case, there are ten different digits/labels/classes.", "outputs": [] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": false, "_cell_guid": "6d110c38-9008-4a28-bde2-b21974a987c4", "_active": false }, "outputs": [], "source": "labels_count = np.unique(labels_flat).shape[0]\n\nprint('labels_count => {0}'.format(labels_count))", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "8471b3bd-a115-4102-96ea-3ee4a6dea50d", "_active": false }, "source": "For most classification problems \"one-hot vectors\" are used. A one-hot vector is a vector that contains a single element equal to 1 and the rest of the elements equal to 0. In this case, the *nth* digit is represented as a zero vector with 1 in the *nth* position.", "outputs": [] }, { "cell_type": "code", "execution_count": 8, "metadata": { "collapsed": false, "_cell_guid": "ed9cd050-070a-4193-b9f0-605caaa88d42", "_active": false }, "outputs": [], "source": "# convert class labels from scalars to one-hot vectors\n# 0 => [1 0 0 0 0 0 0 0 0 0]\n# 1 => [0 1 0 0 0 0 0 0 0 0]\n# ...\n# 9 => [0 0 0 0 0 0 0 0 0 1]\ndef dense_to_one_hot(labels_dense, num_classes):\n num_labels = labels_dense.shape[0]\n index_offset = np.arange(num_labels) * num_classes\n labels_one_hot = np.zeros((num_labels, num_classes))\n labels_one_hot.flat[index_offset + labels_dense.ravel()] = 1\n return labels_one_hot\n\nlabels = dense_to_one_hot(labels_flat, labels_count)\nlabels = labels.astype(np.uint8)\n\nprint('labels({0[0]},{0[1]})'.format(labels.shape))\nprint ('labels[{0}] => {1}'.format(IMAGE_TO_DISPLAY,labels[IMAGE_TO_DISPLAY]))", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "d7a7bd80-707f-4b9d-a2a3-e6ddd8545188", "_active": false }, "source": "Lastly we set aside data for validation. It's essential in machine learning to have a separate dataset which doesn't take part in the training and is used to make sure that what we've learned can actually be generalised.", "outputs": [] }, { "cell_type": "code", "execution_count": 9, "metadata": { "collapsed": false, "_cell_guid": "50e5644b-382e-4758-aa74-f73a768b8d4d", "_active": false }, "outputs": [], "source": "# split data into training & validation\nvalidation_images = images[:VALIDATION_SIZE]\nvalidation_labels = labels[:VALIDATION_SIZE]\n\ntrain_images = images[VALIDATION_SIZE:]\ntrain_labels = labels[VALIDATION_SIZE:]\n\n\nprint('train_images({0[0]},{0[1]})'.format(train_images.shape))\nprint('validation_images({0[0]},{0[1]})'.format(validation_images.shape))", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "4bd1b546-86a7-41b3-abce-2069433a7067", "_active": false }, "source": "*Data is ready. The neural network structure is next.*\n## TensorFlow graph\nTensorFlow does its heavy lifting outside Python. Therefore, instead of running every single operation independently, TensorFlow allows users to build a whole graph of interacting operations and then runs the workflow in a separate process at once.\n#### Helper functions\nFor this NN model, a lot of weights and biases are created. Generally, weights should be initialised with a small amount of noise for symmetry breaking, and to prevent 0 gradients. \n\nSince we are using [ReLU](https://en.wikipedia.org/wiki/Rectifier_(neural_networks) neurones (ones that contain rectifier function *f(x)=max(0,x)*), it is also good practice to initialise them with a slightly positive initial bias to avoid \"dead neurones\".", "outputs": [] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": false, "_cell_guid": "0aaccbda-07b6-4fe3-849b-63028c07211b", "_active": false }, "outputs": [], "source": "# weight initialization\ndef weight_variable(shape):\n initial = tf.truncated_normal(shape, stddev=0.1)\n return tf.Variable(initial)\n\ndef bias_variable(shape):\n initial = tf.constant(0.1, shape=shape)\n return tf.Variable(initial)", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "ef437f5f-2a5b-411f-9f17-a9fdb10a7986", "_active": false }, "source": "For this problem we use zero padded [convolutions](https://en.wikipedia.org/wiki/Convolutional_neural_network#Convolutional_layer) so that the output is the same size as the input. Stride/step in this case is equal to 1.\n\nIn general, convolution layer is used to get the features of the data. In the case of digit recognition - a shape of each digit. It uses learnable kernels/filters each of which corresponds to one particular shape pattern. The number of the filter can differ for other problems.", "outputs": [] }, { "cell_type": "code", "execution_count": 11, "metadata": { "collapsed": false, "_cell_guid": "56183d2a-6520-4a34-91c4-821bc422c757", "_active": false }, "outputs": [], "source": "# convolution\ndef conv2d(x, W):\n return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "db14b194-762d-48d6-a8cf-e6ea6524c570", "_active": false }, "source": "[Pooling](https://en.wikipedia.org/wiki/Convolutional_neural_network#Pooling_layer) is plain max pooling over 2x2 blocks.\n\nPooling is used for downsampling of the data. 2x2 max-pooling splits the image into square 2-pixel blocks and only keeps maximum value for each of those blocks. ", "outputs": [] }, { "cell_type": "code", "execution_count": 12, "metadata": { "collapsed": false, "_cell_guid": "73626dfb-9c94-45f4-b03a-5ed822a90171", "_active": false }, "outputs": [], "source": "# pooling\n# [[0,3],\n# [4,2]] => 4\n\n# [[0,1],\n# [1,1]] => 1\n\ndef max_pool_2x2(x):\n return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "35117654-ebc2-4125-9d33-53be250be6a5", "_active": false }, "source": "*We'll get back to convolutions and pooling in more detail below.*\n\nThe good thing about neural networks that any NN can be used as a layer in a large multilayer NN meaning that output of one can be used as input for another. This sequential approach can create very sophisticated NN with multiple layers. They are also called Deep Neural Networks.\n\nIn this case, we use two convolution layers with pooling in between them, then densely connected layer followed by dropout and lastly readout layer.", "outputs": [] }, { "cell_type": "code", "execution_count": 13, "metadata": { "collapsed": false, "_cell_guid": "fbe5eb08-65a8-4d9e-97ea-6ce4360c88dd", "_active": false }, "outputs": [], "source": "# input & output of NN\n\n# images\nx = tf.placeholder('float', shape=[None, image_size])\n# labels\ny_ = tf.placeholder('float', shape=[None, labels_count])", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "32c848f7-79a7-4990-9b91-060ea32907ba", "_active": false }, "source": "The first layer is a convolution, followed by max pooling. The convolution computes 32 features for each 5x5 patch. Its weight tensor has a shape of [5, 5, 1, 32]. The first two dimensions are the patch size, the next is the number of input channels (1 means that images are grayscale), and the last is the number of output channels. There is also a bias vector with a component for each output channel.\n\nTo apply the layer, we reshape the input data to a 4d tensor, with the first dimension corresponding to the number of images, second and third - to image width and height, and the final dimension - to the number of colour channels.\n\nAfter the convolution, pooling reduces the size of the output from 28x28 to 14x14.", "outputs": [] }, { "cell_type": "code", "execution_count": 14, "metadata": { "collapsed": false, "_cell_guid": "fa77ca6c-0e8d-474b-bdbd-be8900c5e7ba", "_active": false }, "outputs": [], "source": "# first convolutional layer\nW_conv1 = weight_variable([5, 5, 1, 32])\nb_conv1 = bias_variable([32])\n\n# (40000,784) => (40000,28,28,1)\nimage = tf.reshape(x, [-1,image_width , image_height,1])\n#print (image.get_shape()) # =>(40000,28,28,1)\n\n\nh_conv1 = tf.nn.relu(conv2d(image, W_conv1) + b_conv1)\n#print (h_conv1.get_shape()) # => (40000, 28, 28, 32)\nh_pool1 = max_pool_2x2(h_conv1)\n#print (h_pool1.get_shape()) # => (40000, 14, 14, 32)\n\n\n# Prepare for visualization\n# display 32 fetures in 4 by 8 grid\nlayer1 = tf.reshape(h_conv1, (-1, image_height, image_width, 4 ,8)) \n\n# reorder so the channels are in the first dimension, x and y follow.\nlayer1 = tf.transpose(layer1, (0, 3, 1, 4,2))\n\nlayer1 = tf.reshape(layer1, (-1, image_height*4, image_width*8)) ", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "b487d468-2389-4513-9698-71647b6100e3", "_active": false }, "source": "The second layer has 64 features for each 5x5 patch. Its weight tensor has a shape of [5, 5, 32, 64]. The first two dimensions are the patch size, the next is the number of input channels (32 channels correspond to 32 featured that we got from previous convolutional layer), and the last is the number of output channels. There is also a bias vector with a component for each output channel.\n\nBecause the image is down-sampled by pooling to 14x14 size second convolutional layer picks up more general characteristics of the images. Filters cover more space of the picture. Therefore, it is adjusted for more generic features while the first layer finds smaller details.", "outputs": [] }, { "cell_type": "code", "execution_count": 15, "metadata": { "collapsed": false, "_cell_guid": "11e62c22-06d4-4b97-9565-5c553f71a1e6", "_active": false }, "outputs": [], "source": "# second convolutional layer\nW_conv2 = weight_variable([5, 5, 32, 64])\nb_conv2 = bias_variable([64])\n\nh_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)\n#print (h_conv2.get_shape()) # => (40000, 14,14, 64)\nh_pool2 = max_pool_2x2(h_conv2)\n#print (h_pool2.get_shape()) # => (40000, 7, 7, 64)\n\n# Prepare for visualization\n# display 64 fetures in 4 by 16 grid\nlayer2 = tf.reshape(h_conv2, (-1, 14, 14, 4 ,16)) \n\n# reorder so the channels are in the first dimension, x and y follow.\nlayer2 = tf.transpose(layer2, (0, 3, 1, 4,2))\n\nlayer2 = tf.reshape(layer2, (-1, 14*4, 14*16)) ", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "9cec1b97-5d1d-41c6-9f66-3c428fd821b7", "_active": false }, "source": "Now that the image size is reduced to 7x7, we add a [fully-connected layer](https://en.wikipedia.org/wiki/Convolutional_neural_network#Fully_Connected_layer) with 1024 neurones to allow processing on the entire image (each of the neurons of the fully connected layer is connected to all the activations/outpus of the previous layer)", "outputs": [] }, { "cell_type": "code", "execution_count": 16, "metadata": { "collapsed": false, "_cell_guid": "c044c176-76fa-40ca-9560-3364bc820a43", "_active": false }, "outputs": [], "source": "# densely connected layer\nW_fc1 = weight_variable([7 * 7 * 64, 1024])\nb_fc1 = bias_variable([1024])\n\n# (40000, 7, 7, 64) => (40000, 3136)\nh_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])\n\nh_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)\n#print (h_fc1.get_shape()) # => (40000, 1024)", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "f29888a9-f767-40b9-94c9-cfae1ca5675e", "_active": false }, "source": "To prevent overfitting, we apply [dropout](https://en.wikipedia.org/wiki/Convolutional_neural_network#Dropout) before the readout layer.\n\nDropout removes some nodes from the network at each training stage. Each of the nodes is either kept in the network with probability *keep_prob* or dropped with probability *1 - keep_prob*. After the training stage is over the nodes are returned to the NN with their original weights.", "outputs": [] }, { "cell_type": "code", "execution_count": 17, "metadata": { "collapsed": false, "_cell_guid": "d1d72fb6-00e7-4b49-a12a-58b751524b06", "_active": false }, "outputs": [], "source": "# dropout\nkeep_prob = tf.placeholder('float')\nh_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "42c37c5c-6dfd-48ee-9a09-f079fcdc8d1e", "_active": false }, "source": "Finally, we add a softmax layer, the same one if we use just a simple [softmax regression](https://en.wikipedia.org/wiki/Softmax_function).", "outputs": [] }, { "cell_type": "code", "execution_count": 18, "metadata": { "collapsed": false, "_cell_guid": "f8428f96-47b5-4cdc-acd9-cbb798e9e546", "_active": false }, "outputs": [], "source": "# readout layer for deep net\nW_fc2 = weight_variable([1024, labels_count])\nb_fc2 = bias_variable([labels_count])\n\ny = tf.nn.softmax(tf.matmul(h_fc1_drop, W_fc2) + b_fc2)\n\n#print (y.get_shape()) # => (40000, 10)", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "cc181890-65f3-4939-8f0c-cd7621e3f9a0", "_active": false }, "source": "To evaluate network performance we use [cross-entropy](https://en.wikipedia.org/wiki/Cross_entropy) and to minimise it [ADAM optimiser](http://arxiv.org/pdf/1412.6980v8.pdf) is used. \n\nADAM optimiser is a gradient based optimization algorithm, based on adaptive estimates, it's more sophisticated than steepest gradient descent and is well suited for problems with large data or many parameters.", "outputs": [] }, { "cell_type": "code", "execution_count": 19, "metadata": { "collapsed": false, "_cell_guid": "53d9cf6a-f69e-4aa3-bb42-99f53ab08448", "_active": false }, "outputs": [], "source": "# cost function\ncross_entropy = -tf.reduce_sum(y_*tf.log(y))\n\n\n# optimisation function\ntrain_step = tf.train.AdamOptimizer(LEARNING_RATE).minimize(cross_entropy)\n\n# evaluation\ncorrect_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))\n\naccuracy = tf.reduce_mean(tf.cast(correct_prediction, 'float'))", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "5f6f872e-603a-49ea-8684-9f62eb3ff2b9", "_active": false }, "source": "To predict values from test data, highest probability is picked from \"one-hot vector\" indicating that chances of an image being one of the digits are highest.", "outputs": [] }, { "cell_type": "code", "execution_count": 20, "metadata": { "collapsed": false, "_cell_guid": "b3ce64c7-9363-46f9-8ff5-a05e75fcda7e", "_active": false }, "outputs": [], "source": "# prediction function\n#[0.1, 0.9, 0.2, 0.1, 0.1 0.3, 0.5, 0.1, 0.2, 0.3] => 1\npredict = tf.argmax(y,1)", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "af8e5f0e-dba6-43af-a7c1-bdaaf30689e8", "_active": false }, "source": "*Finally neural network structure is defined and TensorFlow graph is ready for training.*\n## Train, validate and predict\n#### Helper functions\n\nIdeally, we should use all data for every step of the training, but that's expensive. So, instead, we use small \"batches\" of random data. \n\nThis method is called [stochastic training](https://en.wikipedia.org/wiki/Stochastic_gradient_descent). It is cheaper, faster and gives much of the same result.", "outputs": [] }, { "cell_type": "code", "execution_count": 21, "metadata": { "collapsed": false, "_cell_guid": "2b37ba55-0c97-4992-82b2-2095ba56a4cc", "_active": false }, "outputs": [], "source": "epochs_completed = 0\nindex_in_epoch = 0\nnum_examples = train_images.shape[0]\n\n# serve data by batches\ndef next_batch(batch_size):\n \n global train_images\n global train_labels\n global index_in_epoch\n global epochs_completed\n \n start = index_in_epoch\n index_in_epoch += batch_size\n \n # when all trainig data have been already used, it is reorder randomly \n if index_in_epoch > num_examples:\n # finished epoch\n epochs_completed += 1\n # shuffle the data\n perm = np.arange(num_examples)\n np.random.shuffle(perm)\n train_images = train_images[perm]\n train_labels = train_labels[perm]\n # start next epoch\n start = 0\n index_in_epoch = batch_size\n assert batch_size <= num_examples\n end = index_in_epoch\n return train_images[start:end], train_labels[start:end]", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "0ddee95b-c86c-4774-b985-e764b68f7b16", "_active": false }, "source": "Now when all operations for every variable are defined in TensorFlow graph all computations will be performed outside Python environment.", "outputs": [] }, { "cell_type": "code", "execution_count": 24, "metadata": { "collapsed": false, "_cell_guid": "2a3968ee-6712-45f8-858f-f14cafed1d65", "_active": false }, "outputs": [], "source": "# start TensorFlow session\n#init = tf.initialize_all_variables()\ninit = tf.global_variables_initializer()\nsess = tf.InteractiveSession()\n\nsess.run(init)", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "1dc97fe5-e7db-4c20-af00-27ba38e83160", "_active": false }, "source": "Each step of the loop, we get a \"batch\" of data points from the training set and feed it to the graph to replace the placeholders. In this case, it's: *x, y* and *dropout.*\n\nAlso, once in a while, we check training accuracy on an upcoming \"batch\".\n\nOn the local environment, we recommend [saving training progress](https://www.tensorflow.org/versions/master/api_docs/python/state_ops.html#Saver), so it can be recovered for further training, debugging or evaluation.", "outputs": [] }, { "cell_type": "code", "execution_count": 25, "metadata": { "collapsed": false, "_cell_guid": "406bd2d2-2bc3-416e-be1b-d9a92bca70b5", "_active": false }, "outputs": [], "source": "# visualisation variables\ntrain_accuracies = []\nvalidation_accuracies = []\nx_range = []\n\ndisplay_step=1\n\nfor i in range(TRAINING_ITERATIONS):\n\n #get new batch\n batch_xs, batch_ys = next_batch(BATCH_SIZE) \n\n # check progress on every 1st,2nd,...,10th,20th,...,100th... step\n if i%display_step == 0 or (i+1) == TRAINING_ITERATIONS:\n \n train_accuracy = accuracy.eval(feed_dict={x:batch_xs, \n y_: batch_ys, \n keep_prob: 1.0}) \n if(VALIDATION_SIZE):\n validation_accuracy = accuracy.eval(feed_dict={ x: validation_images[0:BATCH_SIZE], \n y_: validation_labels[0:BATCH_SIZE], \n keep_prob: 1.0}) \n print('training_accuracy / validation_accuracy => %.2f / %.2f for step %d'%(train_accuracy, validation_accuracy, i))\n \n validation_accuracies.append(validation_accuracy)\n \n else:\n print('training_accuracy => %.4f for step %d'%(train_accuracy, i))\n train_accuracies.append(train_accuracy)\n x_range.append(i)\n \n # increase display_step\n if i%(display_step*10) == 0 and i:\n display_step *= 10\n # train on batch\n sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys, keep_prob: DROPOUT})", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "6ad525f8-fa02-422b-b2e8-0bbe56615f1b", "_active": false }, "source": "After training is done, it's good to check accuracy on data that wasn't used in training.", "outputs": [] }, { "cell_type": "code", "execution_count": 26, "metadata": { "collapsed": false, "_cell_guid": "fa9eb858-e45c-4252-8c8e-119f2d66f563", "_active": false }, "outputs": [], "source": "# check final accuracy on validation set \nif(VALIDATION_SIZE):\n validation_accuracy = accuracy.eval(feed_dict={x: validation_images, \n y_: validation_labels, \n keep_prob: 1.0})\n print('validation_accuracy => %.4f'%validation_accuracy)\n plt.plot(x_range, train_accuracies,'-b', label='Training')\n plt.plot(x_range, validation_accuracies,'-g', label='Validation')\n plt.legend(loc='lower right', frameon=False)\n plt.ylim(ymax = 1.1, ymin = 0.7)\n plt.ylabel('accuracy')\n plt.xlabel('step')\n plt.show()", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "cd19ca67-7684-470f-a988-43fd28b24da1", "_active": false }, "source": "When, we're happy with the outcome, we read test data from *test.csv* and predict labels for provided images.\n\nTest data contains only images and labels are missing. Otherwise, the structure is similar to training data.\n\nPredicted labels are stored into CSV file for future submission.", "outputs": [] }, { "cell_type": "code", "execution_count": 27, "metadata": { "collapsed": false, "_cell_guid": "b28bf251-fa4d-4a36-bc0a-0b0b736a2cf8", "_active": false }, "outputs": [], "source": "# read test data from CSV file \ntest_images = pd.read_csv('../input/test.csv').values\ntest_images = test_images.astype(np.float)\n\n# convert from [0:255] => [0.0:1.0]\ntest_images = np.multiply(test_images, 1.0 / 255.0)\n\nprint('test_images({0[0]},{0[1]})'.format(test_images.shape))\n\n\n# predict test set\n#predicted_lables = predict.eval(feed_dict={x: test_images, keep_prob: 1.0})\n\n# using batches is more resource efficient\npredicted_lables = np.zeros(test_images.shape[0])\nfor i in range(0,test_images.shape[0]//BATCH_SIZE):\n predicted_lables[i*BATCH_SIZE : (i+1)*BATCH_SIZE] = predict.eval(feed_dict={x: test_images[i*BATCH_SIZE : (i+1)*BATCH_SIZE], \n keep_prob: 1.0})\n\n\nprint('predicted_lables({0})'.format(len(predicted_lables)))\n\n# output test image and prediction\ndisplay(test_images[IMAGE_TO_DISPLAY])\nprint ('predicted_lables[{0}] => {1}'.format(IMAGE_TO_DISPLAY,predicted_lables[IMAGE_TO_DISPLAY]))\n\n# save results\nnp.savetxt('submission_softmax.csv', \n np.c_[range(1,len(test_images)+1),predicted_lables], \n delimiter=',', \n header = 'ImageId,Label', \n comments = '', \n fmt='%d')", "execution_state": "idle" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "b7cbcf6e-cd08-4b4f-89b0-779f09a9f6ee", "_active": false }, "source": "## Appendix\nAs it was mentioned before, it is good to output some variables for a better understanding of the process. \n\nHere we pull an output of the first convolution layer from TensorFlow graph. 32 features are transformed into an image grid, and it's quite interesting to see how filters picked by NN outline characteristics of different digits.", "outputs": [] }, { "cell_type": "code", "execution_count": 28, "metadata": { "collapsed": false, "_cell_guid": "69fedd7e-1eb4-442a-8fc3-6f6b24fb1de2", "_active": false }, "outputs": [], "source": "layer1_grid = layer1.eval(feed_dict={x: test_images[IMAGE_TO_DISPLAY:IMAGE_TO_DISPLAY+1], keep_prob: 1.0})\nplt.axis('off')\nplt.imshow(layer1_grid[0], cmap=cm.seismic )", "execution_state": "idle" }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "_cell_guid": "672ec2cc-9da0-4f80-8db5-2b2707cc4e35", "_active": false }, "outputs": [], "source": "sess.close()" }, { "cell_type": "markdown", "metadata": { "_cell_guid": "060e3e6a-2600-416c-afad-c546c755f785", "_active": false }, "source": "## Reference\n- [Deep MNIST for Experts](https://www.tensorflow.org/versions/master/tutorials/mnist/pros/index.html#deep-mnist-for-experts)\n- [A Convolutional Network implementation example using TensorFlow library](https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3%20-%20Neural%20Networks/convolutional_network.ipynb)\n- [Digit recognizer in Python using CNN](https://www.kaggle.com/kobakhit/digit-recognizer/digit-recognizer-in-python-using-cnn)\n- [Deep Learning in a Nutshell: Core Concepts](http://devblogs.nvidia.com/parallelforall/deep-learning-nutshell-core-concepts/)", "outputs": [] } ] }