{"cells":[{"metadata":{},"cell_type":"markdown","source":"![](https://cdn.analyticsvidhya.com/wp-content/uploads/2020/02/Comp-1.gif)"},{"metadata":{},"cell_type":"markdown","source":"# Setup"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"pip install mathutils","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"#Import Libraries\nimport numpy as np\nimport math\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom pylab import rcParams\n\nfrom preprocessing import *\nfrom mathutils import *\n\n%matplotlib inline\n\nsns.set(style='whitegrid')\n\nrcParams['figure.figsize'] = 12, 6\n\nRANDOM_SEED = 42\n\nnp.random.seed(RANDOM_SEED)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"![](https://s3.amazonaws.com/assets.datacamp.com/blog_assets/Keras+Python+Tutorial/content_content_neuron.png)"},{"metadata":{},"cell_type":"markdown","source":"# Let us start by answering this key question: What is a neural network? \nIt is a biologically-inspired method of building computer programs that are able to learn and independently find connections in data. As Figure shows, nets are a collection of software ‘neurons’ arranged in layers, connected together in a way that allows communication.\n\n# Single neuron\n\nEach neuron receives a set of x-values (numbered from 1 to n) as an input and compute the predicted y-hat value. Vector x actually contains the values of the features in one of m examples from the training set. What is more each of units has its own set of parameters, usually referred to as w (column vector of weights) and b (bias) which changes during the learning process. In each iteration, the neuron calculates a weighted average of the values of the vector x, based on its current weight vector w and adds bias. Finally, the result of this calculation is passed through a non-linear activation function g. I will mention a bit about the most popular activation functions in the following part of the article."},{"metadata":{},"cell_type":"markdown","source":"![](https://hackernoon.com/drafts/e8m3z48.png)"},{"metadata":{},"cell_type":"markdown","source":"# Perceptrons \nInvented by Frank Rosenblatt in 1957, are the simplest neural network that consist of n number of inputs, only one neuron and one output, where n is the number of features of our dataset. The process of passing the data through the neural network is know as forward propagation and the forward propagation carried out in a Perceptron is explained in the following three steps:\n\n**Step 1** : For each input, multiply the input value xᵢ with weights wᵢ and sum all the multiplied values. Weights — represent the strength of the connection between neurons and decides how much influence the given input will have on the neuron’s output. If the weight w₁ has higher value than the weight w₂, then the input x₁ will have higher influence on the output than w₂.\n\n![](https://hackernoon.com/photos/0s78blBiawOe4UYlnA9SeCIgjbA3-cjpc3zpj)\n\nThe row vectors of the inputs and weights are x = [x₁, x₂, … , xₙ] and w =[w₁, w₂, … , wₙ] respectively and their dot product is given by\n\n![](https://hackernoon.com/photos/0s78blBiawOe4UYlnA9SeCIgjbA3-tas83zhq)\n\nHence, the summation is equal to the dot product of the vectors x and w\n\n![](https://hackernoon.com/photos/0s78blBiawOe4UYlnA9SeCIgjbA3-h4ud3zb3)\n\n**Step 2** : Add bias b to the summation of multiplied values and let’s call this z. Bias — also know as offset is necessary in most of the cases, to move the entire activation function to the left or right to generate the required output values .\n\n![](https://hackernoon.com/photos/0s78blBiawOe4UYlnA9SeCIgjbA3-gixa3z5p)\n\n**Step 3** : Pass the value of z to a non-linear activation function. Activation functions — are used to introduce non-linearity into the output of the neurons, without which the neural network will just be a linear function. Moreover, they have a significant impact on the learning speed of the neural network. Perceptrons have binary step function as their activation function. However, we shall use Sigmoid — also know as logistic function as our activation function.\n\n![](https://hackernoon.com/photos/0s78blBiawOe4UYlnA9SeCIgjbA3-i3y23z0r)\n\nwhere, σ denotes the Sigmoid activation function and the output we get after the forward prorogation is know as the predicted value ŷ."},{"metadata":{},"cell_type":"markdown","source":"# Learning Algorithm\nThe learning algorithm consist of two parts — Backpropagation and Optimization.\nBackpropagation : Backpropagation, short for backward propagation of errors, refers to the algorithm for computing the gradient of the loss function with respect to the weights. However, the term is often used to refer to the entire learning algorithm. The backpropagation carried out in a Perceptron is explained in the following two steps: \n\n**Step 1** : To know an estimation of how far are we from our desired solution a loss function is used. Generally, Mean Squared Error is chosen as loss function for regression problems and cross entropy for classification problems. Let’s take a regression problem and its loss function be Mean Squared Error, which squares the difference between actual (yᵢ) and predicted value ( ŷᵢ ).\n\n![](https://hackernoon.com/photos/0s78blBiawOe4UYlnA9SeCIgjbA3-v2z33z6t)\n\nLoss function is calculated for the entire training dataset and their average is called the Cost function C.\n\n![](https://hackernoon.com/photos/0s78blBiawOe4UYlnA9SeCIgjbA3-uy1263zz5)"},{"metadata":{},"cell_type":"markdown","source":"**Step 2** : In order to find the best weights and bias for our Perceptron, we need to know how the cost function changes in relation to weights and bias. This is done with the help the gradients (rate of change) — how one quantity changes in relation to another quantity. In our case, we need to find the gradient of the cost function with respect to the weights and bias.\nLet’s calculate the gradient of cost function C with respect to the weight wᵢ using partial derivation. Since the cost function is not directly related to the weight wᵢ, let’s use the chain rule.\n\n![](https://hackernoon.com/photos/0s78blBiawOe4UYlnA9SeCIgjbA3-9j13t3zkz)\n\nNow we need to find the following three gradients\n\n![](https://hackernoon.com/photos/0s78blBiawOe4UYlnA9SeCIgjbA3-y014q3zg9)\n\nLet’s start with the gradient of the Cost function (C) with respect to the predicted value ( ŷ )\n\n![](https://hackernoon.com/photos/0s78blBiawOe4UYlnA9SeCIgjbA3-wp15t3za8)\n\nLet y = [y₁ , y₂ , … yₙ] and ŷ =[ ŷ₁ , ŷ₂ , … ŷₙ] be the row vectors of actual and predicted values. Hence the above equation is simplifies as\n\n![](https://hackernoon.com/photos/0s78blBiawOe4UYlnA9SeCIgjbA3-y916m3zu2)\n\nNow let’s find the the gradient of the predicted value with respect to the z. This will be a bit lengthy.\n\n![](https://hackernoon.com/photos/0s78blBiawOe4UYlnA9SeCIgjbA3-731773zg4)\n\nThe gradient of z with respect to the weight wᵢ is\n\n![](https://hackernoon.com/photos/0s78blBiawOe4UYlnA9SeCIgjbA3-gp1863zl1)\n\nTherefore we get,\n\n![](https://hackernoon.com/photos/0s78blBiawOe4UYlnA9SeCIgjbA3-wf18z3zw3)\n\nWhat about Bias? — Bias is theoretically considered to have an input of constant value 1. Hence,\n\n![](https://hackernoon.com/photos/0s78blBiawOe4UYlnA9SeCIgjbA3-p819q3z83)"},{"metadata":{},"cell_type":"markdown","source":"# Optimization : \nOptimization is the selection of a best element from some set of available alternatives, which in our case, is the selection of best weights and bias of the perceptron. Let’s choose gradient descent as our optimization algorithm, which changes the weights and bias, proportional to the negative of the gradient of the Cost function with respect to the corresponding weight or bias. Learning rate (α) is a hyperparameter which is used to control how much the weights and bias are changed.\n\nThe weights and bias are updated as follows and the Backporpagation and gradient descent is repeated until convergence.\n\n![](https://hackernoon.com/photos/0s78blBiawOe4UYlnA9SeCIgjbA3-sh1ah3z3v)"},{"metadata":{},"cell_type":"markdown","source":"# Single layer\n\nNow let’s zoom out a little and consider how calculations are performed for a whole layer of the neural network. We will use our knowledge of what is happening inside a single unit and vectorize across full layer to combine those calculations in into matrix equations. To unify the notation, the equations will be written for the selected layer. By the way, subscript i mark the index of a neuron in that layer.\n\n![](https://miro.medium.com/max/1400/1*uj8zOKLfcsyba3wjqQeF3w.png)"},{"metadata":{},"cell_type":"markdown","source":"# One more important remark: \n\nWhen we wrote the equations for a single unit, we used x and y-hat, which were respectively the column vector of features and the predicted value. When switching to the general notation for layer, we use the vector a — meaning the activation of the corresponding layer. The x vector is therefore the activation for layer 0 — input layer. Each neuron in the layer performs a similar calculation according to the following equations:\n\n![](https://miro.medium.com/max/626/1*EFl7IyV_w_oNB_capLSqmw.gif)\n\nFor the sake of clarity, let’s write down the equations for example for layer 2:\n\n![](https://miro.medium.com/max/630/1*yPUUwhGKd1jIU_nmDRIz6Q.gif)"},{"metadata":{},"cell_type":"markdown","source":"As you can see, for each of the layers we have to perform a number of very similar operations. Using for-loop for this purpose is not very efficient, so to speed up the calculation we will use vectorization. First of all, by stacking together horizontal vectors of weights w (transposed) we will build matrix W. Similarly, we will stack together bias of each neuron in the layer creating vertical vector b. Now there is nothing to stop us from building a single matrix equations that allows us to perform calculations for all the neurons of the layer at once. Let’s also write down the dimensions of the matrices and vectors we have used.\n\n![](https://miro.medium.com/max/1400/1*wab5q35QBAxZ-4sMGKxcFA.png)\n\n![](https://miro.medium.com/max/660/1*UmrwDSFuBv2XMrf6lY8dww.gif)\n\n![](https://miro.medium.com/max/1400/1*TH4MWXi0QPEF2eKrcaWZkw.png)"},{"metadata":{},"cell_type":"markdown","source":"# Vectorizing across multiple examples\n\nThe equation that we have drawn up so far involves only one example. During the learning process of a neural network, you usually work with huge sets of data, up to millions of entries. The next step will therefore be vectorisation across multiple examples. Let’s assume that our data set has m entries with nx features each. First of all, we will put together the vertical vectors x, a, and z of each layer creating the X, A and Z matrices, respectively. Then we rewrite the previously laid-out equation, taking into account the newly created matrices.\n\n![](https://miro.medium.com/max/1400/1*95uERWPdhqRSmnLzWurZ_w.png)\n\n![](https://miro.medium.com/max/674/1*RA5I9ZG5Lsaj2W40o15omg.gif)"},{"metadata":{},"cell_type":"markdown","source":"# What is activation function and why do we need it?\n\nActivation functions are one of the key elements of the neural network. Without them, our neural network would become a combination of linear functions, so it would be just a linear function itself. Our model would have limited expansiveness, no greater than logistic regression. The non-linearity element allows for greater flexibility and creation of complex functions during the learning process. The activation function also has a significant impact on the speed of learning, which is one of the main criteria for their selection. Figure 6 shows some of the commonly used activation functions. Currently, the most popular one for hidden layers is probably ReLU. We still sometimes use sigmoid, especially in the output layer, when we are dealing with a binary classification and we want the values returned from the model to be in the range from 0 to 1.\n\n![](https://miro.medium.com/max/1400/1*Lj65eT-MaWtXKPpwEaE24g.png)"},{"metadata":{},"cell_type":"markdown","source":"# Sigmoid (and it’s derivative)\n\nThe sigmoid function is used quite commonly in the realm of deep learning, at least it was until recently. It has distinct S shape and it is a differentiable real function for any real input value. Additionally, it has a positive derivative at each point. More importantly, we will use it as an activation function for the hidden layer of our model. Here’s how it is defined:"},{"metadata":{},"cell_type":"markdown","source":"![](https://miro.medium.com/max/556/1*-PkQzu0E21YEbI9wvqvQzg.png)"},{"metadata":{},"cell_type":"markdown","source":"It’s first derivative (which we will use during the backpropagation step of our training algorithm) has the following formula:"},{"metadata":{},"cell_type":"markdown","source":"![](https://miro.medium.com/max/910/1*yT0ToBoL4o9eTgph6BWx4Q.png)"},{"metadata":{},"cell_type":"markdown","source":"So, the derivative can be expressed using the original sigmoid function. Pretty cool, right? Don’t like formulas? Let’s look at a picture:"},{"metadata":{},"cell_type":"markdown","source":"![](https://i.stack.imgur.com/inMoa.png)"},{"metadata":{},"cell_type":"markdown","source":"The derivative shows us the rate of change of a function. We can use it to determine the “slope” of that function. The highest rate of change for the sigmoid function is when x=0x=0, as it is evident from the derivative graph (in green)."},{"metadata":{},"cell_type":"markdown","source":"# Softmax\n\nThe softmax function can be easily differentiated, it is pure (output depends only on input) and the elements of the resulting vector sum to 1. \nHere it is:"},{"metadata":{},"cell_type":"markdown","source":"![](https://miro.medium.com/max/1146/1*gplkMOHmezbBphrtC0HbRQ.png)"},{"metadata":{},"cell_type":"markdown","source":"In probability theory, the output of the softmax function is sometimes used as a representation of a categorical distribution. Let’s see an example result:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def softmax(x):\n \"\"\"Compute softmax values for each sets of scores in x.\"\"\"\n e_x = np.exp(x - np.max(x))\n return e_x / e_x.sum()\nsoftmax(np.array([[2, 4, 6, 8]]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The output has most of its weight corresponding to the input 8. The softmax function highlights the largest value(s) and suppresses the smaller ones."},{"metadata":{},"cell_type":"markdown","source":"# Loss function\n\nThe basic source of information on the progress of the learning process is the value of the loss function. Generally speaking, the loss function is designed to show how far we are from the ‘ideal’ solution. In our case we used binary crossentropy, but depending on the problem we are dealing with different functions can be applied. The function used by us is described by the following formula, and the change of its value during the learning process is visualised in the next Figure. It shows how with each iteration the value of the loss function decreases and accuracy increases.\n\n![](https://miro.medium.com/max/634/1*3L6_FuyP3HfgfJK-5Lx6qw.gif)\n\n![](https://miro.medium.com/max/1400/1*QGFFaNSUE8SzxSq0mmPs2A.gif)"},{"metadata":{},"cell_type":"markdown","source":"# How do neural networks learn?\n\nThe learning process is about changing the values of the W and b parameters so that the loss function is minimized. In order to achieve this goal, we will turn for help to calculus and use gradient descent method to find a function minimum. In each iteration we will calculate the values of the loss function partial derivatives with respect to each of the parameters of our neural network. For those who are less familiar with this type of calculations, I will just mention that the derivative has a fantastic ability to describe the slope of the function. Thanks to that we know how to manipulate variables in order to move downhill in the graph. Aiming to form an intuition about how the gradient descent works (and stop you from falling asleep once again) I prepared a small visualization. You can see how with each successive epoch we are heading towards the minimum. In our NN it works in the same way — the gradient calculated on each iteration shows us the direction in which we should move. The main difference is that in our exemplary neural network, we have many more parameters to manipulate. Exactly… How to calculate such complex derivatives?\n\n![](https://miro.medium.com/max/1152/1*KteGDjA9gflRcRcEThpy4w.gif)"},{"metadata":{},"cell_type":"markdown","source":"# Backpropagation\n\nBackpropagation is an algorithm that allows us to calculate a very complicated gradient, like the one we need. The parameters of the neural network are adjusted according to the following formulae.\n\n![](https://miro.medium.com/max/162/1*mFBqYn_t3j8ehwDSqOOZGQ.gif)\n\nIn the equations above, α represents learning rate - a hyperparameter which allows you to control the value of performed adjustment. Choosing a learning rate is crucial — we set it too low, our NN will be learning very slowly, we set it too high and we will not be able to hit the minimum. dW and db are calculated using the chain rule, partial derivatives of loss function with respect to W and b. The size of dW and db are the same as that of W and b respectively. The next figure shows the sequence of operations within the neural network. We see clearly how forward and backward propagation work together to optimize the loss function.\n\n![](https://miro.medium.com/max/268/1*ta45UODxvyHSG64N7uksvg.gif)\n\n![](https://miro.medium.com/max/1400/1*wjtcuthF07Bhql0lc4deow.png)\n\nBackpropagation is the backbone of almost anything we do when using Neural Networks. The algorithm consists of 3 subtasks:\n* Make a forward pass\n* Calculate the error\n* Make backward pass (backpropagation) \n\nIn the first step, backprop uses the data and the weights of the network to compute a prediction. Next, the error is computed based on the prediction and the provided labels. The final step propagates the error through the network, starting from the final layer. Thus, the weights get updated based on the error, little by little."},{"metadata":{},"cell_type":"markdown","source":"# We will try to create a Neural Network (NN) that can properly predict values from the XOR function. \nHere is its truth table:"},{"metadata":{},"cell_type":"markdown","source":"![](https://miro.medium.com/max/1400/1*kmGgyHl3oj-iZOXPkfo0RA.png)"},{"metadata":{},"cell_type":"markdown","source":"visual representation:"},{"metadata":{},"cell_type":"markdown","source":"![](https://miro.medium.com/max/1400/1*WHajEhvp_7ZdXhuUY5V6Lw.png)"},{"metadata":{},"cell_type":"markdown","source":"Let's start by defining some parameters:"},{"metadata":{"trusted":true},"cell_type":"code","source":"epochs = 60000 # Number of iterations\ninputLayerSize, hiddenLayerSize, outputLayerSize = 2, 3, 1\nLR = 0.1 # learning rate","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Our data\nX = np.array([[0,0], [0,1], [1,0], [1,1]])\ny = np.array([ [0], [1], [1], [0]])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Initialize the weights of our NN to random numbers :"},{"metadata":{"trusted":true},"cell_type":"code","source":"# weights on layer inputs\nw_hidden = np.random.uniform(size=(inputLayerSize, hiddenLayerSize))\nw_output = np.random.uniform(size=(hiddenLayerSize,outputLayerSize))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, implementation of the Backprop algorithm:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def sigmoid (x): return 1/(1 + np.exp(-x)) # activation function\ndef sigmoid_prime(x): return x * (1 - x) # derivative of sigmoid","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for epoch in range(epochs):\n \n # Forward\n act_hidden = sigmoid(np.dot(X, w_hidden))\n output = np.dot(act_hidden, w_output)\n \n # Calculate error\n error = y - output\n \n if epoch % 5000 == 0:\n print(f'error sum {sum(error)}')\n\n # Backward\n dZ = error * LR\n w_output += act_hidden.T.dot(dZ)\n dH = dZ.dot(w_output.T) * sigmoid_prime(act_hidden)\n w_hidden += X.T.dot(dH) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That error seems to be decreasing! Yay! And the implementation is not that scary, isn’t it? We just multiply the matrix containing our training data with the matrix of the weights of the hidden layer. Then, we apply the activation function (sigmoid) to the result and multiply that with the weight matrix of the output layer.\n\nThe error is computed by doing simple subtraction. During the backpropagation step, we adjust the weight matrices using the already computed error and use the derivative of the sigmoid function.\n\nLet’s try to predict using our trained model (doing just the forward step):"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_test = X[1] # [0, 1]\n\nact_hidden = sigmoid(np.dot(X_test, w_hidden))\nnp.round(np.dot(act_hidden, w_output))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"What is this sorcery? The prediction is correct! You can try some of the other input examples."},{"metadata":{},"cell_type":"markdown","source":"# Building our own Neural Network Classifier"},{"metadata":{},"cell_type":"markdown","source":"The “hello world” dataset MNIST (“Modified National Institute of Standards and Technology”), released in 1999, contains images of handwritten digits. Our goal is to build a model that correctly identify digits from a dataset of tens of thousands of handwritten digits.\n\nWe will build our own “vanilla” Neural Network classifier that learns from raw pixels using only Python and NumPy. Let’s start by reading the data:"},{"metadata":{"trusted":true,"_kg_hide-output":true,"collapsed":true},"cell_type":"code","source":"from mlxtend.data import loadlocal_mnist\nimport pandas as pd\nimport os\nimport random\n\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model"},{"metadata":{},"cell_type":"markdown","source":"Let’s define a class, called NNClassifier that does all the dirty work for us. We will implement a somewhat more sophisticated version of our training algorithm shown above along with some handy methods:"},{"metadata":{},"cell_type":"markdown","source":"# Loading data & Showing image: "},{"metadata":{"trusted":true},"cell_type":"code","source":"from os.path import join\nimport struct\nfrom array import array\nclass MnistDataloader(object):\n def __init__(self, training_images_filepath,training_labels_filepath,\n test_images_filepath, test_labels_filepath):\n self.training_images_filepath = training_images_filepath\n self.training_labels_filepath = training_labels_filepath\n self.test_images_filepath = test_images_filepath\n self.test_labels_filepath = test_labels_filepath\n \n def read_images_labels(self, images_filepath, labels_filepath): \n labels = []\n with open(labels_filepath, 'rb') as file:\n magic, size = struct.unpack(\">II\", file.read(8))\n if magic != 2049:\n raise ValueError('Magic number mismatch, expected 2049, got {}'.format(magic))\n labels = array(\"B\", file.read()) \n \n with open(images_filepath, 'rb') as file:\n magic, size, rows, cols = struct.unpack(\">IIII\", file.read(16))\n if magic != 2051:\n raise ValueError('Magic number mismatch, expected 2051, got {}'.format(magic))\n image_data = array(\"B\", file.read()) \n images = []\n for i in range(size):\n images.append([0] * rows * cols)\n for i in range(size):\n img = np.array(image_data[i * rows * cols:(i + 1) * rows * cols])\n img = img.reshape(28, 28)\n images[i][:] = img \n \n return images, labels\n \n def load_data(self):\n x_train, y_train = self.read_images_labels(self.training_images_filepath, self.training_labels_filepath)\n x_test, y_test = self.read_images_labels(self.test_images_filepath, self.test_labels_filepath)\n return (x_train, y_train),(x_test, y_test) \n\n# Set file paths based on added MNIST Datasets\ntraining_images_filepath = '/kaggle/input/mnist-dataset/train-images.idx3-ubyte'\ntraining_labels_filepath = '/kaggle/input/mnist-dataset/train-labels.idx1-ubyte'\ntest_images_filepath = '/kaggle/input/mnist-dataset/t10k-images.idx3-ubyte'\ntest_labels_filepath = '/kaggle/input/mnist-dataset/t10k-labels.idx1-ubyte'\n\n\n# Load MINST dataset\nprint('Loading MNIST dataset...')\nmnist_dataloader = MnistDataloader(training_images_filepath, training_labels_filepath, test_images_filepath, test_labels_filepath)\n(x_train, y_train), (x_test, y_test) = mnist_dataloader.load_data()\nprint('MNIST dataset loaded.')\n\n# Show example images\ndef show_images(images, title_texts):\n cols = 5\n rows = int(len(images)/cols) + 1\n plt.figure(figsize=(28, 28))\n index = 1 \n for x in zip(images, title_texts): \n image = x[0] \n title_text = x[1]\n plt.subplot(rows, cols, index) \n plt.imshow(image, cmap=plt.cm.gray)\n if (title_text != ''):\n plt.title(title_text, fontsize = 15); \n index += 1\n\nrandom_images = []\nfor i in range(0, 10):\n r = random.randint(1, 60000)\n random_images.append((x_train[r], 'training image [' + str(r) + '] = ' + str(y_train[r])))\nfor i in range(0, 5):\n r = random.randint(1, 10000)\n random_images.append((x_test[r], 'test image [' + str(r) + '] = ' + str(y_test[r])))\n\nshow_images(list(map(lambda x: x[0], random_images)), list(map(lambda x: x[1], random_images)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Building functions:"},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef softmax_crossentropy_with_logits(logits, reference_answers):\n # Compute crossentropy from logits[batch,n_classes] and ids of correct answers \n logits_for_answers = logits[np.arange(len(logits)), reference_answers] \n xentropy = - logits_for_answers + np.log(np.sum(np.exp(logits), axis=-1)) \n return xentropy\n\ndef grad_softmax_crossentropy_with_logits(logits, reference_answers):\n # Compute crossentropy gradient from logits[batch,n_classes] and ids of correct answers\n ones_for_answers = np.zeros_like(logits)\n ones_for_answers[np.arange(len(logits)), reference_answers] = 1 \n softmax = np.exp(logits) / np.exp(logits).sum(axis=-1,keepdims=True) \n return (- ones_for_answers + softmax) / logits.shape[0]\n\n# A building block. Each layer is capable of performing two things:\n# - Process input to get output: output = layer.forward(input)\n# - Propagate gradients through itself: grad_input = layer.backward(input, grad_output)\n# Some layers also have learnable parameters which they update during layer.backward.\nclass Layer(object):\n def __init__(self): \n pass\n \n def forward(self, input):\n # Takes input data of shape [batch, input_units], returns output data [batch, output_units]\n # A dummy layer just returns whatever it gets as input.\n return input\n \n def backward(self, input, grad_output):\n # Performs a backpropagation step through the layer, with respect to the given input.\n # To compute loss gradients w.r.t input, we need to apply chain rule (backprop):\n # d loss / d x = (d loss / d layer) * (d layer / d x)\n # Luckily, we already receive d loss / d layer as input, so you only need to multiply it by d layer / d x.\n # If our layer has parameters (e.g. dense layer), we also need to update them here using d loss / d layer\n # The gradient of a dummy layer is precisely grad_output, but we'll write it more explicitly \n num_units = input.shape[1]\n d_layer_d_input = np.eye(num_units)\n return np.dot(grad_output, d_layer_d_input) # chain rule\n\n\nclass ReLU(Layer):\n def __init__(self):\n # ReLU layer simply applies elementwise rectified linear unit to all inputs\n pass\n \n def forward(self, input):\n # Apply elementwise ReLU to [batch, input_units] matrix\n relu_forward = np.maximum(0, input)\n return relu_forward\n \n def backward(self, input, grad_output):\n # Compute gradient of loss w.r.t. ReLU input\n relu_grad = input > 0\n return grad_output * relu_grad\n\nclass Dense(Layer):\n def __init__(self, input_units, output_units, learning_rate = 0.1):\n # A dense layer is a layer which performs a learned affine transformation: f(x) = + b\n self.learning_rate = learning_rate\n self.weights = np.random.normal(loc=0.0, scale = np.sqrt(2 / (input_units + output_units)), size = (input_units, output_units))\n self.biases = np.zeros(output_units)\n \n def forward(self, input):\n # Perform an affine transformation: f(x) = + b \n # input shape: [batch, input_units]\n # output shape: [batch, output units] \n return np.dot(input, self.weights) + self.biases\n \n def backward(self, input, grad_output):\n # compute d f / d x = d f / d dense * d dense / d x where d dense/ d x = weights transposed\n grad_input = np.dot(grad_output, self.weights.T)\n \n # compute gradient w.r.t. weights and biases\n grad_weights = np.dot(input.T, grad_output)\n grad_biases = grad_output.mean(axis=0) * input.shape[0]\n assert grad_weights.shape == self.weights.shape and grad_biases.shape == self.biases.shape\n \n # Here we perform a stochastic gradient descent step. \n self.weights = self.weights - self.learning_rate * grad_weights\n self.biases = self.biases - self.learning_rate * grad_biases\n \n return grad_input\n\n \nclass MCP(object):\n def __init__(self):\n self.layers = []\n \n def add_layer(self, layer):\n self.layers.append(layer)\n \n def forward(self, X):\n # Compute activations of all network layers by applying them sequentially.\n # Return a list of activations for each layer. \n activations = []\n input = X\n \n # Looping through each layer\n for l in self.layers:\n activations.append(l.forward(input))\n # Updating input to last layer output\n input = activations[-1]\n \n assert len(activations) == len(self.layers)\n return activations\n \n \n def train_batch(self, X, y):\n # Train our network on a given batch of X and y.\n # We first need to run forward to get all layer activations.\n # Then we can run layer.backward going from last to first layer.\n # After we have called backward for all layers, all Dense layers have already made one gradient step.\n \n layer_activations = self.forward(X)\n layer_inputs = [X] + layer_activations # layer_input[i] is an input for layer[i]\n logits = layer_activations[-1]\n \n # Compute the loss and the initial gradient \n y_argmax = y.argmax(axis=1) \n loss = softmax_crossentropy_with_logits(logits, y_argmax)\n loss_grad = grad_softmax_crossentropy_with_logits(logits, y_argmax)\n \n # Propagate gradients through the network\n # Reverse propogation as this is backprop\n for layer_index in range(len(self.layers))[::-1]:\n layer = self.layers[layer_index] \n loss_grad = layer.backward(layer_inputs[layer_index], loss_grad) # grad w.r.t. input, also weight updates\n \n return np.mean(loss)\n \n def train(self, X_train, y_train, n_epochs = 25, batch_size = 32):\n train_log = [] \n \n for epoch in range(n_epochs): \n for i in range(0, X_train.shape[0], batch_size):\n # Get pair of (X, y) of the current minibatch/chunk\n x_batch = np.array([x.flatten() for x in X_train[i:i + batch_size]])\n y_batch = np.array([y for y in y_train[i:i + batch_size]]) \n self.train_batch(x_batch, y_batch)\n \n train_log.append(np.mean(self.predict(X_train) == y_train.argmax(axis=-1))) \n print(f\"Epoch: {epoch + 1}, Train accuracy: {train_log[-1]}\") \n return train_log\n \n def predict(self, X):\n # Compute network predictions. Returning indices of largest Logit probability\n logits = self.forward(X)[-1]\n return logits.argmax(axis=-1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Building Model and Fitting :"},{"metadata":{"trusted":true},"cell_type":"code","source":"def normalize(X):\n X_normalize = (X - np.min(X)) / (np.max(X) - np.min(X))\n return X_normalize \n\ndef one_hot(a, num_classes):\n return np.squeeze(np.eye(num_classes)[a.reshape(-1)]) \n\nX_train = normalize(np.array([np.ravel(x) for x in x_train]))\nX_test = normalize(np.array([np.ravel(x) for x in x_test]))\nY_train = np.array([one_hot(np.array(y, dtype=int), 10) for y in y_train], dtype=int)\nY_test = np.array([one_hot(np.array(y, dtype=int), 10) for y in y_test], dtype=int)\n\nprint('X_train.shape', X_train.shape)\nprint('Y_train.shape', Y_train.shape)\ninput_size = X_train.shape[1]\noutput_size = Y_train.shape[1]\n\nnetwork = MCP()\nnetwork.add_layer(Dense(input_size, 100, learning_rate = 0.05))\nnetwork.add_layer(ReLU())\nnetwork.add_layer(Dense(100, 200, learning_rate = 0.05))\nnetwork.add_layer(ReLU())\nnetwork.add_layer(Dense(200, output_size))\n\ntrain_log = network.train(X_train, Y_train, n_epochs = 32, batch_size = 64)\nplt.plot(train_log,label = 'train accuracy')\nplt.legend(loc='best')\nplt.grid()\nplt.show()\n\n\ntest_corrects = len(list(filter(lambda x: x == True, network.predict(X_test) == Y_test.argmax(axis=-1))))\ntest_all = len(X_test)\ntest_accuracy = test_corrects/test_all #np.mean(test_errors)\nprint(f\"Test accuracy = {test_corrects}/{test_all} = {test_accuracy}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Evaluation"},{"metadata":{"trusted":true},"cell_type":"code","source":"network.predict(X_test[1:2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You can “clearly” see that the most probable digit is 2\n\nLet’s look at the image itself:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def visualize_input(img, ax):\n ax.imshow(img, cmap='gray')\n width, height = img.shape\n thresh = img.max()/2.5\n for x in range(width):\n for y in range(height):\n ax.annotate(str(round(img[x][y],2)), xy=(y,x),\n horizontalalignment='center',\n verticalalignment='center',\n color='white' if img[x][y]