{"cells":[{"metadata":{"_uuid":"bfbdf454376e0fe4cdc1023820d482e5aadba67b"},"cell_type":"markdown","source":"### **CNN model using keras. Test set Accuracy Achieved is 99.685%. **\nHi all,   I was fascinated by the book written by Michael Nielsen on neural networks sometime ago and since than I have been a ML enthusiast. :)  Happy to share with you my first kernel in digit recognition. \n* In this kernel I have used various ideas shared before in this platform. I am just returning the benefits I have earned here."},{"metadata":{"_uuid":"b85ed854b61c18b6a39dee556ae0a95d8d9bdc6f"},"cell_type":"markdown","source":"### Here we go !!\nI have used techniques like Convolutional Neural Network with Pooling layers and batch normalisation and dropout. Used Adam Optimization for optimizing the cost. I also have used Data Augmentation to improve accuracy by atleast 0.371% . \nI have not used learning rate decay function and also not used any validation set to measure the accuracy. Trained my model on 42000 images and I have achieved an accuracy of 99.685% with dara Augmentation ON and epochs 75. Without Data Augmentation I achieved test set accuracy of 99.314% [here](https://www.kaggle.com/hengulkakaty/digit-recogniotion-in-cnn)"},{"metadata":{"_uuid":"f11fa83a1692ef032a7cd05ea3d7187fe6b08ecc"},"cell_type":"markdown","source":"### Calling the required function.\n* Keras is a Library which gives us liberty to call different function to use ML algorithms. It is easy to learn. Keras       documentations are easy enough to follow.\n* matplotlib to plot some images and lines.\n* Pandas to load the excell and do simple pre processing.\n* numpy is an awsome mathematical library"},{"metadata":{"trusted":true,"_uuid":"15fa7c42a2437df344ed6533cfea396b558e707c","_kg_hide-output":true},"cell_type":"code","source":"import numpy as np\nfrom keras import layers\nfrom keras.layers import Input, Dense, Activation, ZeroPadding2D, BatchNormalization, Flatten, Conv2D\nfrom keras.layers import AveragePooling2D, MaxPooling2D, Dropout, GlobalMaxPooling2D, GlobalAveragePooling2D\nfrom keras.models import Model\nfrom keras.preprocessing import image\nfrom keras.utils import layer_utils\nfrom keras.utils.data_utils import get_file\nfrom keras.applications.imagenet_utils import preprocess_input\nimport pydot\nfrom IPython.display import SVG\nfrom keras.utils.vis_utils import model_to_dot\nfrom keras.utils import plot_model\nfrom keras.utils import to_categorical\nimport matplotlib.pyplot as plt \nimport pandas as pd\nnp.set_printoptions(suppress=True)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b0f931cce332b288383d48d95aecc48ddf51d8f2"},"cell_type":"markdown","source":"###  Loading the Train and Test Set. paths are set by default if you use kaggle kernel"},{"metadata":{"trusted":true,"_uuid":"5ed840c9a1949c7efd927fcc7f387e39cd6a06e0"},"cell_type":"code","source":"df1 = pd.read_csv('../input/train.csv')\ndf2 = pd.read_csv('../input/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8161cdeade8235683f8bc5f663a28d61e20eadd5"},"cell_type":"markdown","source":" Here we are just formatting the data into an array."},{"metadata":{"trusted":true,"_uuid":"f0a5ecddb4014ec4ce4e3033000e7a5b0b523d6b"},"cell_type":"code","source":"train_data = np.array(df1)\ntest_data  = np.array(df2)\n\nX_train_orig    = np.asarray( train_data[:, 1::], dtype=np.float32 )\nY_train_orig    = np.asarray( train_data[:, 0], dtype=np.float32 )\n\nX_test_orig     = np.asarray( test_data[:, 0::], dtype=np.float32 )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"547ec1329bedd08bf43a34dac662f529df761d44"},"cell_type":"markdown","source":"###  Normalizing the input. \nThe data what we have in our training set are scattered from 0 to 255. So we are dividing each inoput by 255 so that we have a range of values from 0 to 1. This helps in improve accuracy and faster to converge."},{"metadata":{"trusted":true,"_uuid":"f16624c65811f1c30d85c983935e6ab189cb8460"},"cell_type":"code","source":"X_train = X_train_orig / 255.\nX_test = X_test_orig / 255.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3b175a79c81b1c1beafb8d89cf6886ae9cddde92"},"cell_type":"code","source":"X_train.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aa85d40c01afc50db96b75581ae823a7751dfaca"},"cell_type":"markdown","source":"### Reshaping the arrays into desired format.\nIn Keras we need to reshape the arrays into [no of images, images size in x & y, no of channels]\nWe have 42000 examples in our training set\nEach Image is 28x28 size\nEach image have only 1 dimension because we only have grey scale images. This value would have been 3 if we have a colored image. Colored image have three channel which is best known as RGB channels.\n\nSo Our Training set have become  [42000,  28,  28,  1]"},{"metadata":{"trusted":true,"_uuid":"133b574c5d070fe9d0dda559dfd150e9c3c3796a"},"cell_type":"code","source":"X_train = X_train.reshape(-1,28,28,1)\n\nX_test = X_test.reshape(-1,28,28,1)\n\nY_train = Y_train_orig.reshape(-1, 1)\n\nY_train = to_categorical(Y_train, num_classes=10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c74382534b5b8bc3bb8a0cef3a9b0325ba53ec17"},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nX_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size = 0.2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"27c92f19f8e7d560d180cbe0d9094099af01b546"},"cell_type":"markdown","source":"### Lets see the dimension of each array\nTill now we have 4 array. \n* X_train contains the pixels values of each images from 42000 set. On this set our model will train and learn to recognise digits.\n* Y_train contains the labels of each corresponding images. \n* X_test contains the pixels values of each images from 28000 set. We will test our model on this set. This set does not have labels. We will predict the labels here.\n\nLets look:"},{"metadata":{"trusted":true,"_uuid":"0f4e0d01c3498c9dc9a99f60adad38dc00120aee"},"cell_type":"code","source":"print (\"number of training examples = \" + str(X_train.shape[0]))\nprint (\"number of test examples = \" + str(X_test.shape[0]))\nprint (\"X_train shape: \" + str(X_train.shape))\nprint (\"Y_train shape: \" + str(Y_train.shape))\nprint (\"X_test shape: \" + str(X_test.shape))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3fdf74e062dfc03dbc0324c8ff3b8d26bc4c4d55"},"cell_type":"markdown","source":"### Let's look into our data.\nAs our data has been well pre processed now lets look at our data once before proceeding to train our model.\nWe have already turned our Y_train into categorical values. This means that each label has been turned into a vector of 10 values. If our label is 1 than our vector will be like [0, 1, 0, 0, 0, 0, 0, 0, 0, 0]. All zeros except the position 1. In python count always starts from 0. This process is famously known as one hot encoding. We had to do this as we have 10 different classes to predict. 10 different classes are from 0, 1, 2, 3, 4, .....  10."},{"metadata":{"trusted":true,"_uuid":"881be256a6e2e0c4d55a41ef7a81aacb5ec97f03"},"cell_type":"code","source":"plt.figure(figsize=(10,10))   # to fix a shape for each image print\nfor ax in range(10):          # using a for loop to display a number of images\n    plt.subplot(5, 5, ax+1) # we need to use this function to print an array of pictures \n    plt.imshow(X_train[ax,:,:,0],cmap=plt.cm.binary) # this will call the images from train set one by one\n    print(\" Ground Truth Label = \", Y_train[ax,:])   # Lets also look into the labels \n    plt.axis('off')                                  # axis has been turned off to have clear view","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"236e5ebc4b1e2f1faa22ae98fc3bd8395639472a"},"cell_type":"markdown","source":"## Data Augmentation:\n#### Let us do some data Augmentation and see what happens.\nIn data Augmentation we are giving a tilt of 15 degree, zooming 10%, shifting the widhth and height by 10% out of total fraction.\nAll this process is done selecting random images in the data set. \n* We achieved 99.314 % accuracy without Data Augmentation.\n* And with Data Augmentation we achieved 99.685%. "},{"metadata":{"trusted":true,"_uuid":"57f0f715b16313a33bba3a0ce6cc31c7d425ed8d"},"cell_type":"code","source":"from keras.preprocessing.image import ImageDataGenerator\ndatagen = ImageDataGenerator(\n        rotation_range=15,  # randomly rotate images in the range (degrees, 0 to 180)\n        zoom_range = 0.1, # Randomly zoom image \n        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)\n        height_shift_range=0.1)  # randomly shift images vertically (fraction of total height)\n\ndatagen.fit(X_train)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9b5143cdb57239ea705e6ee3eb37cff177222ff6"},"cell_type":"markdown","source":"## Build the CNN model:\n### Conv2D(32 filter with 3x3 kernel, stride 1 padding same) * 2 ---> MaxPool ---> Dropout(0.4) ---> Conv2D(64 filter with 3x3 kernel, stride 1 padding same) ---> MaxPool ---> Dropout(0.4) ---> Flatten ---> Dense(1024) ---> Dense(256) ---> Dropout(0.5) ---> Softmax\n\nLets try to define each term above in simple language:\n* Convolution is a process of looking into the image part by part. Part means a small window. In our case Window is of size 3x3. We will use such 3x3 windown 32 times in one single CNN layer. This number 32 is called filter. Filters are cool to detect the edges whether vertical or horizontal. More number of filters means model is capable to handling more number of edges.\n\n* Pooling is a concept same as convolutional layer but here we take the maximum or average of all the number along the size of the window. No gradient decent while implementing pooling layers. No parameters to learn if we implement Pooling.\nWhy we use Pooling… No one realy knows. In many experiments, it was found that Pooling really works well. Dimension concepts are same as in Convolutional Layer. When using Pooling Padding are rarely used.\n\n* Padding is a concept of putting some extra layer around our image. This helps in keeping our dimensions correct. While we do Convolution or pooling the dimensions of the picture shrinks. But with the use of padding some bits(0 or 1) around the picture we used to retain the dimension of the picture.\n\n* How Dimensions are effected.. If Image size is n x n and window size is 3 x 3 and we use padding p than our output image size will be:\n    n + 2p - f + 1     x      n + 2p - f + 1  \nWe also use padding when we need to retain the input size of the image. To do this we choose padding carefully as :               p  =  (f-1) / 2  ,    f is almost always odd.\n\n* What is stride... When we use Stride of 2 we jump over by 2 step while doing convolution. While using Stride we use the below formula for the output image: (  ((n + 2p - f ) / s)  +  1  )  x   (  ((n + 2p - f ) / s)  +  1  )\n\n* Batchnormalization are used to normalize the activations. Normaly we do normalize the input values and with batch normalization we are doing the same thing to the activations. Simply saying we do normalization in the hidden layers too. This helps in improving accuracy and Speed up tarining process etc.\n\n* Dropout is a cool thing where we nullify the effect of some neuron so as to improve the accuracy of other neuron. Basicaly we set some random weights to zero and thus It helps in a overfitting model and also helps in generalization. Overfitting Model is a model where Training Accuracy is more but Test accuracy is poor. Generalization means how well our model predicts for unseen data.\n\n* At last we flatten our output from CNN so as to get a vector. Lets think Vector as a single column in an excel. We do this to use our data in Fully connected layers. Here we have introduced two FC layers of 1024 neurons and 256 neurons.\n\n* To get the Output we have used a softmax function. This function helps us to give 10 different classes.\n\n## So thats it.. Now lets build our model in keras.\nIn Keras there are two ways to build our model. One is sequential model and another is the functional API. I think sequential model is a bit popular but here we will use functional API model. "},{"metadata":{"trusted":true,"_uuid":"492999686725550e438332efe1477761d042ca1a"},"cell_type":"code","source":"def Keras_Model(input_shape):    \n    \n    X_input = Input(input_shape)\n    \n    # First Convolutional Layer\n    X = Conv2D(64, (3, 3), strides = (1, 1), padding = 'same', name = 'conv0')(X_input) \n    X = BatchNormalization(axis = 3, name = 'bn0')(X)\n    X = Activation('relu')(X)\n    \n    # Second Convolutional Layer\n    X = Conv2D(64, (3, 3), strides = (1, 1), padding = 'same', name = 'conv1')(X) \n    X = BatchNormalization(axis = 3, name = 'bn1')(X)\n    X = Activation('relu')(X)\n    \n    # First Pooling Layer\n    X = MaxPooling2D((2, 2), name='max_pool_1')(X)\n                           \n    X = Dropout(0.35)(X)\n    \n    # Third Convolutional Layer\n    X = Conv2D(128, (3, 3), strides = (1, 1), padding = 'same', name = 'conv2')(X) \n    X = BatchNormalization(axis = 3, name = 'bn2')(X)\n    X = Activation('relu')(X)\n    \n    # Fourth Convolutional Layer\n    X = Conv2D(128, (3, 3), strides = (1, 1), padding = 'same', name = 'conv3')(X) \n    X = BatchNormalization(axis = 3, name = 'bn3')(X)\n    X = Activation('relu')(X)\n    \n    # Second Pooling Layers\n    X = MaxPooling2D((2, 2), name='max_pool_2')(X)                       \n                           \n    X = Dropout(0.35)(X)     \n        \n    \n    # Fifth Convolutional Layer\n    X = Conv2D(256, (3, 3), strides = (1, 1), padding = 'same', name = 'conv4')(X) \n    X = BatchNormalization(axis = 3, name = 'bn4')(X)\n    X = Activation('relu')(X)\n\n    X = Dropout(0.35)(X) \n \n    # Flatten the data.\n    X = Flatten()(X)\n    # Dense Layer\n    X = Dense(1000, activation='relu', name='fc0')(X)\n    X = Dropout(0.5)(X) \n    X = Dense(256, activation='relu', name='fc2')(X)\n    \n    # Using softmax function to get the output\n    X = Dense(10, activation='softmax', name='fc3')(X)\n    \n    model = Model(inputs = X_input, outputs = X, name='model')\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"303a6c3d5dbe688eb303a26e14cbe73aff917550"},"cell_type":"markdown","source":"### Let's call the above function with our input shape. Our Input is X_train right...."},{"metadata":{"trusted":true,"_uuid":"fdb4fa23cd6341eb7eeb3509699c66243312f506"},"cell_type":"code","source":"Keras_Model = Keras_Model(X_train.shape[1:4])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dca2e5bc7769fb0ff91dfdadbb9f1085ceb3b8bc"},"cell_type":"markdown","source":"## Adam optimizer:\nThere are several Optimizing function to use. this time I preferred Adam Optimizer. This is widely used and fast for converging. We are keeping our Learning rate to 0.0001"},{"metadata":{"trusted":true,"_uuid":"897e0788960461647cf1dc5c12a451dbfd00f88f"},"cell_type":"code","source":"from keras.optimizers import Adam\nepochs = 100\nbatch_size = 64\nlrate = 0.0001\ndecay = lrate/epochs\noptimizer = Adam(lr=lrate, epsilon=1e-08, decay = 0.00)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e2e4f4774f0959f87f70544df7e1818aeb50634e"},"cell_type":"markdown","source":"We need to compile the model before training. This step is related to keras. We used categorical crossentropy function as we have One hot encoded our labels."},{"metadata":{"trusted":true,"_uuid":"abefba21496fa6a1670c6320325e15d7d081065d"},"cell_type":"code","source":"Keras_Model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2b75afd80b2b00cdf94f2119a966d9cd6074b128"},"cell_type":"code","source":"from keras.callbacks import ReduceLROnPlateau\nlearning_rate_reduction = ReduceLROnPlateau(monitor='val_acc', \n                                            patience=2, \n                                            verbose=1, \n                                            factor=0.5, \n                                            min_lr=0.0000001)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a9675766a12068bacdd5a3955870514723820775"},"cell_type":"code","source":"from keras.callbacks import ModelCheckpoint, Callback, EarlyStopping\nearly_stopping = EarlyStopping(monitor='val_loss', patience=3, verbose=1, mode='auto')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"243aa61e63f375bd955bc0920f0eda7fd72791e2"},"cell_type":"markdown","source":"## Train the Model\nLets train the model with batch size of 32 and epoch of 75. As we have used data augmentation we need to use a special function called fit generator. I have trained the model to 75 number of epochs beyond which I believe model will overfit the data. Though you can try different number of epochs. I have tried different batch size but it barely effects the accuracy. batch size impacts on speed of training."},{"metadata":{"trusted":true,"_uuid":"6e05b2bf1680d27443032c4109686f0e1e3f6e48"},"cell_type":"code","source":"history = Keras_Model.fit(x = X_train, y = Y_train, batch_size = batch_size, \n                        epochs=epochs, verbose=1, \n                        validation_data = (X_val, Y_val),\n                        \n                          steps_per_epoch= None, validation_steps=None,\n                                  callbacks=[learning_rate_reduction, early_stopping] )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"866eeda8892005350865a3a0880b74dcfdd64727"},"cell_type":"code","source":"preds = Keras_Model.evaluate(X_train, Y_train)\nprint (\"Loss = \" + str(preds[0]))\nprint (\"Train set Accuracy = \" + str(preds[1]))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5df9eefd14e19a95f3eb4d748bf8d4afab1ac2f6"},"cell_type":"markdown","source":"## Train Accuracy:\n\nI have achieved almost same train accuracy with or without data Augmentation. But the model with data augmentation generalizes well. Please see the kernel without data augmentation [here](http://www.kaggle.com/hengulkakaty/digit-recogniotion-in-cnn?scriptVersionId=6090971)"},{"metadata":{"trusted":true,"_uuid":"86160d6648d4180ada0eb6b9fa568449f3aa9835","scrolled":true},"cell_type":"code","source":"preds = Keras_Model.evaluate(X_val, Y_val)\nprint (\"Loss on Val set= \" + str(preds[0]))\nprint (\"Val set Accuracy = \" + str(preds[1]))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"efee31d08eb1edaa5c8aec51ff095a8297b4a145"},"cell_type":"markdown","source":"#### History is used to find out accuracy and loss in Tran as well as Validation set."},{"metadata":{"trusted":true,"_uuid":"5ab50985e8ff90b0a0bf63930fc00230653fc69b"},"cell_type":"code","source":"history_dict = history.history\nhistory_dict.keys()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d2fe1f6dcb97640454cabfc7fdbe011f4a1a7563"},"cell_type":"code","source":"val_loss = history_dict['val_loss']\nval_acc = history_dict['val_acc']\nloss = history_dict['loss']\nacc = history_dict['acc']\nepochs = range(1,len(history_dict['val_loss'])+1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"75760d895d960213a3646aac475f0d2c9cb6efb1"},"cell_type":"code","source":"plt.plot(epochs,acc,'b-')\nplt.title('Accuracy of Model')\nplt.xlabel('epochs')\nplt.ylabel('Accuracy')\n\nplt.plot(epochs,val_acc,'b-', color = 'red')\nplt.title('Accuracy of Model')\nplt.xlabel('epochs')\nplt.ylabel('Accuracy')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f565eeb582df00b09fda31fb3a2b85ec24b8d893"},"cell_type":"markdown","source":"In the above graph we have seen  that our validation accuracy(RED Line) is slightly on higher side than Training accuracy.  So we can expect a good generalisation."},{"metadata":{"trusted":true,"_uuid":"a4008b766c7eed4b06f5d29899c502543a8fa32e"},"cell_type":"code","source":"plt.plot(epochs,loss,'b-')\nplt.title('loss function')\nplt.xlabel('epochs')\nplt.ylabel('Loss')\n\nplt.plot(epochs,val_loss,'b-', color = 'red')\nplt.title('loss function')\nplt.xlabel('epochs')\nplt.ylabel('val_loss')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ad66515174396fd16815e3e6c9b694340d57487b"},"cell_type":"markdown","source":"Above Graphs shows the Loss on Train and Validation set. RED line suggesting Loss on Validation is lower that train loss."},{"metadata":{"_uuid":"840c23af30d9fb4de6418152dfb9c652ddb8eaae"},"cell_type":"markdown","source":"### Let's see where our model is going wrong.\n* STEP 1:  We will predict on Validation set and convert data to integer value using argmax and store it in a variable Y_val_pred_label.\n* STEP 2:  We will find out the integer value from the Ground truth label from Validation set as well and store it in a variable Y_val_True_label.\n* STEP 3:  So Predicted label and True label should be equal to each other in ideal case. If they are different than that is our point of concern.\n* STEP 4:  When there is a difference than we will check the probability. If model is wrong in predicting the label and probability of prediction is quite high than we will check those instance.\n* STEP 5:  We will priint the corresponding image using X_val and also print the predicted label and true label to get an idea how our model is going wrong."},{"metadata":{"trusted":true,"_uuid":"c21197f382190dcec21d1da94aff1dbce693c339"},"cell_type":"code","source":"# STEP 1:\npredicted_val_probability = Keras_Model.predict(X_val, batch_size=32)\nY_val_pred_label = np.argmax(predicted_val_probability, axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a895a2df5ddb0ac2b7abaff09fa82c33599ad890"},"cell_type":"code","source":"# STEP 2:\nY_val_True_label = np.argmax(Y_val, axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d3d2c8b7c4aa7421f7ecaefcb6f179dcee9196d2"},"cell_type":"code","source":"# STEP 3: \nj = []\nmax_probability = []\nfor i in range(len(Y_val_pred_label)):\n    if Y_val_pred_label[i] != Y_val_True_label[i]:     \n        j.append(i)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"51f6db09fc4882aac207c8ffac08d91bfe67d18c"},"cell_type":"code","source":"# STEP 4: \nl = []\nfor ele in j:\n    if np.max(predicted_val_probability[ele]) > 0.95:\n        l.append(ele)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"35d91b97eb9ee8c1fb4e464a61909c39882c3b2a"},"cell_type":"code","source":"# STEP 5: \nc = 1\nplt.figure(figsize=(10,10))   # to fix a shape for each image print\nfor ax in l[:]:          # using a for loop to display a number of images\n    plt.subplot(6,5, c) # we need to use this function to print an array of pictures \n    plt.imshow(X_val[ax,:,:,0],cmap=plt.cm.binary) # this will call the images from train set one by one\n    plt.title('True label:{}\\nPredicted label:{}'.format(Y_val_True_label[ax],Y_val_pred_label[ax]))  # Lets also look into the labels \n    plt.axis('off') \n    c = c+1","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fdec8805328e462f5b15856d1e8e615720d1d97e"},"cell_type":"markdown","source":"From the above picture it is very clear that some data are inconsistent. Even human can go wrong in these kind of cases. So we can conclude that our model is doing prity well in predicting hand written digits."},{"metadata":{"_uuid":"fe8d4676733c82faae8d0035ac36d095fae38ed9"},"cell_type":"markdown","source":"## Prediction on Test set:\nLets look at our prediction. rememeber that we have one hot encoded our labels and hence here we will get 10 different results for each test set examples. In the next step we will find out the highest number(probability) in a single row."},{"metadata":{"trusted":true,"_uuid":"8d32b1c82ada5d0e6297dd5d08a45beb797edc52"},"cell_type":"code","source":"classes = Keras_Model.predict(X_test, batch_size=32)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f6016e30ba69c5ab68778049dc19aaa30f4509c6"},"cell_type":"markdown","source":"we have used argmax function to find out the highest probability in a row. axis=1 doing the trick here."},{"metadata":{"trusted":true,"_uuid":"2b90e69dc04761953b99a69e0af3fa9a114961f7"},"cell_type":"code","source":"class_test_set = np.argmax(classes, axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"237fec2fe1171269a9b130bb169b4f72deed2a11"},"cell_type":"markdown","source":"## Lets prepare the submission excell sheet as desired and commit this kernel to see how we have done."},{"metadata":{"trusted":true,"_uuid":"b040c94d2177d7c939ad9ac9700adc020e6c7812"},"cell_type":"code","source":"prediction = pd.DataFrame()\nprediction['ImageId'] = np.asarray(range(1,28001))\nprediction['Label'] = class_test_set\n\nprediction.to_csv('submission.csv', index = False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e990ce5449302a7012367c1e10c4828b7e06d30a"},"cell_type":"markdown","source":"## Thank You\nThanks to kaggle for giving this platform to connect to some wonderfull people around the globe. Also Thanks a lot to Mr. Andrew Ng who motivatated milions of people including me to learn and Practice Deep Learning.\n* Please do vote and comment on this kernel. Namaste !!"},{"metadata":{"trusted":true,"_uuid":"0ce6289db71885446ebfbcd7775fde6fe410365f"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}