{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Digit Recognition Using LeNet 99% Accuracy🔥","metadata":{}},{"cell_type":"markdown","source":"## Please Upvote👆🏼 if u like my effort!!","metadata":{}},{"cell_type":"code","source":"from IPython.display import Image\nImage(\"../input/image-data/Digit.jpeg\")","metadata":{"execution":{"iopub.status.busy":"2022-06-07T20:23:11.226044Z","iopub.execute_input":"2022-06-07T20:23:11.226819Z","iopub.status.idle":"2022-06-07T20:23:11.253565Z","shell.execute_reply.started":"2022-06-07T20:23:11.226749Z","shell.execute_reply":"2022-06-07T20:23:11.252565Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##  MNIST Data\n\n**CONTEXT**\n\n1. MNIST (\"Modified National Institute of Standards and Technology\") is the de facto “hello world” dataset of computer vision. Since its release in 1999, this classic dataset of handwritten images has served as the basis for benchmarking classification algorithms. As new machine learning techniques emerge, MNIST remains a reliable resource for researchers and learners alike.\n\n2. The data files train.csv and test.csv contain gray-scale images of hand-drawn digits, from zero through nine. Each image is 28 pixels in height and 28 pixels in width, for a total of 784 pixels in total. This pixel-value is an integer between 0 and 255, inclusive.\n\n3. The test data set, (test.csv), is the same as the training set, except that it does not contain the \"label\" column.\n\n\n**AIM**\n\n1. LeNet is Used to design a digit recognition Algorithm that correctly classifies the digits 0 to 9.\n2. Simple and efficient system with high accuracy.","metadata":{}},{"cell_type":"markdown","source":"## Importing Necessary Libraries","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport random\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport tensorflow as tf\nfrom tensorflow.keras import models\nfrom tensorflow.keras import layers\nfrom tensorflow.keras.utils import to_categorical\nsns.set_style(\"darkgrid\")","metadata":{"execution":{"iopub.status.busy":"2022-06-07T20:23:29.365504Z","iopub.execute_input":"2022-06-07T20:23:29.365959Z","iopub.status.idle":"2022-06-07T20:23:32.244668Z","shell.execute_reply.started":"2022-06-07T20:23:29.365923Z","shell.execute_reply":"2022-06-07T20:23:32.24368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Importing Data\ntrain = pd.read_csv(\"../input/digit-recognizer/train.csv\")\ntest = pd.read_csv(\"../input/digit-recognizer/test.csv\")\n\n#Separating labels and predictors\nX = train.drop(\"label\",axis=1)\ny = train[[\"label\"]]","metadata":{"execution":{"iopub.status.busy":"2022-06-07T20:23:35.695456Z","iopub.execute_input":"2022-06-07T20:23:35.696362Z","iopub.status.idle":"2022-06-07T20:23:39.969205Z","shell.execute_reply.started":"2022-06-07T20:23:35.69632Z","shell.execute_reply":"2022-06-07T20:23:39.967998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Dimensions\nprint(X.shape)\nprint(test.shape)\nprint(y.shape)","metadata":{"execution":{"iopub.status.busy":"2022-06-07T20:23:41.932259Z","iopub.execute_input":"2022-06-07T20:23:41.932687Z","iopub.status.idle":"2022-06-07T20:23:41.938424Z","shell.execute_reply.started":"2022-06-07T20:23:41.932651Z","shell.execute_reply":"2022-06-07T20:23:41.937628Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Preprocessing Step","metadata":{}},{"cell_type":"code","source":"def preprocess_data(X,test,y):\n    \n    #Reshaping\n    X = np.array(X).reshape(X.shape[0],28,28,1)\n    test = np.array(test).reshape(test.shape[0],28,28,1)\n\n    #To categorical\n    y = to_categorical(y)\n\n    #Rescaling\n    X    = X/255.0\n    test = test/255.0\n    return(X,test,y)","metadata":{"execution":{"iopub.status.busy":"2022-06-07T20:23:43.663978Z","iopub.execute_input":"2022-06-07T20:23:43.664989Z","iopub.status.idle":"2022-06-07T20:23:43.671535Z","shell.execute_reply.started":"2022-06-07T20:23:43.664941Z","shell.execute_reply":"2022-06-07T20:23:43.670375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X,test,y = preprocess_data(X,test,y)","metadata":{"execution":{"iopub.status.busy":"2022-06-07T20:23:45.085126Z","iopub.execute_input":"2022-06-07T20:23:45.085914Z","iopub.status.idle":"2022-06-07T20:23:45.561394Z","shell.execute_reply.started":"2022-06-07T20:23:45.085861Z","shell.execute_reply":"2022-06-07T20:23:45.56052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Visualizing the MNIST Data","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10,5))\nfor i in range(1,9):\n  ind = random.randint(0, len(X))\n  plt.subplot(2,4,i)\n  plt.imshow(tf.squeeze(X[ind]))","metadata":{"execution":{"iopub.status.busy":"2022-06-07T20:23:46.413097Z","iopub.execute_input":"2022-06-07T20:23:46.413799Z","iopub.status.idle":"2022-06-07T20:23:47.400603Z","shell.execute_reply.started":"2022-06-07T20:23:46.413748Z","shell.execute_reply":"2022-06-07T20:23:47.399628Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Splitting the data to train and validation ","metadata":{}},{"cell_type":"code","source":"#Splitting the data\n\n#Generating random integers\nval_size = int(X.shape[0]*0.2)\nval_ind = random.sample(range(1,X.shape[0]),val_size)\n\n#Getting the val and train data\n\n#Validation Data\nx_val = X[val_ind]\ny_val = y[val_ind]\n\n#Training Data\nx_train = np.delete(X, val_ind, axis = 0)\ny_train = np.delete(y, val_ind, axis = 0)","metadata":{"execution":{"iopub.status.busy":"2022-06-07T20:23:48.660368Z","iopub.execute_input":"2022-06-07T20:23:48.661Z","iopub.status.idle":"2022-06-07T20:23:48.97378Z","shell.execute_reply.started":"2022-06-07T20:23:48.660956Z","shell.execute_reply":"2022-06-07T20:23:48.972689Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Dimensions\nprint(x_train.shape)\nprint(y_train.shape)\n\nprint(x_val.shape)\nprint(y_val.shape)\n","metadata":{"execution":{"iopub.status.busy":"2022-06-07T20:23:49.911255Z","iopub.execute_input":"2022-06-07T20:23:49.911716Z","iopub.status.idle":"2022-06-07T20:23:49.917505Z","shell.execute_reply.started":"2022-06-07T20:23:49.911676Z","shell.execute_reply":"2022-06-07T20:23:49.916399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Visualizing Train and Validation data","metadata":{}},{"cell_type":"code","source":"#Train Data\nplt.figure(figsize=(10,5))\nfor i in range(8):\n  ind = random.randint(0, len(x_train))\n  plt.subplot(240+1+i)\n  plt.imshow(tf.squeeze(x_train[ind]))","metadata":{"execution":{"iopub.status.busy":"2022-06-07T20:23:51.105438Z","iopub.execute_input":"2022-06-07T20:23:51.106162Z","iopub.status.idle":"2022-06-07T20:23:52.068106Z","shell.execute_reply.started":"2022-06-07T20:23:51.106115Z","shell.execute_reply":"2022-06-07T20:23:52.066921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Validation Data\nplt.figure(figsize=(10,5))\nfor i in range(8):\n  ind = random.randint(0, len(x_val))\n  plt.subplot(240+1+i)\n  plt.imshow(tf.squeeze(x_val[ind]))","metadata":{"execution":{"iopub.status.busy":"2022-06-07T20:23:52.835399Z","iopub.execute_input":"2022-06-07T20:23:52.840929Z","iopub.status.idle":"2022-06-07T20:23:53.97092Z","shell.execute_reply.started":"2022-06-07T20:23:52.839899Z","shell.execute_reply":"2022-06-07T20:23:53.969884Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# LeNet Architecture","metadata":{}},{"cell_type":"code","source":"Image(\"../input/image-data/LeNet1.png\")","metadata":{"execution":{"iopub.status.busy":"2022-06-07T20:23:56.195149Z","iopub.execute_input":"2022-06-07T20:23:56.196137Z","iopub.status.idle":"2022-06-07T20:23:56.206997Z","shell.execute_reply.started":"2022-06-07T20:23:56.196089Z","shell.execute_reply":"2022-06-07T20:23:56.205782Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Structure\n\n-  In the figure above, The First layer is a input layer followed by a convolutions, pooling and fully connected layers.\n\n- The input is images of size 28 × 28.\n\n- C1 is the first convolutional layer with 6 convolution kernels of size 5×5. Followed by the pooling layer that outputs 6 channels of 14 × 14 images. The pooling is of size 2 × 2.\n\n- C2 is a convolutional layer with 16 convolution kernels of size 5 × 5. Hence, the output of this layer is 16 feature images of size 10 × 10. Followed by a pooling layer with a pooling size of 2 × 2. Hence, the dimension of images through this layer is halved, it outputs 16 feature images of size 5 × 5.\n\n- F3 is the fully connected layer with 120 neurons. Since the inputs of this layer have the same size as the kernel, then the output size of this layer is 1 × 1. The number of channels in output equals the channel number of kernels, which is 120. Hence the output of this layer is 120 feature images of size  1 × 1.\n\n- F4 is a fully connected layer with 84 neurons which are all connected to the output of F3.\n\n- The output layer consists of 10 neurons corresponding to the number of classes (numbers from 0 to 9).","metadata":{}},{"cell_type":"code","source":"#Model Definition\ndigit_model = models.Sequential()\n\n#First Layer 6 kernels of size 5,5\ndigit_model.add(layers.Conv2D(6,(5,5), input_shape = (28,28,1), activation = 'relu', padding = \"same\"))\n#max pooling with poolind size 2,2\ndigit_model.add(layers.MaxPooling2D(pool_size = (2,2)))\n#Adding Batch Normalization\ndigit_model.add(layers.BatchNormalization())\n#Dropout\ndigit_model.add(layers.Dropout(0.1))\n\n#Second Layer 16 kernels of 5,5\ndigit_model.add(layers.Conv2D(16,(5,5), activation = 'relu'))\n#Pooling\ndigit_model.add(layers.MaxPooling2D(2,2))\n#Batch Norm\ndigit_model.add(layers.BatchNormalization())\n#Dropout\ndigit_model.add(layers.Dropout(0.3))\n\n#Flattening\ndigit_model.add(layers.Flatten())\n\n#NN with 120-84-10 layers\ndigit_model.add(layers.Dense(120, activation='relu'))\ndigit_model.add(layers.Dense(84, activation='relu'))\ndigit_model.add(layers.Dense(10, activation = \"softmax\"))","metadata":{"execution":{"iopub.status.busy":"2022-06-07T20:23:58.374999Z","iopub.execute_input":"2022-06-07T20:23:58.37562Z","iopub.status.idle":"2022-06-07T20:23:58.483288Z","shell.execute_reply.started":"2022-06-07T20:23:58.375579Z","shell.execute_reply":"2022-06-07T20:23:58.482372Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Model Summary\ndigit_model.summary()","metadata":{"execution":{"iopub.status.busy":"2022-06-07T20:23:59.567106Z","iopub.execute_input":"2022-06-07T20:23:59.568154Z","iopub.status.idle":"2022-06-07T20:23:59.576553Z","shell.execute_reply.started":"2022-06-07T20:23:59.568105Z","shell.execute_reply":"2022-06-07T20:23:59.575326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Training the Model with Early Stopping with patience for 5 epochs.\ncallback = tf.keras.callbacks.EarlyStopping(monitor='accuracy',patience=5)\ndigit_model.compile(optimizer = \"rmsprop\",loss = tf.keras.losses.categorical_crossentropy, metrics=[\"accuracy\"])","metadata":{"execution":{"iopub.status.busy":"2022-06-07T20:24:01.56365Z","iopub.execute_input":"2022-06-07T20:24:01.564226Z","iopub.status.idle":"2022-06-07T20:24:01.585724Z","shell.execute_reply.started":"2022-06-07T20:24:01.564192Z","shell.execute_reply":"2022-06-07T20:24:01.584651Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Model Fit\nLeNet_mod = digit_model.fit(epochs = 50, x = x_train, y= y_train,  verbose = 1, validation_data=(x_val,y_val),callbacks=callback)","metadata":{"execution":{"iopub.status.busy":"2022-06-07T20:24:02.349668Z","iopub.execute_input":"2022-06-07T20:24:02.350273Z","iopub.status.idle":"2022-06-07T20:31:24.557774Z","shell.execute_reply.started":"2022-06-07T20:24:02.350235Z","shell.execute_reply":"2022-06-07T20:31:24.556446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10,5))\nplt.plot(LeNet_mod.history[\"accuracy\"], label = \"Training Accuracy\")\nplt.plot(LeNet_mod.history[\"val_accuracy\"], label = \"Validation Accuracy\");\nplt.title('Accuracy for Training and Validation Data')\nplt.ylabel('Accuracy')\nplt.xlabel('No. of epoch')\nplt.legend(loc = \"lower right\");","metadata":{"execution":{"iopub.status.busy":"2022-06-07T20:35:49.185531Z","iopub.execute_input":"2022-06-07T20:35:49.185919Z","iopub.status.idle":"2022-06-07T20:35:49.450024Z","shell.execute_reply.started":"2022-06-07T20:35:49.18589Z","shell.execute_reply":"2022-06-07T20:35:49.44885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Loss\nplt.figure(figsize=(10,5))\nplt.plot(LeNet_mod.history[\"loss\"], label = \"Training Loss\")\nplt.plot(LeNet_mod.history[\"val_loss\"], label = \"Validation Loss\");\nplt.title('Loss for Training and Validation Data')\nplt.ylabel('Loss')\nplt.xlabel('No. of epoch')\nplt.legend(loc = \"upper right\");","metadata":{"execution":{"iopub.status.busy":"2022-06-07T20:36:46.772306Z","iopub.execute_input":"2022-06-07T20:36:46.772773Z","iopub.status.idle":"2022-06-07T20:36:47.058725Z","shell.execute_reply.started":"2022-06-07T20:36:46.772726Z","shell.execute_reply":"2022-06-07T20:36:47.057325Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Final Prediction\nprediction = np.argmax(digit_model.predict(test), axis=-1)\n#Generating Index\nImageid = [i for i in range(1,28000+1)]\n#Creating data frame\nd = {\"ImageId\":Imageid,\"Label\":prediction}\npred_df = pd.DataFrame(d)\npred_df.index = pred_df[\"ImageId\"]\npred_df.drop(\"ImageId\",axis=1,inplace=True)\npred_df.head()\npred_df.to_csv(\"/kaggle/working/sample_submission.csv\",index=False)","metadata":{"execution":{"iopub.status.busy":"2022-06-08T11:11:36.868527Z","iopub.execute_input":"2022-06-08T11:11:36.869625Z","iopub.status.idle":"2022-06-08T11:11:36.953257Z","shell.execute_reply.started":"2022-06-08T11:11:36.869508Z","shell.execute_reply":"2022-06-08T11:11:36.951941Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Wait, Hang on!! Time to **Up-Vote!!👆🏼**","metadata":{}}]}