{"cells":[{"metadata":{},"cell_type":"markdown","source":"# What is the best CNN architecture    \nThere are so many choices for CNN architecture. How do we choose the best one? First we must define what best means. The best may be the simplest, or it may be the most efficient at producing accuracy while minimizing computational complexity. In this kernel, we will run experiments to find the most accurate and efficient CNN architecture for classifying MNIST handwritten digits.\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Loading Data","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Here I'm using mnist original dataset","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd# data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Plot ad hoc mnist instances\nfrom keras.datasets import mnist\nimport matplotlib.pyplot as plt\n# load (downloaded if needed) the MNIST dataset\n(X_train, y_train), (X_test, y_test) = mnist.load_data()\n# plot 4 images as gray scale\nplt.subplot(221)\nplt.imshow(X_train[0], cmap=plt.get_cmap('gray'))\nplt.subplot(222)\nplt.imshow(X_train[1], cmap=plt.get_cmap('gray'))\nplt.subplot(223)\nplt.imshow(X_train[2], cmap=plt.get_cmap('gray'))\nplt.subplot(224)\nplt.imshow(X_train[3], cmap=plt.get_cmap('gray'))\n# show the plot\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# visualization","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def visualize_input(img, ax):\n    ax.imshow(img, cmap='gray')\n    width, height = img.shape\n    thresh = img.max()/2.5\n    for x in range(width):\n        for y in range(height):\n            ax.annotate(str(round(img[x][y],2)), xy=(y,x),\n                        horizontalalignment='center',\n                        verticalalignment='center',\n                        color='white' if img[x][y]<thresh else 'black')\n\nfig = plt.figure(figsize = (12,12)) \nax = fig.add_subplot(111)\nvisualize_input(X_train[1].reshape(28,28), ax)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns\n\ng = sns.countplot(y_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Different layers","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The CNN can isolate features that are useful everywhere from these transformed images (feature maps).","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 1. **convolutional (Conv2D) layer**:    \nIt is like a set of learnable filters. Each filter transforms a part of the image (defined by the kernel size) using the kernel filter. The kernel filter matrix is applied on the whole image. Filters can be seen as a transformation of the image.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://sds-platform-private.s3-us-east-2.amazonaws.com/uploads/35_blog_image_12.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Conv2D class: ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://raw.githubusercontent.com/soham1024/learn-cnn/master/Screenshot%20from%202020-05-09%2012-34-21.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"This layer creates a convolution kernel that is convolved with the layer input to produce a tensor of outputs. If use_bias is True, a bias vector is created and added to the outputs. Finally, if activation is not None, it is applied to the outputs as well.                     \n                         \nWhen using this layer as the first layer in a model, provide the keyword argument input_shape (tuple of integers, does not include the sample axis), e.g. input_shape=(128, 128, 3) for 128x128 RGB pictures in data_format=\"channels_last\".","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://raw.githubusercontent.com/soham1024/learn-cnn/master/Screenshot%20from%202020-05-09%2013-12-22.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Arguments:                     \n\n*     filters: Integer, the dimensionality of the output space (i.e. the number of output filters in the convolution).\n*     kernel_size: An integer or tuple/list of 2 integers, specifying the height and width of the 2D convolution window. Can be a single integer to specify the same value for all spatial dimensions.\n*     strides: An integer or tuple/list of 2 integers, specifying the strides of the convolution along the height and width. Can be a single integer to specify the same value for all spatial dimensions. Specifying any stride value != 1 is incompatible with specifying any dilation_rate value != 1.\n*     padding: one of \"valid\" or \"same\" (case-insensitive).\n*     data_format: A string, one of channels_last (default) or channels_first. The ordering of the dimensions in the inputs. channels_last corresponds to inputs with shape (batch_size, height, width, channels) while channels_first corresponds to inputs with shape (batch_size, channels, height, width). It defaults to the image_data_format value found in your Keras config file at ~/.keras/keras.json. If you never set it, then it will be \"channels_last\".\n*     dilation_rate: an integer or tuple/list of 2 integers, specifying the dilation rate to use for dilated convolution. Can be a single integer to specify the same value for all spatial dimensions. Currently, specifying any dilation_rate value != 1 is incompatible with specifying any stride value != 1.\n*     activation: Activation function to use. If you don't specify anything, no activation is applied ( see keras.activations).\n*    use_bias: Boolean, whether the layer uses a bias vector.\n*    kernel_initializer: Initializer for the kernel weights matrix ( see keras.initializers).\n*    bias_initializer: Initializer for the bias vector ( see keras.initializers).\n*    kernel_regularizer: Regularizer function applied to the kernel weights matrix (see keras.regularizers).\n*    bias_regularizer: Regularizer function applied to the bias vector ( see keras.regularizers).\n*    activity_regularizer: Regularizer function applied to the output of the layer (its \"activation\") ( see keras.regularizers).\n*    kernel_constraint: Constraint function applied to the kernel matrix ( see keras.constraints).\n*    bias_constraint: Constraint function applied to the bias vector ( see keras.constraints).\n\n**Input shape**\n\n*    4D tensor with shape: (batch_size, channels, rows, cols) if data_format='channels_first' or 4D tensor with shape: (batch_size, rows, cols, channels) if data_format='channels_last'.\n\n**Output shape**\n\n*    4D tensor with shape: (batch_size, filters, new_rows, new_cols) if data_format='channels_first' or 4D tensor with shape: (batch_size, new_rows, new_cols, filters) if data_format='channels_last'. rows and cols values might have changed due to padding.\n\n**Returns**\n\n*    A tensor of rank 4 representing activation(conv2d(inputs, kernel) + bias).\n\n**Raises**\n\n*    ValueError: if padding is \"causal\".\n*    ValueError: when both strides > 1 and dilation_rate > 1.\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 2. **pooling (MaxPool2D) layer**:         \nThis layer simply acts as a downsampling filter. It looks at the 2 neighboring pixels and picks the maximal value. These are used to reduce computational cost, and to some extent also reduce overfitting. We have to choose the pooling size (i.e the area size pooled each time) more the pooling dimension is high, more the downsampling is important.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://qph.fs.quoracdn.net/main-qimg-40cdeb3b43594f4b1b1b6e2c137e80b7.webp)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Combining convolutional and pooling layers, CNN are able to combine local features and learn more global features of the image.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://raw.githubusercontent.com/soham1024/learn-cnn/master/Screenshot%20from%202020-05-09%2013-07-34.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Max pooling operation for 2D spatial data.                 \n                        \nDownsamples the input representation by taking the maximum value over the window defined by pool_size for each dimension along the features axis. The window is shifted by strides in each dimension. The resulting output when using \"valid\" padding option has a shape(number of rows or columns) of: output_shape = (input_shape - pool_size + 1) / strides)                        \n                                   \nThe resulting output shape when using the \"same\" padding option is: output_shape = input_shape / strides","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://raw.githubusercontent.com/soham1024/learn-cnn/master/Screenshot%20from%202020-05-09%2013-08-21.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://raw.githubusercontent.com/soham1024/learn-cnn/master/Screenshot%20from%202020-05-09%2014-30-54.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://raw.githubusercontent.com/soham1024/learn-cnn/master/Screenshot%20from%202020-05-09%2014-32-24.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://raw.githubusercontent.com/soham1024/learn-cnn/master/Screenshot%20from%202020-05-09%2014-34-12.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Arguments\n\n*    pool_size: integer or tuple of 2 integers, window size over which to take the maximum. (2, 2) will take the max value over a 2x2 pooling window. If only one integer is specified, the same window length will be used for both dimensions.\n*    strides: Integer, tuple of 2 integers, or None. Strides values. Specifies how far the pooling window moves for each pooling step. If None, it will default to pool_size.\n*    padding: One of \"valid\" or \"same\" (case-insensitive). \"valid\" adds no zero padding. \"same\" adds padding such that if the stride is 1, the output shape is the same as input shape.\n*    data_format: A string, one of channels_last (default) or channels_first. The ordering of the dimensions in the inputs. channels_last corresponds to inputs with shape (batch, height, width, channels) while channels_first corresponds to inputs with shape (batch, channels, height, width). It defaults to the image_data_format value found in your Keras config file at ~/.keras/keras.json. If you never set it, then it will be \"channels_last\".\n\n**Input shape**\n\n*    If data_format='channels_last': 4D tensor with shape (batch_size, rows, cols, channels).\n*    If data_format='channels_first': 4D tensor with shape (batch_size, channels, rows, cols).\n\n**Output shape**\n\n*    If data_format='channels_last': 4D tensor with shape (batch_size, pooled_rows, pooled_cols, channels).\n*    If data_format='channels_first': 4D tensor with shape (batch_size, channels, pooled_rows, pooled_cols).\n\n**Returns**\n\n*    A tensor of rank 4 representing the maximum pooled values. See above for output shape.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 3. **Dropout**:      \nIt is a regularization method, where a proportion of nodes in the layer are randomly ignored (setting their wieghts to zero) for each training sample. This drops randomly a propotion of the network and forces the network to learn features in a distributed way. This technique also improves generalization and reduces the overfitting.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://www.oreilly.com/library/view/deep-learning-for/9781788295628/assets/d4d20bd7-192c-48e7-9da2-6d3ddc7929e7.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://raw.githubusercontent.com/soham1024/learn-cnn/master/Screenshot%20from%202020-05-09%2014-39-46.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The Dropout layer randomly sets input units to 0 with a frequency of rate at each step during training time, which helps prevent overfitting. Inputs not set to 0 are scaled up by 1/(1 - rate) such that the sum over all inputs is unchanged.\n\nNote that the Dropout layer only applies when training is set to True such that no values are dropped during inference. When using model.fit, training will be appropriately set to True automatically, and in other contexts, you can set the kwarg explicitly to True when calling the layer.\n\n(This is in contrast to setting trainable=False for a Dropout layer. trainable does not affect the layer's behavior, as Dropout does not have any variables/weights that can be frozen during training.)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://raw.githubusercontent.com/soham1024/learn-cnn/master/Screenshot%20from%202020-05-09%2014-58-44.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Arguments:\n\n*    rate: Float between 0 and 1. Fraction of the input units to drop.\n*    noise_shape: 1D integer tensor representing the shape of the binary dropout mask that will be multiplied with the input. For instance, if your inputs have shape (batch_size, timesteps, features) and you want the dropout mask to be the same for all timesteps, you can use noise_shape=(batch_size, 1, features).\n *   seed: A Python integer to use as random seed.\n\n**Call arguments:**\n\n    inputs: Input tensor (of any rank).\n    training: Python boolean indicating whether the layer should behave in training mode (adding dropout) or in inference mode (doing nothing).\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# **relu**:     \nIt is the rectifier (activation function max(0,x). The rectifier activation function is used to add non linearity to the network.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://qph.fs.quoracdn.net/main-qimg-07bc0ec05532caf5ebe8b4c82d0f5ca3)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://i.pinimg.com/originals/bd/31/d2/bd31d2c58e90916640168e31014595cf.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://raw.githubusercontent.com/soham1024/learn-cnn/master/Screenshot%20from%202020-05-09%2015-15-06.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Rectified Linear Unit activation function.\n\nWith default values, it returns element-wise max(x, 0).\n\nOtherwise, it follows:","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://raw.githubusercontent.com/soham1024/learn-cnn/master/Screenshot%20from%202020-05-09%2015-17-34.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Arguments :\n\n*    max_value: Float >= 0. Maximum activation value. Default to None, which means unlimited.\n*    negative_slope: Float >= 0. Negative slope coefficient. Default to 0.\n*    threshold: Float. Threshold value for thresholded activation. Default to 0.\n\n**Input shape**\n\n*    Arbitrary. Use the keyword argument input_shape (tuple of integers, does not include the batch axis) when using this layer as the first layer in a model.\n\n**Output shape**\n\n*    Same shape as the input.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 4. Flatten layer:    \nIt is use to convert the final feature maps into a one single 1D vector. This flattening step is needed so that you can make use of fully connected layers after some convolutional/maxpool layers. It combines all the found local features of the previous convolutional layers.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://sds-platform-private.s3-us-east-2.amazonaws.com/uploads/73_blog_image_2.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Flattens the input. Does not affect the batch size.\n\nNote: If inputs are shaped (batch,) without a feature axis, then flattening adds an extra channel dimension and output shape is (batch, 1).","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://raw.githubusercontent.com/soham1024/learn-cnn/master/Screenshot%20from%202020-05-09%2015-22-21.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://raw.githubusercontent.com/soham1024/learn-cnn/master/Screenshot%20from%202020-05-09%2015-24-51.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Arguments:\n\n*    data_format: A string, one of channels_last (default) or channels_first. The ordering of the dimensions in the inputs. channels_last corresponds to inputs with shape (batch, ..., channels) while channels_first corresponds to inputs with shape (batch, channels, ...). It defaults to the image_data_format value found in your Keras config file at ~/.keras/keras.json. If you never set it, then it will be \"channels_last\".\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 5. Dense layer:        \nIt is just artificial neural networks (ANN) classifier.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://deeplizard.com/images/deep%20neural%20network%20with%204%20layers.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://raw.githubusercontent.com/soham1024/learn-cnn/master/Screenshot%20from%202020-05-09%2015-27-03.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Just your regular densely-connected NN layer.\n\nDense implements the operation: output = activation(dot(input, kernel) + bias) where activation is the element-wise activation function passed as the activation argument, kernel is a weights matrix created by the layer, and bias is a bias vector created by the layer (only applicable if use_bias is True).\n\nNote: If the input to the layer has a rank greater than 2, then Dense computes the dot product between the inputs and the kernel along the last axis of the inputs and axis 1 of the kernel (using tf.tensordot). For example, if input has dimensions (batch_size, d0, d1), then we create a kernel with shape (d1, units), and the kernel operates along axis 2 of the input, on every sub-tensor of shape (1, 1, d1) (there are batch_size * d0 such sub-tensors). The output in this case will have shape (batch_size, d0, units).\n\nBesides, layer attributes cannot be modified after the layer has been called once (except the trainable attribute).","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://raw.githubusercontent.com/soham1024/learn-cnn/master/Screenshot%20from%202020-05-09%2015-29-12.png)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Arguments: \n\n*    units: Positive integer, dimensionality of the output space.\n*    activation: Activation function to use. If you don't specify anything, no activation is applied (ie. \"linear\" activation: a(x) = x).\n*    use_bias: Boolean, whether the layer uses a bias vector.\n*    kernel_initializer: Initializer for the kernel weights matrix.\n*    bias_initializer: Initializer for the bias vector.\n*    kernel_regularizer: Regularizer function applied to the kernel weights matrix.\n*    bias_regularizer: Regularizer function applied to the bias vector.\n*    activity_regularizer: Regularizer function applied to the output of the layer (its \"activation\").\n*    kernel_constraint: Constraint function applied to the kernel weights matrix.\n*    bias_constraint: Constraint function applied to the bias vector.\n\n**Input shape**\n\n*    N-D tensor with shape: (batch_size, ..., input_dim). The most common situation would be a 2D input with shape (batch_size, input_dim).\n\n**Output shape**\n\n*    N-D tensor with shape: (batch_size, ..., units). For instance, for a 2D input with shape (batch_size, input_dim), the output would have shape (batch_size, units).","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"####################################################################################################################","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**I've used the Keras Sequential API, where you just have to add one layer at a time, starting from the input.**\n\nIn the last layer(Dense(X,activation=\"softmax\")) the net outputs distribution of probability of each class.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"![](https://miro.medium.com/max/3600/1*dOv2a1ctNrHDo8Zks30Bbw.png)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.datasets import mnist\nfrom keras.models import Sequential\nfrom keras.layers import Dense\nfrom keras.layers import Dropout\nfrom keras.utils import np_utils","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#load data\n(X_train, y_train), (X_test, y_test) = mnist.load_data()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Flatten","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#flatten 28*28 images to a 784 vector for each image\nnum_pixels = X_train.shape[1] * X_train.shape[2]\nX_train = X_train.reshape((X_train.shape[0], num_pixels)).astype('float32')\nX_test = X_test.reshape((X_test.shape[0], num_pixels)).astype('float32')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Normalize","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# normalize inputs from 0-255 to 0-1\nX_train = X_train / 255\nX_test = X_test / 255","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# One Hot Encode","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# one hot encode outputs\ny_train = np_utils.to_categorical(y_train)\ny_test = np_utils.to_categorical(y_test)\nnum_classes = y_test.shape[1]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Baseline Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# define baseline model\ndef baseline_model():\n\t# create model\n\tmodel = Sequential()\n\tmodel.add(Dense(num_pixels, input_dim=num_pixels, kernel_initializer='normal', activation='relu'))\n\tmodel.add(Dense(num_classes, kernel_initializer='normal', activation='softmax'))\n\t# Compile model\n\tmodel.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n\treturn model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# build the model\nmodel = baseline_model()\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Fit the model\nmodel.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=10, batch_size=200, verbose=2)\n# Final evaluation of the model\nscores = model.evaluate(X_test, y_test, verbose=0)\nprint(\"Baseline Error: %.2f%%\" % (100-scores[1]*100))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Simple CNN","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Simple CNN for the MNIST Dataset\nfrom keras.datasets import mnist\nfrom keras.models import Sequential\nfrom keras.layers import Dense\nfrom keras.layers import Dropout\nfrom keras.layers import Flatten\nfrom keras.layers.convolutional import Conv2D\nfrom keras.layers.convolutional import MaxPooling2D\nfrom keras.utils import np_utils\n\n# load data\n(X_train, y_train), (X_test, y_test) = mnist.load_data()\n\n# reshape to be [samples][width][height][channels]\nX_train = X_train.reshape((X_train.shape[0], 28, 28, 1)).astype('float32')\nX_test = X_test.reshape((X_test.shape[0], 28, 28, 1)).astype('float32')\n\n# normalize inputs from 0-255 to 0-1\nX_train = X_train / 255\nX_test = X_test / 255\n\n# one hot encode outputs\ny_train = np_utils.to_categorical(y_train)\ny_test = np_utils.to_categorical(y_test)\nnum_classes = y_test.shape[1]\n\n# define a simple CNN model\ndef baseline_model():\n\t# create model\n\tmodel = Sequential()\n\tmodel.add(Conv2D(32, (5, 5), input_shape=(28, 28, 1), activation='relu'))\n\tmodel.add(MaxPooling2D())\n\tmodel.add(Dropout(0.2))\n\tmodel.add(Flatten())\n\tmodel.add(Dense(128, activation='relu'))\n\tmodel.add(Dense(num_classes, activation='softmax'))\n\t# Compile model\n\tmodel.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n\treturn model\n\n# build the model\nmodel_simple = baseline_model()\nmodel_simple.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Fit the model\nmodel_simple.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=10, batch_size=200)\n\n# Final evaluation of the model\nscores = model_simple.evaluate(X_test, y_test, verbose=1)\n\nprint(\"CNN Error: %.2f%%\" % (100-scores[1]*100))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Large CNN","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Large CNN for the MNIST Dataset\nfrom keras.datasets import mnist\nfrom keras.models import Sequential\nfrom keras.layers import Dense\nfrom keras.layers import Dropout\nfrom keras.layers import Flatten\nfrom keras.layers.convolutional import Conv2D\nfrom keras.layers.convolutional import MaxPooling2D\nfrom keras.utils import np_utils\n\n# load data\n(X_train, y_train), (X_test, y_test) = mnist.load_data()\n\n# reshape to be [samples][width][height][channels]\nX_train = X_train.reshape((X_train.shape[0], 28, 28, 1)).astype('float32')\nX_test = X_test.reshape((X_test.shape[0], 28, 28, 1)).astype('float32')\n\n# normalize inputs from 0-255 to 0-1\nX_train = X_train / 255\nX_test = X_test / 255\n\n# one hot encode outputs\ny_train = np_utils.to_categorical(y_train)\ny_test = np_utils.to_categorical(y_test)\nnum_classes = y_test.shape[1]\n\n# define the larger model\ndef large_model():\n\t# create model\n\tmodel = Sequential()\n\tmodel.add(Conv2D(30, (5, 5), input_shape=(28, 28, 1), activation='relu'))\n\tmodel.add(MaxPooling2D())\n\tmodel.add(Conv2D(15, (3, 3), activation='relu'))\n\tmodel.add(MaxPooling2D())\n\tmodel.add(Dropout(0.2))\n\tmodel.add(Flatten())\n\tmodel.add(Dense(128, activation='relu'))\n\tmodel.add(Dense(50, activation='relu'))\n\tmodel.add(Dense(num_classes, activation='softmax'))\n\t# Compile model\n\tmodel.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n\treturn model\n\n# build the model\nmodel_large = large_model()\nmodel_large.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Fit the model\nmodel_large.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=10, batch_size=200)\n# Final evaluation of the model\n\nscores = model_large.evaluate(X_test, y_test, verbose=1)\n\nprint(\"Large CNN Error: %.2f%%\" % (100-scores[1]*100))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Larger CNN","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Larger CNN for the MNIST Dataset\nfrom keras.datasets import mnist\nfrom keras.models import Sequential\nfrom keras.layers import Dense\nfrom keras.layers import Dropout\nfrom keras.layers import Flatten\nfrom keras.layers.convolutional import Conv2D\nfrom keras.layers.convolutional import MaxPooling2D\nfrom keras.utils import np_utils\n\n# load data\n(X_train, y_train), (X_test, y_test) = mnist.load_data()\n\n# reshape to be [samples][width][height][channels]\nX_train = X_train.reshape((X_train.shape[0], 28, 28, 1)).astype('float32')\nX_test = X_test.reshape((X_test.shape[0], 28, 28, 1)).astype('float32')\n\n# normalize inputs from 0-255 to 0-1\nX_train = X_train / 255\nX_test = X_test / 255\n\n# one hot encode outputs\ny_train = np_utils.to_categorical(y_train)\ny_test = np_utils.to_categorical(y_test)\nnum_classes = y_test.shape[1]\n\n# define the larger model\ndef larger_model():\n    # create model\n    model = Sequential()\n    model.add(Conv2D(filters=64, kernel_size=3, padding='same', activation='relu', input_shape=(28,28,1)))\n    model.add(Conv2D(filters=64, kernel_size=3, padding='same', activation='relu'))\n    model.add(Conv2D(filters=128, kernel_size=3, padding='same', activation='relu'))\n    model.add(MaxPooling2D(pool_size=2))\n    model.add(Dropout(0.1))\n    model.add(Conv2D(filters=128, kernel_size=3, padding='same', activation='relu'))\n    model.add(Conv2D(filters=192, kernel_size=3, padding='same', activation='relu'))\n    model.add(MaxPooling2D(pool_size=2))\n    model.add(Dropout(0.1))\n    model.add(Conv2D(filters=192, kernel_size=5, padding='same', activation='relu'))\n    model.add(MaxPooling2D(pool_size=2, padding='same'))\n    model.add(Flatten())\n    model.add(Dense(256, activation='relu'))\n    model.add(Dense(10, activation='softmax'))\n    # Compile model\n    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n    return model\n\n# build the model\nmodel_larger = larger_model()\nmodel_larger.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Fit the model\nmodel_larger.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=10, batch_size=100)\n# Final evaluation of the model\n\nscores = model_larger.evaluate(X_test, y_test, verbose=1)\n\nprint(\"Larger CNN Error: %.2f%%\" % (100-scores[1]*100))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Choosing final Model\nFrom above all the cases as we can see the last model is having the best accuracy. \nTraining convolutional neural networks is a random process. This makes experiments difficult because each time you run the same experiment, you get different results. Therefore, you must run your experiments dozens of times and take an average. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nX_test = pd.read_csv('/kaggle/input/digit-recognizer/test.csv').values.astype('float32')\nX_test = X_test.reshape(-1, 28, 28, 1)\nX_test = X_test.astype('float32')/255\ntestY = model_larger.predict_classes(X_test, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.read_csv('/kaggle/input/digit-recognizer/sample_submission.csv')\nsub['Label'] = testY\nsub.to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"to know how to make neural network without using any deep learning tools(Like: Tensorflow, Pytorch,etc) got through my another notebook: [Create Neural Network From Scratch](https://www.kaggle.com/soham1024/create-neural-network-from-scratch)     \n\n### how inside the Neural Network work is done: ","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from IPython.display import YouTubeVideo\nYouTubeVideo('3JQ3hYko51Y', width=800, height=450)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"reference: \n* https://keras.io/\n* https://www.kaggle.com/cdeotte/how-to-choose-cnn-architecture-mnist\n* https://www.kaggle.com/yassineghouzam/introduction-to-cnn-keras-0-997-top-6","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# **If you find this notebook helpful or you just liked it , some upvotes would be very much appreciated - That will keep me motivated :)**","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}