{"cells":[{"metadata":{"_uuid":"2889ef6a4303142d4505aebfc779f4ff399ceff6"},"cell_type":"markdown","source":"# **Digit Classifier using Estimators**\n\nEstimator class is used to train and evaluate TensorFlow models. It wraps a model which is specified by a model_fn, which, given inputs and a number of other parameters, returns the operations necessary to perform training, evaluation, and predictions.\n\nWe will use DNNClassifier to classify the hand-written digits in the MNIST dataset.\n\nStart by importing the required packages we need to build the model"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport seaborn as sns\n\nfrom __future__ import absolute_import, division, print_function\nfrom keras.preprocessing.image import ImageDataGenerator\nimport tensorflow as tf\nimport numpy as np\n\ntf.logging.set_verbosity(tf.logging.INFO)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"## **Data Preparation and Visualization**\n\nLoad the training and test data"},{"metadata":{"trusted":true,"_uuid":"50cd9ded137c004d128fb8f51905f28d7abb6732"},"cell_type":"code","source":"# Load the training and test data\ntrain = pd.read_csv(\"../input/train.csv\")\ntest = pd.read_csv(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"acbe6bd4dbca7ec66cadb5531733745b6bd92a7c"},"cell_type":"code","source":"# separate out label into a different data frames\ntrain_y = train['label']\ntrain_x = train.drop(labels = [\"label\"],axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"779f35782a4ffc8d796c05dd0c92f6a82dc1b5e0"},"cell_type":"code","source":"# print the number of labels \ntrain_y.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"00766d326bcd5a49abb429060d0fbc002d31ee8b"},"cell_type":"markdown","source":"This shows that we have approximately same number of training data for each digit. "},{"metadata":{"_uuid":"411c4573a2315b5e14d2850560d35a19fed777e9"},"cell_type":"markdown","source":"### **Normalization**\n\nNormalization is a process that changes the range of pixel intensity values. The motivation is to achieve consistency in dynamic range for a set of data, signals, or images to avoid mental distraction."},{"metadata":{"trusted":true,"_uuid":"8981056228445615720861e5ca77ceaca1fc2ffd"},"cell_type":"code","source":"# Normalize the data\ntrain_x = train_x / 255.0\ntest = test / 255.0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"549baa164e0a45150417b2859c5783adfa0899f3"},"cell_type":"code","source":"# Reshape image in 3 dimensions (28 height x 28 width x 1 channel for gray)\ntrain_x = train_x.values.reshape(-1, 28, 28, 1)\ntest = test.values.reshape(-1, 28, 28, 1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3bafe27862bb9c42423f800d35ab567ab1bbf352"},"cell_type":"markdown","source":"### **Augmentation**\n\nWe will use Data Augmentation technique to increase the training data set to avoid overfitting the model. The motivation is to generate variations of the training dataset (for eg, zooming the image, or shiting the number of the left/right, or rotating the image in any random direction)****"},{"metadata":{"trusted":true,"_uuid":"49b6876f0fb59fcb25b8cab3ff9bc2419729750b"},"cell_type":"code","source":"datagen = ImageDataGenerator(zoom_range = 0.1,\n                            height_shift_range = 0.1,\n                            zca_whitening=False,\n                            width_shift_range = 0.1,\n                            rotation_range = 10)\n\n# this is not being used for now\ndatagen.fit(train_x)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4b4e60ede70873ead40a78e7cf0c089129050a59"},"cell_type":"markdown","source":"## **Building the CNN Classifier**\n\nCNN (Convolutional Neural Network) is the state of the art model whenever it comes to classifying images. The layers in CNN learns the high level features of the given image which is later used for classification.\n\nThe CNN consists of three major components:\n\n1. Convolutional Layer: Applies the convolutional filter to a given image. This means it applies mathematical operations for a sub region to produce a single value in a feature output map.\n2. Pooling Layer: It is used to reduce the dimensionality of the image to improve the training time.\n3. Dense Layer: This is the final layer which performs the classification.\n\nSince now we have a overall idea of what CNN is, let's try building a CNN model using the above mentioned layers"},{"metadata":{"trusted":true,"_uuid":"d0cd8c34447551627190b85ac1734bb755169e20"},"cell_type":"code","source":"def cnn_model(features, labels, mode):\n    # Input Layer\n    input_layer = tf.reshape(features[\"x\"], [-1, 28, 28, 1])\n\n    # 1st Convolutional Layer\n    conv1 = tf.layers.conv2d(inputs=input_layer, filters=32, \n                           kernel_size=[5, 5], padding=\"same\",\n                           activation=tf.nn.relu)\n\n    # 1st Pooling Layer\n    pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)\n\n    # 2nd Convolutional Layer\n    # This takes the output of previous pool layer as it's input\n    conv2 = tf.layers.conv2d(inputs=pool1, filters=64, \n                           kernel_size=[5, 5],padding=\"same\", \n                           activation=tf.nn.relu)\n    \n    # 2nd Pooling Layer\n    pool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2)\n\n    # Before connecting the layer, we'll flatten the feature map\n    dense = tf.layers.dense(inputs=tf.reshape(pool2, [-1, 7 * 7 * 64]), units=1024, activation=tf.nn.relu)\n    \n    # to improve the results apply dropout regularization to the layer to reduce overfitting\n    dropout = tf.layers.dropout(inputs=dense, rate=0.4, training=mode == tf.estimator.ModeKeys.TRAIN)\n\n    # Logits Layer\n    logits = tf.layers.dense(inputs=dropout, units=10)\n\n    predictions = {\n          \"classes\": tf.argmax(input=logits, axis=1),\n          \"probabilities\": tf.nn.softmax(logits, name=\"softmax_tensor\")\n    }\n\n    if mode == tf.estimator.ModeKeys.PREDICT:\n        return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)\n\n    # Calculate Loss (for both TRAIN and EVAL modes)\n    loss = tf.losses.sparse_softmax_cross_entropy(labels=labels, logits=logits)\n\n    # Configure the Training Op (for TRAIN mode)\n    if mode == tf.estimator.ModeKeys.TRAIN:\n        optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001)\n        train_op = optimizer.minimize(loss=loss, global_step=tf.train.get_global_step())\n        \n        return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op)\n\n    # Add evaluation metrics (for EVAL mode)\n    eval_metric_ops = {\n      \"accuracy\": tf.metrics.accuracy(labels=labels, predictions=predictions[\"classes\"])\n    }\n\n    return tf.estimator.EstimatorSpec(mode=mode, loss=loss, eval_metric_ops=eval_metric_ops)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4f695ff1fe5299388745bbc4a99fb08c4c0a2ed9"},"cell_type":"markdown","source":"### **Breaking it down**\n\nThis is how the model looks like: **Conv Layer -> Pool Layer -> Conv Layer -> Pool Layer -> Flatten -> Dropout Layer -> Dense Output Layer**\n\n**Input Layer**\n\n> input_layer = tf.reshape(features[\"x\"], [-1, 28, 28, 1])\n\nThe Convolutional and pooling layer expects the inputs to be in format of [batch_size, image_height, image_width, channels]. Since we have 28(width) x 28(height) x 1(channel) image dataset, hence the shape of the input layer.\n\n**1st Convolutional Layer**\n\n> conv1 = tf.layers.conv2d(inputs=input_layer, filters=32, \n>                            kernel_size=[5, 5], padding=\"same\",\n>                            activation=tf.nn.relu)\n\nThe first convolutional layer applies 32 5x5 filters to the input layer with ReLU as the activation function (to learn non-learnier features)\n\n**1st Pooling Layer**\n\n> pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)\n\nThe pooling layers applies a pool size of 2 to the output of the 1st convolutional layer to reduce the dimensionality of the inputs. The strides argument specifies the size of the stride. Stride of 2 means that the subregions extracted by the filter should be separated by 2 pixels.\n\n**2nd Convolutional Layer**\n\n> conv2 = tf.layers.conv2d(inputs=pool1, filters=64, \n>                            kernel_size=[5, 5],padding=\"same\", \n>                            activation=tf.nn.relu)\n\nThe convolutional layer of the same parameters are applied to the output of the 1st pooling layer.\n\n**2nd Pooling Layer**\n\n> pool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2)\n\nThe pooling layer of the same parameters are applied to the output of the 2nd convolutional layer.\n\n**Flattent Layer**\n\n> dense = tf.layers.dense(inputs=tf.reshape(pool2, [-1, 7 * 7 * 64]), units=1024, activation=tf.nn.relu)\n\nBefore generating the dense layer, we will have to flatten the feature map\n\n**Dropout Layer**\n\n> dropout = tf.layers.dropout(inputs=dense, rate=0.4, training=mode == tf.estimator.ModeKeys.TRAIN)\n\nTo help improve the results of the model we  apply dropout regularization to our dense flattened layer. The rate argument specifies the dropout rate; here, we use 0.4, which means 40% of the elements will be randomly dropped out during the training phase.\n\n**Dense Logits Layer**\n\n> logits = tf.layers.dense(inputs=dropout, units=10)\n\nThe final layer is the logits layer, which will return the raw values for the predictions."},{"metadata":{"trusted":true,"_uuid":"918be88c0112040a886465fcc9a06b5eaaf4ce13"},"cell_type":"code","source":"import numpy as np\nimport tensorflow as tf\nimport logging\nfrom tensorflow.python.training import session_run_hook\n\nclass EarlyStoppingHook(session_run_hook.SessionRunHook):\n    \"\"\"Hook that requests stop at a specified step.\"\"\"\n\n    def __init__(self, monitor='val_loss', min_delta=0, patience=0,\n                 mode='auto'):\n        \"\"\"\n        \"\"\"\n        self.monitor = monitor\n        self.patience = patience\n        self.min_delta = min_delta\n        self.wait = 0\n        if mode not in ['auto', 'min', 'max']:\n            logging.warning('EarlyStopping mode %s is unknown, '\n                            'fallback to auto mode.', mode, RuntimeWarning)\n            mode = 'auto'\n\n        if mode == 'min':\n            self.monitor_op = np.less\n        elif mode == 'max':\n            self.monitor_op = np.greater\n        else:\n            if 'acc' in self.monitor:\n                self.monitor_op = np.greater\n            else:\n                self.monitor_op = np.less\n\n        if self.monitor_op == np.greater:\n            self.min_delta *= 1\n        else:\n            self.min_delta *= -1\n\n        self.best = np.Inf if self.monitor_op == np.less else -np.Inf\n\n    def begin(self):\n        # Convert names to tensors if given\n        graph = tf.get_default_graph()\n        self.monitor = graph.as_graph_element(self.monitor)\n        if isinstance(self.monitor, tf.Operation):\n            self.monitor = self.monitor.outputs[0]\n\n    def before_run(self, run_context):  # pylint: disable=unused-argument\n        return session_run_hook.SessionRunArgs(self.monitor)\n\n    def after_run(self, run_context, run_values):\n        current = run_values.results\n\n        if self.monitor_op(current - self.min_delta, self.best):\n            self.best = current\n            self.wait = 0\n        else:\n            self.wait += 1\n            if self.wait >= self.patience:\n                run_context.request_stop()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3da7dbf18ca265b58ecf7e5ce426a19ca01a0a8f"},"cell_type":"markdown","source":"### **Training the classifier**\n\nSince we have the model ready, let's train the model using the Estimator API's"},{"metadata":{"trusted":true,"_uuid":"7d17b1427b7afc887898d5a357065a7075d1f4ed"},"cell_type":"code","source":"# Create the Estimator\nclassifier = tf.estimator.Estimator(model_fn=cnn_model, model_dir=\"/tmp/model\")\n\nearly_stopping_hook = EarlyStoppingHook(monitor='sparse_softmax_cross_entropy_loss/value', patience=10)\n\n# Train the model\ntrain_input_fn = tf.estimator.inputs.numpy_input_fn(x={\"x\": train_x},\n                                                    y=train_y,\n                                                    batch_size=64,\n                                                    num_epochs=200,\n                                                    shuffle=True)\n\nclassifier.train(input_fn=train_input_fn, steps=30000)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bd12d02f00b4ce77ed8c1b5fd6abc0b6674e43fd"},"cell_type":"markdown","source":"### **Predicting values**"},{"metadata":{"trusted":true,"_uuid":"0c6ecde9d4a944801c8df2cff162f5ea718a1016"},"cell_type":"code","source":"predict_fn = tf.estimator.inputs.numpy_input_fn(x={\"x\": test},\n                                                shuffle=False)\n\neval_results = classifier.predict(input_fn=predict_fn)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"341a4454c1a391e998077271f8633830ee7da329"},"cell_type":"code","source":"# predict results\nresult = []\nfor i in eval_results:\n    result.append(i['classes'])\n\nresults = pd.Series(result,name=\"Label\")\nsubmission = pd.concat([pd.Series(range(1,28001),name = \"ImageId\"),results],axis = 1)\nsubmission.to_csv(\"submission.csv\",index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"21d5f242e77ac0ea44e6daa162492c04f63046c2"},"cell_type":"markdown","source":"## **Pending Tasks**\n\nThis model is a baseline model. There is a lot of scope of impovement here like tuning the hyperparameters, trying out different types of learning optimizers, and more\n\n1. Hyperparameter tuning: Tuning the learning rate, with different optimizers and loss functions\n2. Data Augementation: Enable data augmentation to reduce overfitting\n3. Adding early stop ie train till the model stops learning and thus auto stops\n4. Defining the confusion matrix\n5. Analysing ROC curve to evaluate the performance of classification"},{"metadata":{"trusted":true,"_uuid":"2d3e72e49650976efbcf1b66351cd08f26fb2855"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}