{"cells":[{"metadata":{"_uuid":"962956c62c78554725770866d976495e242c0a20"},"cell_type":"markdown","source":"## **Digit Recognizer with Logistic Regression**"},{"metadata":{"_uuid":"1d780cacc6a29068e1f8200e73ebb36671cedaef"},"cell_type":"markdown","source":"### **Content**\n* [Data Exploration](#1)\n* [Normalization](#2)\n* [Train-Test Split](#3)\n* [Initialization of Weights and Bias](#4)\n* [Sigmoid Function For Logistic Regression](#5)\n* [Forward - Backward Propagation](#6)\n* [Updating Parameters Using Forward and Backward Propagation](#7)\n* [Prediction Method For Logistic Regression Model](#8)\n* [Defining The Logistic Regression Method](#9)\n* [Conclusion](#10)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport matplotlib.cm as cm\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\nimport warnings\nwarnings.filterwarnings('ignore')\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9baedb65e3ccb68007e2bb88d8e8f8ee53d88433"},"cell_type":"markdown","source":"<a id=\"1\"></a> \n**Data Exploration**"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"trainData = pd.read_csv('../input/train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f61a9a87d0638784ba63f23aed5c7c994a8af2dc"},"cell_type":"code","source":"trainData.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"98efb74c5cafa621cf749f1210de16909543d24b"},"cell_type":"code","source":"trainData.info() # We have 42000 samples with 785 columns including label column","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0e48242c8996f63ecd11fd911d37d9dfa4251f4c"},"cell_type":"code","source":"# Visualization of our digits\nsns.countplot(trainData.label)\nplt.title('Digit Count', color = 'blue', fontsize = 15)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5bc2e26e2c366d7f8a0eb4aef81b0711d2f860d7"},"cell_type":"markdown","source":"* We will use two digits for binary classification. Lets take two label from train data. (3 and 8)"},{"metadata":{"trusted":true,"_uuid":"f0fd75dd544b822b3b7d4f8af8177773645333f6"},"cell_type":"code","source":"trainData = trainData[(trainData[\"label\"] == 3) | (trainData[\"label\"] == 8)] # Filtering the train data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a27f9e2dbee44b05df56ce7b213a5752bf8d87d2"},"cell_type":"code","source":"trainData.head() # Now we have only 3 and 8 labeled numbers","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6f84d60aa0755e1b795acc9f63e23c3e2969fd87"},"cell_type":"code","source":"trainData.label = [1 if each == 3 else 0 for each in trainData.label] # We will call 1 for number three and 0 for number eight. We need binary results for logistic regression.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"86d9a6d1b470ac34335a04343e7e4d1c690f42fe"},"cell_type":"code","source":"y = trainData.label.values # Only labels for model training\nx_data = trainData.drop([\"label\"], axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"41c7ddadd0dda32ee685595f20ba9c72f5ef865b"},"cell_type":"code","source":"x_data.head() # Train data wihtout label.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cc2345c0f5b70b6285968fced45374f9fc1d9d5d"},"cell_type":"markdown","source":"<a id=\"2\"></a> \n**Normalization** <br/>\n\n&emsp; * Each pixel value is in the range 0..256. We will convert this range to 0..1*"},{"metadata":{"trusted":true,"_uuid":"187d9fc70a259f081e05f6ee5879c266ea70605b"},"cell_type":"code","source":"x = (x_data / 255.0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b555f01733cd6f0d1fa78b582f611e330eeb964a"},"cell_type":"markdown","source":"We have 8414 images. 28x28 pixels consisting of three and eight"},{"metadata":{"trusted":true,"_uuid":"e5b59c863e26788dad7494cc3987aeb321ded31f"},"cell_type":"code","source":"print('digits({0[0]},{0[1]})'.format(x.shape))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"605e47c3cd227c08c2d26dad29d0ec93f80b08c1"},"cell_type":"markdown","source":"*Lets look some sample of digits.*"},{"metadata":{"trusted":true,"_uuid":"1b17abb7cb3d9bcd62b6d5e8bf20fb6854ca185e"},"cell_type":"code","source":"def showDigit(index):\n    sampleDigit = x.iloc[:,0:].values[index]\n    digit = sampleDigit.reshape(28, 28)\n    plt.axis('off')\n    plt.imshow(digit, cmap = cm.binary) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"24c1e82c41c449633ee9c9d553c00534d189531c"},"cell_type":"code","source":"showDigit(26) # 25 - 26 is random member of our array. Just for check the how the numbers are look","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"defc548d02558847281d0df6f57ebc53fb38aa67"},"cell_type":"code","source":"showDigit(25)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"941d7f3f61fbd43ac625c80408c9e2308faf0030"},"cell_type":"markdown","source":"<a id=\"3\"></a> \n***Train-Test Split***"},{"metadata":{"trusted":true,"_uuid":"df263b1a032cdb67c9ebf3e9e232244833d5f23e"},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\nx_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.2, random_state = 42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9525aa731e0010c7e1b04602b5462cb362356d73"},"cell_type":"code","source":"x_train = x_train.T\nx_test = x_test.T\ny_train = y_train.T\ny_test = y_test.T\n\nprint(\"x_train : \", x_train.shape)\nprint(\"x_test : \", x_test.shape)\nprint(\"y_train : \", y_train.shape)\nprint(\"y_test : \", y_test.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"518d09463aa8b026c2fb7f0fa0f3bad559386bfc"},"cell_type":"markdown","source":"<a id=\"4\"></a> \n**Initialization of weights and bias.**"},{"metadata":{"trusted":true,"_uuid":"8996f4414452d833d7420cbe313a7ab39bb15f3c"},"cell_type":"code","source":"def initialize_weights_and_bias(dimension):\n    w = np.full((dimension, 1), 0.01)\n    b = 0.0\n    return w,b","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7fe5f26525a5aaed24eb6ea26cb1561c129569e7"},"cell_type":"markdown","source":"<a id=\"5\"></a> \n**Sigmoid function for logistic regression**"},{"metadata":{"trusted":true,"_uuid":"fa1cd02633fc9670c3ef531ab33bf113e7da76a6"},"cell_type":"code","source":"def sigmoid(z):\n    y_head = 1/(1 + np.exp(-z))\n    return y_head","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"507ae1ffc59e7755020f06e1495c6f05bc2e6c27"},"cell_type":"markdown","source":"<a id=\"6\"></a> \n**Forward - Backward Propagation**"},{"metadata":{"trusted":true,"_uuid":"8f3a9ee783a028d4e7be6cb8d97704936118f6d2"},"cell_type":"code","source":"def forward_backward_propogation(w, b, x_train, y_train):\n    \n    #forward propogation z = (w.T)x + b \n    z = np.dot(w.T, x_train) + b\n    y_head = sigmoid(z)\n    loss = -y_train * np.log(y_head) - (1 - y_train) * np.log(1 - y_head)\n    cost = (np.sum(loss)) / x_train.shape[1]  # x_train.shape[1] is for scaling\n    \n    #backward propogation\n    derivative_weight = (np.dot(x_train, ((y_head - y_train).T))) / x_train.shape[1] # x_train.shape[1] is for scaling\n    derivative_bias = np.sum(y_head - y_train) / x_train.shape[1] # x_train.shape[1] is for scaling\n    gradients = {\"derivative_weight\" : derivative_weight, \"derivative_bias\" : derivative_bias}\n    \n    return cost, gradients","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"96bb92852d82535ea5719305ccec81ac35b08ea3"},"cell_type":"markdown","source":"<a id=\"7\"></a> \n**Updating Parameters Using Forward and Backward Propagation**"},{"metadata":{"trusted":true,"_uuid":"128e0252fdfe573f4f6bd03ef496c83d99215a6c"},"cell_type":"code","source":"def update(w, b, x_train, y_train, learning_rate, number_of_iteration):\n    costList = []\n    costListForPlot = []\n    index = []\n    \n    # updating(learning) parameters is number_of_iteration times\n    for i in range(number_of_iteration):\n        # make forward and backward propogation and find costs and gradients\n        cost,gradients = forward_backward_propogation(w, b, x_train, y_train)\n        costList.append(cost)\n        #lets update\n        w = w - learning_rate * gradients[\"derivative_weight\"]\n        b = b - learning_rate * gradients[\"derivative_bias\"]\n        if i % 100 == 0:\n            costListForPlot.append(cost)\n            index.append(i)\n            print(\"Cost after iteration %i: %f\" %(i, cost))\n            \n    parameters = {\"weight\" : w, \"bias\" : b}\n    plt.plot(index, costListForPlot)\n    plt.xticks(index, rotation = 'vertical')\n    plt.xlabel(\"Number of Iteration\")\n    plt.ylabel(\"Cost\")\n    plt.show()\n    return parameters, gradients, costList","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3bdec63b6dac895ea71c5e4c51a6e91575eed428"},"cell_type":"markdown","source":"<a id=\"8\"></a> \n**Prediction Method for Logistic Regression Model**"},{"metadata":{"trusted":true,"_uuid":"bfde02beb029b52d13c1729d16c5ce5e1f653013"},"cell_type":"code","source":"# After forward and backward propagation. We will predict results from out model.\n\ndef predict(w,b, x_test):\n    z = sigmoid(np.dot(w.T, x_test) + b)\n    Y_prediction = np.zeros((1, x_test.shape[1]))\n    for i in range(z.shape[1]):\n        if z[0, i] <= 0.5:\n            Y_prediction[0, i] = 0\n        else:\n            Y_prediction[0, i] = 1\n    \n    return Y_prediction","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d67ce1d800d6583a951ce871038938af862e8de5"},"cell_type":"markdown","source":"<a id=\"9\"></a> \n**Defining The Logistic Regression Method**"},{"metadata":{"trusted":true,"_uuid":"ccde84d9a8fa0cf146d24a958c475ece3b523b28"},"cell_type":"code","source":"def logistic_regression(x_train, y_train, x_test, y_test, learning_rate, num_iterations):\n    \n    # initialize\n    dimension = x_train.shape[0]\n    w,b = initialize_weights_and_bias(dimension)\n    \n    parameters, gradients, cost_list = update(w, b, x_train, y_train, learning_rate, num_iterations) \n    y_prediction_test = predict(parameters[\"weight\"], parameters[\"bias\"], x_test)\n    y_prediction_train = predict(parameters[\"weight\"],parameters[\"bias\"],x_train)\n    \n    print(\"train accuracy: {} %\".format(100 - np.mean(np.abs(y_prediction_train - y_train)) * 100))\n    print(\"test accuracy : {} %.\".format(100 - np.mean(np.abs(y_prediction_test - y_test)) * 100))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8cb6e396cbdd0a76fca9a7caf728af6b0517ac11"},"cell_type":"markdown","source":"* Lets test our model. For 1000 iterations with 0.1 learning rate."},{"metadata":{"trusted":true,"_uuid":"334b8ffcf2d997e41048b088e3a399bbf8c1d712"},"cell_type":"code","source":"logistic_regression(x_train, y_train, x_test, y_test, learning_rate = 0.1, num_iterations = 1000) ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2b07801f591a93a318a7bbdad3323924eb85f699"},"cell_type":"markdown","source":"<a id=\"10\"></a> \n## Conclusion\n\n* The above transactions are called simple neural network (logistic regression). These are made without sklearn library for learn logic behind logistic regression. <br/>\n\nThanks in advance for suggestions."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}