{ "cells": [ { "cell_type": "markdown", "metadata": { "_cell_guid": "84d4608d-4cc3-fcbb-57fb-61f07ad7d020" }, "source": [ "*Poonam Ligade*\n", "\n", "*1st Feb 2017*\n", "\n", "\n", "----------\n", "\n", "\n", "This notebook is like note to self.\n", "\n", "I am trying to understand various components of Artificial Neural Networks aka Deep Learning.\n", "\n", "Hope it might be useful for someone else here.\n", "\n", "I am designing neural net on MNIST handwritten digits images to identify their correct label i.e number in image.\n", "\n", "You must have guessed its an image recognition task.\n", "\n", "MNIST is called Hello world of Deep learning.\n", "\n", "Lets start!!\n", "\n", "This notebook is inspired from [Jeremy's][1] [Deep Learning][2] mooc and [Deep learning with python][3] book by Keras author [Fran\u00e7ois Chollet][4] .\n", "\n", "\n", " [1]: https://www.linkedin.com/in/howardjeremy/\n", " [2]: http://course.fast.ai/\n", " [3]: https://www.manning.com/books/deep-learning-with-python\n", " [4]: https://research.google.com/pubs/105096.html" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "654456b6-e648-0379-0d66-1cc97af6d00d" }, "source": [ "**Import all required libraries**\n", "===============================" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "e5b02688-c589-5a89-e11c-837c6a99eb6e" }, "outputs": [], "source": [ " # This Python 3 environment comes with many helpful analytics libraries installed\n", "# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n", "# For example, here's several helpful packages to load in \n", "\n", "import numpy as np # linear algebra\n", "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n", "\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline\n", "\n", "from keras.models import Sequential\n", "from keras.layers import Dense , Dropout , Lambda, Flatten\n", "from keras.optimizers import Adam ,RMSprop\n", "from sklearn.model_selection import train_test_split\n", "\n", "# Input data files are available in the \"../input/\" directory.\n", "# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n", "\n", "from subprocess import check_output\n", "print(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n", "\n", "# Any results you write to the current directory are saved as output." ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "22a7fd70-ab61-432d-24cb-93e558414495" }, "source": [ "**Load Train and Test data**\n", "============================" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "05226b08-226a-1a00-044d-a0e6b2101388" }, "outputs": [], "source": [ "# create the training & test sets, skipping the header row with [1:]\n", "train = pd.read_csv(\"../input/train.csv\")\n", "\n", "test_images = (pd.read_csv(\"../input/test.csv\").values).astype('float32')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "2ec570a6-b41a-2139-5e0e-4941c4f0a9d0" }, "outputs": [], "source": [ "train_images = (train.ix[:,1:].values).astype('float32')\n", "train_labels = train.ix[:,0].values.astype('int32')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "1ae10fe0-dde9-7659-f53d-1a1bd625cfb1" }, "outputs": [], "source": [ "train_images.shape" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "60957d82-c76f-4822-28ff-def7011a34fa" }, "source": [ "Lets look at 3 images from dataset with their labels." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "1541678d-a08b-d2b2-1e1e-eabf882baaec" }, "outputs": [], "source": [ "#Convert train datset to (num_images, img_rows, img_cols) format \n", "\n", "train_images = train_images.reshape(train_images.shape[0], 28, 28)\n", "\n", "for i in range(6, 9):\n", " plt.subplot((i+1)+330)\n", " plt.imshow(train_images[i], cmap=plt.get_cmap('gray'))\n", " plt.title(train_labels[i]);" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "6be2f3e9-42eb-85b6-9162-c25e4d706155" }, "outputs": [], "source": [ "train_images = train_images.reshape((42000, 28 * 28))\n", "\n", "test_images.shape" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "b8e93d32-8918-3ba9-ab2e-b44eb857172e" }, "source": [ "The output variable is an integer from 0 to 9. \n", "This is a multiclass classification problem." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "6949468c-fd27-19c5-15c7-0b357a961003" }, "outputs": [], "source": [ "train_labels.shape" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "c8c07f11-e6dd-3a74-6694-e655fd3e2194" }, "outputs": [], "source": [ "train_labels" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "1232c385-3cb2-56fd-4d1d-f027df7bc78e" }, "source": [ "**Preprocessing the digit images**\n", "==================================" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "6fcc1f9e-1586-e393-49ba-50c73564e0ed" }, "source": [ "**Feature Standardization**\n", "-------------------------------------\n", "\n", "Its good to normalize pixel values." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "a3f837ef-0373-8d91-46e6-30992cf73166" }, "outputs": [], "source": [ "train_images = train_images / 255\n", "test_images = test_images / 255" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "725c55fc-9742-a63c-9822-c67ab0c773ee" }, "source": [ "*One Hot encoding of labels.*\n", "-----------------------------\n", "\n", "A one-hot vector is a vector which is 0 in most dimensions, and 1 in a single dimension. In this case, the nth digit will be represented as a vector which is 1 in the nth dimension. \n", "\n", "For example, 3 would be [0,0,0,1,0,0,0,0,0,0]." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "c879f076-b3dd-6cb1-e2d9-2f404f2ed132" }, "outputs": [], "source": [ "from keras.utils.np_utils import to_categorical\n", "train_labels = to_categorical(train_labels)\n", "num_classes = train_labels.shape[1]\n", "num_classes" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "4d76fb04-57fc-e802-6d91-06ece552686b" }, "source": [ "Lets plot 10th label." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "1c927e75-08d2-d539-54f3-71ab0308fec1" }, "outputs": [], "source": [ "plt.title(train_labels[9])\n", "plt.plot(train_labels[9])\n", "plt.xticks(range(10));" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "4e130661-9f09-d9a9-d49b-7274ef13927f" }, "source": [ "Oh its 3 !" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "6a89dcdd-7b68-6ed1-2c39-b3a1edb3e7be" }, "source": [ "**Designing neural network architecture**\n", "=========================================" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "39107235-d87a-af4d-44fb-80c9c3aa0212" }, "outputs": [], "source": [ "# fix random seed for reproducibility\n", "seed = 43\n", "np.random.seed(seed)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "5dbe450c-845f-aaa2-dbde-21414a91d8c1" }, "outputs": [], "source": [ "train_images.shape" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "5c3f674f-f3fc-9614-f2d4-056c3e3ad633" }, "outputs": [], "source": [ "train_labels.shape" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "a2c27783-3cfa-e907-4749-1e340a513f26" }, "outputs": [], "source": [ "from keras.models import Sequential\n", "from keras.layers import Dense , Dropout\n", "\n", "model=Sequential()\n", "model.add(Dense(32,activation='relu',input_dim=(28 * 28)))\n", "model.add(Dense(16,activation='relu'))\n", "model.add(Dense(10,activation='softmax'))" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "260645fb-61b7-68e9-6826-047b97436c14" }, "source": [ "**Compile network**\n", "===================\n", "\n", "Before making network ready for training we have to make sure to add below things:\n", "\n", " 1. A loss function: to measure how good the network is\n", " \n", " 2. An optimizer: to update network as it sees more data and reduce loss\n", " value\n", " \n", " 3. Metrics: to monitor performance of network" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "9d1d1af9-b2a8-e3b9-6eaf-100d08fe83aa" }, "outputs": [], "source": [ "from keras.optimizers import RMSprop\n", "model.compile(optimizer=RMSprop(lr=0.001),\n", " loss='categorical_crossentropy',\n", " metrics=['accuracy'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "db3b4be6-4f72-c6cc-65cd-b45978db2462" }, "outputs": [], "source": [ "history=model.fit(train_images, train_labels, validation_split = 0.05, \n", " epochs=25, batch_size=64)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "9f344366-c372-0b04-b7e0-860778d4bfd3" }, "outputs": [], "source": [ "history_dict = history.history\n", "history_dict.keys()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "df40f5fc-586a-1fae-025e-ee508a8d9b71" }, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "%matplotlib inline\n", "loss_values = history_dict['loss']\n", "val_loss_values = history_dict['val_loss']\n", "epochs = range(1, len(loss_values) + 1)\n", "\n", "# \"bo\" is for \"blue dot\"\n", "plt.plot(epochs, loss_values, 'bo')\n", "# b+ is for \"blue crosses\"\n", "plt.plot(epochs, val_loss_values, 'b+')\n", "plt.xlabel('Epochs')\n", "plt.ylabel('Loss')\n", "\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "1ed6b756-00c2-d08c-c596-0ce496ec3d04" }, "outputs": [], "source": [ "plt.clf() # clear figure\n", "acc_values = history_dict['acc']\n", "val_acc_values = history_dict['val_acc']\n", "\n", "plt.plot(epochs, acc_values, 'bo')\n", "plt.plot(epochs, val_acc_values, 'b+')\n", "plt.xlabel('Epochs')\n", "plt.ylabel('Accuracy')\n", "\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "925711cc-6c54-7363-ebbc-c64bb68eff94" }, "source": [ "From the graphs we can see that training loss is decreasing and training accuracy is increasing slowly. Thats what we intended to do using gradient descent.\n", "But thats not the case with validation set\n", "after 15th epoch val_loss is increasing and val_acc is decreasing.\n", "That is called as **overfitting**.\n", "\n", "after the second epoch, we are over optimising on the training data, and we ended up learning representations that are specific to the training data and do not generalize to data outside of the training set\n", "\n", "To avoid this we will simply stop training after 15 epochs." ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "cb401848-406d-9753-41c3-131a001a90b7" }, "source": [ "**creating model again from scratch**\n", "=====================================" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "e9c49f42-26af-73dc-9061-90c50c06f16e" }, "outputs": [], "source": [ "model = Sequential()\n", "model.add(Dense(64, activation='relu',input_dim=(28 * 28)))\n", "model.add(Dense(128, activation='relu'))\n", "model.add(Dropout(0.15))\n", "model.add(Dense(64, activation='relu'))\n", "model.add(Dropout(0.15))\n", "model.add(Dense(10, activation='softmax'))\n", "\n", "\n", "model.compile(optimizer=Adam(lr=0.0001), loss='categorical_crossentropy',\n", " metrics=['accuracy'])\n", "\n", "history=model.fit(train_images, train_labels, \n", " epochs=15, batch_size=64)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "c2841d54-f3dd-1ee8-a30d-4457dec0a67a" }, "outputs": [], "source": [ "predictions = model.predict_classes(test_images, verbose=0)\n", "\n", "submissions=pd.DataFrame({\"ImageId\": list(range(1,len(predictions)+1)),\n", " \"Label\": predictions})\n", "submissions.to_csv(\"DR.csv\", index=False, header=True)" ] }, { "cell_type": "markdown", "metadata": { "_cell_guid": "d74ac4ce-9fd5-f8d7-fa1c-b6b64b25e882" }, "source": [ "More to come . Please upvote if you find it useful." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "9a2f810d-f04b-3c77-5cbd-4cc06208854e" }, "outputs": [], "source": [ "model=Sequential()\n", "model.add(Dense(128,activation='relu',input_dim=(28*28)))\n", "model.add(Dense(64,activation='relu'))\n", "model.add(Dropout(0.2))\n", "model.add(Dense(10,activation='softmax'))\n", "\n", "model.compile(optimizer=Adam(lr=0.001),loss='categorical_crossentropy',metrics=['accuracy'])\n", "\n", "history=model.fit(train_images,train_labels,epochs=20,batch_size=128)\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "3e4a7566-54fd-f80d-c50c-9e00e62aaacc" }, "outputs": [], "source": [ "history_dict=history.history\n", "history_dict.keys()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "d7240940-ec4e-08bc-f1c3-25138f2b0290" }, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "%matplotlib inline\n", "loss_values = history_dict['loss']\n", "val_acc = history_dict['acc']\n", "epochs = range(1, len(loss_values) + 1)\n", "\n", "# \"bo\" is for \"blue dot\"\n", "plt.plot(epochs, loss_values, 'bo')\n", "# b+ is for \"blue crosses\"\n", "plt.plot(epochs, val_acc, 'b+')\n", "plt.xlabel('Epochs')\n", "plt.ylabel('Loss')\n", "\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "_cell_guid": "40da0816-c00a-27c2-cc35-bed5c5b87360" }, "outputs": [], "source": [ "predictions = model.predict_classes(test_images, verbose=0)\n", "\n", "submissions=pd.DataFrame({\"ImageId\": list(range(1,len(predictions)+1)),\n", " \"Label\": predictions})\n", "submissions.to_csv(\"DR.csv\", index=False, header=True)" ] } ], "metadata": { "_change_revision": 0, "_is_fork": false, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.0" } }, "nbformat": 4, "nbformat_minor": 0 }