{"cells":[{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"Deep learning neural networks are very easy to create and evaluate in Python with Keras, but we must follow a strict model life-cycle. In this kernel we will discover the step-by-step life-cycle for creating, training and evaluating deep learning neural networks in Keras and how to make predictions with a trained model."},{"metadata":{},"cell_type":"markdown","source":"## <u>Keras Model Life-Cycle </u>\n1. Define Network\n2. Compile Network\n3. Fit Network\n4. Evaluate Network\n5. Make Predictions"},{"metadata":{},"cell_type":"markdown","source":"### <u> Step 1. Define Network </u>\nNeural networks are defined in Keras as a sequence of layers. The container for these layers is the Sequential class. The first step is to create an instance of the Sequential class. Then we can create our layers and add them in the order that they should be connected."},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.models import Sequential\nfrom keras.layers import Dense, Activation","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Sequential model with one Dense layer with 2 neurons.\n\n# Create the layers and add them in the order that they should be connected.\nmodel = Sequential()\nmodel.add(Dense(2))\n\n# OR\n\n# Create an array of layers and pass it to the constructor of the Sequential class.\n\nlayers = [Dense(2)]\nmodel = Sequential(layers)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The first layer in the network must define the number of inputs to expect. For a Multilayer Perceptron model this is specified by \"input_dim\" attribute."},{"metadata":{"trusted":true},"cell_type":"code","source":"# MLP with 2 inputs in the visible layer, 5 neurons in the hidden layer and one neuron in the output layer.\n# Activation functions are defined separately from layers.\n\nmodel = Sequential()\nmodel.add(Dense(5, input_dim=2))\nmodel.add(Activation('relu'))\nmodel.add(Dense(1))\nmodel.add(Activation('sigmoid'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The choice of activation function is most important for the output layer as it will define the format that predictions will take.\n\n- <b> Regression </b>: Linear activation function, or linear, and the number of neurons matching the number of outputs.\n- <b> Binary Classification (2 class) </b>: Logistic activation function, or sigmoid, one neuron at the output layer.\n- <b> Multiclass Classification (>2 class) </b>: Softmax activation function, or softmax, and one output neuron per class value, assuming one hot encoded output pattern."},{"metadata":{},"cell_type":"markdown","source":"### <u> Step 2. Compile Network </u>\n\nCompilation requires the specification of the optimization algorithm to use to train the network and the loss function used to evaluate the network that is minimized by the optimization algorithm."},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras import optimizers\n\nmodel.compile(optimizer='sgd', loss='mean_squared_error')\n\n# OR\n\nalgorithm = optimizers.SGD(lr=0.1, momentum=0.3)\nmodel.compile(optimizer=algorithm, loss='mean_squared_error')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Standard loss functions for different predictive model types:\n- <b> Regression </b>: Mean Squared Error.\n- <b> Binary Classification (2 class) </b>: Logarithmic Loss, also called cross entropy or binary crossentropy.\n- <b> Multiclass Classification (>2 class) </b>: Multiclass Logarithmic Loss or categorical crossentropy.\n\nCommonly used optimization algorithms:\n- <b> Stochastic Gradient Descent </b> - Requires the tuning of a learning rate and momentum.\n- <b> Adam </b> - Requires the tuning of learning rate.\n- <b> RMSprop </b> - Requires the tuning of learning rate.\n\nFinally, we can also specify metrics to while fitting our model in addition to the loss function. Generally, the most useful additional metric to collect is \"accuracy\" for classification problems."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Defining metrics when compiling the model.\n\nmodel.compile(optimizer='sgd', loss='mean_squared_error', metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### <u> Step 3. Fit Network </u>\n\nOnce the network is compiled, it can be fit to adapt the weights on a training dataset. Fitting the network requires Matrix \"train_X\", and array \"train_y\". The network is trained using the backpropagation algorithm and optimized according to the optimization algorithm and loss function specified when compiling the model.\n\nThe backpropagation algorithm requires that the network be trained for a specified number of epochs. Each epoch can be partitioned into groups of input-output pattern pairs called batches.Weights are updated after each epoch."},{"metadata":{"trusted":true},"cell_type":"code","source":"# history = model.fit(X, y, batch_size=10, epochs=100)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Once fit, a history object is returned that provides a summary of the performance of the model during training. This includes both the loss and any additional metrics specified when compiling the model, recorded each epoch."},{"metadata":{},"cell_type":"markdown","source":"### <u>Step 4. Evaluate Network </u>\n\nOnce the network is trained, it can be evaluated. We evaluate the performance of the network on a separate dataset, unseen during training.\n\nThe model evaluates the loss across all of the test patterns, as well as any other metrics specified when the model was compiled, like classification accuracy. A list of evaluation metrics is returned. For example, for a model compiled with the accuracy metric, we could evaluate it on a new dataset as follows:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# loss, accuracy = model.evaluate(test_X, test_y)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### <u>Step 5. Make Predictions </u>\n\nOnce we are satisfied with the performance of our fit model, we can use it to make predictions on new data."},{"metadata":{"trusted":true},"cell_type":"code","source":"# predictions = model.predict(test_X)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The predictions will be returned in the format provided by the output layer of the network. In the case of a regression problem, these predictions may be in the format of the problem directly, provided by a linear activation function. For a binary classification problem, the predictions may be an array of probabilities for the first class that can be converted to a 1 or 0 by rounding.\n\nFor a multiclass classification problem, the results may be in the form of an array of probabilities (assuming a one hot encoded output variable) that may need to be converted to a single class output prediction using the argmax() NumPy function. Alternately, for classification problems, we can use the predict classes() function that will automatically convert uncrisp predictions to crisp integer class values."},{"metadata":{"trusted":true},"cell_type":"code","source":"# predictions = model.predict_classes(test_X)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## <u> Keras Functional Models </u>\n\nThe sequential API allows us to create models layer-by-layer for most problems. It is limited in that it does not allow us to create models that share layers or have multiple inputs or outputs. The functional API in Keras is an alternate way of creating models that offers a lot more flexibility, including creating more complex models.\n\nIt specifically allows us to define multiple input or output models as well as models that share layers. It also allows us to define ad hoc acyclic network graphs."},{"metadata":{},"cell_type":"markdown","source":"### <u> Defining Input </u>\n\n\nUnlike the Sequential model, we must create and define a standalone Input layer that specifies the shape of input data. The input layer takes a shape argument(tuple) that indicates the dimensionality of the input data. When input data is one-dimensional,(for a Multilayer Perceptron) the shape must explicitly leave room for the shape of the mini-batch size used when splitting the data when training the network. Therefore, the shape tuple is always defined with a hanging last dimension (2,)."},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.layers import Input\n\nvisible = Input(shape=(2,))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### <u>Connecting Layers </u>\nThe layers in the model are connected pairwise. This is done by specifying where the input comes from when defining each new layer.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create the input layer, then create a hidden layer as a Dense that receives input only from the input layer.\n# (visible) after the creation of the Dense layer connects the input layer’s output as the input to the Dense hidden layer.\n\nfrom keras.layers import Input\nfrom keras.layers import Dense\n\nvisible = Input(shape=(2,))\nhidden = Dense(2)(visible)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### <u> Creating the Model </u>\n\nAfter creating the model layers and connecting them together, we must define the model. As with the Sequential API, the model is the thing we can summarize, fit, evaluate, and use to make predictions. Keras provides a Model class that we can use to create a model from our created layers. It requires that we only specify the input and output layers."},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.models import Model\nfrom keras.layers import Input\nfrom keras.layers import Dense\n\nvisible = Input(shape=(2,))\nhidden = Dense(2)(visible)\nmodel = Model(inputs=visible, outputs=hidden)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## <u> Standard Network Models </u>"},{"metadata":{},"cell_type":"markdown","source":"### <u> Multilayer Perceptron </u>"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Multilayer Perceptron\n\nfrom keras.utils import plot_model\nfrom keras.models import Model\nfrom keras.layers import Input\nfrom keras.layers import Dense\n\n# 10 inputs\nvisible = Input(shape=(10,))\n\n# 3 hidden layers with 10, 20, and 10 neurons\nhidden1 = Dense(10, activation='relu')(visible)\nhidden2 = Dense(20, activation='relu')(hidden1)\nhidden3 = Dense(10, activation='relu')(hidden2)\n\n# Output layer with 1 output\noutput = Dense(1, activation='sigmoid')(hidden3)\n\nmodel = Model(inputs=visible, outputs=output)\n\n# summarize layers\nmodel.summary()\n\n# plot graph\nplot_model(model, to_file='multilayer_perceptron_graph.png')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### <u> Convolutional Neural Network </u>\n\nConvolutional neural network for image classification. The model receives black and white 64 x 64 images as input, then has a sequence of two convolutional and pooling layers as feature extractors, followed by a fully connected layer to interpret the features and an output layer with a sigmoid activation for two-class predictions."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Convolutional Neural Network\nfrom keras.utils import plot_model\nfrom keras.models import Model\nfrom keras.layers import Input\nfrom keras.layers import Dense\nfrom keras.layers.convolutional import Conv2D\nfrom keras.layers.pooling import MaxPooling2D\n\n# Model receives black and white 64 x 64 images as input\nvisible = Input(shape=(64,64,1))\n\n# Sequence of two convolutional and pooling layers as feature extractors\nconv1 = Conv2D(32, kernel_size=4, activation='relu')(visible)\npool1 = MaxPooling2D(pool_size=(2, 2))(conv1)\nconv2 = Conv2D(16, kernel_size=4, activation='relu')(pool1)\npool2 = MaxPooling2D(pool_size=(2, 2))(conv2)\n\n# Fully connected layer to interpret the features\nhidden1 = Dense(10, activation='relu')(pool2)\n\n# Output layer with a sigmoid activation for two-class predictions\noutput = Dense(1, activation='sigmoid')(hidden1)\n\nmodel = Model(inputs=visible, outputs=output)\n\n# summarize layers\nmodel.summary()\n\n# plot graph\nplot_model(model, to_file='convolutional_neural_network.png')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### <u> Recurrent Neural Network </u>\n\nLong short-term memory recurrent neural network for sequence classification."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Recurrent Neural Network\nfrom keras.utils import plot_model\nfrom keras.models import Model\nfrom keras.layers import Input\nfrom keras.layers import Dense\nfrom keras.layers.recurrent import LSTM\n\n# 100 time steps of one feature as input\nvisible = Input(shape=(100,1))\n\n# Single LSTM hidden layer to extract features from the sequence\nhidden1 = LSTM(10)(visible)\n\n# Fully connected layer to interpret the LSTM output\nhidden2 = Dense(10, activation='relu')(hidden1)\n\n# Output layer for making binary predictions\noutput = Dense(1, activation='sigmoid')(hidden2)\n\nmodel = Model(inputs=visible, outputs=output)\n\n# summarize layers\nmodel.summary()\n\n# plot graph\nplot_model(model, to_file='recurrent_neural_network.png')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### End\nIf you reached this far please comment and upvote this kernel, feel free to make improvements on the kernel and please share if you found anything useful !"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}