{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## INTRODUCTION\n- It’s a Python based scientific computing package targeted at two sets of audiences:\n    - A replacement for NumPy to use the power of GPUs\n    - Deep learning research platform that provides maximum flexibility and speed\n- pros: \n    - Iinteractively debugging PyTorch. Many users who have used both frameworks would argue that makes pytorch significantly easier to debug and visualize.\n    - Clean support for dynamic graphs\n    - Organizational backing from Facebook\n    - Blend of high level and low level APIs\n- cons:\n    - Much less mature than alternatives\n    - Limited references / resources outside of the official documentation\n- I accept you know neural network basics. If you do not know check my tutorial. Because I will not explain neural network concepts detailed, I only explain how to use pytorch for neural network\n- Neural Network tutorial: https://www.kaggle.com/kanncaa1/deep-learning-tutorial-for-beginners \n- The most important parts of this tutorial from matrices to ANN. If you learn these parts very well, implementing remaining parts like CNN or RNN will be very easy. \n<br>\n<br>**Content:**\n1. Basics of Pytorch, Linear Regression, Logistic Regression, Artificial Neural Network (ANN), Convolutional Neural Network (CNN)\n    - https://www.kaggle.com/kanncaa1/pytorch-tutorial-for-deep-learning-lovers/code\n1. [Recurrent Neural Network (RNN)](#1)\n1. Long-Short Term Memory (LSTM)\n    - https://www.kaggle.com/kanncaa1/long-short-term-memory-with-pytorch","metadata":{"_cell_guid":"16db9e81-fe38-44f0-8a4f-3dba9eba0409","_uuid":"d0469858bd9b00ece464f6267cfad7dc450d5d54"}},{"cell_type":"markdown","source":"<a id=\"1\"></a> <br>\n### Recurrent Neural Network (RNN)\n- **RNN is essentially repeating ANN but information get pass through from previous non-linear activation function output**.\n- **Steps of RNN:**\n    1. Import Libraries\n    1. Prepare Dataset\n    1. Create RNN Model\n        - hidden layer dimension is 100 (100 nodes)\n        - number of hidden layer is 1 \n    1. Instantiate Model\n    1. Instantiate Loss\n        - Cross entropy loss\n        - It also has softmax(logistic function) in it.\n    1. Instantiate Optimizer\n        - SGD Optimizer\n    1. Traning the Model\n    1. Prediction","metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.status.busy":"2022-07-09T17:57:33.631885Z","iopub.execute_input":"2022-07-09T17:57:33.632106Z","iopub.status.idle":"2022-07-09T17:57:33.642915Z","shell.execute_reply.started":"2022-07-09T17:57:33.632056Z","shell.execute_reply":"2022-07-09T17:57:33.642128Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**What is nn in torch nn?**\n\nPyTorch provides the **torch.nn** *module* to help us in creating and training of the deep neural network. \n\n**What does torch Autograd Variable do?**\n\nAutograd is a PyTorch *package* for the differentiation for all operations on Tensors. It performs the backpropagation starting from a variable. In deep learning, this variable often holds the value of the cost function. Backward executes the backward pass and computes all the backpropagation gradients automatically\n\n**What does Variable () do PyTorch?**\n\nA PyTorch Variable is a wrapper around a PyTorch Tensor, and represents a node in a computational graph. If x is a Variable then *x.data* is a Tensor giving its value, and *x.grad* is another Variable holding the gradient of x with respect to some scalar value\n\n**What is Torch utils data dataloader?**\n\nData loader. Combines a dataset and a sampler, and provides an iterable over the given dataset. The DataLoader supports both map-style and iterable-style datasets with single- or multi-process loading, customizing loading order and optional automatic batching (collation) and memory pinning\n\n**Why is dataloader used?**\n\nCreating a PyTorch Dataset and managing it with Dataloader keeps your data manageable and helps to simplify your machine learning pipeline. a Dataset stores all your data, and Dataloader is can be used to iterate through the data, manage batches, transform the data, and much more\n\n","metadata":{}},{"cell_type":"code","source":"# Import Libraries\nimport torch\nimport torch.nn as nn\nfrom torch.autograd import Variable\nfrom sklearn.model_selection import train_test_split\nfrom torch.utils.data import DataLoader, TensorDataset","metadata":{"_cell_guid":"482946c2-72d8-4489-a094-d6cb8993a912","_uuid":"ceffbb7fe5381f0d2f5f234ea37d1f834843edee","execution":{"iopub.status.busy":"2022-07-09T17:58:19.207676Z","iopub.execute_input":"2022-07-09T17:58:19.207994Z","iopub.status.idle":"2022-07-09T17:58:19.920479Z","shell.execute_reply.started":"2022-07-09T17:58:19.207935Z","shell.execute_reply":"2022-07-09T17:58:19.919817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**What is the difference between epoch batch and iteration?**\n\nIteration is one time processing for forward and backward for a batch of images (say one batch is defined as 16, then 16 images are processed in one iteration). Epoch is once all images are processed one time individually of forward and backward to the network, then that is one epoch\n\nThe number of batches is equal to number of iterations for one epoch. Let's say we have 2000 training examples that we are going to use . We can divide the dataset of 2000 examples into batches of 500 then it will take 4 iterations to complete 1 epoch\n\nhttps://towardsdatascience.com/epoch-vs-iterations-vs-batch-size-4dfb9c7ce9c9#:~:text=Note%3A%20The%20number%20of%20batches,iterations%20to%20complete%201%20epoch.\n\n","metadata":{}},{"cell_type":"code","source":"# Prepare Dataset\n# load data\ntrain = pd.read_csv(r\"../input/train.csv\",dtype = np.float32)\n\n# split data into features(pixels) and labels(numbers from 0 to 9)\ntargets_numpy = train.label.values\nfeatures_numpy = train.loc[:,train.columns != \"label\"].values/255 # normalization\n\n# train test split. Size of train data is 80% and size of test data is 20%. \nfeatures_train, features_test, targets_train, targets_test = train_test_split(features_numpy,\n                                                                             targets_numpy,\n                                                                             test_size = 0.2,\n                                                                             random_state = 42) \n\n# create feature and targets tensor for train set. As you remember we need variable to accumulate gradients. \n# Therefore first we create tensor, then we will create variable\n\nfeaturesTrain = torch.from_numpy(features_train) # Creates a Tensor from a numpy.ndarray . \n                                                #The returned tensor and ndarray share the same memory\ntargetsTrain = torch.from_numpy(targets_train).type(torch.LongTensor) # data type is long\n\n# create feature and targets tensor for test set.\nfeaturesTest = torch.from_numpy(features_test)\ntargetsTest = torch.from_numpy(targets_test).type(torch.LongTensor) # data type is long\n\n# batch_size, epoch and iteration\nbatch_size = 100\nn_iters = 10000\nnum_epochs = n_iters / (len(features_train) / batch_size)\nnum_epochs = int(num_epochs)\n\n# Pytorch train and test sets\ntrain = TensorDataset(featuresTrain,targetsTrain)\ntest = TensorDataset(featuresTest,targetsTest)\n\n# data loader\ntrain_loader = DataLoader(train, batch_size = batch_size, shuffle = False)\ntest_loader = DataLoader(test, batch_size = batch_size, shuffle = False)\n\n# visualize one of the images in data set\nplt.imshow(features_numpy[10].reshape(28,28))\nplt.axis(\"off\")\nplt.title(str(targets_numpy[10]))\nplt.savefig('graph.png')\nplt.show()","metadata":{"_cell_guid":"55dd8ffd-6011-49a3-a1fe-c6933c4187b7","_uuid":"840f7b1c60d1a2d5b2222a7c53b2b9d08aac9169","execution":{"iopub.status.busy":"2022-07-09T17:58:29.447127Z","iopub.execute_input":"2022-07-09T17:58:29.447395Z","iopub.status.idle":"2022-07-09T17:58:34.225453Z","shell.execute_reply.started":"2022-07-09T17:58:29.447349Z","shell.execute_reply":"2022-07-09T17:58:34.224709Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train\n\n# The Dataset class is an abstract class that is used to define new types of (customs) datasets. \n# Instead, the TensorDataset is a ready to use class to represent your data as list of tensors","metadata":{"execution":{"iopub.status.busy":"2022-07-09T17:59:10.650824Z","iopub.execute_input":"2022-07-09T17:59:10.651114Z","iopub.status.idle":"2022-07-09T17:59:10.658419Z","shell.execute_reply.started":"2022-07-09T17:59:10.651064Z","shell.execute_reply":"2022-07-09T17:59:10.657537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.data_tensor","metadata":{"execution":{"iopub.status.busy":"2022-07-09T18:01:20.269175Z","iopub.execute_input":"2022-07-09T18:01:20.269470Z","iopub.status.idle":"2022-07-09T18:01:20.957703Z","shell.execute_reply.started":"2022-07-09T18:01:20.269414Z","shell.execute_reply":"2022-07-09T18:01:20.956918Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.target_tensor","metadata":{"execution":{"iopub.status.busy":"2022-07-09T18:01:42.357288Z","iopub.execute_input":"2022-07-09T18:01:42.357577Z","iopub.status.idle":"2022-07-09T18:01:42.387889Z","shell.execute_reply.started":"2022-07-09T18:01:42.357515Z","shell.execute_reply":"2022-07-09T18:01:42.387318Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"featuresTrain","metadata":{"execution":{"iopub.status.busy":"2022-07-09T18:02:32.163748Z","iopub.execute_input":"2022-07-09T18:02:32.164054Z","iopub.status.idle":"2022-07-09T18:02:32.825490Z","shell.execute_reply.started":"2022-07-09T18:02:32.164001Z","shell.execute_reply":"2022-07-09T18:02:32.824846Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"targetsTrain","metadata":{"execution":{"iopub.status.busy":"2022-07-09T18:03:08.704599Z","iopub.execute_input":"2022-07-09T18:03:08.704898Z","iopub.status.idle":"2022-07-09T18:03:08.735405Z","shell.execute_reply.started":"2022-07-09T18:03:08.704842Z","shell.execute_reply":"2022-07-09T18:03:08.734799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_loader","metadata":{"execution":{"iopub.status.busy":"2022-07-09T18:03:40.739545Z","iopub.execute_input":"2022-07-09T18:03:40.739818Z","iopub.status.idle":"2022-07-09T18:03:40.746166Z","shell.execute_reply.started":"2022-07-09T18:03:40.739765Z","shell.execute_reply":"2022-07-09T18:03:40.745292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"import torch.autograd as autograd-----------# computation graph\n\nfrom torch import Tensor--------------------# tensor node in the computation graph\n\nimport torch.nn as nn-----------------------# neural networks\n\nimport torch.nn.functional as F-------------# layers, activations and more\n\nimport torch.optim as optim-----------------# optimizers e.g. gradient descent, ADAM, etc.\n\nfrom torch.jit import script, trace---------# hybrid frontend decorator and tracing jit\n\nnn contains different classess that help you build neural network models. All models in PyTorch inherit from the subclass nn.Module , which has useful methods like parameters() , __call__() and others. This module torch.nn also has various layers that you can use to build your neural network.\n\nhttps://www.cs.toronto.edu/~lczhang/360/lec/w03/nn.html#:~:text=nn%20contains%20different%20classess%20that,to%20build%20your%20neural%20network.","metadata":{}},{"cell_type":"code","source":"# Create RNN Model\n\nclass RNNModel(nn.Module):\n    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim):\n        super(RNNModel, self).__init__()\n        \n        # Number of hidden dimensions\n        self.hidden_dim = hidden_dim\n        \n        # Number of hidden layers\n        self.layer_dim = layer_dim\n        \n        # RNN\n        self.rnn = nn.RNN(input_dim, hidden_dim, layer_dim, batch_first=True, nonlinearity='relu')\n        \n        # Readout layer\n        self.fc = nn.Linear(hidden_dim, output_dim)\n    \n    def forward(self, x):\n        \n        # Initialize hidden state with zeros\n        h0 = Variable(torch.zeros(self.layer_dim, x.size(0), self.hidden_dim))\n            \n        # One time step\n        out, hn = self.rnn(x, h0)\n        out = self.fc(out[:, -1, :]) \n        return out\n\n# batch_size, epoch and iteration\nbatch_size = 100\nn_iters = 8000\nnum_epochs = n_iters / (len(features_train) / batch_size)\nnum_epochs = int(num_epochs)\n\n# Pytorch train and test sets\ntrain = TensorDataset(featuresTrain,targetsTrain)\ntest = TensorDataset(featuresTest,targetsTest)\n\n# data loader\ntrain_loader = DataLoader(train, batch_size = batch_size, shuffle = False)\ntest_loader = DataLoader(test, batch_size = batch_size, shuffle = False)\n    \n# Create RNN\ninput_dim = 28    # input dimension\nhidden_dim = 100  # hidden layer dimension\nlayer_dim = 1     # number of hidden layers\noutput_dim = 10   # output dimension\n\nmodel = RNNModel(input_dim, hidden_dim, layer_dim, output_dim)\n\n# Cross Entropy Loss \nerror = nn.CrossEntropyLoss()\n\n# SGD Optimizer\nlearning_rate = 0.05\noptimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)","metadata":{"_cell_guid":"7fbe419e-7ce2-4d72-bb31-8b27e8161f1b","_uuid":"bb1b6d4fb5504400ed7678d8e95d0a4478b5f409","execution":{"iopub.status.busy":"2022-07-09T18:25:10.450512Z","iopub.execute_input":"2022-07-09T18:25:10.451970Z","iopub.status.idle":"2022-07-09T18:25:10.549339Z","shell.execute_reply.started":"2022-07-09T18:25:10.451902Z","shell.execute_reply":"2022-07-09T18:25:10.548680Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"seq_dim = 28  \nloss_list = []\niteration_list = []\naccuracy_list = []\ncount = 0\nfor epoch in range(num_epochs):\n    for i, (images, labels) in enumerate(train_loader):\n\n        train  = Variable(images.view(-1, seq_dim, input_dim))\n        labels = Variable(labels )\n            \n        # Clear gradients\n        optimizer.zero_grad()\n        \n        # Forward propagation\n        outputs = model(train)\n        \n        # Calculate softmax and ross entropy loss\n        loss = error(outputs, labels)\n        \n        # Calculating gradients\n        loss.backward()\n        \n        # Update parameters\n        optimizer.step()\n        \n        count += 1\n        \n        if count % 250 == 0:\n            # Calculate Accuracy         \n            correct = 0\n            total = 0\n            # Iterate through test dataset\n            for images, labels in test_loader:\n                images = Variable(images.view(-1, seq_dim, input_dim))\n                \n                # Forward propagation\n                outputs = model(images)\n                \n                # Get predictions from the maximum value\n                predicted = torch.max(outputs.data, 1)[1]\n                \n                # Total number of labels\n                total += labels.size(0)\n                \n                correct += (predicted == labels).sum()\n            \n            accuracy = 100 * correct / float(total)\n            \n            # store loss and iteration\n            loss_list.append(loss.data)\n            iteration_list.append(count)\n            accuracy_list.append(accuracy)\n            if count % 500 == 0:\n                # Print Loss\n                print('Iteration: {}  Loss: {}  Accuracy: {} %'.format(count, loss.data[0], accuracy))","metadata":{"_cell_guid":"32786a5c-0388-412d-b6da-ee5ace604eda","_uuid":"9c935ac4a1d1964b85513da422ebf60085dca0e3","execution":{"iopub.status.busy":"2022-07-09T18:26:56.695055Z","iopub.execute_input":"2022-07-09T18:26:56.695335Z","iopub.status.idle":"2022-07-09T18:28:32.841744Z","shell.execute_reply.started":"2022-07-09T18:26:56.695286Z","shell.execute_reply":"2022-07-09T18:28:32.840969Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Learn PyTorch on Github:\n\nhttps://github.com/HabibMrad/pytorch-tutorial\n\nhttps://github.com/HabibMrad/tutorials\n\nhttps://github.com/HabibMrad/PyTorch-Tutorial-1","metadata":{}},{"cell_type":"code","source":"# visualization loss \nplt.plot(iteration_list,loss_list)\nplt.xlabel(\"Number of iteration\")\nplt.ylabel(\"Loss\")\nplt.title(\"RNN: Loss vs Number of iteration\")\nplt.show()\n\n# visualization accuracy \nplt.plot(iteration_list,accuracy_list,color = \"red\")\nplt.xlabel(\"Number of iteration\")\nplt.ylabel(\"Accuracy\")\nplt.title(\"RNN: Accuracy vs Number of iteration\")\nplt.savefig('graph.png')\nplt.show()","metadata":{"_cell_guid":"0e527a85-b600-4e40-a0ef-850537db2ab1","_uuid":"0cb7130ea6e22093d6d5cb1284822b0b76b8d66c","execution":{"iopub.status.busy":"2022-07-09T18:29:45.715469Z","iopub.execute_input":"2022-07-09T18:29:45.715763Z","iopub.status.idle":"2022-07-09T18:29:45.983243Z","shell.execute_reply.started":"2022-07-09T18:29:45.715711Z","shell.execute_reply":"2022-07-09T18:29:45.982567Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Conclusion\nIn this tutorial, we learn: \n1. Basics of pytorch\n1. Linear regression with pytorch\n1. Logistic regression with pytorch\n1. Artificial neural network with with pytorch\n1. Convolutional neural network with pytorch\n    - https://www.kaggle.com/kanncaa1/pytorch-tutorial-for-deep-learning-lovers/code\n1. Recurrent neural network with pytorch\n1. Long-Short Term Memory (LSTM)\n    - https://www.kaggle.com/kanncaa1/long-short-term-memory-with-pytorch\n\n<br> If you have any question or suggest, I will be happy to hear it ","metadata":{"_cell_guid":"eeaf09ff-e125-42ee-99fa-e231e97c4308","_uuid":"0771ccf728b05cf6a5e3804e7d9bc5fa376e7ef8"}},{"cell_type":"markdown","source":"https://towardsdatascience.com/convolution-neural-networks-a-beginners-guide-implementing-a-mnist-hand-written-digit-8aa60330d022\n\nhttps://towardsdatascience.com/learn-how-recurrent-neural-networks-work-84e975feaaf7\n\nhttp://karpathy.github.io/2015/05/21/rnn-effectiveness/\n\nhttp://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/\n\nhttps://blog.paperspace.com/recurrent-neural-networks-part-1-2/\n\nhttps://www.youtube.com/watch?v=Keqep_PKrY8&t=1080s\n\nhttp://web.stanford.edu/class/cs224n/\n\nhttps://www.tensorflow.org/tutorials/text/text_generation\n\nhttps://arxiv.org/pdf/1506.00019.pdf\n\nhttps://www.w3schools.com/python/python_ml_getting_started.asp\n","metadata":{}}]}