{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Using Convolutional Neural Networks (CNN) to classify Images\n\nWe will use the **[PyTorch](https://pytorch.org/)** open source Python distributio  to define a Convolutional Neaural Network that will be trained on the Natural Images dataset [1] by *Prasun Roy*.\n\n\n## Convolutional Neural Network (CNN)\n\nA **convolutional neural network (CNN)** consists of an input and an output layer, as well as multiple hidden layers. The hidden layers of a CNN typically consist of a series of **convolutional layers** that convolve with a multiplication or other dot product. The activation function is commonly a *REctified Linear Unit (RELU) layer*, and is subsequently followed by additional convolutions such as **pooling layers**, **fully connected layers** and **normalization layers**, referred to as *hidden layers* because their inputs and outputs are masked by the activation function and final convolution. The final convolution, in turn, often involves backpropagation in order to more accurately weight the end product.[2]\n\nTo recap, a CNN have\n1. Convolutional Layers\n2. Pooling Layers\n3. Fully Connected Layers\n4. Nornalization Layers\nwhere 2,3,4 are *hidden layers*. \n\n![a](https://upload.wikimedia.org/wikipedia/commons/6/63/Typical_cnn.png)\n\n\n\n### Convolutional Layers \n\nWhen programming a CNN, the input is a tensor with shape (number of images, (image width , image height), image depth). Then after passing through a convolutional layer, the image becomes abstracted to a feature map, with shape (number of images, (feature map width, feature map height) , feature map channels). A **convolutional layer** within a neural network should have the following attributes:\n    * Convolutional kernels defined by a width and height (hyper-parameters).\n    * The number of input channels and output channels (hyper-parameter).\n    * The depth of the Convolution filter (the input channels) must be equal to the number channels (depth) of the input feature map.\n\nConvolutional layers convolve the input and pass its result to the next layer. The convolution operation brings a solution to the problem arising from the presence of a huge number of input data (i.e. the number of pixels) as it reduces the number of free parameters, allowing the network to be deeper with fewer parameters.\n\nThe convolutional layer is the core building block of a CNN. The layer's parameters consist of a set of learnable filters (or kernels), which have a small receptive field, but extend through the full depth of the input volume. During the forward pass, each filter is convolved across the width and height of the input volume, computing the dot product between the entries of the filter and the input and producing a 2-dimensional activation map of that filter. As a result, the network learns filters that activate when it detects some specific type of feature at some spatial position in the input. \n\nStacking the activation maps for all filters along the depth dimension forms the full output volume of the convolution layer. Every entry in the output volume can thus also be interpreted as an output of a neuron that looks at a small region in the input and shares parameters with neurons in the same activation map.\n\n### Pooling \n\nConvolutional networks may include **local** or **global pooling layers** to streamline the underlying computation. Pooling layers *reduce the dimensions of the data* by combining the outputs of neuron clusters at one layer into a single neuron in the next layer. Local pooling combines small clusters, typically 2 x 2. \n\nGlobal pooling acts on all the neurons of the convolutional layer. In addition, pooling may compute a max or an average. *Max pooling* uses the maximum value from each of a cluster of neurons at the prior layer. *Average pooling* uses the average value from each of a cluster of neurons at the prior layer.\n\n![a](https://upload.wikimedia.org/wikipedia/commons/e/e9/Max_pooling.png)\n\n### Fully Connected Layers\n\n**Fully connected layers** connect every neuron in one layer to every neuron in another layer. It is in principle the same as the traditional multi-layer perceptron neural network (MLP). The flattened matrix goes through a fully connected layer to classify the images.\n\n#### Receptive Fields\n\nIn neural networks, each neuron receives input from some number of locations in the previous layer. In a *fully connected layer*, *each* neuron receives input from every element of the previous layer. In a *convolutional layer*, neurons receive input from *only a restricted subarea* of the previous layer. Typically the subarea is of a square shape (e.g., size 5 by 5). The input area of a neuron is called its **receptive field**. \n\nSo: \n\n1. in a **fully connected layer**, the receptive field is the *entire previous layer*. \n2. In a **convolutional layer**, the receptive area is smaller than the entire previous layer.\n\n### Weights\n\nEach neuron in a neural network computes an output value by applying a specific function to the input values coming from the receptive field in the previous layer (e.g. *perceptron* [3]). The function that is applied to the input values is determined by a vector of weights and a bias (typically real numbers). Learning, in a neural network, progresses by making iterative adjustments to these biases and weights.\n\nThe vector of weights and the bias are called **filters** and represent particular features of the input (e.g., a particular shape). \n\nA distinguishing feature of CNNs is that many neurons can *share the same filter*. This reduces memory footprint because a single bias and a single vector of weights are used across all receptive fields sharing that filter, as opposed to each receptive field having its own bias and vector weighting. \n\n\n### Backpropagation\n\n**Backpropagation** is an algorithm widely used in the training of feedforward neural networks for supervised learning; generalizations exist for other artificial neural networks (ANNs), and for functions generally. Backpropagation efficiently computes the gradient of the loss function with respect to the weights of the network for a single input-output example. This makes it feasible to use gradient methods for training multi-layer networks, updating weights to minimize loss; commonly one uses gradient descent or variants such as stochastic gradient descent. The backpropagation algorithm works by computing the gradient of the loss function with respect to each weight by the chain rule, iterating backwards one layer at a time from the last layer to avoid redundant calculations of intermediate terms in the chain rule; this is an example of dynamic programming. [4], [5].\n\nWhen we train a CNN, we perform mulitple passes forward through the network of layers, and then use a *loss function* to measure the difference between the output values (which you may recall are probability predictions for each class) and the actual values for the known image classes used to train the model (in other words, 1 for the correct class and 0 for all the others). For example, if our CNN have three possibile classes $[C_1, C_2, C_3]$ and, e.g.,  predicted probabilities are 0.15 for $C_1$, 0.8 for $C_2$, and 0.05 for $C_2$. Let's suppose that the image in question is an example of $C_2$, so the expected output is actually 0 for$C_1$, 1 for $C_2$, and 0 for $C_3$. The error (or *loss*) represents how far from the expected values our results are.\n\nHaving calculated the loss, the training process uses a specified *optimizer* to calculate the derivitive of the loss function wit respect to the weights and biases used in the network layers, and determine how best to adjust them to reduce the loss. We then go backwards through the network, adjusting the weights before the next forward pass. The degree to which we adjust the weights is determined by the *learning rate* - the larger the learning rate, the bigger the adjustments made to the weights.\n\n### Counteracting Overfitting: Data Augmentation and Drop Layers\n\n#### Overfitting\n\nIn statistics, overfitting is \n> the production of an analysis that corresponds too closely or exactly to a particular set of data, and may therefore fail to fit additional data or predict future observations reliably.\n\nAn overfitted model is a statistical model that contains more parameters than can be justified by the data. The essence of overfitting is to have unknowingly extracted some of the residual variation (i.e. the noise) as if that variation represented underlying model structure. [6] \n\n![b](https://upload.wikimedia.org/wikipedia/commons/1/1f/Overfitting_svg.svg)\n\nIn the image above: *training error* is shown in blue, *validation error* in red, both as a function of the number of training epochs. If the validation error increases (positive slope) while the training error steadily decreases (negative slope), then a situation of overfitting may have occurred. \n\nThe best predictive and fitted model would be where the validation error has its global minimum.\n\n#### Data Augmentation\n\nOne way to mitigate the overfitting problem is to perform *data augmentation* by making random transformations of the training images; for example by *flipping*, *rotating*, or *cropping* the images. \n\nBecause these data augmentation transformations are randomly applied during training, the same image might be presented differently from batch to batch, creating more variation in the training data and helping the model to learn features based the same objects at different orientations or scales.\n\n#### Drop Layers\n\nDuring the training process, the convolution and pooling layers in the feature extraction section of the model generate lots of feature maps from the training images. Randomly dropping some of these feature maps helps vary the features that are extracted in each batch, ensuring the model doesn't become overly-reliant on any one dominant feature in the training data. [7]\n\n\n-----\n\n### App. A: Basics of Artificial Neural Networks\n\n**Artificial neural networks** (ANN or NN)  are computing systems that are inspired by, but not identical to, biological neural networks that constitute animal brains. Such systems *learn* to perform tasks by considering examples, generally without being programmed with task-specific rules.\n\nA NN is based on a collection of connected units or nodes called **artificial neurons**, which loosely model the neurons in a biological brain. The basic example is the **perceptron**[a] . Each connection, like the synapses in a biological brain, can transmit a signal to other neurons. An artificial neuron that receives a signal then processes it and can signal neurons connected to it.\n\n![a](https://www.allaboutcircuits.com/uploads/articles/how-to-perform-classification-using-a-neural-network-a-simple-perceptron-example_rk_aac_image2.jpg)\n\nIn ANN implementations, the \"signal\" at a connection is a real number, and the output of each neuron is computed by some non-linear function of the sum of its inputs. The connections are called edges. Neurons and edges typically have a weight that adjusts as learning proceeds. The weight increases or decreases the strength of the signal at a connection. Neurons may have a threshold such that a signal is sent only if the aggregate signal crosses that threshold. Typically, neurons are aggregated into layers. Different layers may perform different transformations on their inputs. Signals travel from the first layer (the input layer), to the last layer (the output layer), possibly after traversing the layers multiple times. [b], \n\n#### Single-layer and Multi-layer perceptrons\n\nA **single layer perceptron (SLP)** is a feed-forward network based on a threshold transfer function. SLP is the simplest type of artificial neural networks and can only classify linearly separable cases with a binary target (1, 0). [c], [d]\n\nBecause SLP is a linear classifier and if the cases are not linearly separable the learning process will never reach a point where all the cases are classified properly. The most famous example of the inability of perceptron to solve problems with linearly non-separable cases is the XOR problem.\n\nA **multi-layer perceptron (MLP)** has the same structure of a single layer perceptron with one or more hidden layers. The backpropagation algorithm consists of two phases: the forward phase where the activations are propagated from the input to the output layer, and the backward phase, where the error between the observed actual and the requested nominal value in the output layer is propagated backwards in order to modify the weights and bias values. \t\n\n![a](https://www.saedsayad.com/images/Perceptron_bkp_1.png)\n\n-----\n\n### About the Dataset\n#### Natural Images\nThis dataset is created as a benchmark dataset for the work on Effects of Degradations on Deep Neural Network Architectures.\nThe source code is publicly available on GitHub.\n\n#### Description\nThis dataset contains 6,899 images from 8 distinct classes compiled from various sources (see Acknowledgements). The classes include airplane, car, cat, dog, flower, fruit, motorbike and person.\n\n#### Acknowledgements\n\n1. Airplane images obtained from http://host.robots.ox.ac.uk/pascal/VOC\n2. Car images obtained from https://ai.stanford.edu/~jkrause/cars/car_dataset.html\n3. Cat images obtained from https://www.kaggle.com/c/dogs-vs-cats\n4. Dog images obtained from https://www.kaggle.com/c/dogs-vs-cats\n5. Flower images obtained from http://www.image-net.org\n6. Fruit images obtained from https://www.kaggle.com/moltean/fruits\n7. Motorbike images obtained from http://host.robots.ox.ac.uk/pascal/VOC\n8. Person images obtained from http://www.briancbecker.com/blog/research/pubfig83-lfw-dataset\n\n\n-----\n[1] https://www.kaggle.com/prasunroy/natural-images\n\n[2] https://en.wikipedia.org/wiki/Convolutional_neural_network\n\n[3] https://en.wikipedia.org/wiki/Perceptron\n\n[4] https://en.wikipedia.org/wiki/Backpropagation\n\n[5] https://becominghuman.ai/back-propagation-in-convolutional-neural-networks-intuition-and-code-714ef1c38199\n\n[6] https://en.wikipedia.org/wiki/Overfitting\n\n[7] https://machinelearningmastery.com/dropout-for-regularizing-deep-neural-networks/\n\n[a] https://en.wikipedia.org/wiki/Perceptron\n\n[b] https://en.wikipedia.org/wiki/Artificial_neural_network\n\n[c] https://www.saedsayad.com/artificial_neural_network_bkp.htm\n\n[d] https://iamtrask.github.io/2015/07/12/basic-python-network/","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nfrom matplotlib import image as mp_image\nimport seaborn as sns\n\n# Required magic to display matplotlib plots in notebooks\n%matplotlib inline\n\nfrom sklearn import metrics\nfrom sklearn.metrics import accuracy_score, confusion_matrix\nfrom sklearn.model_selection import train_test_split\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nimport shutil\ncount = 0\nfor dirname, _, filenames in os.walk('input/eda-folder2'):\n    for filename in filenames:\n        # print(os.path.join(dirname, filename))\n        count += 1\nprint(count)\n# Any results you write to the current directory are saved as output.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-05-26T16:27:27.084554Z","iopub.execute_input":"2021-05-26T16:27:27.084927Z","iopub.status.idle":"2021-05-26T16:27:27.097132Z","shell.execute_reply.started":"2021-05-26T16:27:27.084874Z","shell.execute_reply":"2021-05-26T16:27:27.096318Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# The images are in a folder named 'input/natural-images/natural_images'\ntraining_folder_name = 'input/eda-folder2/EDA_folder2'\n\n# All images are 128x128 pixels\nimg_size = (96,96)\n\n# The folder contains a subfolder for each class of shape\nclasses = [\"no_cancer\",\"cancer\"]\nprint(classes)","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","execution":{"iopub.status.busy":"2021-05-26T16:27:27.099547Z","iopub.execute_input":"2021-05-26T16:27:27.099898Z","iopub.status.idle":"2021-05-26T16:27:27.115420Z","shell.execute_reply.started":"2021-05-26T16:27:27.099834Z","shell.execute_reply":"2021-05-26T16:27:27.114468Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Import PyTorch libraries\nimport torch\nimport torchvision\nimport torchvision.transforms as transforms\nimport torch.nn as nn\nimport torch.optim as optim\nimport torch.nn.functional as F\n\nprint(\"Libraries imported - ready to use PyTorch\", torch.__version__)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:27:27.117028Z","iopub.execute_input":"2021-05-26T16:27:27.117531Z","iopub.status.idle":"2021-05-26T16:27:27.126868Z","shell.execute_reply.started":"2021-05-26T16:27:27.117349Z","shell.execute_reply":"2021-05-26T16:27:27.125812Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* # Preliminaries I: standardize the images' size\n\n1. We have a huge amount of images with different sizes and shapes. We will this define a resizing function **resize_image** that resize consistently the image to a shape passed to the function by the user (by default is (128,128)), as done in my notebook on [image pre-treatment](https://www.kaggle.com/androbomb/image-pre-treatment). ","metadata":{}},{"cell_type":"code","source":"from PIL import Image\n\n# function to resize image\ndef resize_image(src_image, size=(128,128), bg_color=\"white\"): \n    from PIL import Image, ImageOps \n    \n    # resize the image so the longest dimension matches our target size\n    src_image.thumbnail(size, Image.ANTIALIAS)\n    \n    # Create a new square background image\n    new_image = Image.new(\"RGB\", size, bg_color)\n    \n    # Paste the resized image into the center of the square background\n    new_image.paste(src_image, (int((size[0] - src_image.size[0]) / 2), int((size[1] - src_image.size[1]) / 2)))\n  \n    # return the resized image\n    return new_image","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:27:27.128695Z","iopub.execute_input":"2021-05-26T16:27:27.129328Z","iopub.status.idle":"2021-05-26T16:27:27.138391Z","shell.execute_reply.started":"2021-05-26T16:27:27.129270Z","shell.execute_reply":"2021-05-26T16:27:27.137572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"training_folder_name = 'input/edafolder/EDA folder'\n\n# New location for the resized images\ntrain_folder = 'working/data/edadatafolder'\n\n\n# Create resized copies of all of the source images\nsize = (96,96)\n\n# Create the output folder if it doesn't already exist\nif os.path.exists(train_folder):\n    shutil.rmtree(train_folder)\n\n# Create a dictionary with the file names and their value of cancer\ncancer_dict = pd.read_csv('input/edacsv/EDAlabels.csv', header=None, index_col=0, squeeze=True).to_dict()\n# print(cancer_dict)\n\n# Loop through each subfolder in the input folder\nprint('Transforming images...')\nfor root, folders, files in os.walk(training_folder_name):\n    file_names = os.listdir(root)\n    for file_name in file_names:\n        result = cancer_dict[file_name[:-4]]\n        saveFolder = os.path.join(train_folder, classes[result])\n        print(saveFolder)\n        \n        if not os.path.exists(saveFolder):\n            os.makedirs(saveFolder)\n        \n        file_path = os.path.join(root, file_name)\n        print(\"reading \" + file_path)\n        \n        image = Image.open(file_path)\n        resized_image = resize_image(image, size)\n        saveAs = os.path.join(saveFolder, file_name)\n        print(\"writing \" + saveAs)\n        resized_image.save(saveAs)\n\nprint('Done.')","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:27:27.140112Z","iopub.execute_input":"2021-05-26T16:27:27.140374Z","iopub.status.idle":"2021-05-26T16:27:27.179981Z","shell.execute_reply.started":"2021-05-26T16:27:27.140319Z","shell.execute_reply":"2021-05-26T16:27:27.177970Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preliminaries II: prepare and augment the data\n\nPyTorch [1]  includes functions for loading and transforming data, as **[torchvision.datasets.ImageFolder](https://pytorch.org/docs/stable/torchvision/datasets.html)** .\n\nWe will use these to create an iterative loader for *training data*, and a second iterative loader for *test data*. \n\nThe loaders will transform the image data into *tensors*, which are the core data structure used in PyTorch, and normalize them so that the pixel values are in a scale with a mean of 0.5 and a standard deviation of 0.5.\n\nAt this point we can add transformations to randomly modify the images as they are added to a training batch. In this case, we will flip images horizontally at random. This is done to prevent **overfitting**. The transformations are done using **torchvision.transforms** [2]. The basic transformations available are\n1. transforms.RandomVerticalFlip(p=0.5)\n2. transforms.RandomHorizontalFlip(p=0.5)\n3. transforms.RandomRotation(degrees, resample=False, expand=False, center=None, fill=0)\n4. transforms.RandomResizedCrop(size, scale=(0.08, 1.0), ratio=(0.75, 1.3333333333333333), interpolation=2)\n    * Crop the given PIL Image to random size and aspect ratio.\n\n-----\n[1] https://pytorch.org/\n\n[2] https://pytorch.org/docs/stable/torchvision/transforms.html","metadata":{}},{"cell_type":"code","source":"def load_dataset(data_path):\n    import torch\n    import torchvision\n    import torchvision.transforms as transforms\n    # Load all the images\n    transformation = transforms.Compose([\n        # Randomly augment the image data\n            # Random horizontal flip\n        transforms.RandomHorizontalFlip(0.5),\n            # Random vertical flip\n        transforms.RandomVerticalFlip(0.3),\n        # transform to tensors\n        transforms.ToTensor(),\n        # Normalize the pixel values (in R, G, and B channels)\n        transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5])\n    ])\n\n    # Load all of the images, transforming them\n    full_dataset = torchvision.datasets.ImageFolder(\n        root=data_path,\n        transform=transformation\n    )\n    \n    \n    # Split into training (70% and testing (30%) datasets)\n    train_size = int(0.7 * len(full_dataset))\n    test_size = len(full_dataset) - train_size\n    \n    # use torch.utils.data.random_split for training/test split\n    train_dataset, test_dataset = torch.utils.data.random_split(full_dataset, [train_size, test_size])\n    \n    # define a loader for the training data we can iterate through in 50-image batches\n    train_loader = torch.utils.data.DataLoader(\n        train_dataset,\n        batch_size=3,\n        num_workers=0,\n        shuffle=False\n    )\n    \n    # define a loader for the testing data we can iterate through in 50-image batches\n    test_loader = torch.utils.data.DataLoader(\n        test_dataset,\n        batch_size=3,\n        num_workers=0,\n        shuffle=False\n    )\n        \n    return train_loader, test_loader\n\n\n\n\n#####################################################################################################\n\n\n\n\n# Recall that we have resized the images and saved them into\ntrain_folder = '../working/data/edadatafolder'\n\n# Get the iterative dataloaders for test and training data\ntrain_loader, test_loader = load_dataset(train_folder)\nbatch_size = train_loader.batch_size\nprint(\"Data loaders ready to read\", train_folder)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:27:27.181534Z","iopub.status.idle":"2021-05-26T16:27:27.182339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Defining the Convolutional Neural Network\n\nIn PyTorch, you define a neural network model as a class that is derived from the **[nn.Module](https://pytorch.org/docs/stable/_modules/torch/nn/modules/module.html)** base class. \n\nYour class must define the layers in the network, and provide a *forward* method that is used to process data through the layers of the network.\n\nAs in the kernel [MNIST_ale_CNN](https://www.kaggle.com/androbomb/mnist-ale-cnn), where we used keras to create a CNN to apply it to the standard MNIST database, we build a CNN as following:\n\nWe now need to create our Convolutional Neural Network model. In order to do so, we need to choose the convolution and poolying layers.\n\nWe will thus:\n\n1. Define the model as a sequential layers\n2. Introduce the Convolutions. \n    * they are important to create a feature map\n3. Introduce the Poolings\n    * MaxPooling is used to reduce dimensionality. In MaxPooling, the output value is just the maximum of the input values in each patch (for ex. The maximum pixel in a span of 3 pixels).\n4. Flatten the data in order to have a np.array to feed the NN\n\nIn particoular, we will apply\n\n1. 16 3x3 Convolution matrix with activation='relu', i.e. a Rectified Linear Unit (ReLU) activation.\n2. A 2x2 MaxPooling\n3. 32 3x3 Convolution matrix with activation='relu', i.e. a Rectified Linear Unit (ReLU) activation.\n4. A 2x2 MaxPooling\n5. 64 3x3 Convolution matrix with activation='relu', i.e. a Rectified Linear Unit (ReLU) activation.\n6. A 2x2 MaxPooling\n7. 64 3x3 Convolution matrix with activation='relu', i.e. a Rectified Linear Unit (ReLU) activation.\n8. A 2x2 MaxPooling\n\nAs a optimaizer, we decided to use the **ADAM (ADAptive Moment estimation)** optimization algorithm, that is an extension to stochastic gradient descent that has recently seen broader adoption for deep learning applications in computer vision and natural language processing. For more on optimizers in PyTorch, see [1]\n\nNotice that we have a layer that randomly drops 20% of the features to prevent **overfitting**. \n\n-----\n[1] https://pytorch.org/docs/stable/optim.html#algorithms","metadata":{}},{"cell_type":"code","source":"# Create a neural net class\nclass Net(nn.Module):\n    \n    \n    # Defining the Constructor\n    def __init__(self, num_classes=3):\n        super(Net, self).__init__()\n        \n        # In the init function, we define each layer we will use in our model\n        \n        # Our images are RGB, so we have input channels = 3. \n        # We will apply 12 filters in the first convolutional layer\n        self.conv1 = nn.Conv2d(in_channels=3, out_channels=12, kernel_size=3, stride=1, padding=1)\n        \n        # A second convolutional layer takes 12 input channels, and generates 24 outputs\n        self.conv2 = nn.Conv2d(in_channels=12, out_channels=24, kernel_size=3, stride=1, padding=1)\n        \n        # We in the end apply max pooling with a kernel size of 2\n        self.pool = nn.MaxPool2d(kernel_size=2)\n        \n        # A drop layer deletes 20% of the features to help prevent overfitting\n        self.drop = nn.Dropout2d(p=0.2)\n        \n        # Our 128x128 image tensors will be pooled twice with a kernel size of 2. 128/2/2 is 32.\n        # This means that our feature tensors are now 32 x 32, and we've generated 24 of them\n        \n        # We need to flatten these in order to feed them to a fully-connected layer\n        self.fc = nn.Linear(in_features=13824, out_features=num_classes)\n\n    def forward(self, x):\n        # In the forward function, pass the data through the layers we defined in the init function\n        \n        # Use a ReLU activation function after layer 1 (convolution 1 and pool)\n        x = F.relu(self.pool(self.conv1(x))) \n        \n        # Use a ReLU activation function after layer 2\n        x = F.relu(self.pool(self.conv2(x)))  \n        \n        # Select some features to drop to prevent overfitting (only drop during training)\n        x = F.dropout(self.drop(x), training=self.training)\n        \n        # Flatten\n        x = x.view(-1, 13824)\n        # Feed to fully-connected layer to predict class\n        x = self.fc(x)\n        # Return class probabilities via a log_softmax function \n        return torch.log_softmax(x, dim=1)\n    \ndevice = \"cpu\"\nif (torch.cuda.is_available()):\n    # if GPU available, use cuda (on a cpu, training will take a considerable length of time!)\n    device = \"cuda\"\n\n# Create an instance of the model class and allocate it to the device\nmodel = Net(num_classes=len(classes)).to(device)\n\nprint(model)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:27:27.184025Z","iopub.status.idle":"2021-05-26T16:27:27.184882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Training function\n\nTraining consists of an iterative series of forward passes in which the training data is processed in batches by the layers in the network, and the optimizer goes back and adjusts the weights. We will also use a separate set of test images to test the model at the end of each *epoch*, so we can track the performance improvement as the training process progresses. \n\n\n\nThe traning function we need to define needs the following steps: \n\n1. Set the model to training mode;\n2. Process the images in batches; we will iterate over images in batches. Inside each batches, we have to\n    1.  Import labels and features;\n    2. Reset the optimizer\n    3. Push the data forward through the layers of the model\n    4. compute the loss\n    5. Backpropagate\n3.  Compute the Average Loss of the Model during the Epoch\n\nWe will thus calling eath once per epoch. ","metadata":{}},{"cell_type":"code","source":"def train(model, device, train_loader, optimizer, epoch):\n    # Set the model to training mode\n    model.train()\n    train_loss = 0\n    print(\"Epoch:\", epoch)\n    # Process the images in batches\n    for batch_idx, (data, target) in enumerate(train_loader):\n        # Use the CPU or GPU as appropriate\n        # Recall that GPU is optimized for the operations we are dealing with\n        data, target = data.to(device), target.to(device)\n        \n        # Reset the optimizer\n        optimizer.zero_grad()\n        \n        # Push the data forward through the model layers\n        output = model(data)\n        \n        # Get the loss\n        loss = loss_criteria(output, target)\n\n        # Keep a running total\n        train_loss += loss.item()\n        \n        # Backpropagate\n        loss.backward()\n        optimizer.step()\n        \n        # Print metrics so we see some progress\n        print('\\tTraining batch {} Loss: {:.6f}'.format(batch_idx + 1, loss.item()))\n            \n    # return average loss for the epoch\n    avg_loss = train_loss / (batch_idx+1)\n    print('Training set: Average loss: {:.6f}'.format(avg_loss))\n    return avg_loss","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:27:27.186232Z","iopub.status.idle":"2021-05-26T16:27:27.187162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Test function: \n\nHere we need the model in evaluation mode to get the accuray by confronting with the labels (we don't propagate anything here). So\n\n1. Set the model to evaluation mode;\n2. Process the images in batches; we will iterate over images in batches. Inside each batches, we have to\n    1. Get the prediction for each image in the batch\n    2. Calculate the loss for the batch\n    3. Calculate the accuracy for this batch\n3. Calculate the average accuracy and loss for the epoch","metadata":{}},{"cell_type":"code","source":"def test(model, device, test_loader):\n    # Switch the model to evaluation mode (so we don't backpropagate or drop)\n    model.eval()\n    test_loss = 0\n    correct = 0\n    with torch.no_grad():\n        batch_count = 0\n        for data, target in test_loader:\n            batch_count += 1\n            data, target = data.to(device), target.to(device)\n            \n            # Get the predicted classes for this batch\n            output = model(data)\n            \n            # Calculate the loss for this batch\n            test_loss += loss_criteria(output, target).item()\n            \n            # Calculate the accuracy for this batch\n            _, predicted = torch.max(output.data, 1)\n            correct += torch.sum(target==predicted).item()\n\n    # Calculate the average loss and total accuracy for this epoch\n    avg_loss = test_loss / batch_count\n    print('Validation set: Average loss: {:.6f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n        avg_loss, correct, len(test_loader.dataset),\n        100. * correct / len(test_loader.dataset)))\n    \n    # return average loss for the epoch\n    return avg_loss","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:27:27.188545Z","iopub.status.idle":"2021-05-26T16:27:27.189376Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training the Model\n\n\n#### Adam optimizer\nWhen training the Model, we use the **ADAM** optimizer [1], [2], that is an adaptive learning rate optimization algorithm that’s been designed specifically for training deep neural networks. \n\nAdam can be looked at as a combination of RMSprop and Stochastic Gradient Descent with momentum. It uses the squared gradients to scale the learning rate like RMSprop and it takes advantage of momentum by using moving average of the gradient instead of gradient itself like SGD with momentum. Let’s take a closer look at how it works.\nAdam is an adaptive learning rate method, which means, it computes individual learning rates for different parameters. Its name is derived from adaptive moment estimation, and the reason it’s called that is because Adam uses estimations of first and second moments of gradient to adapt the learning rate for each weight of the neural network.\n\n\n#### Loss Criteria\nAs a loss criteria we use the **Cross Entropy Loss** (log loss), that measures the performance of a classification model whose output is a probability value between 0 and 1. Cross-entropy loss increases as the predicted probability diverges from the actual label. So predicting a probability of .012 when the actual observation label is 1 would be bad and result in a high loss value. A perfect model would have a log loss of 0. [3]\n\n\n------\n[1] https://arxiv.org/abs/1412.6980\n\n[2] https://towardsdatascience.com/adam-latest-trends-in-deep-learning-optimization-6be9a291375c\n\n[3] https://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html","metadata":{}},{"cell_type":"code","source":"# Use an \"Adam\" optimizer to adjust weights\noptimizer = optim.Adam(model.parameters(), lr=0.01)\n\n# Specify the loss criteria\nloss_criteria = nn.CrossEntropyLoss()\n\n# Track metrics in these arrays\nepoch_nums = []\ntraining_loss = []\nvalidation_loss = []\n\n# Train over 10 epochs (We restrict to 10 for time issues)\nepochs = 10\nprint('Training on', device)\nfor epoch in range(1, epochs + 1):\n        train_loss = train(model, device, train_loader, optimizer, epoch)\n        test_loss = test(model, device, test_loader)\n        epoch_nums.append(epoch)\n        training_loss.append(train_loss)\n        validation_loss.append(test_loss)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:27:27.190878Z","iopub.status.idle":"2021-05-26T16:27:27.191603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## View Loss History","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(15,15))\nplt.plot(epoch_nums, training_loss)\nplt.plot(epoch_nums, validation_loss)\nplt.xlabel('epoch')\nplt.ylabel('loss')\nplt.legend(['training', 'validation'], loc='upper right')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:27:27.192835Z","iopub.status.idle":"2021-05-26T16:27:27.193556Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Evaluate the Model","metadata":{}},{"cell_type":"code","source":"print (\"hello\")\n# Defining Labels and Predictions\ntruelabels = []\npredictions = []\nmodel.eval()\nprint(\"Getting predictions from test set...\")\nprint(test_loader)\nfor data, target in test_loader:\n    print(target)\n    print(data)\n    for label in target.data.numpy():\n        print (label)\n        truelabels.append(label)\n    for prediction in model(data.to(device)).cpu():\n        p = prediction.data.numpy()\n        print (p.shape)\n        predictions.append(p.argmax(0))\n\n# Plot the confusion matrix\ncm = confusion_matrix(truelabels, predictions)\ntick_marks = np.arange(len(classes))\nprint(cm)\ndf_cm = pd.DataFrame(cm, index = classes, columns = classes)\nprint(\"So help me god\")\nplt.figure(figsize = (7,7))\nsns.heatmap(df_cm, annot=True, cmap=plt.cm.Blues, fmt='g')\nplt.xlabel(\"Predicted Shape\", fontsize = 20)\nplt.ylabel(\"True Shape\", fontsize = 20)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:27:27.194992Z","iopub.status.idle":"2021-05-26T16:27:27.195797Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that there is a huge mislassification among dog and cats. It is somewhat expected, since dogs and cat looks alike, and we need a well deeper CNN to properly classify them. \n\n\nA somewhat unexpected classification error is the superposition of Flowers upon Dog/Cats, instead of fruits that are well classified!","metadata":{}},{"cell_type":"markdown","source":"Here we insert a code to cancel the images we have resized and saved into the '../working' folder due to the fact that Kaggle limits to 500 the number of output files:","metadata":{}},{"cell_type":"code","source":"import shutil\nshutil.rmtree(\"../working/data/edadatafolder\")","metadata":{"execution":{"iopub.status.busy":"2021-05-26T16:27:27.197150Z","iopub.status.idle":"2021-05-26T16:27:27.197921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I hope you enjoyed the reading!","metadata":{}}]}