{"cells":[{"metadata":{},"cell_type":"markdown","source":"
\n\n

Let's set the Agenda!\n

\n\nPurpose of this Kernel is to serve as a Tutorial for the beginners.\n \nMake you understand the basic concepts of a CNN, build a simple CNN Model(with & without Regularization) for a Binary Image Classification problem.\n \nThis Tutorial will help you understand the different pieces in the architechture of a CNN, different layers used & different parameters associated with each and every layer."},{"metadata":{},"cell_type":"markdown","source":"
\n\n

CNN - Convolutional Neural Network\n

\n\nIn Deep Learning, a `Convolutional Neural Network` (CNN, or ConvNet) is a class of Deep Neural Networks, most commonly applied to Analyzing Visual Imagery.\n \n### Why do we even need a CNN when we have Traditional Neural Networks?\nI will assume that you are already familiar with traditional neural networks called the multilayer perceptron (MLP). \n\nIf you are not familiar with these, there are hundreds of tutorials on Medium outlining how MLPs work. \n\nThese are modeled on the human brain, whereby neurons are stimulated by connected nodes and are only activated when a certain threshold value is reached. A standard multilayer perceptron (traditional neural network) is shown in the image below:\n \n\n \nThere are several drawbacks of MLP’s, especially when it comes to Image Processing. \n\nMLPs use one Perceptron for each input (e.g. pixel in an image, multiplied by 3 in RGB case). \n\nThe amount of weights rapidly becomes unmanageable for large images. For a 224 x 224 pixel image with 3 color channels there are around 150,000 weights that must be trained! As a result, difficulties arise whilst training and overfitting can occur.\n\nAnother common problem is that MLPs react differently to an input (images) and its shifted version — they are not translation invariant. For example, if a picture of a cat appears in the top left of the image in one picture and the bottom right of another picture, the MLP will try to correct itself and assume that a cat will always appear in this section of the image.\n \nClearly, MLPs are not the best idea to use for image processing. One of the main problems is that spatial information is lost when the image is flattened into an MLP. \n \nNodes that are close together are important because they help to define the features of an image. We thus need a way to leverage the spatial correlation of the image features (pixels) in such a way that we can see the cat in our picture no matter where it may appear. \n \nA CNN can help us overcoming the problems that we discussed above, let's see how!"},{"metadata":{},"cell_type":"markdown","source":"
\n\n

Basic Building Blocks of a CNN Model\n

\n\n#### Convolution layer:\nThe convolution operation is a dot product of the original pixel values with weights defined in the `filter`(more about filters later).A “filter”, sometimes called a “kernel”, is passed over the image, viewing a few pixels at a time (for example, 3X3 or 5X5).The results are summed up into one number that represents all the pixels the filter observed. \n \n#### Pooling layer:\n“pooling” is the process of further downsampling and reducing the size of the matrix. A filter is passed over the results of the previous layer and selects one number out of each group of values (typically the maximum, this is called max pooling). This allows the network to train much faster, focusing on the most important information in each feature of the image.\n\n#### Fully connected layer:\nA traditional multilayer perceptron structure. Its input is a one-dimensional vector representing the output of the previous layers. Its output is a list of probabilities for different possible labels attached to the image (e.g. dog, cat, bird). The label that receives the highest probability is the classification decision.\n \n\n \n
\nWe will discuss all these building blocks in details one by one. \n \nLets start with the operations performed in a Convolutional Layer."},{"metadata":{},"cell_type":"markdown","source":"
\n\n

Operations in a Convolutional Layer \n

\n \nBefore we proceed further, it is very important that we understand the meaning of following operations that happen in a Convolutional Layer:\n* Applying Filters\n* Creating Feature Maps\n* Padding"},{"metadata":{},"cell_type":"markdown","source":"
\n\n

1) Filters\n

\n\nWe analyze the influence of nearby pixels by using something called a `filter`. CNN’s leverage the fact that nearby pixels are more strongly related than distant ones\n\nLet's see the image below:\n\n\n \nA filter is exactly what you think it is, in our situation, we take a filter of a size specified by the user (a rule of thumb is 3x3 or 5x5) and we move this across the image from top left to bottom right. For each point on the image, a value is calculated based on the filter using a convolution operation.\n\nA filter could be related to anything, for pictures of humans, one filter could be associated with seeing noses, and our nose filter would give us an indication of how strongly a nose seems to appear in our image, and how many times and in what locations they occur. This reduces the number of weights that the neural network must learn compared to an MLP, and also means that when the location of these features changes it does not throw the neural network off.\n\nAn example of the convolution operation is shown below:\n\n \nIf you are wondering how the different features are learned by the network, and whether it is possible that the network will learn the same features (having 10 nose filters would be kind of redundant), this is highly unlikely to happen. When building the network, we randomly specify values for the filters, which then continuously update themselves as the network is trained. It is very very unlikely that two filters that are the same will be produced unless the number of chosen filters is extremely large.\n\nSome examples of filters, or kernels as we call them, are given below:\n"},{"metadata":{},"cell_type":"markdown","source":"
\n\n

2) Feature Maps\n

\n \nAfter the filters have passed over the image, a `feature map` is generated for each filter. \n\nThese are then taken through an activation function, which decides whether a certain feature is present at a given location in the image. \n \nWe can then do a lot of things, such as adding more filtering layers and creating more feature maps, which become more and more abstract as we create a deeper CNN. \n \nWe can also use pooling layers in order to select the largest values on the feature maps and use these as inputs to subsequent layers. \n \nIn theory, any type of operation can be done in pooling layers, but in practice, only max pooling is used because we want to find the outliers — these are when our network sees the feature!\n \n\n \n Above image shows an example of CNN with two convolutional layers, two pooling layers, and a fully connected layer which decides the final classification of the image into one of several categories."},{"metadata":{},"cell_type":"markdown","source":"
\n\n

3) Padding\n

\n \nCNN’s are also composed of layers, but those layers are not fully connected: they have filters, sets of cube-shaped weights that are applied throughout the image. Each 2D slice of the filters are called kernels. These filters introduce translation invariance and parameter sharing. How are they applied? Convolutions! \n \nExample of how convolutions are applied to images using kernel filters is shown in the image below:\n \n \n \nA good question to have right now is what happens at the edges of the image? \n \nIf we apply convolutions on a normal image, the result will be down-sampled by an amount depending on the size of the filter. \n \nWhat do we do if we don’t want this to happen? We can use padding.\n\nPadding essentially makes the feature maps produced by the filter kernels the same size as the original image. \n \nThis is very useful for deep CNN’s as we don’t want the output to be reduced so that we only have a 2x2 region left at the end of the network upon which to predict our result.\n \nBelow image shows how padding works:\n \n"},{"metadata":{},"cell_type":"markdown","source":"
\n\n

Pooling Layers\n

\n \nWe will now see the different type Pooling Layers and how they work.\n\n### Definition of Pooling:\nPooling is basically “downscaling” the image obtained from the previous layers. It can be compared to shrinking an image to reduce its pixel density.\n \n### What are pooling operations?\n\nFirst Convolutional Layer in a CNN can be represented as follows(assume we are working on an Image Classification Problem):\n \n\n
\nThe inputs for this layer are images, of height H, width W and with three channels. \n
\nThus, they’re likely RGB images. Using a 3x3x3 kernel, a convolution operation is performed over the input image, generating N “feature maps” of size H_{fm} \\times W_{fm}\n\n
\nOne feature map learns one particular feature present in the image.` \n
\n \nIn the first layer, feature maps captures very low-level elements within the image, such as curves and edges, a.k.a. the details.\n\nHowever, we cannot see the higher-level patterns with just one convolutional layer. We need many, stacked together, to learn these patterns. This is also called building a spatial hierarchy. Good spatial hierarchies summarize the data substantially when moving from bottom to top, and they’re like a pyramid. Here’s a good one(left hand side image) versus a bad one(right hand side image):\n \n\n \n
\n \n### There are 4 types of Pooling Layers:\n \n* Max Pooling\n* Average Pooling\n* Global Max Pooling\n* Global Average Pooling\n \nLet's take a look at these one by one.\n \n### 1) Max Pooling\nSuppose that this is one of the 4 x 4 pixels feature maps from our CNN:\n\n\nIf we want to downsample it, we can use a pooling operation what is known as “max pooling” (more specifically, this is two-dimensional max pooling). \n \nIn this pooling operation, a H×W “block” slides over the input data, where H is the height and W the width of the block. \n \nThe stride (i.e. how much it steps during the sliding operation) is often equal to the pool size, so that its effect equals a reduction in height and width.\n\nFor each block, or “pool”, the operation simply involves computing the max value, like this:\n\n \nDoing so for each pool, we get a nicely downsampled outcome, greatly benefiting the spatial hierarchy we need:\n\n\nMax Pooling comes in a `one-dimensional`, `two-dimensional` and `three-dimensional` variant The one-dimensional variant can be used together with Conv1D layers, and thus for temporal data.\n
\nMaxPooling Layer provided by KERAS API(dont worry about the parameters, we will discuss the important ones in detail later):\nkeras.layers.MaxPooling1D(pool_size=2, strides=None, padding='valid', data_format='channels_last')\nkeras.layers.MaxPooling2D(pool_size=(2, 2), strides=None, padding='valid', data_format=None)\nkeras.layers.MaxPooling3D(pool_size=(2, 2, 2), strides=None, padding='valid', data_format=None)\n \n### 2) Average Pooling:\nAverage Pooling is different from Max Pooling in the sense that it retains much information about the “less important” elements of a block, or pool. Whereas Max Pooling simply throws them away by picking the maximum value, Average Pooling blends them in. This can be useful in a variety of situations, where such information is useful.\n \nkeras.layers.AveragePooling1D(pool_size=2, strides=None, padding='valid', data_format='channels_last')\n
\nkeras.layers.AveragePooling2D(pool_size=(2, 2), strides=None, padding='valid', data_format=None)\n
\nkeras.layers.AveragePooling3D(pool_size=(2, 2, 2), strides=None, padding='valid', data_format=None)\n\n#### Which one is better then? Max Pooling or Average Pooling?\nThere is no fix answer to this question. It all depends on the task you are performing. If the position of objects is not important, Max Pooling seems to be the better choice. If it is, it seems that better results can be achieved with Average Pooling.\n \n### 3) Global Max Pooling\nHere, we set the pool size equal to the input size, so that the max of the entire input is computed as the output value.\n\n \nOr, visualizing it differently:\n\n \n \nGlobal pooling layers can be used in a variety of cases. Primarily, it can be used to reduce the dimensionality of the feature maps output by some convolutional layer, to replace Flattening and sometimes even Dense layers in your classifier. \n \nWhat’s more, it can also be used for e.g. word spotting. This is due to the property that it allows detecting noise, and thus “large outputs” (e.g. the value 9 in the exmaple above). However, this is also one of the downsides of Global Max Pooling, and like the regular one.\n\nkeras.layers.GlobalMaxPooling1D(data_format='channels_last')\n
\nkeras.layers.GlobalMaxPooling2D(data_format='channels_last')\n
\nkeras.layers.GlobalMaxPooling3D(data_format='channels_last')\n \n### 4) Global Average Pooling\nWhen applying Global Average Pooling, the pool size is still set to the size of the layer input, but rather than the maximum, the average of the pool is taken:\n\n\nOr, once again when visualized differently:\n\n \n\nThey’re often used to replace the fully-connected or densely-connected layers in a classifier. \n \nInstead, the model ends with a convolutional layer that generates as many feature maps as the number of target classes, and applies global average pooling to each in order to convert each feature map into one value. \n \nAs feature maps can recognize certain elements within the input data, the maps in the final layer effectively learn to “recognize” the presence of a particular class in this architecture. By feeding the values generated by global average pooling into a Softmax activation function, you once again obtain the multiclass probability distribution that you want.\n\nWhat’s more, this approach might improve model performance because of the nativeness of the “classifier” to the “feature extractor” (they’re both convolutional instead of convolutional/dense), and reduce overfitting because of the fact that there is no parameter to be learnt in the global average pooling layer. Isn't this interesting? You can try exprenting this on your own once we are done with buiding CNN model.\n \nkeras.layers.GlobalAveragePooling1D(data_format='channels_last')\n
\nkeras.layers.GlobalAveragePooling2D(data_format='channels_last')\n
\nkeras.layers.GlobalAveragePooling3D(data_format='channels_last')\n \n \n#### Plesae Note: We are going to use 2D Layers in our implementation below"},{"metadata":{},"cell_type":"markdown","source":"
\n\n

Comparison of different layers in a CNN Network\n

\n \nSo, far we have learnt the Basic Building blocks of a CNN model, different kind of layers present in the CNN, different operations performed in Convolution Layers, different type of Pooling Layers available to us. It's time to see how these layers are different from each other.\n \nAs discussed above, There are three types of layers in a convolutional neural network: \n\n* Convolutional layer\n* Pooling layer\n* Fully connected layer\n\nEach of these layers has different parameters(we will learn about these parameters later in this kernel) that can be optimized and performs a different task on the input data.\n \n### Convolutional layers: \n\nConvolutional layers are the layers where filters are applied to the original image, or to other feature maps in a deep CNN. \n\nThis is where most of the user-specified parameters are in the network. The most important parameters are the number of `kernels` and the `size of the kernels`.\n\n\n \n

\n \n### Pooling Layers\n \nPooling layers are similar to convolutional layers, but they perform a specific function such as max pooling, which takes the maximum value in a certain filter region, or average pooling, which takes the average value in a filter region. These are typically used to reduce the dimensionality of the network.\n\n\nMore about Pooling layers later! \n
\n \n### Pooling Vs Convolution\nAs discussed earlier,In a CNN, a small block slides over the entire input image, taking element-wise multiplications with the part of the image it currently slides over. This is a relatively expensive operation. Can’t this be done in a simpler way? Do we really need to have a hierarchy built up from convolutions only? The answer is no, and pooling operations prove this.\n\nPooling is performed though a hardcoded tensor operation such as max, rather than through a learned transformation, we don’t need the relatively expensive operation of learning the weights. This way, we get a nice and possibly useful spatial hierarchy at a fraction of the cost.\n \n### Fully Connected Layers\n\nFully Connected Layers are placed before the Classification Output of a CNN and are used to flatten the results before classification. This is similar to the output layer of an MLP.\n \n\n

\n \n### Image showing a Fully Connected Dense Layer below:\n "},{"metadata":{},"cell_type":"markdown","source":"
\n\n

Let's Implement CNN from scratch!\n

\n \nWe will build a CNN for the given Image Classification problem.\n \nWe have images available in multiple formats here, Jpeg, tfrecords, dicom files. We are going to use Jpeg images to train our CNN model.\n \nThe task is to basically to identify whether a given image is benign(target=0), or malignant(target=1).\n \nIf you want to know more about this competition, feel free to refer to my other notebook [here](https://www.kaggle.com/jagdmir/siim-melanoma-classification-modelling)"},{"metadata":{},"cell_type":"markdown","source":"### Load libraries"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport glob\nimport cv2\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train_dir='/kaggle/input/siim-isic-melanoma-classification/jpeg/train/'\ntrain=pd.read_csv('/kaggle/input/siim-isic-melanoma-classification/train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Add image path to the training dataset itself"},{"metadata":{"trusted":true},"cell_type":"code","source":"train['path'] = train_dir + train.image_name + \".jpg\"\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Let's load a sample image"},{"metadata":{"trusted":true},"cell_type":"code","source":"img=cv2.imread('../input/siim-isic-melanoma-classification/jpeg/train/ISIC_0015719.jpg') \nimg = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\nplt.imshow(img)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Class Distribution"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.target.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will take 600 images from for target = 0 and will take all 584 = 1184 images for target = 1\nso, we will have total of 600 + 584 images in total to train our model.\n\nThis number is obviously not enough to train a image classification model, but since this is just a demo, we will take a small number of images to train the model quickly."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_0=train[train['target']==0].sample(600)\ndf_1=train[train['target']==1]\ntrain=pd.concat([df_0,df_1])\ntrain=train.reset_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# we will resize the given images to 128 x 128 size images for faster processing\nIMG_DIM = (128, 128)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Keras provides some amazing libraries to work with images, lets import them\nfrom keras.preprocessing.image import ImageDataGenerator, load_img, img_to_array, array_to_img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# We will reserve 20% of our training data for the validation purpose\nfrom sklearn.model_selection import train_test_split\nX_train, X_val, y_train, y_val = train_test_split(train, train.target, test_size=0.2, random_state=42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# save image path\ntrain_files = X_train.path\nval_files = X_val.path\n\n# load images using load_img function from keras preprocessing \n# target_size is used to load the images with smaller size\n# img_to_array will tranform the loaded image to an array\ntrain_imgs = [img_to_array(load_img(img, target_size=IMG_DIM)) for img in train_files]\nvalidation_imgs = [img_to_array(load_img(img, target_size=IMG_DIM)) for img in val_files]\n\n# convert the list of arrays to array\ntrain_imgs = np.array(train_imgs)\ntrain_labels = y_train\n\nvalidation_imgs = np.array(validation_imgs)\nval_labels = y_val\n\n\nprint('Train dataset shape:', train_imgs.shape, \n '\\tValidation dataset shape:', validation_imgs.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Scale Images\nscale each image with pixel values between (0, 255) to values between (0, 1) because deep learning models work really well with small input values."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_imgs_scaled = train_imgs.astype('float32')\n\nvalidation_imgs_scaled = validation_imgs.astype('float32')\n\n# divide the pixels by 255 to scale the pixels between 0 and 1\ntrain_imgs_scaled /= 255\nvalidation_imgs_scaled /= 255\n\nprint(train_imgs[0].shape)\n\n# array_to_img function will convert the given array to image\narray_to_img(train_imgs[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n\n

Model Building\n

"},{"metadata":{},"cell_type":"markdown","source":"### The following terminology is very important with regard to training our model:\n* The `batch_size` indicates the total number of images passed to the model per iteration.\n\n* The `weights` of the units in layers are updated after each iteration.\n\n* The total number of iterations is always equal to the total number of training samples divided by the batch_size\n\n* An `epoch` is when the complete dataset has passed through the network once, that is, all the iterations are completed based on data batches."},{"metadata":{},"cell_type":"markdown","source":"We will use a batch_size of 30 and our training data has a total of 947 samples, which indicates that there will be a total of 947/30 = 32(approx) iterations per epoch. \n\nWe train the model for a total of 30 epochs and validate it consequently on our validation set of 237 images."},{"metadata":{"trusted":true},"cell_type":"code","source":"# setup basic configuration\nbatch_size = 30\nnum_classes = 2\nepochs = 30\ninput_shape = (128, 128, 3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### CNN Architecture and Parameters"},{"metadata":{},"cell_type":"markdown","source":"We will start by building a basic CNN model with three convolutional layers, coupled with max pooling for auto-extraction of features from our images and also downsampling the output convolution feature maps."},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"
\n\n

Parameters used in a Conv2D (Convolutional) Layer\n

"},{"metadata":{},"cell_type":"markdown","source":"Generally, a Conv2D layer can be defined as below:\n\nConv2D(\n filters,\n kernel_size,\n strides=(1, 1),\n padding=\"valid\",\n data_format=None,\n dilation_rate=(1, 1),\n groups=1,\n activation=None,\n use_bias=True,\n kernel_initializer=\"glorot_uniform\",\n bias_initializer=\"zeros\",\n kernel_regularizer=None,\n bias_regularizer=None,\n activity_regularizer=None,\n kernel_constraint=None,\n bias_constraint=None,\n **kwargs\n)\n\nYou can see a lot of parameters used in a Convolutional layer. We will discuss few of them here,for the rest, feel free to check KERAS documentation."},{"metadata":{},"cell_type":"markdown","source":"### filters \n\nThe Keras Conv2D parameter, `filters` determines the number of kernels to convolve with the input volume. \n\nLayers early in the network architecture (i.e., closer to the actual input image) learn fewer convolutional filters while layers deeper in the network (i.e., closer to the output predictions) will learn more filters.\n\nNotice at as our output spatial volume is decreasing our number of filters learned is increasing — this is a common practice in designing CNN architectures \n\nYou may need to tune the exact value depending on:\n* the complexity of your dataset\n* the depth of your neural network, but I recommend starting with filters in the range [32, 64, 128] in the earlier and increasing up to [256, 512, 1024] in the deeper layers.\n\nAgain, the exact range of the values may be different for you, but start with a smaller number of filters and only increase when necessary."},{"metadata":{},"cell_type":"markdown","source":"\n### kernel_size\n\n`kernel_size` is a 2-tuple specifying the width and height of the 2D convolution window.\n\nTypical values for kernel_size include: (1, 1) , (3, 3) , (5, 5) , (7, 7) . It’s rare to see kernel sizes larger than 7×7.\n\nSo, when do you use each?\n\nIf your input images are greater than 128×128 you may choose to use a kernel size > 3 to help:\n* learn larger spatial filters\n* to help reduce volume size\n\nOther networks, such as VGGNet, exclusively use (3, 3) filters throughout the entire network.\n\nMore advanced architectures such as Inception, ResNet, and SqueezeNet design entire micro-architectures which are “modules” inside the network that learn local features at different scales (i.e., 1×1, 3×3, and 5×5) and then combine the outputs."},{"metadata":{},"cell_type":"markdown","source":"### strides\nThe `strides` parameter is a 2-tuple of integers, specifying the “step” of the convolution along the x and y axis of the input volume.\n\nThe strides value defaults to (1, 1) , implying that:\n\nA given convolutional filter is applied to the current location of the input volume\n\nThe filter takes a 1-pixel step to the right and again the filter is applied to the input volume\n\nThis process is performed until we reach the far-right border of the volume in which we move our filter one pixel down and then start again from the far left\n\nTypically you’ll leave the strides parameter with the default (1, 1) value; \n\nhowever, you may occasionally increase it to (2, 2) to help reduce the size of the output volume (since the step size of the filter is larger)."},{"metadata":{},"cell_type":"markdown","source":"### padding\nThe Keras Conv2D `padding` parameter accepts either \"valid\" (no padding) or \"same\" (padding + preserving spatial dimensions).\n\nWith the valid parameter the input volume is not zero-padded and the spatial dimensions are allowed to reduce via the natural application of convolution.\n\nIf you instead want to preserve the spatial dimensions of the volume such that the output volume size matches the input volume size, then you would want to supply a value of same for the padding\n\nWhile the default Keras Conv2D value is valid I will typically set it to same for the majority of the layers in my network and then either reduce spatial dimensions of my volume by either:\n\n1) Max pooling\n\n2) Strided convolution\n\nNote: See [this](https://www.pyimagesearch.com/2016/07/25/convolutions-with-opencv-and-python/) tutorial on the basics of convolution if you need help understanding how and why spatial dimensions naturally reduce when applying convolutions."},{"metadata":{},"cell_type":"markdown","source":"### Other Parameters\n\nIf you like to learn about other parameters, you can go through Keras Documentation\n* data_format\n* dilation_rate\n* activation\n* use_bias\n* kernel_initializer\n* bias_initializer\n* kernel_regularizer\n* bias_regularizer\n* activity_regularizer\n* kernel_constraint \n* bias_constraint"},{"metadata":{},"cell_type":"markdown","source":"
\n\n

Parameters used in a MaxPooling2d (Max Pooling) Layer\n

\n \n#### keras.layers.MaxPooling2D(pool_size=(2, 2), strides=None, padding='valid', data_format=None)\n \n`pool size`: \nInteger or tuple of 2 integers, window size over which to take the maximum. (2, 2) will take the max value over a 2x2 pooling window. If only one integer is specified, the same window length will be used for both dimensions.\n \n`strides`:\nInteger, tuple of 2 integers, or None. Strides values. Specifies how far the pooling window moves for each pooling step. If None, it will default to pool_size.\n \n`padding`:\nOne of \"valid\" or \"same\" (case-insensitive). \"valid\" means no padding. \"same\" results in padding evenly to the left/right or up/down of the input such that output has the same height/width dimension as the input.\n \n`data format`:\nA string, one of channels_last (default) or channels_first. The ordering of the dimensions in the inputs. channels_last corresponds to inputs with shape (batch, height, width, channels) while channels_first corresponds to inputs with shape (batch, channels, height, width). It defaults to the image_data_format value found in your Keras config file at ~/.keras/keras.json. If you never set it, then it will be \"channels_last\"."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Here we will import the necessary libraries\nfrom keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout\n\n# we will import sequential model and add different layers to it\nfrom keras.models import Sequential\n\n# import optimizers, please go through online tutorials if you want to learn what is the purpose of an optimizer\nfrom keras import optimizers\n\n# creating and instance of Sequential\nmodel = Sequential()\n\n# add Conv2D layer(this is the convolutional layer we discussed earlier),filter size,kernel size,activation and padding are the parameters used\n# This layer would create feature maps for each and every filter used\n# feature maps created here are then taken through an activation function(relu here), which decides whether a certain feature is present \n# at a given location in the image.\nmodel.add(Conv2D(16, kernel_size=(3, 3), activation='relu', \n input_shape=input_shape))\n# Pooling layer used here will select the largest values on the feature maps and use these as inputs to subsequent layers\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\n\n# another set of Convolutional & Max Pooling layers\nmodel.add(Conv2D(64, kernel_size=(3, 3), activation='relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\nmodel.add(Conv2D(128, kernel_size=(3, 3), activation='relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\nmodel.add(Flatten())\n# Finally the Dense Layer\nmodel.add(Dense(512, activation='relu'))\n# sigmoid function here will help us in perform binary classification\nmodel.add(Dense(1, activation='sigmoid'))\n\n\nmodel.compile(loss='binary_crossentropy',\n optimizer=optimizers.RMSprop(),\n metrics=['accuracy'])\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n\n

Flatten Layer\n

\n \nWe have used one more layer above - Flatten Layer, which we have not yet discussed.\n\nLets see what it exactly does:\n\nFlatten is the function that converts the pooled feature map to a single column that is passed to the fully connected layer. \n\nIn other words, Flattening is converting the data into a 1-dimensional array for inputting it to the next layer. We flatten the output of the convolutional layers to create a single long feature vector. And it is connected to the final classification model, which is called a fully-connected layer, as we can seen in our implementation above."},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit(x=train_imgs_scaled, y=train_labels,\n validation_data=(validation_imgs_scaled, val_labels),\n batch_size=batch_size,\n epochs=epochs,\n verbose=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n\n

Let's check Model's performance\n

"},{"metadata":{"trusted":true},"cell_type":"code","source":"f, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))\nt = f.suptitle('Basic CNN Performance', fontsize=12)\nf.subplots_adjust(top=0.85, wspace=0.3)\n\nepoch_list = list(range(1,31))\nax1.plot(epoch_list, history.history['accuracy'], label='Train Accuracy')\nax1.plot(epoch_list, history.history['val_accuracy'], label='Validation Accuracy')\nax1.set_xticks(np.arange(0, 31, 5))\nax1.set_ylabel('Accuracy Value')\nax1.set_xlabel('Epoch')\nax1.set_title('Accuracy')\nl1 = ax1.legend(loc=\"best\")\n\nax2.plot(epoch_list, history.history['loss'], label='Train Loss')\nax2.plot(epoch_list, history.history['val_loss'], label='Validation Loss')\nax2.set_xticks(np.arange(0, 31, 5))\nax2.set_ylabel('Loss Value')\nax2.set_xlabel('Epoch')\nax2.set_title('Loss')\nl2 = ax2.legend(loc=\"best\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It is very much evident from the plots above that our model is not doing very well and it is overfitting after few epochs.\n\nTraining accuracy is increasing after every epoch but validation accuracy is decreasing.\n\nTraining loss is decreasing with each epoch but validation loss is consistently increasing after few epochs.\n\n### Let's handle overfitting by introducing some regularization (drop outs) to the model."},{"metadata":{},"cell_type":"markdown","source":"
\n\n

CNN Model with Regularization\n

\n\nLet’s improve upon our base CNN model by adding in one more convolution layer, another dense hidden layer. \n\nBesides this, we will add dropout of 0.3 after each hidden dense layer to enable regularization. \n\nBasically, dropout is a powerful method of regularizing in deep neural nets. It can be applied separately to both input layers and the hidden layers. \n\nDropout randomly masks the outputs of a fraction of units from a layer by setting their output to zero (in our case, it is 30% of the units in our dense layers)."},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Sequential()\n\nmodel.add(Conv2D(16, kernel_size=(3, 3), activation='relu',\n input_shape=input_shape))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\nmodel.add(Conv2D(64, kernel_size=(3, 3), activation='relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\nmodel.add(Conv2D(128, kernel_size=(3, 3), activation='relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\nmodel.add(Conv2D(128, kernel_size=(3, 3), activation='relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\nmodel.add(Flatten())\nmodel.add(Dense(512, activation='relu'))\nmodel.add(Dropout(0.3))\nmodel.add(Dense(512, activation='relu'))\nmodel.add(Dropout(0.3))\nmodel.add(Dense(1, activation='sigmoid'))\n\n\nmodel.compile(loss='binary_crossentropy',\n optimizer=optimizers.RMSprop(),\n metrics=['accuracy'])\n \n \nhistory = model.fit(x=train_imgs_scaled, y=train_labels,\n validation_data=(validation_imgs_scaled, val_labels),\n batch_size=batch_size,\n epochs=epochs,\n verbose=1) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"f, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))\nt = f.suptitle('CNN Model with Regularization', fontsize=12)\nf.subplots_adjust(top=0.85, wspace=0.3)\n\nepoch_list = list(range(1,31))\nax1.plot(epoch_list, history.history['accuracy'], label='Train Accuracy')\nax1.plot(epoch_list, history.history['val_accuracy'], label='Validation Accuracy')\nax1.set_xticks(np.arange(0, 31, 5))\nax1.set_ylabel('Accuracy Value')\nax1.set_xlabel('Epoch')\nax1.set_title('Accuracy')\nl1 = ax1.legend(loc=\"best\")\n\nax2.plot(epoch_list, history.history['loss'], label='Train Loss')\nax2.plot(epoch_list, history.history['val_loss'], label='Validation Loss')\nax2.set_xticks(np.arange(0, 31, 5))\nax2.set_ylabel('Loss Value')\nax2.set_xlabel('Epoch')\nax2.set_title('Loss')\nl2 = ax2.legend(loc=\"best\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Results are little better but not much improvement, model is still overfitting.\n\nThe reason for model overfitting is because we have much less training data and the model keeps seeing the same instances over time across each epoch. \n\n### A way to combat this would be to leverage an image augmentation strategy to augment our existing training data with images that are slight variations of the existing images."},{"metadata":{},"cell_type":"markdown","source":"
\n\n

Model with Image Augmentation\n

\n\nLet’s improve upon our regularized CNN model by adding in more data using a proper image augmentation strategy. \n\nSince our previous model was trained on the same small sample of data points each time, it wasn’t able to generalize well and ended up overfitting after a few epochs. \n\nThe idea behind image augmentation is that we follow a set process of taking in existing images from our training dataset and applying some image transformation operations to them, such as rotation, shearing, translation, zooming, and so on, to produce new, altered versions of existing images. Due to these random transformations, we don’t get the same images each time, and we will leverage Python generators to feed in these new images to our model during training.\n\nThe Keras framework has an excellent utility called ImageDataGenerator that can help us in doing all the preceding operations. Let’s initialize two of the data generators for our training and validation datasets."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_datagen = ImageDataGenerator(rescale=1./255, zoom_range=0.3, rotation_range=50,\n width_shift_range=0.2, height_shift_range=0.2, shear_range=0.2, \n horizontal_flip=True, fill_mode='nearest')\n\nval_datagen = ImageDataGenerator(rescale=1./255)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are a lot of options available in ImageDataGenerator and we have just utilized a few of them. Feel free to check out the documentation to get a more detailed perspective. \n\nIn our training data generator, we take in the raw images and then perform several transformations on them to generate new images. \n\nThese include the following:\n\n`Zooming` the image randomly by a factor of 0.3 using the zoom_range parameter.\n\n`Rotating` the image randomly by 50 degrees using the rotation_range parameter.\n\n`Translating` the image randomly horizontally or vertically by a 0.2 factor of the image’s width or height using the width_shift_range and the height_shift_range parameters.\n\nApplying `shear-based transformations` randomly using the shear_range parameter.\n\nRandomly `flipping` half of the images horizontally using the horizontal_flip parameter.\n\nLeveraging the `fill_mode` parameter to fill in new pixels for images after we apply any of the preceding operations (especially rotation or translation). In this case, we just fill in the new pixels with their nearest surrounding pixel values."},{"metadata":{},"cell_type":"markdown","source":"Let’s see how some of these generated images might look so that you can understand them better. We will take two sample images from our training dataset to illustrate the same."},{"metadata":{"trusted":true},"cell_type":"code","source":"# lets take a random image and see how transformated images actually looks\nimg_id = 1\n\nimg_generator = train_datagen.flow(train_imgs[img_id:img_id+1], train_labels[img_id:img_id+1],\n batch_size=1)\n\nimg = [next(img_generator) for i in range(0,5)]\n\nfig, ax = plt.subplots(1,5, figsize=(16, 6))\nprint('Labels:', [item[1][0] for item in img])\nl = [ax[i].imshow(img[i][0][0]) for i in range(0,5)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You can clearly see in the previous output that we generated a new version of our training image each time (with translations, rotations, and zoom) and also we assign relevant label to it so that the model can extract relevant features from these images and also remember their labels."},{"metadata":{},"cell_type":"markdown","source":"This shows us how image augmentation can help in creating new images, and how training a model on them should help in combating overfitting. \n\nRemember for our validation generator, we just need to send the validation images (original ones) to the model for evaluation; hence, we just scale the image pixels (between 0–1) and do not apply any transformations. \n\nWe just apply image augmentation transformations only on our training images. \n\nLet’s now train a CNN model with regularization using the image augmentation data generators we created. We will use the same model architecture as before."},{"metadata":{},"cell_type":"markdown","source":"
\n\n

Little change in approach\n

\n\nWe will reduce the default learning rate by a factor of 10 here for our optimizer to prevent the model from getting stuck in a local minima or overfit, as we will be sending a lot of images with random transformations. \n\nTo train the model, we need to slightly modify our approach now, since we are using data generators. We will leverage the fit_generator(…) function from Keras to train this model. \n\nThe train_generator generates 30 images each time, so we will use the steps_per_epoch parameter and set it to 32 to train the model on 987 randomly generated images from the training data for each epoch. \n\nOur val_generator generates 20 images each time so we will set the validation_steps parameter to 12 to validate our model accuracy on all the 237 validation images (remember we are not augmenting our validation dataset)."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_generator = train_datagen.flow(train_imgs, train_labels, batch_size=30)\nval_generator = val_datagen.flow(validation_imgs, val_labels, batch_size=20)\n\ninput_shape = input_shape\n\nmodel = Sequential()\n\nmodel.add(Conv2D(16, kernel_size=(3, 3), activation='relu', \n input_shape=input_shape))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\nmodel.add(Conv2D(64, kernel_size=(3, 3), activation='relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\nmodel.add(Conv2D(128, kernel_size=(3, 3), activation='relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\nmodel.add(Conv2D(128, kernel_size=(3, 3), activation='relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\nmodel.add(Flatten())\nmodel.add(Dense(512, activation='relu'))\nmodel.add(Dropout(0.3))\nmodel.add(Dense(512, activation='relu'))\nmodel.add(Dropout(0.3))\nmodel.add(Dense(1, activation='sigmoid'))\n\nmodel.compile(loss='binary_crossentropy',\n optimizer=optimizers.RMSprop(lr=1e-4),\n metrics=['accuracy'])\n \nhistory = model.fit_generator(train_generator, steps_per_epoch=32, epochs=100,\n validation_data=val_generator, validation_steps=12, \n verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"f, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))\nt = f.suptitle('CNN with Regularization & Augmentation', fontsize=12)\nf.subplots_adjust(top=0.85, wspace=0.3)\n\nepoch_list = list(range(1,101))\nax1.plot(epoch_list, history.history['accuracy'], label='Train Accuracy')\nax1.plot(epoch_list, history.history['val_accuracy'], label='Validation Accuracy')\nax1.set_xticks(np.arange(0, 101, 5))\nax1.set_ylabel('Accuracy Value')\nax1.set_xlabel('Epoch')\nax1.set_title('Accuracy')\nl1 = ax1.legend(loc=\"best\")\n\nax2.plot(epoch_list, history.history['loss'], label='Train Loss')\nax2.plot(epoch_list, history.history['val_loss'], label='Validation Loss')\nax2.set_xticks(np.arange(0, 101, 5))\nax2.set_ylabel('Loss Value')\nax2.set_xlabel('Epoch')\nax2.set_title('Loss')\nl2 = ax2.legend(loc=\"best\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"If you compare above two graphs with the earlier once, you would clearly notice an improvement in the performance of the model.\n\nTraining and validation accuracies are much similar now, which proves that the model is not overfitting anymore.\n\nEven though there are some spikes in both the validation accuracy and loss, we can still say that this particular model with image augmentation is doing much better than the earlier models where we did not perform the augmentation. "},{"metadata":{},"cell_type":"markdown","source":"
\n\n

How to improve the Model's performance further?\n

\n \n### More Images to Train the model\nFirst and the foremost, we need more images to train our model, we trained our model with less than 1000 images, which is obviously not enough for any image classifier. We need atleast 10-20 thousands images to train the model. \n\n### Fine Tune Hyper Parameters \nWe should look to fine tune `Filter Size`, `No. Of Filters`, `Stride` & `Padding` parameters in order to get better results from the model\n \n### Visualise the Filters \nWhen we talk about “filters”, we basically refer to it's learned weights of the convolutions. \n\nFor example, a single 3x3 convolution is called a “filter” and that filter has a total of 10 weights (9 + 1 bias).\n \nBy visualising the learned weights we can get some idea as to how well our network has learned. For example, if we see a lot of zeros then we’ll know we have many dead filters that aren’t going much for our network — a great opportunity to do some pruning for model compression.\n \nLet's take a look at the code for visualising the filters in a while."},{"metadata":{"trusted":true},"cell_type":"code","source":"# lets check the layers we added to our model\n\nfor l in model.layers:\n print(l.name,l)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Visualise Filters\n\n# lets save layer names and layers to a dictionary \nlayer_dict = dict([(layer.name, layer) for layer in model.layers])\n\n# lets pick one layer and visualise it\nlayer_name = 'conv2d_7'\nfilter_index = 0 # index of the filter we will visualize\n\n# Grab the filter and bias weights for the selected layer\nfilters, biases = layer_dict[layer_name].get_weights()\n\n# Normalize filter values to a range of 0 to 1 so we can visualize them\n# This will help create a clear visualisation when we show the weights as colours on the screen.\nf_min, f_max = np.amin(filters), np.amax(filters)\nfilters = (filters - f_min) / (f_max - f_min)\n\n# Plot first few filters\nn_filters, index = 6, 1\nfor i in range(n_filters):\n f = filters[:, :, :, i]\n \n # Plot each channel separately\n for j in range(3):\n\n ax = plt.subplot(n_filters, 3, index)\n ax.set_xticks([])\n ax.set_yticks([])\n \n plt.imshow(f[:, :, j], cmap='viridis') \n index += 1\n \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For the “viridis” colour map we selected, yellow represents a value of 1 and dark blue a value of 0.\n\nAs discussed above, if we see a lot of zeros then we’ll know we have many dead filters that aren’t going much for our network — a great opportunity to do some pruning for model compression."},{"metadata":{},"cell_type":"markdown","source":"#### Closing Remarks:-\nYou might need to do a lots of research and expriments before coming up with a great model for Image Classification kind of problems. And sometimes it is not feasible to carry out experiments with the whole dataset, as it might contain thousands of images,\nTraing a model on such a huge dataset would be time consuming.\n\nYou can follow a simple approach to reduce training time, you may take a subset of the training data (as I did in this example), train your model on this subset of data and see how it works with different sets of parameters. And then once you are satisfied with your results, you can scale you model to the whole data set.\n\nOf course this approach will have its pros and cons but this is worth giving a try.\n\nThanks for Reading my Kernel, hope you find it helpful!\n\n# If you like this kernel, kindly upvote :-)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}