{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# **Introduction**\n* This kernel is my project for the course - Machine Learning => CSE4020\n* In this kernel I will first go through the dataset, perform the exploratory data analysis and then run 3 approaches to convolutional neural networks - a simple model created by me, the NASNET mobile model which provides a learning architecture which scales from smaller datasets to larger datasets and the DenseNet 121 model provided by Fast.ai"},{"metadata":{"_uuid":"8b688a42a02b9ac8b7c2276e6c92876cbf7f9ba5"},"cell_type":"markdown","source":"# **Contents**\n* Importing modules and an explanation of their use\n* Explanation of the competition and the description of the dataset\n* Loading the data and the exploratory data analysis\n* Creating, training and validating the model\n* Creating, training and validating the NASNET Mobile model\n* Creating, training and validating the DenseNet 121 model\n* Choosing the model with the best results and creating a submission"},{"metadata":{"_uuid":"62b4abc905af29a13ba75f87cf71eb205133286e"},"cell_type":"markdown","source":"# **Importing the modules and an explanation of their use**\nWe will use multiple libraries and modules in the kernel. They are all powerful modules which will be used for a variety of jobs such as reading files, storing the data efficiently, a deep learning API, image processing and plotting functionalities! These modules are:\n* Numpy - The math module that makes matrix multiplication and other complicated mathematical operations easier\n* Pandas - The module that provides us with efficient and powerful data structures which can help us store and use our data efficiently.\n* Matplotlib - A powerful module that will help us plot and visualize our data.\n* Opencv - Imported as cv2, it is used for image processing and in this case, we will use it to for loading images.\n* Keras - A high level API used by most of the world for Deep Learning. It is very useful as it provides us with a high degree of functionality and makes using Tensorflow easier.\n* Glob - A module that helps us load and match filenames easily.\n* TQDM - A useful library which provides with a progress bar while training.\n* Torchvision - A module that holds the popular datasets and models while using PyTorch.\n* Sklearn - A very powerful module that provides with tools for Machine Learning and Data Analysis in Python.\n* Imgaug - A machine learning oriented package that helps us create a larger set of images which are altered from a smaller set of images\n"},{"metadata":{"trusted":true,"_uuid":"e9cb8017824e5ee75e4566218ae5a8c73d16cdaf"},"cell_type":"code","source":"# Modules required for simple model\nfrom glob import glob \nimport numpy as np\nimport pandas as pd\nimport cv2,os\nimport keras\nfrom keras.utils import np_utils\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.models import Sequential\nfrom keras.layers import Dense\nfrom keras.layers import Dropout\nfrom keras.layers import Flatten\nfrom keras.layers import BatchNormalization\nfrom keras.layers import Activation\nfrom keras.layers import Conv2D\nfrom keras.layers import MaxPool2D\nfrom keras.layers import BatchNormalization\nfrom tqdm import tqdm_notebook,trange\nimport matplotlib.pyplot as plt\nimport gc\nscale = 70\nseed = 7\n# Modules required for NASNET Mobile Model\nfrom random import shuffle\nfrom sklearn.model_selection import train_test_split\nfrom imgaug import augmenters as iaa\nimport imgaug as ia\n# Modules required for DenseNet 121 Model\n#from fastai.vision import *\n#import torchvision","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"15730a381b317da875129d8b889688fbbb025aa0"},"cell_type":"markdown","source":"# **Explaning the competition and Dataset Description**\n* The competition is basically a binary classification problem for images - which means in this kernel, we will be dividing the images into 2 separate classes.\n* The dataset consists of microscopic images of lymph node tissue with a resolution of 96 pixels by 96 pixels and we have to predict or classify the images based on whether the images show metastatic cancer tissue in the 32 pixel by 32 pixel centre region of the image.\n* The dataset has 220000 images for training our models and 57000 images in the testing set."},{"metadata":{"_uuid":"532b70c8842ad49c9759a154bd97eaa40e57f248"},"cell_type":"markdown","source":"# **Loading the data and exploratory data analysis**\n* We will first create a DataFrame which will store the path of the files in the training folder and then read the labels of the images from the given csv file.\n* We will then load the images using the uint8 format - this will reduce the size of the images and thereby letting the data fit in the 14GB of memory provided to us.\n* We will then perform the Exploratory Data Analysis where will \"SEE\" our data for the first time. We will also check for dataset imbalance and look for image features like RGB channels, HSV channels, and other features."},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"4a5265604bbe2bbaf50c155fd3e7c2c008b1276a"},"cell_type":"code","source":"#set paths to training and test data\npath = \"../input/\"\ntrain_path = path + 'train/'\ntest_path = path + 'test/'\ndf = pd.DataFrame({'path': glob(os.path.join(train_path,'*.tif'))})\ndf['id'] = df.path.map(lambda x: x.split('/')[3].split(\".\")[0])\nlabels = pd.read_csv(path+\"train_labels.csv\")\n# merge labels and filepaths\ndf = df.merge(labels, on = \"id\") \n# Loading the images\ndef load_data(N,df):\n    # allocate a numpy array for the images - 3 channels\n    X = np.zeros([N,96,96,3],dtype=np.uint8) \n    #convert the labels to an array\n    y = np.squeeze(df.as_matrix(columns=['label']))[0:N]\n    #read images one by one, tdqm notebook displays a progress bar\n    for i, row in tqdm_notebook(df.iterrows(), total=N):\n        if i == N:\n            break\n        X[i] = cv2.imread(row['path'])\n    return X,y\nN = 10000\nX,y = load_data(N=N,df=df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2426d8a8d21487e545fe63449d87a5ad3a4f0162"},"cell_type":"code","source":"# Displaying the loaded images\nfig = plt.figure(figsize=(10, 4), dpi=150)\nnp.random.seed(100)\nfor plotNr,idx in enumerate(np.random.randint(0,N,8)):\n    ax = fig.add_subplot(2, 8//2, plotNr+1, xticks=[], yticks=[])\n    plt.imshow(X[idx])\n    ax.set_title('Label: ' + str(y[idx]))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"372a700499ee4deb68a445ac8e34d293c8367148"},"cell_type":"markdown","source":"So we can see some sample images from the dataset - we can also see how it is impossible for computer scientists like us to discern the cancer containing images!"},{"metadata":{"_uuid":"16c0e02059651737c06d895df338c65bc1fc0dac"},"cell_type":"markdown","source":"## Data Distribution\nLet us see how the class distribution is in the dataset"},{"metadata":{"trusted":true,"_uuid":"fed6d12ebbe40bc85ae97eb6462ce46aaf6d0bf5"},"cell_type":"code","source":"fig = plt.figure(figsize=(4, 2),dpi=150)\nplt.bar([1,0], [(y==0).sum(), (y==1).sum()]); #plot a bar chart of the label frequency\nplt.xticks([1,0],[\"Negative (N={})\".format((y==0).sum()),\"Positive (N={})\".format((y==1).sum())]);\nplt.ylabel(\"# of samples\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"015f4dac633493f43be15a7cb8079b358f05db0c"},"cell_type":"markdown","source":"From the above graph, we can see that there is a 40/60 split amongst the positive and negative classes. This shows that even a baseline classification - where all the images are given the more populated class will give an accuracy of 60%!\nWe may have to undersample the negative class or generate more positive sample so that we can avoid a bias and improve classification stability."},{"metadata":{"_uuid":"35fa7f7acc18801754915514fd1400003f239862"},"cell_type":"markdown","source":"## Let us look at each class individually!\nWe will now compare the pixel distribution in each of the BGR channels in both positive and negative samples to see if we can see any features that can be engineered!"},{"metadata":{"trusted":true,"_uuid":"85c16da9a8c92d74b77229d1b0bd3ce15e570471"},"cell_type":"code","source":"#Separating the classes\npositive_samples = X[y == 1]\nnegative_samples = X[y == 0]\n#Binning each pixel value for the histogram\nnr_of_bins = 256 \nfig,axs = plt.subplots(4,2,sharey=True,figsize=(8,8),dpi=150)\n#RGB channels\naxs[0,0].hist(positive_samples[:,:,:,0].flatten(),bins=nr_of_bins,density=True)\naxs[0,1].hist(negative_samples[:,:,:,0].flatten(),bins=nr_of_bins,density=True)\naxs[1,0].hist(positive_samples[:,:,:,1].flatten(),bins=nr_of_bins,density=True)\naxs[1,1].hist(negative_samples[:,:,:,1].flatten(),bins=nr_of_bins,density=True)\naxs[2,0].hist(positive_samples[:,:,:,2].flatten(),bins=nr_of_bins,density=True)\naxs[2,1].hist(negative_samples[:,:,:,2].flatten(),bins=nr_of_bins,density=True)\n#All channels\naxs[3,0].hist(positive_samples.flatten(),bins=nr_of_bins,density=True)\naxs[3,1].hist(negative_samples.flatten(),bins=nr_of_bins,density=True)\n# Labelling the Plots\naxs[0,0].set_title(\"Positive samples (N =\" + str(positive_samples.shape[0]) + \")\");\naxs[0,1].set_title(\"Negative samples (N =\" + str(negative_samples.shape[0]) + \")\");\naxs[0,1].set_ylabel(\"Red\",rotation='horizontal',labelpad=35,fontsize=12)\naxs[1,1].set_ylabel(\"Green\",rotation='horizontal',labelpad=35,fontsize=12)\naxs[2,1].set_ylabel(\"Blue\",rotation='horizontal',labelpad=35,fontsize=12)\naxs[3,1].set_ylabel(\"RGB\",rotation='horizontal',labelpad=35,fontsize=12)\nfor i in range(4):\n    axs[i,0].set_ylabel(\"Relative frequency\")\naxs[3,0].set_xlabel(\"Pixel value\")\naxs[3,1].set_xlabel(\"Pixel value\")\nfig.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8652a6c93aa78a296d7ff89014f386a36787ef77"},"cell_type":"markdown","source":"### Conclusions:\nWe can see some interesting conclusions from the following graphs!\n* Negative samples have higher peaks on the right hand side of the graph - showing brightness is higher in negative samples\n* Positive samples have darker pixels in the green channel.\n"},{"metadata":{"_uuid":"1401140c8e906e664f268396b19a6a8cbe0c84ed"},"cell_type":"markdown","source":"### Mean Brightness Distribution\nThis is the process of taking each images, averaging its pixel values to get a value. This value is plotted for all the classes. This can help us generate another feature!"},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"16397f69f9772465a735c7aefacba4c9bd1c8f7c"},"cell_type":"code","source":"# We use 64 bins to get smooth graphs\nnr_of_bins = 64 \nfig,axs = plt.subplots(1,2,sharey=True, sharex = True, figsize=(8,2),dpi=150)\naxs[0].hist(np.mean(positive_samples,axis=(1,2,3)),bins=nr_of_bins,density=True);\naxs[1].hist(np.mean(negative_samples,axis=(1,2,3)),bins=nr_of_bins,density=True);\naxs[0].set_title(\"Mean brightness, positive samples\");\naxs[1].set_title(\"Mean brightness, negative samples\");\naxs[0].set_xlabel(\"Image mean brightness\")\naxs[1].set_xlabel(\"Image mean brightness\")\naxs[0].set_ylabel(\"Relative frequency\")\naxs[1].set_ylabel(\"Relative frequency\");","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e0e641f69cca203978f18118e93ade742cfbc1a5"},"cell_type":"markdown","source":"### Conclusions:\n* The mean brightness distribution shows that the positive samples have a normal distribution with images having brightness in the range of 100-240 while the negative samples have a biomodal distribution with the pixel values ranging from 50-255.\n"},{"metadata":{"_uuid":"737554802e1980188504dbdf3c3e0af693e0051e"},"cell_type":"markdown","source":"## Creating the model\nLet us create a simple model for this problem - with 3 convolutional blocks. The basic architecture consists of :\n* A convolutional layer followed by \n* A batch normalization layer followed by\n* An activation layer\n* We repeat the above block and follow it up with a maxpooling layer and a dropout layer.\n\nThis is a very standard and basic architecture - which if we repeat three times - can give us above average results!\n\nWe will also divide the data into 80% for training and 20% for validation. We will set up a garbage collector to speed up the process by freeing up the RAM."},{"metadata":{"trusted":true,"_uuid":"ca25d27264054fd20f402fce6d88277eb0b383c2"},"cell_type":"code","source":"# Getting the number of images in the dataset\nN = df[\"path\"].size\nX,y = load_data(N=N,df=df)\n\n# Collecting garbage\npositives_samples = None\nnegative_samples = None\ngc.collect();\n\n# Setting up the training/testing ratio\ntraining_portion = 0.8 \nsplit_idx = int(np.round(training_portion * y.shape[0]))\n\n#Setting seeds to ensure we can repeat this process \nnp.random.seed(42) \nidx = np.arange(y.shape[0])\nnp.random.shuffle(idx)\nX = X[idx]\ny = y[idx]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7101d039b31dd097ac3f1811f678571f17aa1568"},"cell_type":"code","source":"# Network Parameters\nkernel_size = (3,3)\npool_size= (2,2)\nfirst_filters = 32\nsecond_filters = 64\nthird_filters = 128\n\n# Setting up dropout parameters for regularization\ndropout_conv = 0.3\ndropout_dense = 0.5\n\n# Creating model\nmodel = Sequential()\n\n# Convolutional Block 1\nmodel.add(Conv2D(first_filters, kernel_size, input_shape = (96, 96, 3)))\nmodel.add(BatchNormalization())\nmodel.add(Activation(\"relu\"))\nmodel.add(Conv2D(first_filters, kernel_size, use_bias=False))\nmodel.add(BatchNormalization())\nmodel.add(Activation(\"relu\"))\nmodel.add(MaxPool2D(pool_size = pool_size)) \nmodel.add(Dropout(dropout_conv))\n\n# Convolutional Block 2\nmodel.add(Conv2D(second_filters, kernel_size, use_bias=False))\nmodel.add(BatchNormalization())\nmodel.add(Activation(\"relu\"))\nmodel.add(Conv2D(second_filters, kernel_size, use_bias=False))\nmodel.add(BatchNormalization())\nmodel.add(Activation(\"relu\"))\nmodel.add(MaxPool2D(pool_size = pool_size))\nmodel.add(Dropout(dropout_conv))\n\n# Convolutional Block 3\nmodel.add(Conv2D(third_filters, kernel_size, use_bias=False))\nmodel.add(BatchNormalization())\nmodel.add(Activation(\"relu\"))\nmodel.add(Conv2D(third_filters, kernel_size, use_bias=False))\nmodel.add(BatchNormalization())\nmodel.add(Activation(\"relu\"))\nmodel.add(MaxPool2D(pool_size = pool_size))\nmodel.add(Dropout(dropout_conv))\n\n# Dense Layer \nmodel.add(Flatten())\nmodel.add(Dense(256, activation='relu'))\nmodel.add(BatchNormalization())\nmodel.add(Dropout(dropout_dense))\n\n# Softmax - to convert values to 0 or 1\nmodel.add(Dense(1, activation = \"sigmoid\"))\n\nmodel.summary()\n\nbatch_size = 50\n\nmodel.compile(loss=keras.losses.binary_crossentropy,\n              optimizer=keras.optimizers.Adam(0.001), \n              metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b9bc88c1c96e72c317e39c060e5bd630d53b029e"},"cell_type":"markdown","source":"### Training and Validation of the model:\nWe will now train the model for three epochs (should take ~20mins). That means the model will have performed a forward and backward pass for each image in the training exactly three times.\nTo do so, we will split the training data in batches and feed one batch after another into the network. The batch size is a critical parameter for training a neural network.\nKeras can do the splitting automatically for you, but, I thought, this way it is more transparent what is happening."},{"metadata":{"trusted":true,"_uuid":"a1d6f5996056de0f2e94512735dfc9cdc6e440ad"},"cell_type":"code","source":"epochs = 3 #how many epochs we want to perform\nfor epoch in range(epochs):\n    #compute how many batches we'll need\n    iterations = np.floor(split_idx / batch_size).astype(int) #the floor makes us discard a few samples here, I got lazy...\n    loss,acc = 0,0 #we will compute running loss and accuracy\n    with trange(iterations) as t: #display a progress bar\n        for i in t:\n            start_idx = i * batch_size #starting index of the current batch\n            x_batch = X[start_idx:start_idx+batch_size] #the current batch\n            y_batch = y[start_idx:start_idx+batch_size] #the labels for the current batch\n\n            metrics = model.train_on_batch(x_batch, y_batch) #train the model on a batch\n\n            loss = loss + metrics[0] #compute running loss\n            acc = acc + metrics[1] #compute running accuracy\n            t.set_description('Running training epoch ' + str(epoch)) #set progressbar title\n            t.set_postfix(loss=\"%.2f\" % round(loss / (i+1),2),acc=\"%.2f\" % round(acc / (i+1),2)) #display metrics","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c3a7d2a2caad09fc9ccb73c73ee179f99273f44e"},"cell_type":"markdown","source":"Now, to verify that our model also works with data it hasn't seen yet, we will perform a validation epoch, i.e., check the accuracy on the validation set without further training the network."},{"metadata":{"trusted":true,"_uuid":"f4c9c3a27e473351a9427dc92e7fa0d1646ac8e4"},"cell_type":"code","source":"#compute how many batches we'll need\niterations = np.floor((y.shape[0]-split_idx) / batch_size).astype(int) #as above, not perfect\nloss,acc = 0,0 #we will compute running loss and accuracy\nwith trange(iterations) as t: #display a progress bar\n    for i in t:\n        start_idx = i * batch_size #starting index of the current batch\n        x_batch = X[start_idx:start_idx+batch_size] #the current batch\n        y_batch = y[start_idx:start_idx+batch_size] #the labels for the current batch\n        \n        metrics = model.test_on_batch(x_batch, y_batch) #compute metric results for this batch using the model\n        \n        loss = loss + metrics[0] #compute running loss\n        acc = acc + metrics[1] #compute running accuracy\n        t.set_description('Running training') #set progressbar title\n        t.set_description('Running validation')\n        t.set_postfix(loss=\"%.2f\" % round(loss / (i+1),2),acc=\"%.2f\" % round(acc / (i+1),2))\n        \nprint(\"Validation loss:\",loss / iterations)\nprint(\"Validation accuracy:\",acc / iterations)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4e0af93107bc3e1826a53a02e75573ccfb1041b3"},"cell_type":"markdown","source":"## Create a submission\nWell, now that we have a trained a model, we can create a submission by predicting the labels of the test data and see, where we are at in the leaderboards!"},{"metadata":{"trusted":true,"_uuid":"11e47f8f8f0b26d9466e53fbb383ff62267f109e"},"cell_type":"code","source":"X = None\ny = None\ngc.collect();\nbase_test_dir = path + 'test/' #specify test data folder\ntest_files = glob(os.path.join(base_test_dir,'*.tif')) #find the test file names\nsubmission = pd.DataFrame() #create a dataframe to hold results\nfile_batch = 5000 #we will predict 5000 images at a time\nmax_idx = len(test_files) #last index to use\nfor idx in range(0, max_idx, file_batch): #iterate over test image batches\n    print(\"Indexes: %i - %i\"%(idx, idx+file_batch))\n    test_df = pd.DataFrame({'path': test_files[idx:idx+file_batch]}) #add the filenames to the dataframe\n    test_df['id'] = test_df.path.map(lambda x: x.split('/')[3].split(\".\")[0]) #add the ids to the dataframe\n    test_df['image'] = test_df['path'].map(cv2.imread) #read the batch\n    K_test = np.stack(test_df[\"image\"].values) #convert to numpy array\n    predictions = model.predict(K_test,verbose = 1) #predict the labels for the test data\n    test_df['label'] = predictions #store them in the dataframe\n    submission = pd.concat([submission, test_df[[\"id\", \"label\"]]])\nprint(submission.head())\nsubmission.to_csv(\"submission.csv\", index = False, header = True) #create the submission file","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}