{"cells":[{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nfrom mpl_toolkits.axes_grid1 import ImageGrid\nimport random\nimport imageio\nimport cv2\nimport scipy.ndimage as ndi\nfrom keras.models import Sequential\n#Import from keras_preprocessing not from keras.preprocessing\nfrom keras_preprocessing.image import ImageDataGenerator\nfrom keras.layers import Dense, Activation, Flatten, Dropout, BatchNormalization\n# Conv2D are objects to reprsent convolutions \nfrom keras.layers import Conv2D, MaxPooling2D\nfrom keras import regularizers, optimizers\nfrom keras.callbacks import ModelCheckpoint ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Neural Networks\n**Neural Networks** are **parametric**. The model has parameters that we need to optimize to find the optimal value for that parameter. Neural networks are **universal function approximators**. When neurons combine together, they can approximate any function. If a network has more than two hidden layers, it is deep. \n## Design Parameters\n+ Architecture\n+ Number of layers\n+ Number of neurons in a layer\n+ Activation functions: typically use the same type of activation function for all of the neurons in one layer. The output layer activation function is determined by the task we want to do. \n\n\nDeeper layers and a larger number of neurons in a layer mean more model capacity, it can handle bigger, more complex data. If the capacity is much more than the data, it will overfit. Dense layers have a weight for every connection. \n+ Weights = parameters (the strength of the connection between neurons, they need to be determined as result of optimization) \n+ Hyperparameters = design parameters (ex: number of neurons in a layer, number of layers in a network, type of activation function to use in each layer. "},{"metadata":{},"cell_type":"markdown","source":"## Artificial Neural Network \nWe are using the Keras library to create neural networks and train these neural networks to classify images of tumor tissue. The model is the Sequential type. Outputs of one layer are provided as inputs only to the next layer."},{"metadata":{},"cell_type":"markdown","source":"The neural network models we are using to classify the images as having a tumor or not having a tumor, are **Sequential** type. The outputs of one layer are provided as inputs only to the next layer. \n<br>\nCreate a neural network with **Dense** layers meaning that each unit in each layer is connected to all of the units in the previous layer. For example, each unit in the first layer is connected to the pixels in the input images. The Dense layer object receivs the following arguments: numer of units in that layer, activation function for the units, and an input_shape keyword (for the first layer in the network). <br>\nThere are 10 units here to start. More units would increase the complexity of the network and its capacity to represent complex inputs. To facilitate learning, we are using a **rectified linear unit (a relu) as the activation**. The input shape argument keyword argument tells us how many inputs each of these units should expect. "},{"metadata":{"trusted":true},"cell_type":"code","source":"def append_ext(fn):\n    return fn+\".tif\"\ntraindf=pd.read_csv(\"/kaggle/input/histopathologic-cancer-detection/train_labels.csv\",dtype=str)\n#traindf=pd.read_csv(“./trainLabels.csv”,dtype=str)\ntestdf=pd.read_csv(\"/kaggle/input/histopathologic-cancer-detection/sample_submission.csv\",dtype=str)\ntraindf[\"id\"]=traindf[\"id\"].apply(append_ext)\ntestdf[\"id\"]=testdf[\"id\"].apply(append_ext)\ndatagen=ImageDataGenerator(rescale=1./255.,validation_split=0.25)\nprint(len(traindf))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# used this tutorial: https://medium.com/@vijayabhaskar96/tutorial-on-keras-flow-from-dataframe-1fd4493d237c\ntrain_generator=datagen.flow_from_dataframe(\ndataframe=traindf,\ndirectory=\"/kaggle/input/histopathologic-cancer-detection/train/\",\n#directory=\"./train/\",\nx_col=\"id\",\ny_col=\"label\",\nsubset=\"training\",\nbatch_size=32,\nseed=42,\nshuffle=True,\nclass_mode=\"categorical\",\ntarget_size=(96,96))\nvalid_generator=datagen.flow_from_dataframe(\ndataframe=traindf,\ndirectory=\"/kaggle/input/histopathologic-cancer-detection/train/\",\n#directory=\"./train/\",\nx_col=\"id\",\ny_col=\"label\",\nsubset=\"validation\",\nbatch_size=14,\nseed=42,\nshuffle=True,\nclass_mode=\"categorical\",\ntarget_size=(96,96))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"test_datagen=ImageDataGenerator(rescale=1./255.)\ntest_generator=test_datagen.flow_from_dataframe(\ndataframe=testdf,\ndirectory=\"/kaggle/input/histopathologic-cancer-detection/test/\",    \n#directory=\"./test/\",\nx_col=\"id\",\ny_col=\"label\",\n#batch_size=32,\nbatch_size = 2,\nseed=42,\nshuffle=False,\nclass_mode=\"categorical\",\ntarget_size=(96,96))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# initialize a sequential model\nmodel = Sequential()\nmodel.add(Flatten())\nmodel.add(Dense(10, activation='relu',input_shape=(96,96,3)))\n# add another hidden layer, also with relu activation\n# also with 10 units \nmodel.add(Dense(10, activation='relu'))\n# the output is a fully connected layer with a unit for each class of inputs\n# the output unit uses sigmoid function to decide which of the 2 classes was presented\n# initially had softmax but softmax is a better option for multi-class classification\nmodel.add(Dense(2, activation='softmax'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Once the model is constructed, it needs to be compiled before it can be fit to data. Specify the optimizer that will be used to fit the model and the loss function that will be used in optimization. Optionally, you can also specify a list of metrics that the model will keep track of (see the list with 'accuracy' in the code below). The **loss function** is a measure of error, how accurate the prediction was. The loss function is used for traiing optimization. The function should be differentiable so MSE or cross-entropy are good options for classification tasks. (Because Accuracy, TPR, and F1 are not differentiable, we typically use other things.) "},{"metadata":{"trusted":true},"cell_type":"code","source":"# next compile the model\n# categorical_crossentropy loss function is appropriate for classification\n# list of reported merics including binary accuracy and AUC \nmodel.compile(optimizer='adam', \n              loss='categorical_crossentropy',\n              metrics=['accuracy', 'binary_accuracy', 'AUC'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"\nfit the model to training data\nnetwork adjusts its weights through backpropagation\nand gradient descent\n3 epochs means it will go over all of the training data\n3 times \n\"\"\"\nbatch_size = 14\nSTEP_SIZE_TRAIN=train_generator.n//train_generator.batch_size\nSTEP_SIZE_VALID=valid_generator.n//valid_generator.batch_size\nSTEP_SIZE_TEST=test_generator.n//test_generator.batch_size\nmodel.fit_generator(generator=train_generator,\n                    steps_per_epoch=STEP_SIZE_TRAIN,\n                    validation_data=valid_generator,\n                    validation_steps=STEP_SIZE_VALID,\n                    epochs=1\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.evaluate_generator(generator=valid_generator,\nsteps=STEP_SIZE_TEST)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.metrics_names","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(valid_generator.n)\nprint(len(valid_generator.classes))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"STEP_SIZE_TEST=valid_generator.n//valid_generator.batch_size\n\nvalid_generator.reset()\npred=model.predict_generator(valid_generator,\nsteps=STEP_SIZE_TEST,\nverbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(len(pred))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# adapted from: https://gist.github.com/RyanAkilos/3808c17f79e77c4117de35aa68447045\n#Confusion Matrix and Classification Report\nfrom sklearn.metrics import classification_report, confusion_matrix\n#Y_pred = model.predict_generator(validation_generator, num_of_test_samples // batch_size+1)\n#y_pred = np.argmax(Y_pred, axis=1)\ny_pred = np.argmax(pred, axis=1)\nprint('Confusion Matrix')\nprint(confusion_matrix(valid_generator.classes, y_pred))\n#print(confusion_matrix(validation_generator.classes, y_pred))\nprint('Classification Report')\n#target_names = ['Cats', 'Dogs', 'Horse']\ntarget_names = [\"NoTumor\", \"Tumor\"]\n#print(classification_report(validation_generator.classes, y_pred, target_names=target_names))\nprint(classification_report(valid_generator.classes, y_pred, target_names=target_names))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import roc_curve\n#y_pred_keras = keras_model.predict(X_test).ravel()\ny_pred = np.argmax(pred, axis=1)\nfpr, tpr, thresholds = roc_curve(valid_generator.classes, y_pred)\n#fpr_keras, tpr_keras, thresholds_keras = roc_curve(y_test, y_pred_keras)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import auc\nauc = auc(fpr, tpr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(fpr,tpr,label=\"data 1, auc=\"+str(auc))\nplt.legend(loc=4)\nplt.title('ROC Curve', fontsize=18)\nplt.xlabel('False Positive Rate', fontsize=16)\nplt.ylabel('True Positive Rate', fontsize=16)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Evaluation\nThe Kaggle competition notes, \"Submissions are evaluated on area under the ROC curve between the predicted probability and the observed target.\"\n+ tf.keras.metrics.AUC computes the approximate AUC (Area under the curve) for ROC curve via the Riemann sum. "},{"metadata":{},"cell_type":"markdown","source":"It would be ineffective for what we want to do, but classifying all images as not having tumor tissue would have an accuracy of about 60%. For the classifier to have any sort of functionality, it is important to pay attention to other metrics, particularly **AUC** which is how the Kaggle competition scores submissions. <br>\n+ **AUC** area under ROC curve: ROC is a probability curve. AUC represents degree of measure or separability. It tells how well the model does at distinguishing classes. \n+ **Precision** refers to how exact the predictions are. \n+ **Recall** refers to completeness. \n+ **F1 Score** combines precision and recall into a single higher is better metric. In practice, different types of misclassifications incur different costs. Ultimately, when diagnosing cancer, we would not expect precision and recall to have the same relative importance.  \n"},{"metadata":{},"cell_type":"markdown","source":"Resources consulted about Evaluation:\n+ https://neptune.ai/blog/keras-metrics\n+ https://keras.io/api/metrics/accuracy_metrics/\n+ https://towardsdatascience.com/understanding-auc-roc-curve-68b2303cc9c5\n\n### Evaluation Metrics:\n+ binary_accuracy computes the mean accuracy rate across all predictions for binary classification problems."},{"metadata":{},"cell_type":"markdown","source":"Ultimately we want the model to be able to learn how to generalize. "},{"metadata":{},"cell_type":"markdown","source":"## Convolutional Neural Network (CNN)\nIn the first neural network constructed, each unit in the first layer had a unit connecting it separately to a pixel in the image. We know that pixels in most images are not independent from their neighbors. Natural images contain spatial correlations. How can we use these correlations to our advantage? Our own visual system uses these correlations, identifying edges at a particular location in the visual view. The convolution is the fundamental operation that Convolutional Neural Networks use to process images.\nThe kernel slides over the input image. In each location, the window is multiplied by the values in the kernel and added up to create the resulting values for one pixel in the resulting array. The resulting array is called a feature map. The feature map contains a map of the featues in the image represented by the kernel.\nThe convolution of an image with a kernel summarizes a part of the image as the sum of the multiplication of that part of the image with the kernel."},{"metadata":{},"cell_type":"markdown","source":"One of the articles I read gives this helpful definition:\nA convolution is the simple application of a filter to an input that results in an activation. Repeated application of the same filter to an input results in a map of activations called a feature map, indicating the locations and strength of a detected feature in an input, such as an image.<br>\nIt also provides this descripton of the role of a convolution in a **CNN**. \nIn the context of a convolutional neural network, a convolution is a linear operation that involves the multiplication of a set of weights with the input, much like a traditional neural network. Given that the technique was designed for two-dimensional input, the multiplication is performed between an array of input data and a two-dimensional array of weights, called a filter or a kernel.\nhttps://machinelearningmastery.com/convolutional-layers-for-deep-learning-neural-networks/"},{"metadata":{},"cell_type":"markdown","source":"Keras has objects to represent convolutions. Instead of having every unit connected to every unit in the previous layer (like in the Dense layer), it is connected to the previous layer through a convolution. The output of a Convolutional Layer is the convolution of a kernel over the image input. During training of a network that has convolutions, the kernels in each unit would be adjusted using backpropagation.A convolutional layer has fewer weights than a Dense layer. A Dense layer has one weight for each pixel in the image.A Convolution layer has only one weight for each pixel in the kernel. \n<br>\nIf the kernel size is 3, the kernel of each unit has 9 pixels. If the layer has 10 units, it would have 90 parameters. \n<br> Flatten is a connection between convolution and densely connected layers. This takes the output of the convolutional layer that we previously referred to as a feature map and flatens it into a one-dimensional array.This is the expected input into the densely connected layer that is the output layer. <br>\nThe dense layer has two units because there are two classes. We are using the sigmoid activation function. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# initialize the model object\nmodel = Sequential() \n# add a convolutional layer \nmodel.add(Conv2D(10, kernel_size=3, activation='relu',\n                 input_shape=(96,96,3)))\n# flatten the output of the convolutional layer\n# this layer translates between the image processing\n# and classification parts of the network \nmodel.add(Flatten())\n# add an output layer for the 2 categories\nmodel.add(Dense(2,activation='sigmoid'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# next compile the model\n# categorical_crossentropy loss function is appropriate for classification\n# accuracy will be a reported metric \nmodel.compile(optimizer='adam', \n              loss='categorical_crossentropy',\n              metrics=['accuracy', 'binary_accuracy', 'AUC'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 32\nSTEP_SIZE_TRAIN=train_generator.n//train_generator.batch_size\nSTEP_SIZE_VALID=valid_generator.n//valid_generator.batch_size\nSTEP_SIZE_TEST=test_generator.n//test_generator.batch_size\nmodel.fit_generator(generator=train_generator,\n                    steps_per_epoch=STEP_SIZE_TRAIN,\n                    validation_data=valid_generator,\n                    validation_steps=STEP_SIZE_VALID,\n                    epochs=3\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# evaluate on test set held aside during training \nmodel.evaluate_generator(generator=valid_generator,\nsteps=STEP_SIZE_TEST)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.metrics_names","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"STEP_SIZE_TEST=valid_generator.n//valid_generator.batch_size\n\nvalid_generator.reset()\npred=model.predict_generator(valid_generator,\nsteps=STEP_SIZE_TEST,\nverbose=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"With the previous neural network, we needed to reshape the images before feeding them to the network. Here, we want the pixels to retain their spatial relationships so we don't do that. That is why we specified the input shape when we defined the convolutional layer."},{"metadata":{},"cell_type":"markdown","source":"## Why CNN? What do convolution filters do?\n+ Learning filters\n+ Weight sharing: weights are shared across the image \n+ Computational efficiency\n+ Translational invariance\n+ Robust: less overfit: since there are less weights, they are less likely to overfit  "},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}