{"cells":[{"metadata":{},"cell_type":"markdown","source":"# **INTRODUCTION**"},{"metadata":{},"cell_type":"markdown","source":"This notebook was created as part of the Kaggle Histopatholic Cancer Detection competition, which challenged participants to identify metastatic tissue in histopathologic scans of lymph node sections. To view the notebook please follow: https://www.kaggle.com/monicab13/ai-project2"},{"metadata":{},"cell_type":"markdown","source":"# **PROBLEM STATEMENT**"},{"metadata":{},"cell_type":"markdown","source":"The goal of this project is to create a model which can analyze scans of lymph nodes and make predictions about wether or not that image contains metastatic tissue, i.e. cancer and that can be trained relatively quickly."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom glob import glob \nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D\nfrom tensorflow.keras.layers import Dense, Dropout, Flatten, Activation, BatchNormalization\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau, ModelCheckpoint\nfrom tensorflow.keras.optimizers import Adam\nfrom skimage.io import imread\nimport os\nimport cv2\nfrom sklearn.utils import shuffle\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn.model_selection import train_test_split\nimport itertools\nimport shutil\nimport matplotlib.pyplot as plt\nfrom tqdm import tqdm_notebook,trange\nfrom sklearn.metrics import roc_curve, auc, roc_auc_score\nimport matplotlib.pyplot as plt\nfrom IPython.display import HTML\n%matplotlib inline\n\nSAMPLE_SIZE = 10000\nIMAGE_SIZE = 96\nIMAGE_CHANNELS = 3","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **DATA**"},{"metadata":{},"cell_type":"markdown","source":"The dataset contains 220,025 images for training that are labeled with either a 0 or 1. A 0 indicates negative, i.e cancer-free and a 1 indicates positive, i.e. that the image contained cancer. A positive label indicates that the image contains at least one pixel of tumor tissue. Tumor tissue in the outer region of the patch does not influence the label. The dataset contains about 60% negative and 40% positive scans. \n\nThe dataset also contains 57,468 test images which will be used to evaluate our submission. The test samples are unlabeled so they will not be used in building and evaluating our model."},{"metadata":{},"cell_type":"markdown","source":"# **LOADING DATA AND EXPLORATORY DATA ANALYSIS**"},{"metadata":{},"cell_type":"markdown","source":"Here we load in the filenames of the training images and their corresponding lables. A sample of what the resulting dataframe looks like is shown below."},{"metadata":{"trusted":true},"cell_type":"code","source":"#referenced https://www.kaggle.com/gomezp/complete-beginner-s-guide-eda-keras-lb-0-93\npath = \"../input/histopathologic-cancer-detection/\"\ntrain_path = path + 'train/'\ntest_path = path + 'test/'\n\ndf = pd.DataFrame({'path': glob(os.path.join(train_path,'*.tif'))}) \ndf['id'] = df.path.map(lambda x: x.split('/')[4].split(\".\")[0]) \nlabels = pd.read_csv(path+\"train_labels.csv\")\ndf = df.merge(labels, on = \"id\")\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The training dataset consists of 130,908 negatives and 89,117 positives which is approximately 59.5% and 41.5%, respectively as a shown in the pie chart below."},{"metadata":{"trusted":true},"cell_type":"code","source":"df['label'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"labels = 'Negative', 'Positive' \nsizes = df['label'].value_counts()\nfig1, ax1 = plt.subplots()\nax1.pie(sizes, labels=labels, autopct='%1.1f%%',\n        startangle=90)\nax1.axis('equal')  \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since the dataset is very large and training a model usng the whole dataset would take a long time, I decided to trim down the number of training samples that would be used. After some trial and error, I decided on using a total of 20,000 total samples, 10,000 positve and 10,000 negative. Sample sizes greater than 20,000 samples did not result in improved model accuracy or area under the ROC curve. This sample size should be large enough represent the full training dataset, while being able to train the model relatively quickly. I decided to make the number of positive and negative samples the same so that accuracy wouldn't be skewed by simply fitting the model towards the more frequent outcome."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_neg = df[df['label'] == 0].sample(SAMPLE_SIZE, random_state = 113)\ndf_pos = df[df['label'] == 1].sample(SAMPLE_SIZE, random_state = 113)\ndata = pd.concat([df_neg, df_pos], axis=0).reset_index(drop=True)\ndata = shuffle(data)\n\ndata['label'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Once I had the dataset I was going to work with, I read in the first 100 images and then randomly selected a few for viewing to get an idea of what the images I was working with looked like. The sample images can be see below."},{"metadata":{"trusted":true},"cell_type":"code","source":"#referenced https://www.kaggle.com/gomezp/complete-beginner-s-guide-eda-keras-lb-0-93\n\ndef load_images(df, N):\n    \n    X_img = np.zeros([N,IMAGE_SIZE,IMAGE_SIZE,IMAGE_CHANNELS],dtype=np.uint8) \n    #convert the labels to a numpy array too\n    y_img = data['label'].to_numpy()\n    #read images one by one, tdqm notebook displays a progress bar\n   \n        \n    for i, row in tqdm_notebook(df.iterrows(), total = N):\n       if i == N:\n            break\n       X_img[i] = cv2.imread(row['path'])\n    return X_img, y_img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#referenced https://www.kaggle.com/gomezp/complete-beginner-s-guide-eda-keras-lb-0-93\n\ndata.reset_index(inplace = True, drop = True)\nX_img, y_img = load_images(data, 100)\nfig = plt.figure(figsize=(10, 4), dpi=150)\nnp.random.seed(113) #we can use the seed to get a different set of random images\nfor plotNr,idx in enumerate(np.random.randint(0, 100,8)):\n   ax = fig.add_subplot(2, 8//2, plotNr+1, xticks=[], yticks=[]) #add subplots\n   plt.imshow(X_img[idx]) #plot image\n   ax.set_title('Label: ' + str(y_img[idx]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since I didn't have the labels for the test data, I split the 20,000 training samples into a test set and a validation set. I decided to use 90% of the images to train the model and to reserve 10% of the images to validate the model. The samples were split between the train and validation sets randomly and stratified on y. I decided to stratify on y in order to keep the number of positives and negatives in each set even once again so the model would not be skewed in the direction of the most frequent outcome."},{"metadata":{"trusted":true},"cell_type":"code","source":"y = data['label']\n\ndf_train, df_val = train_test_split(data, test_size=0.10, random_state=113, stratify=y)\n\nprint(df_train.shape)\nprint(df_val.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train['label'].value_counts()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_val['label'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The next step was creating directories to store the images. I created a test and a validation directory and then images were sorted to the appropriate directories. This seemed to be easier than keeping lists of what was in each sample population and repeatedly reading everything from the initial train directory."},{"metadata":{"trusted":true},"cell_type":"code","source":"#reference: https://www.kaggle.com/fmarazzi/baseline-keras-cnn-roc-fast-10min-0-925-lb\n\ntrain_path = 'base_dir/train'\nvalid_path = 'base_dir/valid'\ntest_path = '../input/histopathologic-cancer-detection/test'\nfor fold in [train_path, valid_path]:\n    for subf in [\"0\", \"1\"]:\n        os.makedirs(os.path.join(fold, subf))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.set_index('id', inplace=True)\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for image in df_train['id'].values:\n    \n    fname = image + '.tif'\n    label = str(data.loc[image,'label'])\n    src = os.path.join('../input/histopathologic-cancer-detection/train', fname)\n    dst = os.path.join(train_path, label, fname)\n    shutil.copyfile(src, dst) \n\nfor image in df_val['id'].values:\n    fname = image + '.tif'\n    label = str(data.loc[image,'label'])\n    src = os.path.join('../input/histopathologic-cancer-detection/train', fname)\n    dst = os.path.join(valid_path, label, fname)\n    shutil.copyfile(src, dst)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The images were preprocessed using the keras ImageDataGenerator functions, each pixel x was transformed to the value representing how many standard deviations it was from the mean x value. This makes pixels that vary significantly from the average standout, which may be useful in identifying images that contain cancer."},{"metadata":{"trusted":true},"cell_type":"code","source":"#reference: https://www.kaggle.com/fmarazzi/baseline-keras-cnn-roc-fast-10min-0-925-lb\n\nnum_train_samples = len(df_train)\nnum_val_samples = len(df_val)\ntrain_batch_size = 32\nval_batch_size = 32\n\ntrain_steps = np.ceil(num_train_samples / train_batch_size)\nval_steps = np.ceil(num_val_samples / val_batch_size)\n\ndatagen = ImageDataGenerator(preprocessing_function=lambda x:(x - x.mean()) / x.std() if x.std() > 0 else x,\n                            horizontal_flip=True,\n                            vertical_flip=True)\n\ntrain_data = datagen.flow_from_directory(train_path,\n                                        target_size=(IMAGE_SIZE,IMAGE_SIZE),\n                                        batch_size=train_batch_size,\n                                        class_mode='binary')\n\nval_data = datagen.flow_from_directory(valid_path,\n                                        target_size=(IMAGE_SIZE,IMAGE_SIZE),\n                                        batch_size=val_batch_size,\n                                        class_mode='binary')\n\n\ntest_data = datagen.flow_from_directory(valid_path,\n                                        target_size=(IMAGE_SIZE,IMAGE_SIZE),\n                                        batch_size=1,\n                                        class_mode='binary',\n                                        shuffle=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# BUILD THE MODEL"},{"metadata":{},"cell_type":"markdown","source":"A CNN network using the keras library was selected to build the model because CNN networks are commonly used for image processing/analysis problems. Sequential was selected from keras because this allowed the model to be built layer by layer. The adam optimizer was seleted based on a mix of experimentation and the information in the following article https://medium.com/octavian-ai/which-optimizer-and-learning-rate-should-i-use-for-deep-learning-5acb418f9b2. Of the optimizers Adam seems to learn the fastest and it is more stable than the other optimizers in terms of accuracies with various learning rates. I selected a starting learning rate of .01 by testing a number of incremental learning rates including .005, .008, .01, and .015 and seeing which resulted in the lowest loss, highest accuracy and still was relatively time efficient. ‘binary_crossentropy’ was used for the loss function because either it is cancer or it is not. The remaining inputs were selected following guidance from: https://towardsdatascience.com/building-a-convolutional-neural-network-cnn-in-keras-329fbbadc5f5."},{"metadata":{"trusted":true},"cell_type":"code","source":"#reference: https://www.kaggle.com/fmarazzi/baseline-keras-cnn-roc-fast-10min-0-925-lb\n\nkernel_size = (3,3) \npool_size= (2,2)\nfirst_filters = 32\nsecond_filters = 64\nthird_filters = 128\n\ndropout_conv = 0.3\ndropout_dense = 0.5\n\nmodel = Sequential()\nmodel.add(Conv2D(first_filters, kernel_size, activation = 'relu', input_shape = (IMAGE_SIZE, IMAGE_SIZE, 3)))\nmodel.add(Conv2D(first_filters, kernel_size, use_bias=False))\nmodel.add(BatchNormalization())\nmodel.add(Activation(\"relu\")) \nmodel.add(Dropout(dropout_conv))\n\nmodel.add(Conv2D(second_filters, kernel_size, use_bias=False))\nmodel.add(BatchNormalization())\nmodel.add(Activation(\"relu\"))\nmodel.add(Conv2D(second_filters, kernel_size, use_bias=False))\nmodel.add(BatchNormalization())\nmodel.add(Activation(\"relu\"))\nmodel.add(Dropout(dropout_conv))\n\nmodel.add(Conv2D(third_filters, kernel_size, use_bias=False))\nmodel.add(BatchNormalization())\nmodel.add(Activation(\"relu\"))\nmodel.add(Conv2D(third_filters, kernel_size, use_bias=False))\nmodel.add(BatchNormalization())\nmodel.add(Activation(\"relu\"))\nmodel.add(Dropout(dropout_conv))\n\nmodel.add(Flatten())\nmodel.add(Dense(256, use_bias=False))\nmodel.add(BatchNormalization())\nmodel.add(Activation(\"relu\"))\nmodel.add(Dropout(dropout_dense))\nmodel.add(Dense(1, activation = \"sigmoid\"))\n\n# Compile the model\nmodel.compile(Adam(0.01), loss = \"binary_crossentropy\", metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# TRAIN THE MODEL"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n#reference: https://www.kaggle.com/fmarazzi/baseline-keras-cnn-roc-fast-10min-0-925-lb\n\nreduce = ReduceLROnPlateau(monitor='val_loss', patience=1, verbose=1, factor=0.1)\nmodel.fit_generator(train_data, steps_per_epoch=train_steps, \n                    validation_data=val_data,\n                    validation_steps=val_steps,\n                    epochs=10, #did not see improvement in accuracy beyond 10 epochs\n                   callbacks=[reduce])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# ANALYSIS"},{"metadata":{},"cell_type":"markdown","source":"I generated predictions for the validation samples and compared those predictions with the actual outcomes. The ROC curve below shows the True Positive versus False Postive rates. Generally the goal with classification problems is to maximize the area under the ROC curve (AUC), trying to get the value to be as close to 1 as possible. My model results in an AUC of over .9 which is sufficiently high for the goal of this project. Thus I will stop tuning the parameters of the model and use this version for submittal."},{"metadata":{"trusted":true},"cell_type":"code","source":"#reference: https://www.kaggle.com/fmarazzi/baseline-keras-cnn-roc-fast-10min-0-925-lb\n\ny_pred = model.predict_generator(test_data, steps=len(df_val), verbose=1)\nfpr, tpr, thresholds_keras = roc_curve(test_data.classes, y_pred)\nauc = auc(fpr, tpr)\nauc\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(1)\nplt.plot([0, 1], [0, 1])\nplt.plot(fpr, tpr)\nplt.xlabel('False Positive Rate')\nplt.ylabel('True Positive Rate')\nplt.title('ROC Curve')\nplt.legend(loc='best')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# PREPARE SUBMISSION"},{"metadata":{},"cell_type":"markdown","source":"Here we use the actual competition test images and make predictions to submit. You can see a sample of what the submission will look like below."},{"metadata":{"trusted":true},"cell_type":"code","source":"#referenced https://www.kaggle.com/gomezp/complete-beginner-s-guide-eda-keras-lb-0-93\n\nbase_test_dir = '../input/histopathologic-cancer-detection/test'\ntest_samples = glob(os.path.join(base_test_dir,'*.tif'))\nbatch_size = 5000\nidx = len(test_samples)\nfor i in range(0, idx, batch_size):\n    print(\"Indexes: %i - %i\"%(i, i+batch_size))\n    submission_df = pd.DataFrame({'path': test_samples[i:i+batch_size]})\n    submission_df['id'] = submission_df.path.map(lambda x: x.split('/')[4].split(\".\")[0])\n    submission_df['image'] = submission_df['path'].map(imread)\n    test = np.stack(submission_df[\"image\"].values)\n    test = (test - test.mean()) / test.std()\n    predictions = model.predict(test)\n    submission_df['label'] = predictions\n    submission = pd.concat([submission, submission_df[[\"id\", \"label\"]]])\nsubmission.head()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#reference: https://www.kaggle.com/fmarazzi/baseline-keras-cnn-roc-fast-10min-0-925-lb\n\nshutil.rmtree(train_path) #delete directory path\nshutil.rmtree(valid_path) #delete directory path\nsubmission.to_csv(\"submission.csv\", index = False, header = True) #make csv file","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.read_csv(\"submission.csv\") #check contents of csv file","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# CONCLUSION"},{"metadata":{},"cell_type":"markdown","source":"A CNN model can be used to make predictions about cancer from image scans with reasonable accuracy, and a relatively high auc rate. Although you need a decent sized population of sample data to train a model, I learned the benefit from increasing the number of samples starts to taper off around 20,000 samples in this case. Additionally, training epochs beyond 10 did not seem to improve the model performance significantly. For both of these factors, the benefit of a faster model seemed to outweigh any minimal gains by increasing their values. The model I created can be trained in approximately 10 mintues using GPU acceleration while being relatively reliable. Further advances in nueral networks or just myself gaining a more in depth understanding of them could result in improved accuracy and auc numbers. This shows the power of what machine learning can and could do in medical settings."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}