{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Histopathologic Cancer Detection\n\nKaggle Competition : https://www.kaggle.com/c/histopathologic-cancer-detection"},{"metadata":{},"cell_type":"markdown","source":"## Competition Info"},{"metadata":{},"cell_type":"markdown","source":"### Objective\n- create an algorithm to identify metastatic cancer in small image patches taken from larger digital pathology scans on the dataset, that is cleaned for simplicity \n- For each id in the test set, you must predict a probability that center 32x32px region of a patch contains at least one pixel of tumor tissue.\n- binary classification problem - you are predicting labels for images in test folder"},{"metadata":{},"cell_type":"markdown","source":"### Data\n- training data contains images named with an image 'id'\n- train_labels.csv gives the truth about training data images\n- we are only looking at the centre 32x32px region - positive label = at least one pixel of tumor\n- there are no duplicates in the data"},{"metadata":{},"cell_type":"markdown","source":"## First Iteration\n\n- Trying a Convolutional Neural Network[](http://)\n- Using template from https://www.tensorflow.org/tutorials/images/classification and modifiying appropriately"},{"metadata":{},"cell_type":"markdown","source":"1. Examining data and building an image input pipeline using ImageDataGenerator class provided by tf.keras"},{"metadata":{"trusted":true},"cell_type":"code","source":"# import the right packages\n\"\"\"\nos package is used to read files and directory structure\n\n\"\"\"\nfrom __future__ import absolute_import, division, print_function, unicode_literals\n\n#for contructing the model\nimport tensorflow as tf\nfrom tensorflow import keras\n\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Conv2D, Flatten, Dropout, MaxPooling2D\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\n\n#data manipulation/viz\nimport os\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport glob\n\nfrom sklearn.model_selection import train_test_split # for splitting the data into train/val sets\nfrom sklearn.utils import shuffle\nimport shutil # for copy and moving files\n\n\nRND = 1993","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# list avail files and directories\nos.listdir('/kaggle/input/histopathologic-cancer-detection/')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# coutning the num of samples\nprint(\"num test: \", len(os.listdir('/kaggle/input/histopathologic-cancer-detection/test')))\nprint(\"total training set: \", len(os.listdir('/kaggle/input/histopathologic-cancer-detection/train')))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load Data\n# train_labels.csv contains list of all image ids and corresponding label\nlabels = pd.read_csv('/kaggle/input/histopathologic-cancer-detection/train_labels.csv')\nlabels.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# see how many samples of each label there are\nlabels['label'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- 130908 images in the 0 class\n- 89117 images in the 1 class"},{"metadata":{},"cell_type":"markdown","source":"We will first try training the model with unbalanced data, then compare it when we train with balanced data. The balancing will be done as follows:\n\n- source: https://www.kaggle.com/vbookshelf/cnn-how-to-use-160-000-images-without-crashing"},{"metadata":{"trusted":true},"cell_type":"code","source":"# # uncomment this section if you want to balance the data\n\n# # make the num of neg cases equal to num of pos cases\n\n\n# SAMPLE_SIZE = 89117\n\n# # take a random sample of class 0 with size equal to num samples in class 1\n# df_0 = labels[labels['label'] == 0].sample(SAMPLE_SIZE, random_state = RND)\n\n# # filter out class 1\n# df_1 = labels[labels['label'] == 1].sample(SAMPLE_SIZE, random_state = RND)\n\n# # concat the dataframes\n# labels_equal = pd.concat([df_0, df_1], axis=0).reset_index(drop=True)\n\n# # shuffle\n# labels_equal = shuffle(labels_equal) # shuffle is from sklearn\n\n# #verify the pos and neg cases equal\n# labels_equal['label'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# split into train and val sets\ndf_train, df_val = train_test_split(labels, test_size=57458, \n                                    random_state=RND, stratify = labels['label'])\n\n# we want the validation size to be the same as the test size\n# if you want to use the balanced set use \"labels_equal\"\n# by default test size is 0.25\n\n# stratification means that the train_test_split method returns training and test subsets that\n# have the same proportions of class labels as the input dataset.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# num of training samples\ndf_train['label'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# num of val samples\ndf_val['label'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create folders to store data\n\nPATH = '/kaggle/'\n\ntrain_dir = os.path.join(PATH, 'train_set')\nos.mkdir(train_dir)\n\nvalidation_dir = os.path.join(PATH, 'validation_set')\nos.mkdir(validation_dir)\n#test_dir = os.path.join(PATH, 'test_set')\n\n\ntrain_pos_dir = os.path.join(train_dir, 'pos')  # directory with our training cancer positive pictures\nos.mkdir(train_pos_dir)\n\ntrain_neg_dir = os.path.join(train_dir, 'neg')  # directory with our training cancer negative pictures\nos.mkdir(train_neg_dir)\n\n\nvalidation_pos_dir = os.path.join(validation_dir, 'pos')  \nos.mkdir(validation_pos_dir)\n\nvalidation_neg_dir = os.path.join(validation_dir, 'neg') \nos.mkdir(validation_neg_dir)\n\n\n\n# train_pos_dir = os.path.join(train_dir, 'pos')  \n# train_neg_dir = os.path.join(train_dir, 'neg')  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"\ndirectory structure that we are trying to develop:\n\nkaggle/\n|__ train_set\n    |______ pos: [cat.0.jpg, cat.1.jpg, cat.2.jpg ....]\n    |______ neg: [dog.0.jpg, dog.1.jpg, dog.2.jpg ...]\n|__ validation_set\n    |______ pos: [cat.2000.jpg, cat.2001.jpg, cat.2002.jpg ....]\n    |______ neg: [dog.2000.jpg, dog.2001.jpg, dog.2002.jpg ...]\n\n|__ input\n    |______ Histo....\n            |______ test\n            |______ train\n            \n            .\n            .\n            .\n            \nfrom: https://www.tensorflow.org/tutorials/images/classification\n\"\"\"\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Set the id to be the index in labels_equal dataset\nlabels.set_index('id', inplace=True)\n\n# Get a list of train and val images\ntrain_list = list(df_train['id'])\nval_list = list(df_val['id'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Transfer the train images into the appropriate folders created\n\n#looping through id's\nfor image in train_list:\n    \n    # the id in the csv file does not have the .tif extension therefore we add it here\n    fname = image + '.tif'\n    \n    # get the label for a certain image\n    target = labels.loc[image,'label']\n    \n    # these must match the folder names\n    if target == 0:\n        label = 'neg'\n    if target == 1:\n        label = 'pos'\n    \n    # source path to image\n    src = os.path.join('/kaggle/input/histopathologic-cancer-detection/train', fname)\n    \n    # destination path to image\n    dst = os.path.join(train_dir, label, fname)\n    \n    # copy the image from the source to the destination\n    shutil.copyfile(src, dst)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Transfer the validation images into the appropriate folders created\n\n\n#looping through id's\nfor image in val_list:\n    \n    # the id in the csv file does not have the .tif extension therefore we add it here\n    fname = image + '.tif'\n    \n    # get the label for a certain image\n    target = labels.loc[image,'label']\n    \n    # these must match the folder names\n    if target == 0:\n        label = 'neg'\n    if target == 1:\n        label = 'pos'\n    \n    # source path to image\n    src = os.path.join('/kaggle/input/histopathologic-cancer-detection/train', fname)\n    \n    # destination path to image\n    dst = os.path.join(validation_dir, label, fname)\n    \n    # copy the image from the source to the destination\n    shutil.copyfile(src, dst)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# explore the data\n\n\nnum_pos_tr = len(os.listdir(train_pos_dir))\nnum_neg_tr = len(os.listdir(train_neg_dir))\n\nnum_pos_val = len(os.listdir(validation_pos_dir))\nnum_neg_val = len(os.listdir(validation_neg_dir))\n\ntotal_train = num_pos_tr + num_neg_tr\ntotal_val = num_pos_val + num_neg_val\n\nprint('total training cancer positive images:', num_pos_tr)\nprint('total training cancer negative images:', num_neg_tr)\n\nprint('total validation cancer positive images:', num_pos_val)\nprint('total validation cancer negative images:', num_neg_val)\nprint(\"--\")\nprint(\"Total training images:\", total_train)\nprint(\"Total validation images:\", total_val)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# For convenience, set up variables to use while pre-processing the dataset and training the network.\n\nbatch_size = 128\nepochs = 15\nIMG_HEIGHT = 96\nIMG_WIDTH = 96\nIMG_DEPTH = 3","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Data Preperation\n\n\"\"\"\nFormat the images into appropriately pre-processed floating point tensors before feeding to the network:\n\n-Read images from the disk.\n-Decode contents of these images and convert it into proper grid format as per their RGB content.\n-Convert them into floating point tensors.\n-Rescale the tensors from values between 0 and 255 to values between 0 and 1, as neural networks prefer \nto deal with small input values.\n\"\"\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Because the model overfit in previous runs, I'm going to apply some data augmentation to increase the number of training examples. Namely, zoom and rotating the training samples. The validation images will remain as is so they match the distribution of the test set."},{"metadata":{"trusted":true},"cell_type":"code","source":"# ImageDataGenerator class provided by tf.keras: It can read images from disk and preprocess them into \n# proper tensors. It will also set up generators that convert these images into batches of tensors—helpful \n# when training the network.\n\n# Using this because we do not have enough memory to save *all* training images together to feed to network\n#Generator for our training data\ntrain_image_generator = ImageDataGenerator(rescale=1./255,\n                                          rotation_range=45,\n                                          zoom_range=0.5) # applied zoom and rotation augmentations\nvalidation_image_generator = ImageDataGenerator(rescale=1./255) # Generator for our validation data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data_gen = train_image_generator.flow_from_directory(batch_size=batch_size,\n                                                           directory=train_dir,\n                                                           shuffle=True,\n                                                           target_size=(IMG_HEIGHT, IMG_WIDTH),\n                                                           color_mode = 'rgb',\n                                                           class_mode='binary')\n\n# color mode is rgb by defualt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"val_data_gen = validation_image_generator.flow_from_directory(batch_size=batch_size,\n                                                              directory=validation_dir,\n                                                              target_size=(IMG_HEIGHT, IMG_WIDTH),\n                                                              class_mode='binary')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"2. Visualizing the training images"},{"metadata":{"trusted":true},"cell_type":"code","source":"\nsample_training_images, sample_training_labels = next(train_data_gen)\n\n\"\"\"\nThe next function returns a batch from the dataset. The return value of next \nfunction is in form of (x_train, y_train) where x_train is training features and \ny_train, its labels. Discard the labels to only visualize the training images.\n\"\"\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# This function will plot images in the form of a grid with 1 row and 5 columns where \n# images are placed in each column.\ndef plotImages(images_arr, label):\n    fig, axes = plt.subplots(1, 5, figsize=(20,20))\n    axes = axes.flatten()\n    for img, ax in zip( images_arr, axes):\n        ax.imshow(img)\n        ax.axis('off')\n    plt.tight_layout()\n    plt.show()\n    \nplotImages(sample_training_images[:5], sample_training_labels[:5])\n\nprint(sample_training_labels[:5])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"3. Building the model\n"},{"metadata":{},"cell_type":"markdown","source":"I'm going to introduce dropout in the network to further prevent the model from overfitting"},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"\nThe model consists of three convolution blocks with a max pool layer in each of them. \nThere's a fully connected layer with 512 units on top of it that is activated by a relu \nactivation function.\n\"\"\"\n# # original\n# model = Sequential([\n#     Conv2D(16, 3, padding='same', activation='relu', input_shape=(IMG_HEIGHT, IMG_WIDTH ,3)),\n#     MaxPooling2D(),\n#     Conv2D(32, 3, padding='same', activation='relu'),\n#     MaxPooling2D(),\n#     Conv2D(64, 3, padding='same', activation='relu'),\n#     MaxPooling2D(),\n#     Flatten(),\n#     Dense(512, activation='relu'),\n#     Dense(1)\n# ])\n\n#dropout\n\nmodel = Sequential([\n    Conv2D(16, 3, padding='same', activation='relu', input_shape=(IMG_HEIGHT, IMG_WIDTH ,3)),\n    MaxPooling2D(),\n    Dropout(0.2),\n    \n    Conv2D(32, 3, padding='same', activation='relu'),\n    MaxPooling2D(),\n    \n    Conv2D(64, 3, padding='same', activation='relu'),\n    MaxPooling2D(),\n    Dropout(0.2),\n    \n    Flatten(),\n    Dense(512, activation='relu'),\n    Dropout(0.2),\n\n    Dense(1, activation = \"sigmoid\")\n])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Compile the model\n# additional reference: https://www.tensorflow.org/tutorials/structured_data/imbalanced_data\n\n# METRICS = [\n#       keras.metrics.TruePositives(name='tp'),\n#       keras.metrics.FalsePositives(name='fp'),\n#       keras.metrics.TrueNegatives(name='tn'),\n#       keras.metrics.FalseNegatives(name='fn'), \n#       keras.metrics.BinaryAccuracy(name='accuracy'),\n#       keras.metrics.Precision(name='precision'),\n#       keras.metrics.Recall(name='recall'),\n#       keras.metrics.AUC(name='auc'),\n# ]\n\nmodel.compile(optimizer='adam',\n              loss=tf.keras.losses.BinaryCrossentropy(from_logits=True))\n\n\n# using AUC as our metric since the competition will be judged using that\n# for imbalanced data accuracy is not a good metric\n\n\n# Model Summary\n# View all the layers\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train the model\n    \nhistory = model.fit(\n    train_data_gen,\n    steps_per_epoch=total_train // batch_size,\n    epochs=epochs,\n    validation_data=val_data_gen,\n    validation_steps=total_val // batch_size\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Visualize the training results\n\nloss=history.history['loss']\nval_loss=history.history['val_loss']\n\nepochs_range = range(epochs)\n\nplt.subplot(1, 2, 2)\nplt.plot(epochs_range, loss, label='Training Loss')\nplt.plot(epochs_range, val_loss, label='Validation Loss')\nplt.legend(loc='upper right')\nplt.title('Training and Validation Loss')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Test the performance of the trained model on the validation set"},{"metadata":{"trusted":true},"cell_type":"code","source":"# # Put the predictions into a dataframe.\n# # The columns need to be oredered to match the output of the previous cell\n# predictions = model.predict_generator(test_gen, steps=len(df_val), verbose=1)\n\n# df_preds = pd.DataFrame(predictions, columns=['no_tumor_tissue', 'has_tumor_tissue'])\n\n# # Get the true labels\n# y_true = test_gen.classes\n\n# # Get the predicted labels as probabilities\n# y_pred = df_preds['has_tumor_tissue']\n\n\n# from sklearn.metrics import roc_auc_score\n\n# roc_auc_score(y_true, y_pred)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#reminder of how stratify works\n\n# X = np.arange(10).reshape((5, 2))\n\n# X[0,0] = 0\n# X[1,0] = 0\n# X[2,0] = 1\n# X[3,0] = 1\n# X[4,0] = 0\n\n# X_train,  y_train = train_test_split(\n#     X, test_size=0.33, random_state=42, stratify = X[:,0])\n\n# X_train\n# y_train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}