{"cells":[{"metadata":{"_uuid":"eb49344adfa785e154e7f5cd60643298c51ca819"},"cell_type":"markdown","source":"### Part 2 - Metastatic Model"},{"metadata":{"_uuid":"204b788dc08224218324ee80a6ff9a6d6b1f2070"},"cell_type":"markdown","source":"*Link to Part 1:*
\nhttps://www.kaggle.com/vbookshelf/part-1-breast-cancer-analyzer-web-app"},{"metadata":{"_uuid":"c1e92489350fd6543d60b41cbde9c69f332eb23e"},"cell_type":"markdown","source":"Welcome back. In this section we will create the Metastatic_model. This model will predict whether or not Metastatic cancer tissue is present in histopathologic image patches.\n\n**Dataset **\n\nWe will use Kaggle's version of the PCam (PatchCamelyon) dataset. It's part of the [Histopathologic Cancer Detection competition](https://www.kaggle.com/c/histopathologic-cancer-detection) where the challenge is to identify metastatic tissue in histopathologic scans of lymph node sections. \n\nThe dataset consists of 220,025 image patches of size 96x96 (130,908 Metastatic negative and 89,117 Metastatic positive). \n\nThe images are in tiff format. Many web browsers, including Chrome, don't support the tiff format. Thus the web app wil not be able to accept tiff images. Before training, we will convert these images to png format. This will ensure that the model will be trained on images of similar quality to what we expect a user to submit.\n\n\n**Results**\n\nOur cnn model will achieve an accuracy and F1 score of approximately 0.94.\n\n"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"from numpy.random import seed\nseed(101)\nfrom tensorflow import set_random_seed\nset_random_seed(101)\n\nimport pandas as pd\nimport numpy as np\n\n\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D\nfrom tensorflow.keras.layers import Dense, Dropout, Flatten, Activation\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau, ModelCheckpoint\nfrom tensorflow.keras.optimizers import Adam\n\nimport os\nimport cv2\n\nfrom sklearn.utils import shuffle\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn.model_selection import train_test_split\nimport itertools\nimport shutil\nimport matplotlib.pyplot as plt\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b7bbdd52c81188b8e9c528b88d9fd0da176bf4bc"},"cell_type":"code","source":"\nIMAGE_SIZE = 96\nIMAGE_CHANNELS = 3\n\nSAMPLE_SIZE = 80000 # the number of images we use from each of the two classes\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"25802a26e4afba8f6dc42754f7f61da4c8cfea5c"},"cell_type":"markdown","source":"### What files are available?"},{"metadata":{"trusted":true,"_uuid":"699bb899bde433ba20fb0d086fa0f33a0f61a250"},"cell_type":"code","source":"os.listdir('../input/histopathologic-cancer-detection')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c24d1662f8bd5fc4d8c57c29449990a3520cd96b"},"cell_type":"markdown","source":"### Labels as per csv file\n\n0 = no met tissue
\n1 = has met tissue.
\n"},{"metadata":{"_uuid":"5a285343c286be191aa827ff9b836b67821176a5"},"cell_type":"markdown","source":"### How many training images are in each folder?"},{"metadata":{"trusted":true,"_uuid":"54461212efed65ac377369a468c80e7d708010f4"},"cell_type":"code","source":"\nprint(len(os.listdir('../input/histopathologic-cancer-detection/train')))\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b90854e07d495d9f945a0e40189fd32a0c32bff5"},"cell_type":"markdown","source":"### Create a Dataframe containing all images"},{"metadata":{"trusted":true,"_uuid":"e9c9f40ffab35044641b0dc7d9b18609af1aa25e"},"cell_type":"code","source":"df_data = pd.read_csv('../input/histopathologic-cancer-detection/train_labels.csv')\n\n# removing this image because it caused a training error previously\ndf_data = df_data[df_data['id'] != 'dd6dfed324f9fcb6f93f46f32fc800f2ec196be2']\n\n# removing this image because it's black\ndf_data = df_data[df_data['id'] != '9369c7278ec8bcc6c880d99194de09fc2bd4efbe']\n\n\nprint(df_data.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cfbd53b7f8ea1929952ffed6221b380012618e32"},"cell_type":"markdown","source":"### Check the class distribution"},{"metadata":{"trusted":true,"_uuid":"e18560bf69d3dfc0c4772e7c79bb119fd2eb634b"},"cell_type":"code","source":"df_data['label'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6efe2e5de99c4bf92079b1a7d0b892d30fc9d518"},"cell_type":"markdown","source":"### Display a random sample of train images by class"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"1c5143f227da4262eafce8cf0210a02c8072fb8e"},"cell_type":"code","source":"# source: https://www.kaggle.com/gpreda/honey-bee-subspecies-classification\n\ndef draw_category_images(col_name,figure_cols, df, IMAGE_PATH):\n \n \"\"\"\n Give a column in a dataframe,\n this function takes a sample of each class and displays that\n sample on one row. The sample size is the same as figure_cols which\n is the number of columns in the figure.\n Because this function takes a random sample, each time the function is run it\n displays different images.\n \"\"\"\n \n\n categories = (df.groupby([col_name])[col_name].nunique()).index\n f, ax = plt.subplots(nrows=len(categories),ncols=figure_cols, \n figsize=(4*figure_cols,4*len(categories))) # adjust size here\n # draw a number of images for each location\n for i, cat in enumerate(categories):\n sample = df[df[col_name]==cat].sample(figure_cols) # figure_cols is also the sample size\n for j in range(0,figure_cols):\n file=IMAGE_PATH + sample.iloc[j]['id'] + '.tif'\n im=cv2.imread(file)\n ax[i, j].imshow(im, resample=True, cmap='gray')\n ax[i, j].set_title(cat, fontsize=16) \n plt.tight_layout()\n plt.show()\n ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bd38bcfb5839975e4fee9e70b93d42c29c1b5d2e"},"cell_type":"code","source":"IMAGE_PATH = '../input/histopathologic-cancer-detection/train/' \n\ndraw_category_images('label',4, df_data, IMAGE_PATH)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1da4226777aefe65b1bb3430208ea91ea7ca7d9a"},"cell_type":"markdown","source":"### Create the Train and Val Sets"},{"metadata":{"trusted":true,"_uuid":"547f9f571ee82e20b7647e16ed36de7550046032"},"cell_type":"code","source":"df_data.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c1150500d2772b7f36cdaa5aa5fd7f0fb4a72628"},"cell_type":"markdown","source":"#### Balance the target distribution\nWe will reduce the number of samples in class 0."},{"metadata":{"trusted":true,"_uuid":"270fc18640b552ecc3cb0e1dd3036441db7a4a2b"},"cell_type":"code","source":"# take a random sample of class 0 with size equal to num samples in class 1\ndf_0 = df_data[df_data['label'] == 0].sample(SAMPLE_SIZE, random_state = 101)\n# filter out class 1\ndf_1 = df_data[df_data['label'] == 1].sample(SAMPLE_SIZE, random_state = 101)\n\n# concat the dataframes\ndf_data = pd.concat([df_0, df_1], axis=0).reset_index(drop=True)\n# shuffle\ndf_data = shuffle(df_data)\n\ndf_data['label'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a166dec3ef84c66ad9cd815b63fc1a753df2eb76"},"cell_type":"code","source":"df_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"15ba9792e6a370b7560330af15b3cfe21185c1cb"},"cell_type":"code","source":"# train_test_split\n\n# stratify=y creates a balanced validation set.\ny = df_data['label']\n\ndf_train, df_val = train_test_split(df_data, test_size=0.10, random_state=101, stratify=y)\n\nprint(df_train.shape)\nprint(df_val.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7de70d915a5f1d2599725e00bdb3b9103d947883"},"cell_type":"code","source":"df_train['label'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"392c0eea00be8e43a6e55438d1458650e842030b"},"cell_type":"code","source":"df_val['label'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ba17dd34b75367fc61df6634d51dac94c3ab4951"},"cell_type":"markdown","source":"### Create a Directory Structure"},{"metadata":{"trusted":true,"_uuid":"ff8acc2e92a1b1b5002d6e1bf9a1180c3256f19d","_kg_hide-input":true},"cell_type":"code","source":"# Create a new directory\nbase_dir = 'base_dir'\nos.mkdir(base_dir)\n\n\n#[CREATE FOLDERS INSIDE THE BASE DIRECTORY]\n\n# now we create 2 folders inside 'base_dir':\n\n# train_dir\n # a_no_met_tissue\n # b_has_met_tissue\n\n# val_dir\n # a_no_met_tissue\n # b_has_met_tissue\n\n\n\n# create a path to 'base_dir' to which we will join the names of the new folders\n# train_dir\ntrain_dir = os.path.join(base_dir, 'train_dir')\nos.mkdir(train_dir)\n\n# val_dir\nval_dir = os.path.join(base_dir, 'val_dir')\nos.mkdir(val_dir)\n\n\n\n# [CREATE FOLDERS INSIDE THE TRAIN AND VALIDATION FOLDERS]\n# Inside each folder we create seperate folders for each class\n\n# create new folders inside train_dir\nno_met_tissue = os.path.join(train_dir, 'a_no_met_tissue')\nos.mkdir(no_met_tissue)\nhas_met_tissue = os.path.join(train_dir, 'b_has_met_tissue')\nos.mkdir(has_met_tissue)\n\n\n# create new folders inside val_dir\nno_met_tissue = os.path.join(val_dir, 'a_no_met_tissue')\nos.mkdir(no_met_tissue)\nhas_met_tissue = os.path.join(val_dir, 'b_has_met_tissue')\nos.mkdir(has_met_tissue)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"03ca5d4b8b027c2712d7096314d3a79ef829b23c"},"cell_type":"code","source":"# check that the folders have been created\nos.listdir('base_dir/train_dir')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6a2e56340ba18f3b63c1b129fd995fecfadaa21d"},"cell_type":"markdown","source":"### Transfer the images into the folders"},{"metadata":{"trusted":true,"_uuid":"e84c8a9642b030094b1888af3299063f883112a6"},"cell_type":"code","source":"# Set the id as the index in df_data\ndf_data.set_index('id', inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"afb8969a9ee75c13bddc808a4bcc326611baaaaf","_kg_hide-input":true},"cell_type":"code","source":"\n\n# Get a list of train and val images\ntrain_list = list(df_train['id'])\nval_list = list(df_val['id'])\n\n\n\n# Transfer the train images\n\nfor image in train_list:\n \n # the id in the csv file does not have the .tif extension therefore we add it here\n fname_tif = image + '.tif'\n # get the label for a certain image\n target = df_data.loc[image,'label']\n \n # these must match the folder names\n if target == 0:\n label = 'a_no_met_tissue'\n if target == 1:\n label = 'b_has_met_tissue'\n \n # source path to image\n src = os.path.join('../input/histopathologic-cancer-detection/train', fname_tif)\n # change the new file name to png\n fname_png = image + '.png'\n # destination path to image\n dst = os.path.join(train_dir, label, fname_png)\n\n \n # read the file as an array\n cv2_image = cv2.imread(src)\n # save the image at the destination as a png file\n cv2.imwrite(dst, cv2_image)\n \n\n\n# Transfer the val images\n\nfor image in val_list:\n \n # the id in the csv file does not have the .tif extension therefore we add it here\n fname_tif = image + '.tif'\n # get the label for a certain image\n target = df_data.loc[image,'label']\n \n # these must match the folder names\n if target == 0:\n label = 'a_no_met_tissue'\n if target == 1:\n label = 'b_has_met_tissue'\n \n\n # source path to image\n src = os.path.join('../input/histopathologic-cancer-detection/train', fname_tif)\n # change the new file name to png\n fname_png = image + '.png'\n # destination path to image\n dst = os.path.join(val_dir, label, fname_png)\n\n \n # read the file as an array\n cv2_image = cv2.imread(src)\n # save the image at the destination as a png file\n cv2.imwrite(dst, cv2_image)\n\n\n ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"71532bfc32608289b1f773ffdbc8a7cea1bfb94c"},"cell_type":"code","source":"# check how many train images we have in each folder\n\nprint(len(os.listdir('base_dir/train_dir/a_no_met_tissue')))\nprint(len(os.listdir('base_dir/train_dir/b_has_met_tissue')))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"897e9df543bb65b47bb00019dc681125ca08ee5d"},"cell_type":"code","source":"# check how many val images we have in each folder\n\nprint(len(os.listdir('base_dir/val_dir/a_no_met_tissue')))\nprint(len(os.listdir('base_dir/val_dir/b_has_met_tissue')))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ef780ac9a9fc89b4ff4f042593eb68992f354a1d"},"cell_type":"code","source":"# End of Data Preparation\n### ================================================================================== ###\n# Start of Model Building\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f8dce940ee8a7a42aacb062e4c6b5a4a54dba58f"},"cell_type":"markdown","source":"### Set Up the Generators"},{"metadata":{"trusted":true,"_uuid":"ef4fe7be09f11ff4badfd22d5fd5e03f8521ed58"},"cell_type":"code","source":"train_path = 'base_dir/train_dir'\nvalid_path = 'base_dir/val_dir'\ntest_path = '../input/test'\n\nnum_train_samples = len(df_train)\nnum_val_samples = len(df_val)\ntrain_batch_size = 10\nval_batch_size = 10\n\n\ntrain_steps = np.ceil(num_train_samples / train_batch_size)\nval_steps = np.ceil(num_val_samples / val_batch_size)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"68fbd9d5fbb80859a82f94a12e335ce05a93bd51"},"cell_type":"code","source":"datagen = ImageDataGenerator(rescale=1.0/255)\n\ntrain_gen = datagen.flow_from_directory(train_path,\n target_size=(IMAGE_SIZE,IMAGE_SIZE),\n batch_size=train_batch_size,\n class_mode='categorical')\n\nval_gen = datagen.flow_from_directory(valid_path,\n target_size=(IMAGE_SIZE,IMAGE_SIZE),\n batch_size=val_batch_size,\n class_mode='categorical')\n\n# Note: shuffle=False causes the test dataset to not be shuffled\ntest_gen = datagen.flow_from_directory(valid_path,\n target_size=(IMAGE_SIZE,IMAGE_SIZE),\n batch_size=1,\n class_mode='categorical',\n shuffle=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"79da4d0a66a90cffe40580a596dd4d0e2bc45a9b"},"cell_type":"markdown","source":"### Create the Model Architecture¶"},{"metadata":{"trusted":true,"_uuid":"b9835ea0fd0bca54138904895c39d38227a70c22","_kg_hide-output":true},"cell_type":"code","source":"# Source: https://www.kaggle.com/fmarazzi/baseline-keras-cnn-roc-fast-5min-0-8253-lb\n\nkernel_size = (3,3)\npool_size= (2,2)\nfirst_filters = 32\nsecond_filters = 64\nthird_filters = 128\n\ndropout_conv = 0.3\ndropout_dense = 0.3\n\n\nmodel = Sequential()\nmodel.add(Conv2D(first_filters, kernel_size, activation = 'relu', input_shape = (96, 96, 3)))\nmodel.add(Conv2D(first_filters, kernel_size, activation = 'relu'))\nmodel.add(Conv2D(first_filters, kernel_size, activation = 'relu'))\nmodel.add(MaxPooling2D(pool_size = pool_size)) \nmodel.add(Dropout(dropout_conv))\n\nmodel.add(Conv2D(second_filters, kernel_size, activation ='relu'))\nmodel.add(Conv2D(second_filters, kernel_size, activation ='relu'))\nmodel.add(Conv2D(second_filters, kernel_size, activation ='relu'))\nmodel.add(MaxPooling2D(pool_size = pool_size))\nmodel.add(Dropout(dropout_conv))\n\nmodel.add(Conv2D(third_filters, kernel_size, activation ='relu'))\nmodel.add(Conv2D(third_filters, kernel_size, activation ='relu'))\nmodel.add(Conv2D(third_filters, kernel_size, activation ='relu'))\nmodel.add(MaxPooling2D(pool_size = pool_size))\nmodel.add(Dropout(dropout_conv))\n\nmodel.add(Flatten())\nmodel.add(Dense(256, activation = \"relu\"))\nmodel.add(Dropout(dropout_dense))\nmodel.add(Dense(2, activation = \"softmax\"))\n\nmodel.summary()\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"75cfc4fcb8dd3408d1c4fcf8cd85e0e2f5b611d7"},"cell_type":"markdown","source":"### Train the Model"},{"metadata":{"trusted":true,"_uuid":"9de9715f49a63b55775b10abd2f461b395e23b5d"},"cell_type":"code","source":"model.compile(Adam(lr=0.0001), loss='binary_crossentropy', \n metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"227d84a4f44c0b7256855c06ba04dabd58d89d84"},"cell_type":"code","source":"# Get the labels that are associated with each index\nprint(val_gen.class_indices)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"a746769db61563f226288eba9aa8a6584b9e8e0b"},"cell_type":"code","source":"filepath = \"model.h5\"\ncheckpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, \n save_best_only=True, mode='max')\n\nreduce_lr = ReduceLROnPlateau(monitor='val_acc', factor=0.5, patience=2, \n verbose=1, mode='max', min_lr=0.00001)\n \n \ncallbacks_list = [checkpoint, reduce_lr]\n\nhistory = model.fit_generator(train_gen, steps_per_epoch=train_steps, \n validation_data=val_gen,\n validation_steps=val_steps,\n epochs=20, verbose=1,\n callbacks=callbacks_list)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fa15a8afda3593973726e9087cbd98073041c908"},"cell_type":"markdown","source":"### Evaluate the model using the val set"},{"metadata":{"trusted":true,"_uuid":"70104420ec7f400cd06203f875dbeba30f4d8a96"},"cell_type":"code","source":"# get the metric names so we can use evaulate_generator\nmodel.metrics_names","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"428bdf5b24ff8cef35012205c3f2eb37006fc9e9"},"cell_type":"code","source":"# Here the best epoch will be used.\n\nmodel.load_weights('model.h5')\n\nval_loss, val_acc = \\\nmodel.evaluate_generator(test_gen, \n steps=len(df_val))\n\nprint('val_loss:', val_loss)\nprint('val_acc:', val_acc)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"93556c9e4b6a188cf9cb67a6519c9bc365c60caf"},"cell_type":"markdown","source":"### Plot the Training Curves"},{"metadata":{"trusted":true,"_uuid":"385da8ba94a1079d17909790716b295fc2737584","_kg_hide-input":true},"cell_type":"code","source":"# display the loss and accuracy curves\n\nimport matplotlib.pyplot as plt\n\nacc = history.history['acc']\nval_acc = history.history['val_acc']\nloss = history.history['loss']\nval_loss = history.history['val_loss']\n\nepochs = range(1, len(acc) + 1)\n\nplt.plot(epochs, loss, 'bo', label='Training loss')\nplt.plot(epochs, val_loss, 'b', label='Validation loss')\nplt.title('Training and validation loss')\nplt.legend()\nplt.figure()\n\nplt.plot(epochs, acc, 'bo', label='Training acc')\nplt.plot(epochs, val_acc, 'b', label='Validation acc')\nplt.title('Training and validation accuracy')\nplt.legend()\nplt.figure()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5636e76f23202dd1f2a27ace25e15e09619a5e4e"},"cell_type":"markdown","source":"### Make a prediction on the val set\nWe need these predictions to calculate the AUC score, print the Confusion Matrix and calculate the F1 score."},{"metadata":{"trusted":true,"_uuid":"652d9d6aa51dc1818d1c5171212d10e141ad7de9"},"cell_type":"code","source":"# make a prediction\npredictions = model.predict_generator(test_gen, steps=len(df_val), verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"edf1866df4638ded26de2e0e3d2ba0f5e00e1ace"},"cell_type":"code","source":"predictions.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6bbf4f095fe5412561924c09c96d8dc2ea4adfc0"},"cell_type":"markdown","source":"**A note on Keras class index values**\n\nKeras assigns it's own index value (here 0 and 1) to the classes. It infers the classes based on the folder structure.\nImportant: These index values may not match the index values we were given in the train_labels.csv file.\n\nI've used 'a' and 'b' folder name pre-fixes to get keras to assign index values to match what was in the train_labels.csv file - I guessed that keras is assigning the index value based on folder name alphabetical order."},{"metadata":{"trusted":true,"_uuid":"dc71f69944e7db83329417c5265a5bc31f9c4fc3"},"cell_type":"code","source":"# This is how to check what index keras has internally assigned to each class. \ntest_gen.class_indices","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4a6709d73969f7fd597128223b110be077f84edb"},"cell_type":"code","source":"# Put the predictions into a dataframe.\n# The columns need to be oredered to match the output of the previous cell\n\ndf_preds = pd.DataFrame(predictions, columns=['no_met_tissue', 'has_met_tissue'])\n\ndf_preds.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0954d61a4ef8bc056452b3bbad9456d45c00bed1"},"cell_type":"code","source":"# Get the true labels\ny_true = test_gen.classes\n\n# Get the predicted labels as probabilities\ny_pred = df_preds['has_met_tissue']\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f07c814d213e814cdac4e3d05d0a8db847fbfe28"},"cell_type":"markdown","source":"### What is the AUC Score?"},{"metadata":{"trusted":true,"_uuid":"0b7b7a56c6fa47cc40764d0c06d64860580cbea1"},"cell_type":"code","source":"from sklearn.metrics import roc_auc_score\n\nroc_auc_score(y_true, y_pred)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9d8404e8e6b6008c8989b8184700ec5562b99366"},"cell_type":"markdown","source":"### Create a Confusion Matrix"},{"metadata":{"trusted":true,"_uuid":"91f570e8e5f07126e5361bbf92929d786e853a09","_kg_hide-input":true},"cell_type":"code","source":"# Source: Scikit Learn website\n# http://scikit-learn.org/stable/auto_examples/\n# model_selection/plot_confusion_matrix.html#sphx-glr-auto-examples-model-\n# selection-plot-confusion-matrix-py\n\n\ndef plot_confusion_matrix(cm, classes,\n normalize=False,\n title='Confusion matrix',\n cmap=plt.cm.Blues):\n \"\"\"\n This function prints and plots the confusion matrix.\n Normalization can be applied by setting `normalize=True`.\n \"\"\"\n if normalize:\n cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n print(\"Normalized confusion matrix\")\n else:\n print('Confusion matrix, without normalization')\n\n print(cm)\n\n plt.imshow(cm, interpolation='nearest', cmap=cmap)\n plt.title(title)\n plt.colorbar()\n tick_marks = np.arange(len(classes))\n plt.xticks(tick_marks, classes, rotation=45)\n plt.yticks(tick_marks, classes)\n\n fmt = '.2f' if normalize else 'd'\n thresh = cm.max() / 2.\n for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n plt.text(j, i, format(cm[i, j], fmt),\n horizontalalignment=\"center\",\n color=\"white\" if cm[i, j] > thresh else \"black\")\n\n plt.ylabel('True label')\n plt.xlabel('Predicted label')\n plt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c20bc358dc753020d5a560dc56f2350c7f40a4f9"},"cell_type":"code","source":"# Get the labels of the test images.\n\ntest_labels = test_gen.classes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3a537724473df19c74bb1bf6928f531dd0fcfdb3"},"cell_type":"code","source":"test_labels.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0bb4323e931ff53081994bbf58e82b1ec93ab327"},"cell_type":"code","source":"# argmax returns the index of the max value in a row\ncm = confusion_matrix(test_labels, predictions.argmax(axis=1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ec2058223eb30485898a12c0e904bb170c0aa884"},"cell_type":"code","source":"# Print the label associated with each class\ntest_gen.class_indices","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ba26c7e718df937a18aa2035c4ba883252e44c79"},"cell_type":"code","source":"# Define the labels of the class indices. These need to match the \n# order shown above.\ncm_plot_labels = ['no_met_tissue', 'has_met_tissue']\n\nplot_confusion_matrix(cm, cm_plot_labels, title='Confusion Matrix')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2576c1144cfe93d66f3197396990f3e43addd499"},"cell_type":"markdown","source":"### Create a Classification Report"},{"metadata":{"trusted":true,"_uuid":"91dcace7eb99aca310774b7a3a55535c9127ce55"},"cell_type":"code","source":"from sklearn.metrics import classification_report\n\n# Generate a classification report\n\n# For this to work we need y_pred as binary labels not as probabilities\ny_pred_binary = predictions.argmax(axis=1)\n\nreport = classification_report(y_true, y_pred_binary, target_names=cm_plot_labels)\n\nprint(report)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"36880545abb112946ebe48242d5da6c8517cf61b"},"cell_type":"markdown","source":"**Recall **= Given a class, will the classifier be able to detect it?
\n**Precision **= Given a class prediction from a classifier, how likely is it to be correct?
\n**F1 Score** = The harmonic mean of the recall and precision. Essentially, it punishes extreme values.\n\nFrom the confusion matrix and classification report we see that our model is equally good at detecting both classes."},{"metadata":{"_uuid":"4b0b4f6065e0ba97d9170cc6d9e23ae3d064a73e"},"cell_type":"markdown","source":"### Convert the model to from Keras to Tensorflowjs\nThis conversion needs to be done so that the model can be loaded into the web app."},{"metadata":{"trusted":true,"_uuid":"1510fc5dedb309e4b03c6cbee3acf27f55739ecc"},"cell_type":"code","source":"# Delete the base_dir directory we created to free up disk space to download tensorflowjs\n# and to prevent a Kaggle error.\n# Kaggle allows a max of 500 files to be saved.\n\nshutil.rmtree('base_dir')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true,"_uuid":"28dd1e74473a0523885e50e8d17fd7d50036d9df"},"cell_type":"code","source":"!pip install tensorflowjs","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d565698d4fd2e7479bb0331bf9c9c73fd97fb64d"},"cell_type":"code","source":"# Use the command line conversion tool to convert the model\n\n!tensorflowjs_converter --input_format keras model.h5 tfjs_model_2/model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"daf2016cea165e5aad1ad28f92a81f0ca08d800f"},"cell_type":"markdown","source":"### Lessons learned while building the web app"},{"metadata":{"_uuid":"c9ae43bfacf8c8a579484705e9dce379daf1e54a"},"cell_type":"markdown","source":"**1.**
\nA large part of the web, including the web app for this project uses the Javascript language. I used Javascript to feed the images to the model. The challenge lies in the fact that Javascript is very fast whereas the model is not. This difference in speed can lead to incorrect predictions.\n\nFor example, say we are using a loop to feed ten images to the model for prediction. Because of the realtively slower prediction speed, the model may end up making all of it's ten predictions using only image 1 or image 10. To solve this problem, on each iteration of the loop, we must make the code wait for the model to finish a prediction before the next loop begins i.e. before feeding it the next image.\n\nIn the app I used async/await to achieve this. Because my Javascript knowledge is very basic, you should check the Javascript code thoroughly before using it in one of your own projects.\n\n**2.**
\nI think there is one other potential source of prediction errors. In order to read an image Tensorflowjs needs the image to first be made part of an html image element. It then reads this image element and converts it into a tensor. When using multiple images, delays relating to putting an image into an html element (setting the src attribute) could also cause prediction errors. The programmer needs to keep this in mind especially if the image sizes are large.\n\n**3.**
\nAs mentioned above, most web browsers don't support the tiff image format. This needs to be kept in mind when pre-processing training data if the intention is to build a web app.\n\n**4.**
\nBecause Tensorflowjs is a new technology, web apps bulit using it may not work in some browsers. The user will see a message saying the \"Ai is loading...\" but that message will never go away because the app is actually frozen. It's better to advise users to use the latest version of Chrome.\n\n"},{"metadata":{"_uuid":"eb92189bf34d85d3e5088803fc170297456da902"},"cell_type":"markdown","source":"\n### Resources\n\nI found these resources very helpful:\n\n1. Gabriel Preda, Honey Bee Subspecies Classification
\nhttps://www.kaggle.com/gpreda/honey-bee-subspecies-classification
\n\n2. Beluga, Black and White CNN
\nhttps://www.kaggle.com/gaborfodor/black-white-cnn-lb-0-77\n\n3. Francesco Marazzi, Baseline Keras CNN
\nhttps://www.kaggle.com/fmarazzi/baseline-keras-cnn-roc-fast-5min-0-8253-lb\n\n4. YouTube tutorial by deeplizard on how to use Tensorflowjs
\nhttps://www.youtube.com/watch?v=HEQDRWMK6yY\n\n5. Tutorial by jenkov.com explaining the HTML5 File API
\nhttp://tutorials.jenkov.com/html5/file-api.html\n\n6. Blog post by Anton Lavrenov on how to handle async/await
\nhttps://blog.lavrton.com/javascript-loops-how-to-handle-async-await-6252dd3c795\n\n7. jQuery tutorial by W3Schools
\nhttps://www.w3schools.com/jquery/default.asp\n"},{"metadata":{"_uuid":"2b21291a024b887cef5d17f324c5dc4503f2a0fb"},"cell_type":"markdown","source":"### Conclusion"},{"metadata":{"_uuid":"608870cf40209fac3b55b7af4c972293fd7c8b80"},"cell_type":"markdown","source":"All the html, css, and javascript code used to build the web app is available on Github. \n\nLive web app:
\nhttp://histo.test.woza.work/
\nGithub:
\nhttps://github.com/vbookshelf/Breast-Cancer-Analyzer\n\nThank you for reading."},{"metadata":{"trusted":true,"_uuid":"9b789103dec6faf61148e291c59f9c7f13d1344a"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}