{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Histopathologic Cancer Detection\n\nThis is my entry for the Kaggle playground competition on cancer detection. This is my second machine learning project and was motivated by my completion of Course 4 of the Deeplearning.ai specialisation on Coursera. Pandas, matplotlib, experimenting with different hyper-parameters and transfer learning are among the skills I have practiced during my time doing this.\n\n\n","metadata":{"_uuid":"8ff2f9de1d43b7e2c6d7e8e910913a67ebb0d475"}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport cv2\nimport os\nfrom glob import glob \nimport matplotlib.pyplot as plt\n\nimport random\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import roc_curve, auc, roc_auc_score\n\nfrom keras_preprocessing.image import ImageDataGenerator\n\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, BatchNormalization, Activation\nfrom keras.layers import Conv2D, MaxPool2D\n\nfrom keras.callbacks import EarlyStopping, ReduceLROnPlateau\n\nfrom IPython.display import clear_output\n","metadata":{"_uuid":"e028ae38502c4a4459355d33e079d7689c6f96d0","execution":{"iopub.status.busy":"2022-05-21T17:23:08.425339Z","iopub.execute_input":"2022-05-21T17:23:08.425605Z","iopub.status.idle":"2022-05-21T17:23:10.483927Z","shell.execute_reply.started":"2022-05-21T17:23:08.425559Z","shell.execute_reply":"2022-05-21T17:23:10.483215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Processing the data","metadata":{"_uuid":"a167e8a535edd325007febe644dc3b5875e5ec70"}},{"cell_type":"markdown","source":"Now it's time to import the data.\nWhat I want to do is:\n\n1. Load and display some positive and negative test examples\n2. Split the train data into train and dev sets","metadata":{"_uuid":"3889a45e5fd543405d51683e5953fb108ac6a500"}},{"cell_type":"code","source":"path = \"../input/\" \nlabels = pd.read_csv(path + 'train_labels.csv')\ntrain_path = path + 'train/'\ntest_path = path + 'test/'","metadata":{"_uuid":"350a0d71785d3aa17772d495a3d799080399f72f","execution":{"iopub.status.busy":"2022-05-21T17:23:10.485048Z","iopub.execute_input":"2022-05-21T17:23:10.485319Z","iopub.status.idle":"2022-05-21T17:23:10.832255Z","shell.execute_reply.started":"2022-05-21T17:23:10.485275Z","shell.execute_reply":"2022-05-21T17:23:10.831448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create a dataframe which contains every training examples path, id and label:","metadata":{"_uuid":"1d2ba183604216f4f58470837e474edbf72e5ed8"}},{"cell_type":"code","source":"df = pd.DataFrame({'path': glob(os.path.join(train_path,'*.tif'))})\ndf['id'] = df.path.map(lambda x: ((x.split(\"n\")[2].split('.')[0])[1:]))\ndf = df.merge(labels, on = \"id\")\ndf.head(3)","metadata":{"scrolled":true,"_uuid":"18b7b8e894af8283799d8b11ee5fe6693a4d5221","execution":{"iopub.status.busy":"2022-05-21T17:23:10.833405Z","iopub.execute_input":"2022-05-21T17:23:10.833660Z","iopub.status.idle":"2022-05-21T17:23:15.952635Z","shell.execute_reply.started":"2022-05-21T17:23:10.833618Z","shell.execute_reply":"2022-05-21T17:23:15.951906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Choose 4 random positve and negative examples, find their respective path then display them in a subplot:","metadata":{"_uuid":"b955187f2e82207066107721423275d5a2c4756e"}},{"cell_type":"code","source":"def readImage(path):\n    # OpenCV reads the image in bgr format by default\n    bgr_img = cv2.imread(path)\n    # We flip it to rgb for visualization purposes\n    b,g,r = cv2.split(bgr_img)\n    rgb_img = cv2.merge([r,g,b])\n    return rgb_img","metadata":{"_uuid":"1af05b00ea18b0281b8db8f05c84b9aec3d2b891","execution":{"iopub.status.busy":"2022-05-21T17:23:15.953792Z","iopub.execute_input":"2022-05-21T17:23:15.954208Z","iopub.status.idle":"2022-05-21T17:23:15.958660Z","shell.execute_reply.started":"2022-05-21T17:23:15.954159Z","shell.execute_reply":"2022-05-21T17:23:15.957946Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"positive_indices = list(np.where(df[\"label\"] == True)[0])\nnegative_indices = list(np.where(df[\"label\"] == False)[0])\nrand_pos_inds = random.sample(positive_indices, 4)\nrand_neg_inds = random.sample(negative_indices, 4)\n\nfig, ax = plt.subplots(2,4, figsize=(20,8))\nfig.suptitle('Histopathologic scans of lymph node sections',fontsize=20, fontweight='bold')\n\nfor i in range(0, 4):\n    ax[0,i].imshow(readImage(df.iloc[rand_pos_inds[i],0]))\n    ax[0,i].set_title(\"Positive Example\", fontweight='bold')\n    \n    ax[1,i].imshow(readImage(df.iloc[rand_neg_inds[i],0]))\n    ax[1,i].set_title(\"Negative Example\", fontweight='bold')","metadata":{"_uuid":"89ae9498a74f1919e42e62b6f4f0ba23e97aa217","execution":{"iopub.status.busy":"2022-05-21T17:23:15.959834Z","iopub.execute_input":"2022-05-21T17:23:15.960282Z","iopub.status.idle":"2022-05-21T17:23:17.846424Z","shell.execute_reply.started":"2022-05-21T17:23:15.960223Z","shell.execute_reply":"2022-05-21T17:23:17.845701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train/Validation Split and Loading the Data","metadata":{"_uuid":"b6aec2347b757e81761ec94caa8d89f35ab96af0"}},{"cell_type":"markdown","source":"Increasing the size of the image results in a much higher performance.","metadata":{}},{"cell_type":"code","source":"IMG_SIZE = 196\nBATCH_SIZE = 128","metadata":{"execution":{"iopub.status.busy":"2022-05-21T17:23:17.847394Z","iopub.execute_input":"2022-05-21T17:23:17.847639Z","iopub.status.idle":"2022-05-21T17:23:17.851806Z","shell.execute_reply.started":"2022-05-21T17:23:17.847598Z","shell.execute_reply":"2022-05-21T17:23:17.851074Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_list = os.listdir(test_path)\ntrain_list = os.listdir(train_path)\nprint(\"There are \" + str(len(train_list)) + \" training examples.\")\nprint(\"There are \" + str(len(test_list)) + \" test examples.\")","metadata":{"_uuid":"342a29282f3e38c1856937b74acdc31982918118","execution":{"iopub.status.busy":"2022-05-21T17:23:17.853002Z","iopub.execute_input":"2022-05-21T17:23:17.853454Z","iopub.status.idle":"2022-05-21T17:23:19.239757Z","shell.execute_reply.started":"2022-05-21T17:23:17.853399Z","shell.execute_reply":"2022-05-21T17:23:19.239079Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Going to split 20% of the training set into a validation set.","metadata":{"_uuid":"34a98be4485e3edf5fb085eb9282ad337f81bec2"}},{"cell_type":"code","source":"df['label'] = df['label'].astype(str)\ntrain, valid = train_test_split(df, test_size=0.2, stratify = df['label'])\n","metadata":{"_uuid":"fdd11fc9d5cdf331ccad0ee0baec066af7f81429","execution":{"iopub.status.busy":"2022-05-21T17:23:19.241139Z","iopub.execute_input":"2022-05-21T17:23:19.241446Z","iopub.status.idle":"2022-05-21T17:23:20.011932Z","shell.execute_reply.started":"2022-05-21T17:23:19.241378Z","shell.execute_reply":"2022-05-21T17:23:20.011183Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I originally thought about cropping the image to the central region but that proved to be ineffective.","metadata":{}},{"cell_type":"code","source":"def crop_centre(image, crop_length):\n    original_size = image.shape[0]\n    centre = original_size // 2\n    lower_bound = centre - crop_length // 2 \n    upper_bound = centre + crop_length // 2\n    image = image[(lower_bound):(upper_bound),(lower_bound):(upper_bound)]\n    return image","metadata":{"_uuid":"88c10cdf4e33a9b4d0fad96bf05eeb28b0675981","execution":{"iopub.status.busy":"2022-05-21T17:23:20.013177Z","iopub.execute_input":"2022-05-21T17:23:20.013451Z","iopub.status.idle":"2022-05-21T17:23:20.018813Z","shell.execute_reply.started":"2022-05-21T17:23:20.013403Z","shell.execute_reply":"2022-05-21T17:23:20.018102Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\ntrain_datagen = ImageDataGenerator(rescale=1./255,\n                                  vertical_flip = True,\n                                  horizontal_flip = True,\n                                  rotation_range=90,\n                                  zoom_range=0.2, \n                                  width_shift_range=0.1,\n                                  height_shift_range=0.1,\n                                  shear_range=0.05,\n                                  channel_shift_range=0.1)\n\ntest_datagen = ImageDataGenerator(rescale = 1./255) ","metadata":{"_uuid":"e6bef8f94cbe078da658cd9f39af33a2ee79f79e","execution":{"iopub.status.busy":"2022-05-21T17:23:20.020193Z","iopub.execute_input":"2022-05-21T17:23:20.020716Z","iopub.status.idle":"2022-05-21T17:23:20.028464Z","shell.execute_reply.started":"2022-05-21T17:23:20.020659Z","shell.execute_reply":"2022-05-21T17:23:20.027616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\ntrain_generator = train_datagen.flow_from_dataframe(dataframe = train, \n                                                    directory = None,\n                                                    x_col = 'path', \n                                                    y_col = 'label',\n                                                    target_size = (IMG_SIZE,IMG_SIZE),\n                                                    class_mode = \"binary\",\n                                                    batch_size=BATCH_SIZE,\n                                                    seed = 110318,\n                                                    shuffle = True)","metadata":{"_uuid":"adaeaf8dd2872dcf29d0905f70a80bd6221370b6","execution":{"iopub.status.busy":"2022-05-21T17:23:20.029817Z","iopub.execute_input":"2022-05-21T17:23:20.030344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"valid_generator = test_datagen.flow_from_dataframe(dataframe = valid,\n                                                   directory = None,\n                                                   x_col = 'path',\n                                                   y_col = 'label',\n                                                   target_size = (IMG_SIZE,IMG_SIZE),\n                                                   class_mode = 'binary',\n                                                   batch_size = BATCH_SIZE,\n                                                   shuffle = False)","metadata":{"_uuid":"3d1061fd15cdc493e2cf7a3e8cf2a0e87081ebcf","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Creating the model","metadata":{"_uuid":"a2823f65e4d1a782a2503abe67999ac3282e9ce7"}},{"cell_type":"code","source":"from keras.applications.resnet50 import ResNet50\n\ndropout_fc = 0.5\n\nconv_base = ResNet50(weights = 'imagenet', include_top = False, input_shape = (IMG_SIZE,IMG_SIZE,3))\n\nmy_model = Sequential()\n\nmy_model.add(conv_base)\nmy_model.add(Flatten())\nmy_model.add(Dense(256, use_bias=False))\nmy_model.add(BatchNormalization())\nmy_model.add(Activation(\"relu\"))\nmy_model.add(Dropout(dropout_fc))\nmy_model.add(Dense(1, activation = \"sigmoid\"))\n","metadata":{"_uuid":"ae37e0b06806be46190ea940f78c697ec8ea879a","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"my_model.summary()","metadata":{"_uuid":"fa05e6c5f7f0f6bb1ac6532ab8a85e21ffe5258b","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we're using ResNet50 trained on ImageNet, we're going to need to train the last few layers instead of the just the last one.  Cell images are quite different to what you see on ImageNet. ","metadata":{}},{"cell_type":"code","source":"conv_base.Trainable=True\n\nset_trainable=False\nfor layer in conv_base.layers:\n    if layer.name == 'res5a_branch2a':\n        set_trainable = True\n    if set_trainable:\n        layer.trainable = True\n    else:\n        layer.trainable = False","metadata":{"_uuid":"a3f35f658bfee3ae5c28a0ce7e02c5b3d7f6c0f2","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"_uuid":"a4758de74e465c8339643eb303cbf92814c1b37d","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from keras import optimizers\nmy_model.compile(optimizers.Adam(0.001), loss = \"binary_crossentropy\", metrics = [\"accuracy\"])","metadata":{"_uuid":"dbaebaa508beb5c08fca58cfe81229c1994a2310","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_step_size = train_generator.n // train_generator.batch_size\nvalid_step_size = valid_generator.n // valid_generator.batch_size","metadata":{"_uuid":"98783d7eb0b33810c14573c21726e109f2dae67b","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"earlystopper = EarlyStopping(monitor='val_loss', patience=3, verbose=2, restore_best_weights=True)\nreduce = ReduceLROnPlateau(monitor='val_loss', patience=1, verbose=1, factor=0.1)","metadata":{"_uuid":"7ad56b34ecc891b4c7038b94d169011725dee9fc","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = my_model.fit_generator(train_generator,\n                                     steps_per_epoch = train_step_size,\n                                     epochs = 10,\n                                     validation_data = valid_generator,\n                                     validation_steps = valid_step_size,\n                                     callbacks = [reduce, earlystopper],\n                                     verbose = 2)\n\n    ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Analysis\nNow that our model has been trained, it is time to plot some training graphs to see how our accuracies and losses varied over epochs.","metadata":{"_uuid":"97f6cfd9d1eb8a2d0255b4567b008e72fe39574f"}},{"cell_type":"code","source":"epochs = [i for i in range(1, len(history.history['loss'])+1)]\n\nplt.plot(epochs, history.history['loss'], color='blue', label=\"training_loss\")\nplt.plot(epochs, history.history['val_loss'], color='red', label=\"validation_loss\")\nplt.legend(loc='best')\nplt.title('training')\nplt.xlabel('epoch')\nplt.savefig(\"training.png\", bbox_inches='tight')\nplt.show()\n\nplt.plot(epochs, history.history['acc'], color='blue', label=\"training_accuracy\")\nplt.plot(epochs, history.history['val_acc'], color='red',label=\"validation_accuracy\")\nplt.legend(loc='best')\nplt.title('validation')\nplt.xlabel('epoch')\nplt.savefig(\"validation.png\", bbox_inches='tight')\nplt.show()","metadata":{"_uuid":"a3c0d4e6d5d75a88caf81243fb872781404c79a3","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"ROC Plot","metadata":{"_uuid":"96c5c295e3cde7c625c2e0a5fda45e103cdd71df"}},{"cell_type":"code","source":"roc_validation_generator = ImageDataGenerator(rescale=1./255).flow_from_dataframe(valid,\n                                                                                  x_col = 'path',\n                                                                                  y_col = 'label',\n                                                                                  target_size = (IMG_SIZE,IMG_SIZE),\n                                                                                  class_mode = 'binary',\n                                                                                  batch_size = BATCH_SIZE,\n                                                                                  shuffle = False)\npredictions = my_model.predict_generator(roc_validation_generator, steps=len(roc_validation_generator), verbose=2)\nfalse_positive_rate, true_positive_rate, threshold = roc_curve(roc_validation_generator.classes, predictions)\narea_under_curve = auc(false_positive_rate, true_positive_rate)\n\nplt.plot([0, 1], [0, 1], 'k--')\nplt.plot(false_positive_rate, true_positive_rate, label='AUC = {:.3f}'.format(area_under_curve))\nplt.xlabel('False positive rate')\nplt.ylabel('True positive rate')\nplt.title('ROC curve')\nplt.legend(loc='best')\nplt.savefig('ROC_PLOT.png', bbox_inches='tight')\nplt.show()","metadata":{"_uuid":"8c348e99651e82a3a971f87d5ede9f2beb46aae1","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"_uuid":"b97ca14759d26fbb1fda0afaf387dbc6aae0199c","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Predictions","metadata":{"_uuid":"3eaa72e5cba254cf43af52547233f43eb016fbc9"}},{"cell_type":"markdown","source":"For my predictions I'm going to use Test Time Augmentation. For each test image I will augment it 5 ways and average the prediction. I've also used ensemble learning by averaging the results of 3 versions of this model, due to this I was able to achieve my highest leaderboard score of 0.964.","metadata":{"_uuid":"5f39ab9a0d4e5840e7af853d7b1b0a2812a67932","trusted":true}},{"cell_type":"code","source":"testdf = pd.DataFrame({'path': glob(os.path.join(test_path, '*.tif'))})\ntestdf['id'] = testdf.path.map(lambda x: (x.split(\"/\")[3].split('.')[0]))\ntestdf.head(3)","metadata":{"_uuid":"8fdc51bac85542400e47a382b232551d7a5b9f35","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tta_datagen = ImageDataGenerator(rescale=1./255, #Normalise\n                                 vertical_flip = True,\n                                 horizontal_flip = True,\n                                 rotation_range=90,\n                                 zoom_range=0.2, \n                                 width_shift_range=0.1,\n                                 height_shift_range=0.1,\n                                 shear_range=0.05,\n                                 channel_shift_range=0.1)","metadata":{"_uuid":"b67eedd3c1b658814f75f0f1be7d1e24ec2213b0","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tta_steps = 5\nsubmission = pd.DataFrame()\nfor index in range(0, len(testdf)):\n    data_frame = pd.DataFrame({'path': testdf.iloc[index,0]}, index=[index])\n    data_frame['id'] = data_frame.path.map(lambda x: x.split('/')[3].split('.')[0])\n    img_path = data_frame.iloc[0,0]\n    test_img = cv2.imread(img_path)\n    test_img = cv2.resize(test_img,(IMG_SIZE,IMG_SIZE))\n    test_img = np.expand_dims(test_img, axis = 0)  \n    predictionsTTA = []\n    for i in range(0, tta_steps):\n        preds = my_model.predict_generator(tta_datagen.flow_from_dataframe(dataframe = data_frame,\n                                                                           directory = None,\n                                                                           x_col = 'path',\n                                                                           target_size = (IMG_SIZE, IMG_SIZE),\n                                                                           class_mode = None,\n                                                                           batch_size = 1,\n                                                                           shuffle = False), steps = 1)\n        predictionsTTA.append(preds)\n    clear_output()\n    prediction_entry = np.array(np.mean(predictionsTTA, axis=0))\n    data_frame['label'] = prediction_entry\n    submission = pd.concat([submission, data_frame[['id', 'label']]])\n    ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.set_index('id')\nsubmission.head(3)","metadata":{"_uuid":"4544440fde361922da366240c31f6bec42e62e52","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv', index=False, header=True)","metadata":{"_uuid":"67b4c89e80f6b3fd7ad2c720803ca043aeac8490","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# References\nHeavily inspired by these kernels:\n\n1. [https://www.kaggle.com/qitvision/a-complete-ml-pipeline-fast-ai](http://)\n\n2. [https://www.kaggle.com/fadhli/starter-code-keras-resnet50-0-9275-lb](http://)\n\n3. [https://www.kaggle.com/gomezp/complete-beginner-s-guide-eda-keras-lb-0-93](http://)\n\n4. [https://www.kaggle.com/greg115/histopathologic-cancer-detector-lb-0-958](http://)","metadata":{"_uuid":"a8b5425a6f5481951f7c1ca98dd82ab70aa6bd65"}}]}