{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# <center> CS 184A Final Project </center>\n## **Task**: RSNA MICCAI Brain Tumor Radiogenomic Classification Kaggle Competition [(link)](https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/overview)\n\n## **Team**:\n#### Pratyush Muthukumar, 66495041, muthukup@uci.edu\n#### Gabriella Vass, 17792084, gvass@uci.edu\n#### Ramya Sai Swathi Mangu, 50021174, rmangu@uci.edu\n\n\n## Introduction\nThe goal of our project is a binary classification task to classify brain tumor MRI imagery of glioblastoma patients into those with the MGMT promoter gene sequence and those without. The brain tumor MRI imagery is provided as 3D DICOM imagery of multi-parametric MRI (mpMRI) scans. Each patient's data comprised of 4 MRI sequences: \n- Fluid Attenuated Inversion Recovery (FLAIR)\n- T1-weighted pre-contrast (T1w)\n- T1-weighted post-contrast (T1Gd)\n- T2-weighted (T2)\n\nFor this Kaggle competition, the goal is to provide the softmax probabilities for the prevalence of the MGMT promoter gene sequence given binary labels and the MRI imagery. The competition then evaluates your results using the AUC error metric ranging from 0 - 1 where the optimal AUC score is 1. \n\nIn this notebook, we outline our data preprocessing/augmentation pipeline, the implementation of a baseline 2D CNN classifier model, the implementation of our custom fully-convolutional U-Net model, the results of both the models, and our final submission. This notebook provides the raw source code to all results and figures in our final report and video. \n\nWe ran this notebook on Kaggle using the hosted competition dataset and the GPU accelerator. If you are planning on running this notebook, we suggest you do the same.\n\n---\n\n## Data Preprocessing/Augmentation","metadata":{}},{"cell_type":"code","source":"# Imports\nimport os\nimport json\nimport glob\nimport random\nimport collections\n\nimport numpy as np\nimport pandas as pd\nimport pydicom\nfrom pydicom.pixel_data_handlers.util import apply_voi_lut\nimport cv2\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport random\nfrom tqdm.notebook import tqdm\n\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.model_selection import train_test_split\nfrom tensorflow.keras.utils import to_categorical\nfrom tensorflow.keras import layers","metadata":{"execution":{"iopub.status.busy":"2021-12-09T21:38:38.435262Z","iopub.execute_input":"2021-12-09T21:38:38.435806Z","iopub.status.idle":"2021-12-09T21:38:42.330664Z","shell.execute_reply.started":"2021-12-09T21:38:38.435718Z","shell.execute_reply":"2021-12-09T21:38:42.329798Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# MRI sequence types\nTYPES = [\"FLAIR\", \"T1w\", \"T2w\", \"T1wCE\"]\n# Augmentation mask pixel threshold parameter\nWHITE_THRESHOLD = 10 # out of 255\n# The Kaggle competition encourages us to exclude these samples because of inconsistencies\nEXCLUDE = [109, 123, 709]\n\ntrain_df = pd.read_csv(\"../input/rsna-miccai-brain-tumor-radiogenomic-classification/train_labels.csv\")\ntrain_df = train_df[~train_df.BraTS21ID.isin(EXCLUDE)]\ntest_df = pd.read_csv('../input/rsna-miccai-brain-tumor-radiogenomic-classification/sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2021-12-09T21:38:43.878534Z","iopub.execute_input":"2021-12-09T21:38:43.879427Z","iopub.status.idle":"2021-12-09T21:38:43.902075Z","shell.execute_reply.started":"2021-12-09T21:38:43.879378Z","shell.execute_reply":"2021-12-09T21:38:43.901385Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Data loading using the PyDicom Python package\ndef load_dicom(path, size = 224):\n    ''' \n    Reads a DICOM image, standardizes so that the pixel values are between 0 and 1, then rescales to 0 and 255\n    '''\n    dicom = pydicom.read_file(path)\n    data = dicom.pixel_array\n    if np.max(data) != 0:\n        data = data / np.max(data)\n    data = (data * 255).astype(np.uint8)\n    return cv2.resize(data, (size, size))","metadata":{"execution":{"iopub.status.busy":"2021-12-09T21:38:45.084789Z","iopub.execute_input":"2021-12-09T21:38:45.085069Z","iopub.status.idle":"2021-12-09T21:38:45.091407Z","shell.execute_reply.started":"2021-12-09T21:38:45.085040Z","shell.execute_reply":"2021-12-09T21:38:45.090334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_all_image_paths(brats21id, image_type, folder='train'): \n    '''\n    Returns an array of all the images of a particular type for a particular patient ID\n    '''\n    assert(image_type in TYPES)\n    \n    patient_path = os.path.join(\n        \"../input/rsna-miccai-brain-tumor-radiogenomic-classification/%s/\" % folder, \n        str(brats21id).zfill(5),\n    )\n\n    paths = sorted(\n        glob.glob(os.path.join(patient_path, image_type, \"*\")), \n        key=lambda x: int(x[:-4].split(\"-\")[-1]),\n    )\n    \n    num_images = len(paths)\n    \n    start = int(num_images * 0.25)\n    end = int(num_images * 0.75)\n\n    interval = 3\n    \n    if num_images < 10: \n        interval = 1\n    \n    return np.array(paths[start:end:interval])","metadata":{"execution":{"iopub.status.busy":"2021-12-09T21:38:45.951604Z","iopub.execute_input":"2021-12-09T21:38:45.952106Z","iopub.status.idle":"2021-12-09T21:38:45.959110Z","shell.execute_reply.started":"2021-12-09T21:38:45.952073Z","shell.execute_reply":"2021-12-09T21:38:45.958100Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_all_images(brats21id, image_type, folder='train', size=225):\n    return [load_dicom(path, size) for path in get_all_image_paths(brats21id, image_type, folder)]\nIMAGE_SIZE = 128\n\n# Creating training data\ndef get_all_data_for_train(image_type):\n    global train_df\n    \n    X = []\n    y = []\n    train_ids = []\n\n    for i in tqdm(train_df.index):\n        x = train_df.loc[i]\n        images = get_all_images(int(x['BraTS21ID']), image_type, 'train', IMAGE_SIZE)\n        label = x['MGMT_value']\n\n        X += images\n        y += [label] * len(images)\n        train_ids += [int(x['BraTS21ID'])] * len(images)\n        assert(len(X) == len(y))\n    return np.array(X), np.array(y), np.array(train_ids)\n\n# Creating testing data\ndef get_all_data_for_test(image_type):\n    global test_df\n    \n    X = []\n    test_ids = []\n\n    for i in tqdm(test_df.index):\n        x = test_df.loc[i]\n        images = get_all_images(int(x['BraTS21ID']), image_type, 'test', IMAGE_SIZE)\n        X += images\n        test_ids += [int(x['BraTS21ID'])] * len(images)\n\n    return np.array(X), np.array(test_ids)\nX_t1p, y_t1p, trainidt_t1p = get_all_data_for_train('T1wCE')\nX_test_t1p, testidt_t1p = get_all_data_for_test('T1wCE')\n\nX_flair, y_flair, trainidt_flair = get_all_data_for_train('FLAIR')\nX_test_flair, testidt_flair = get_all_data_for_test('FLAIR')\n\nX_t1, y_t1, trainidt_t1 = get_all_data_for_train('T1w')\nX_test_t1, testidt_t1 = get_all_data_for_test('T1w')\n\nX_t2, y_t2, trainidt_t2 = get_all_data_for_train('T2w')\nX_test_t2, testidt_t2 = get_all_data_for_test('T2w')","metadata":{"execution":{"iopub.status.busy":"2021-12-09T21:38:46.682639Z","iopub.execute_input":"2021-12-09T21:38:46.682875Z","iopub.status.idle":"2021-12-09T21:48:52.174313Z","shell.execute_reply.started":"2021-12-09T21:38:46.682850Z","shell.execute_reply":"2021-12-09T21:48:52.173562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For each MRI sequence, there are only 582 samples for training and 87 samples for testing, so clearly we need to do some data augmentation to generate more features or else our models won't be able to learn.","metadata":{}},{"cell_type":"code","source":"# Keras Data Augmentation Layer\n# Perform random flips and random rotations on data to generate more features\nglobal_average_layer = tf.keras.layers.GlobalAveragePooling2D()\ndata_augmentation = tf.keras.Sequential([\n  tf.keras.layers.experimental.preprocessing.RandomFlip('horizontal'),\n  tf.keras.layers.experimental.preprocessing.RandomRotation(0.2),\n])","metadata":{"execution":{"iopub.status.busy":"2021-12-09T21:49:11.983464Z","iopub.execute_input":"2021-12-09T21:49:11.983954Z","iopub.status.idle":"2021-12-09T21:49:12.000464Z","shell.execute_reply.started":"2021-12-09T21:49:11.983915Z","shell.execute_reply":"2021-12-09T21:49:11.999678Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Split into training and valid\nX_train, X_valid, y_train, y_valid, trainidt_train, trainidt_valid = train_test_split(X_t1p, y_t1p, trainidt_t1p, test_size=0.2, random_state=40)\n\nX_train = tf.expand_dims(X_train, axis=-1)\nX_valid = tf.expand_dims(X_valid, axis=-1)\n\ny_train = to_categorical(y_train)\ny_valid = to_categorical(y_valid)","metadata":{"execution":{"iopub.status.busy":"2021-12-09T21:49:13.031045Z","iopub.execute_input":"2021-12-09T21:49:13.031705Z","iopub.status.idle":"2021-12-09T21:49:13.572395Z","shell.execute_reply.started":"2021-12-09T21:49:13.031670Z","shell.execute_reply":"2021-12-09T21:49:13.571627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"X_train\", X_train.shape)\nprint(\"y_train\", y_train.shape)\n\nprint(\"X_valid\", X_valid.shape)\nprint(\"y_valid\", y_valid.shape)","metadata":{"execution":{"iopub.status.busy":"2021-12-09T21:49:22.487345Z","iopub.execute_input":"2021-12-09T21:49:22.487658Z","iopub.status.idle":"2021-12-09T21:49:22.497538Z","shell.execute_reply.started":"2021-12-09T21:49:22.487624Z","shell.execute_reply":"2021-12-09T21:49:22.496583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Through augmenting and preprocessing the DICOM data, we have created 12956 training samples and 3240 validation samples of 128x128 images for each of the 4 MRI sequences. The labels are binary matrices for the binary classification labels of each patient's MRI data.","metadata":{}},{"cell_type":"code","source":"# Visualization of processed dataset\nfig, (ax1, ax2, ax3, ax4) = plt.subplots(1, 4)\nfig.suptitle('No MGMT Promoter (y=0)')\nfig.set_size_inches(18, 4)\nax1.imshow(X_t1p[-1,:,:], cmap=\"gray\")\nax1.set_title(\"T1 Pre-contrast\")\nax2.imshow(X_t1[-1,:,:], cmap=\"gray\")\nax2.set_title(\"T1 Post-contrast\")\nax3.imshow(X_t2[-1,:,:], cmap=\"gray\")\nax3.set_title(\"T2\")\nax4.imshow(X_flair[-1,:,:], cmap=\"gray\")\nax4.set_title(\"FLAIR\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-09T21:49:28.353436Z","iopub.execute_input":"2021-12-09T21:49:28.354335Z","iopub.status.idle":"2021-12-09T21:49:29.101490Z","shell.execute_reply.started":"2021-12-09T21:49:28.354288Z","shell.execute_reply":"2021-12-09T21:49:29.100690Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, (ax1, ax2, ax3, ax4) = plt.subplots(1, 4)\nfig.suptitle('MGMT Promoter (y=1)')\nfig.set_size_inches(18, 4)\nax1.imshow(X_t1p[0,:,:], cmap=\"gray\")\nax1.set_title(\"T1 Pre-contrast\")\nax2.imshow(X_t1[0,:,:], cmap=\"gray\")\nax2.set_title(\"T1 Post-contrast\")\nax3.imshow(X_t2[0,:,:], cmap=\"gray\")\nax3.set_title(\"T2\")\nax4.imshow(X_flair[0,:,:], cmap=\"gray\")\nax4.set_title(\"FLAIR\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-09T21:49:31.622518Z","iopub.execute_input":"2021-12-09T21:49:31.623075Z","iopub.status.idle":"2021-12-09T21:49:32.373808Z","shell.execute_reply.started":"2021-12-09T21:49:31.623037Z","shell.execute_reply":"2021-12-09T21:49:32.373086Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2D CNN Baseline\n\nHere' we utilize the [DenseNet121](http://https://arxiv.org/abs/1608.06993) architecture, a pretrained dense CNN model, as a baseline model. Apart from using this pre-trained model, we train this model as a standard CNN classifier model using tensorflow and keras. ","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras import Input, Model \nfrom tensorflow.keras.layers import Conv2D, GlobalAveragePooling2D, Dense\nfrom tensorflow.keras.applications import *\n\ninput_tensor = keras.Input(shape=(128, 128, 1))\nefnet = DenseNet121(weights=None, include_top = False, input_shape=(128, 128, 3))\nmapping3feat = Conv2D(3, (3, 3), padding='same', use_bias=False)(input_tensor)\n\noutput = efnet(mapping3feat)\noutput = GlobalAveragePooling2D()(output)\noutput = Dense(2, activation='sigmoid')(output)\n\nprint(output.shape)\n\ntf.keras.backend.clear_session()\nmodel = Model(input_tensor, output)\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2021-12-09T21:49:39.298863Z","iopub.execute_input":"2021-12-09T21:49:39.299138Z","iopub.status.idle":"2021-12-09T21:49:41.998127Z","shell.execute_reply.started":"2021-12-09T21:49:39.299109Z","shell.execute_reply":"2021-12-09T21:49:41.997404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Cross entropy loss\n# Also show the training/validation AUC throughout training\nmodel.compile(loss='categorical_crossentropy',\n             optimizer=tf.keras.optimizers.SGD(learning_rate =0.0001),\n             metrics=[tf.keras.metrics.AUC()])\n\nhistory = model.fit(x=X_train, y = y_train, epochs=25, validation_data= (X_valid, y_valid))","metadata":{"execution":{"iopub.status.busy":"2021-12-09T21:49:54.550721Z","iopub.execute_input":"2021-12-09T21:49:54.551163Z","iopub.status.idle":"2021-12-09T22:04:50.743419Z","shell.execute_reply.started":"2021-12-09T21:49:54.551126Z","shell.execute_reply":"2021-12-09T22:04:50.742665Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plotting training loss\nplt.plot(history.history['loss'])\nplt.plot(history.history['val_loss'])\nplt.title('model loss')\nplt.ylabel('loss')\nplt.xlabel('epoch')\nplt.legend(['train', 'validation'], loc='upper left')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-09T22:06:38.226594Z","iopub.execute_input":"2021-12-09T22:06:38.227342Z","iopub.status.idle":"2021-12-09T22:06:38.429637Z","shell.execute_reply.started":"2021-12-09T22:06:38.227303Z","shell.execute_reply":"2021-12-09T22:06:38.428936Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plotting AUC score\nplt.plot(history.history['auc_1'])\nplt.plot(history.history['val_auc_1'])\nplt.title('Model accuracy')\nplt.ylabel('AUC')\nplt.xlabel('epoch')\nplt.legend(['train', 'validation'], loc='upper left')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-09T22:06:53.921773Z","iopub.execute_input":"2021-12-09T22:06:53.922380Z","iopub.status.idle":"2021-12-09T22:06:54.130893Z","shell.execute_reply.started":"2021-12-09T22:06:53.922339Z","shell.execute_reply":"2021-12-09T22:06:54.130111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create submission file\ny_pred=model.predict(X_test_t1p)\npred = np.argmax(y_pred, axis=1)\nresult=pd.DataFrame(testidt_t1p)\nresult[1] = pred\nresult.columns=['BraTS21ID','MGMT_value']\nresult2 = result.groupby('BraTS21ID',as_index=False).mean()\nsample = pd.read_csv('../input/rsna-miccai-brain-tumor-radiogenomic-classification/sample_submission.csv')\nresult2['BraTS21ID'] = sample['BraTS21ID']\nresult2.to_csv('cnn_submission.csv',index=False)","metadata":{"execution":{"iopub.status.busy":"2021-12-09T22:19:13.558775Z","iopub.execute_input":"2021-12-09T22:19:13.559504Z","iopub.status.idle":"2021-12-09T22:19:15.293155Z","shell.execute_reply.started":"2021-12-09T22:19:13.559462Z","shell.execute_reply":"2021-12-09T22:19:15.292316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result2.head()","metadata":{"execution":{"iopub.status.busy":"2021-12-09T22:19:20.463602Z","iopub.execute_input":"2021-12-09T22:19:20.464401Z","iopub.status.idle":"2021-12-09T22:19:20.475186Z","shell.execute_reply.started":"2021-12-09T22:19:20.464350Z","shell.execute_reply":"2021-12-09T22:19:20.474215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## U-Net Implementation\n\nHere, we implement a fully-convolutional U-Net architecture with 4 contracting Conv2D layers, 4 expanding Conv2D layers, and 1 bottleneck layer. A modification we implemented is the residual and convolutional connections implemented in place of the skip connections connecting each pair of layers in the expanding arm. Instead of the concat operation, our custom U-Net model implements residual connection followed by a strided convolution operation for each skip connection, as proposed by the famous [ResNet](https://arxiv.org/abs/1512.03385) architecture. ","metadata":{}},{"cell_type":"code","source":"# Fully-convolutional U-net with 4 contracting, 4 expanding arms and residual skip connections\nfrom tensorflow.keras.applications import *\nimport os, numpy as np, pandas as pd, matplotlib.pyplot as plt\nfrom tensorflow import losses, optimizers\nfrom tensorflow.keras import Input, Model, models, layers, callbacks, utils, metrics\n\n# --- Define kwargs dictionary\nkwargs = {\n    'kernel_size': (3, 3),\n    'padding': 'same'}\n\n# --- Define lambda functions\nconv = lambda x, filters, strides : layers.Conv2D(filters=filters, strides=strides, **kwargs)(x)\nnorm = lambda x : layers.BatchNormalization()(x)\nrelu = lambda x : layers.ReLU()(x)\ntran = lambda x, filters, strides : layers.Conv2DTranspose(filters=filters, strides=strides, **kwargs)(x)\n\n# --- Define stride-1, stride-2 blocks\nconv1 = lambda filters, x : relu(norm(conv(x, filters, strides=1)))\nconv2 = lambda filters, x : relu(norm(conv(x, filters, strides=2)))\ntran2 = lambda filters, x : relu(norm(tran(x, filters, strides=2)))","metadata":{"execution":{"iopub.status.busy":"2021-12-09T22:19:22.570586Z","iopub.execute_input":"2021-12-09T22:19:22.571519Z","iopub.status.idle":"2021-12-09T22:19:22.581004Z","shell.execute_reply.started":"2021-12-09T22:19:22.571467Z","shell.execute_reply":"2021-12-09T22:19:22.580201Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# --- Define contracting layers\nl1 = conv1(32, input_tensor) # 128x128x32\nl2 = conv1(48, norm(conv2(48, l1))) #64x64x48\nl3 = conv1(64, norm(conv2(64, l2))) #32x32x64\nl4 = conv1(128, norm(conv2(128, l3))) #16x16x128\n# --- Bottleneck layer\nl5 = conv1(256, norm(conv2(256, l4))) # 8x8x256\n# --- Define expanding layers\nl6  = tran2(128, l5)# 16x16x128\nl7  = tran2(64, norm(conv1(128, l4+l6)))#32x32x64\nl8  = tran2(48, norm(conv1(64, l3+l7)))#64x64x48\nl9 = tran2(32,  norm(conv1(48, l2+l8)))#128x128x32\n\n# --- Define survival prediction\nh0 = layers.Dropout(0.3)(l9)\nh1 = layers.Flatten()(h0)\nh2 = layers.Dense(128, activation='relu')(h1)\noutput = layers.Dense(2, activation='sigmoid')(h2)\n\n# --- Create model\ntf.keras.backend.clear_session()\nmodel = Model(inputs=input_tensor, outputs=output)\n\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2021-12-09T22:19:23.244437Z","iopub.execute_input":"2021-12-09T22:19:23.245053Z","iopub.status.idle":"2021-12-09T22:19:23.668039Z","shell.execute_reply.started":"2021-12-09T22:19:23.245010Z","shell.execute_reply":"2021-12-09T22:19:23.667330Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see from the model summary, our custom U-Net model is more complex than the baseline 2D CNN model with nearly 10x the amount of trainable parameters.","metadata":{}},{"cell_type":"code","source":"# --- Compile model\nmodel.compile(loss='categorical_crossentropy',\n             optimizer=tf.keras.optimizers.SGD(learning_rate=0.0001),\n             metrics=[tf.keras.metrics.AUC()])\n# --- Train model\nhistory = model.fit(x=X_train, y = y_train, epochs=25, validation_data= (X_valid, y_valid))","metadata":{"execution":{"iopub.status.busy":"2021-12-09T22:20:02.153267Z","iopub.execute_input":"2021-12-09T22:20:02.154061Z","iopub.status.idle":"2021-12-09T22:30:49.403838Z","shell.execute_reply.started":"2021-12-09T22:20:02.154017Z","shell.execute_reply":"2021-12-09T22:30:49.403056Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plotting training loss\nplt.plot(history.history['loss'])\nplt.plot(history.history['val_loss'])\nplt.title('model loss')\nplt.ylabel('loss')\nplt.xlabel('epoch')\nplt.legend(['train', 'validation'], loc='upper left')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-09T22:30:52.047909Z","iopub.execute_input":"2021-12-09T22:30:52.049416Z","iopub.status.idle":"2021-12-09T22:30:52.278781Z","shell.execute_reply.started":"2021-12-09T22:30:52.049370Z","shell.execute_reply":"2021-12-09T22:30:52.278118Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plotting AUC score\nplt.plot(history.history['auc_1'])\nplt.plot(history.history['val_auc_1'])\nplt.title('Model accuracy')\nplt.ylabel('AUC')\nplt.xlabel('epoch')\nplt.legend(['train', 'validation'], loc='upper left')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-09T22:31:02.803600Z","iopub.execute_input":"2021-12-09T22:31:02.803863Z","iopub.status.idle":"2021-12-09T22:31:02.991877Z","shell.execute_reply.started":"2021-12-09T22:31:02.803835Z","shell.execute_reply":"2021-12-09T22:31:02.991126Z"},"trusted":true},"execution_count":null,"outputs":[]}]}