{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# CNN Cancer Detection Kaggle Mini Project\n\nThe goal of this Kaggle competition, is to create an algorithm to identify metastatic cancer \nin small image patches taken from larger digital pathology scans. \nThe data for this competition are 96x96 pixel images separated in both training and test sets.\n\n## Introduction: Step 1\n    \nWe will see how many training and test samples we have as well as verify the shapes of the samples are\n96x96x3 (96x96 pixels, 3 colors for RGB).  Also, as this is a binary classification problem, I will\nwant to make sure the classes are balanced so as not to introduce any bias into the predictions","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"#import necessary packages and training labels\nimport numpy as np\nimport pandas as pd\n\ntrain_labels = pd.read_csv('../input/histopathologic-cancer-detection/train_labels.csv', dtype=str)\nprint(train_labels.shape)","metadata":{"execution":{"iopub.status.busy":"2022-05-23T00:05:09.074510Z","iopub.execute_input":"2022-05-23T00:05:09.075630Z","iopub.status.idle":"2022-05-23T00:05:09.528439Z","shell.execute_reply.started":"2022-05-23T00:05:09.075519Z","shell.execute_reply":"2022-05-23T00:05:09.527380Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-23T00:05:09.530557Z","iopub.execute_input":"2022-05-23T00:05:09.531726Z","iopub.status.idle":"2022-05-23T00:05:09.558595Z","shell.execute_reply.started":"2022-05-23T00:05:09.531656Z","shell.execute_reply":"2022-05-23T00:05:09.557840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels.dtypes","metadata":{"execution":{"iopub.status.busy":"2022-05-23T00:05:09.559871Z","iopub.execute_input":"2022-05-23T00:05:09.560271Z","iopub.status.idle":"2022-05-23T00:05:09.567587Z","shell.execute_reply.started":"2022-05-23T00:05:09.560237Z","shell.execute_reply":"2022-05-23T00:05:09.566849Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels['label'] = train_labels['label'].astype(float)","metadata":{"execution":{"iopub.status.busy":"2022-05-23T00:05:09.569657Z","iopub.execute_input":"2022-05-23T00:05:09.570075Z","iopub.status.idle":"2022-05-23T00:05:09.632848Z","shell.execute_reply.started":"2022-05-23T00:05:09.570044Z","shell.execute_reply":"2022-05-23T00:05:09.631868Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nprint(len(os.listdir('../input/histopathologic-cancer-detection/train/')))\nprint(len(os.listdir('../input/histopathologic-cancer-detection/test/')))","metadata":{"execution":{"iopub.status.busy":"2022-05-23T00:05:09.634613Z","iopub.execute_input":"2022-05-23T00:05:09.634981Z","iopub.status.idle":"2022-05-23T00:05:14.829384Z","shell.execute_reply.started":"2022-05-23T00:05:09.634940Z","shell.execute_reply":"2022-05-23T00:05:14.828145Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So, we see we have 220,025 training samples and 57,458 test samples.","metadata":{}},{"cell_type":"code","source":"len(train_labels)","metadata":{"execution":{"iopub.status.busy":"2022-05-23T00:05:14.830691Z","iopub.execute_input":"2022-05-23T00:05:14.830992Z","iopub.status.idle":"2022-05-23T00:05:14.838336Z","shell.execute_reply.started":"2022-05-23T00:05:14.830960Z","shell.execute_reply":"2022-05-23T00:05:14.837196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Step 2: Exploratory Data Analysis\n\nI will now look to do any data cleaning and figure out my plan of how to approach this task.  Initially I will want to determine if the classes in the training set are unbalanced.","metadata":{}},{"cell_type":"code","source":"train_labels['label'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-05-23T00:05:14.840117Z","iopub.execute_input":"2022-05-23T00:05:14.841115Z","iopub.status.idle":"2022-05-23T00:05:14.870987Z","shell.execute_reply.started":"2022-05-23T00:05:14.841051Z","shell.execute_reply":"2022-05-23T00:05:14.869988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels['label'].value_counts().plot(kind='pie')","metadata":{"execution":{"iopub.status.busy":"2022-05-23T00:05:14.872900Z","iopub.execute_input":"2022-05-23T00:05:14.873387Z","iopub.status.idle":"2022-05-23T00:05:15.075667Z","shell.execute_reply.started":"2022-05-23T00:05:14.873309Z","shell.execute_reply":"2022-05-23T00:05:15.074251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There is a class imbalance in the training data with significantly more non-cancerous images (label '0') versus cancerous (label '1').  I will reduce the size of label 0 by random undersampling of that set in order to match the same size as the set of images with label 1.","metadata":{}},{"cell_type":"code","source":"#split into two sets based on labels\ntrain_labels_pos = train_labels[train_labels['label']==1]\ntrain_labels_neg = train_labels[train_labels['label']==0]","metadata":{"execution":{"iopub.status.busy":"2022-05-23T00:05:15.077733Z","iopub.execute_input":"2022-05-23T00:05:15.078270Z","iopub.status.idle":"2022-05-23T00:05:15.118740Z","shell.execute_reply.started":"2022-05-23T00:05:15.078210Z","shell.execute_reply":"2022-05-23T00:05:15.117470Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#take a random sample of the neg labels of the same size as the set of pos labels\ntrain_labels_neg = train_labels_neg.sample(n = train_labels_pos.shape[0])","metadata":{"execution":{"iopub.status.busy":"2022-05-23T00:05:15.123160Z","iopub.execute_input":"2022-05-23T00:05:15.124073Z","iopub.status.idle":"2022-05-23T00:05:15.152433Z","shell.execute_reply.started":"2022-05-23T00:05:15.124008Z","shell.execute_reply":"2022-05-23T00:05:15.151797Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#confirm both sets are of the same size\nprint(train_labels_neg.shape[0])\nprint(train_labels_pos.shape[0])","metadata":{"execution":{"iopub.status.busy":"2022-05-23T00:05:15.153366Z","iopub.execute_input":"2022-05-23T00:05:15.154186Z","iopub.status.idle":"2022-05-23T00:05:15.159349Z","shell.execute_reply.started":"2022-05-23T00:05:15.154147Z","shell.execute_reply":"2022-05-23T00:05:15.158185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#combine and randomize the two sets\ntrain_labels_balanced = pd.concat([train_labels_neg,train_labels_pos]).sample(frac=1, random_state=12345).reset_index(drop=True)\ntrain_labels_balanced.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-23T00:05:15.160830Z","iopub.execute_input":"2022-05-23T00:05:15.161377Z","iopub.status.idle":"2022-05-23T00:05:15.334290Z","shell.execute_reply.started":"2022-05-23T00:05:15.161336Z","shell.execute_reply":"2022-05-23T00:05:15.333268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#confirm final set has the expected amount and shape\ntrain_labels_balanced.shape","metadata":{"execution":{"iopub.status.busy":"2022-05-23T00:05:15.335694Z","iopub.execute_input":"2022-05-23T00:05:15.335983Z","iopub.status.idle":"2022-05-23T00:05:15.342154Z","shell.execute_reply.started":"2022-05-23T00:05:15.335950Z","shell.execute_reply":"2022-05-23T00:05:15.341132Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#confirm final set has the expected value counts\ntrain_labels_balanced['label'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-05-23T00:05:15.344098Z","iopub.execute_input":"2022-05-23T00:05:15.344492Z","iopub.status.idle":"2022-05-23T00:05:15.362453Z","shell.execute_reply.started":"2022-05-23T00:05:15.344447Z","shell.execute_reply":"2022-05-23T00:05:15.361822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels_balanced['label'].value_counts().plot(kind='pie')","metadata":{"execution":{"iopub.status.busy":"2022-05-23T00:05:15.363826Z","iopub.execute_input":"2022-05-23T00:05:15.364790Z","iopub.status.idle":"2022-05-23T00:05:15.456545Z","shell.execute_reply.started":"2022-05-23T00:05:15.364711Z","shell.execute_reply":"2022-05-23T00:05:15.455528Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now I will look at a several images to see what we are actually trying to classif and confirm they are of the expected size","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\n\nimg = mpimg.imread(f'../input/histopathologic-cancer-detection/train/{train_labels_balanced.iloc[47,0]}.tif')\nimgplot = plt.imshow(img)","metadata":{"execution":{"iopub.status.busy":"2022-05-23T00:05:15.458468Z","iopub.execute_input":"2022-05-23T00:05:15.459127Z","iopub.status.idle":"2022-05-23T00:05:15.686547Z","shell.execute_reply.started":"2022-05-23T00:05:15.459070Z","shell.execute_reply":"2022-05-23T00:05:15.685555Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(img.shape)","metadata":{"execution":{"iopub.status.busy":"2022-05-23T00:05:15.688358Z","iopub.execute_input":"2022-05-23T00:05:15.688883Z","iopub.status.idle":"2022-05-23T00:05:15.693050Z","shell.execute_reply.started":"2022-05-23T00:05:15.688843Z","shell.execute_reply":"2022-05-23T00:05:15.692243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_imgs = np.random.choice(train_labels_balanced.index,15)\n","metadata":{"execution":{"iopub.status.busy":"2022-05-23T00:05:15.694800Z","iopub.execute_input":"2022-05-23T00:05:15.695325Z","iopub.status.idle":"2022-05-23T00:05:15.710348Z","shell.execute_reply.started":"2022-05-23T00:05:15.695279Z","shell.execute_reply":"2022-05-23T00:05:15.708921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(5, 3,figsize=(20,20))\n\nfor i in range(0, sample_imgs.shape[0]):\n    ax = plt.subplot(5, 3, i+1)\n    img = mpimg.imread(f'../input/histopathologic-cancer-detection/train/{train_labels_balanced.iloc[sample_imgs[i],0]}.tif')\n    ax.imshow(img)\n    lab = train_labels_balanced.iloc[sample_imgs[i],1]\n    ax.set_title('Label: %s'%lab)\n    \nplt.tight_layout()","metadata":{"execution":{"iopub.status.busy":"2022-05-23T00:05:15.711949Z","iopub.execute_input":"2022-05-23T00:05:15.712658Z","iopub.status.idle":"2022-05-23T00:05:18.253094Z","shell.execute_reply.started":"2022-05-23T00:05:15.712607Z","shell.execute_reply":"2022-05-23T00:05:18.251612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So, we can see some images are vastly different from others, and with no medical training I cant possibly tell which images are indicative of cancer and which arent.  Hopefully, with deep learning, the model will be able to!\n\n## Step 3: Model Architecture\n\nI will initially attempt to use a model architecture based on the VGGNet model.  This utilizes blocks of convulutions in a [Conv-Conv-MaxPool] set repeated n times.  This is a fairly simple architecture for a beginner like my self to develop and understand so I will stick with that for this assignment.\n\nI will try a couple of different block sizes then look to further tune other hyperparameters such as optimization methods and activation functions.\n\nFor the first step, I will split the training set into a training subset and a validation subset.  I will use these same subsets throughout this process in order to remain consistent.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split","metadata":{"execution":{"iopub.status.busy":"2022-05-23T00:05:18.254534Z","iopub.execute_input":"2022-05-23T00:05:18.255245Z","iopub.status.idle":"2022-05-23T00:05:19.513167Z","shell.execute_reply.started":"2022-05-23T00:05:18.255206Z","shell.execute_reply":"2022-05-23T00:05:19.512111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df, valid_df = train_test_split(train_labels_balanced, test_size=0.25, random_state=1234, stratify=train_labels_balanced.label)","metadata":{"execution":{"iopub.status.busy":"2022-05-23T00:05:19.515082Z","iopub.execute_input":"2022-05-23T00:05:19.515445Z","iopub.status.idle":"2022-05-23T00:05:19.624725Z","shell.execute_reply.started":"2022-05-23T00:05:19.515396Z","shell.execute_reply":"2022-05-23T00:05:19.623790Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#import tensorflow and keras as well as any necessary packages\nimport tensorflow as tf\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow import keras\nfrom keras.layers import Dense, Activation, Flatten, Dropout, BatchNormalization\nfrom keras.layers import Conv2D, MaxPooling2D\nfrom keras import regularizers, optimizers\nfrom keras.layers import PReLU\nfrom keras.initializers import Constant\n\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator","metadata":{"execution":{"iopub.status.busy":"2022-05-23T00:05:19.626373Z","iopub.execute_input":"2022-05-23T00:05:19.626631Z","iopub.status.idle":"2022-05-23T00:05:26.714734Z","shell.execute_reply.started":"2022-05-23T00:05:19.626595Z","shell.execute_reply":"2022-05-23T00:05:26.712875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df['id'] = train_df['id']+'.tif'\nvalid_df['id'] = valid_df['id']+'.tif'","metadata":{"execution":{"iopub.status.busy":"2022-05-23T00:05:26.717194Z","iopub.execute_input":"2022-05-23T00:05:26.717535Z","iopub.status.idle":"2022-05-23T00:05:26.790671Z","shell.execute_reply.started":"2022-05-23T00:05:26.717493Z","shell.execute_reply":"2022-05-23T00:05:26.789576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df['label'] = train_df['label'].astype(str)\nvalid_df['label'] = valid_df['label'].astype(str)","metadata":{"execution":{"iopub.status.busy":"2022-05-23T00:05:26.792164Z","iopub.execute_input":"2022-05-23T00:05:26.792409Z","iopub.status.idle":"2022-05-23T00:05:26.911605Z","shell.execute_reply.started":"2022-05-23T00:05:26.792382Z","shell.execute_reply":"2022-05-23T00:05:26.910618Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#create the training and validation subsets\ntrain_datagen=ImageDataGenerator(rescale=1/255)\n\ntrain_generator=train_datagen.flow_from_dataframe(dataframe=train_df,directory=\"../input/histopathologic-cancer-detection/train/\",\n                x_col=\"id\",y_col=\"label\",batch_size=64,seed=1234,shuffle=True,\n                class_mode=\"binary\",target_size=(96,96))\n\nvalid_generator=train_datagen.flow_from_dataframe(dataframe=valid_df,directory=\"../input/histopathologic-cancer-detection/train/\",\n                x_col=\"id\",y_col=\"label\",batch_size=64,seed=1234,shuffle=True,\n                class_mode=\"binary\",target_size=(96,96))\n","metadata":{"execution":{"iopub.status.busy":"2022-05-23T00:05:26.913247Z","iopub.execute_input":"2022-05-23T00:05:26.913498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n#initial model with 4 sets of 2 convolutional layers\nmodel = Sequential()\nmodel.add(Conv2D(32, (3, 3), padding='same',input_shape=(96,96,3)))\nmodel.add(Activation('relu'))\nmodel.add(Conv2D(32, (3, 3)))\nmodel.add(Activation('relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\nmodel.add(BatchNormalization())\n\nmodel.add(Conv2D(64, (3, 3)))\nmodel.add(Activation('relu'))\nmodel.add(Conv2D(64, (3, 3)))\nmodel.add(Activation('relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\nmodel.add(BatchNormalization())\n\nmodel.add(Conv2D(128, (3, 3)))\nmodel.add(Activation('relu'))\nmodel.add(Conv2D(128, (3, 3)))\nmodel.add(Activation('relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\nmodel.add(BatchNormalization())\n\nmodel.add(Conv2D(256, (3, 3)))\nmodel.add(Activation('relu'))\nmodel.add(Conv2D(256, (3, 3)))\nmodel.add(Activation('relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\nmodel.add(BatchNormalization())\n          \nmodel.add(Flatten())\nmodel.add(Dropout(0.25))\nmodel.add(Dense(512))\nmodel.add(Activation('relu'))\n\nmodel.add(Dropout(0.25))\nmodel.add(Dense(256))\nmodel.add(Activation('relu'))\n\nmodel.add(Dropout(0.25))\nmodel.add(Dense(64))\nmodel.add(Activation('relu')) \n\nmodel.add(Dropout(0.25))\nmodel.add(Dense(1, activation='sigmoid'))\nopt = tf.keras.optimizers.Adam(0.001)\nmodel.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])\n\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nSTEP_SIZE_TRAIN=train_generator.n//train_generator.batch_size\nSTEP_SIZE_VALID=valid_generator.n//valid_generator.batch_size\n\nhistory = model.fit_generator(generator=train_generator,\n                    steps_per_epoch=STEP_SIZE_TRAIN,\n                    validation_data=valid_generator,\n                    validation_steps=STEP_SIZE_VALID,\n                    epochs=30, verbose=1\n)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n#next model with 3 sets of 5 convolutional layers\nmodel2 = Sequential()\nmodel2.add(Conv2D(32, (3, 3), padding='same',input_shape=(96,96,3)))\nmodel2.add(Activation('relu'))\nmodel2.add(Conv2D(32, (3, 3)))\nmodel2.add(Activation('relu'))\nmodel2.add(Conv2D(32, (3, 3)))\nmodel2.add(Activation('relu'))\nmodel2.add(Conv2D(32, (3, 3)))\nmodel2.add(Activation('relu'))\nmodel2.add(Conv2D(32, (3, 3)))\nmodel2.add(Activation('relu'))\nmodel2.add(MaxPooling2D(pool_size=(2, 2)))\nmodel2.add(BatchNormalization())\n\nmodel2.add(Conv2D(64, (3, 3)))\nmodel2.add(Activation('relu'))\nmodel2.add(Conv2D(64, (3, 3)))\nmodel2.add(Activation('relu'))\nmodel2.add(Conv2D(64, (3, 3)))\nmodel2.add(Activation('relu'))\nmodel2.add(Conv2D(64, (3, 3)))\nmodel2.add(Activation('relu'))\nmodel2.add(Conv2D(64, (3, 3)))\nmodel2.add(Activation('relu'))\nmodel2.add(MaxPooling2D(pool_size=(2, 2)))\nmodel2.add(BatchNormalization())\n\nmodel2.add(Conv2D(128, (3, 3)))\nmodel2.add(Activation('relu'))\nmodel2.add(Conv2D(128, (3, 3)))\nmodel2.add(Activation('relu'))\nmodel2.add(Conv2D(128, (3, 3)))\nmodel2.add(Activation('relu'))\nmodel2.add(Conv2D(128, (3, 3)))\nmodel2.add(Activation('relu'))\nmodel2.add(Conv2D(128, (3, 3)))\nmodel2.add(Activation('relu'))\nmodel2.add(MaxPooling2D(pool_size=(2, 2)))\nmodel2.add(BatchNormalization())\n          \nmodel2.add(Flatten())\nmodel2.add(Dropout(0.25))\nmodel2.add(Dense(512))\nmodel2.add(Activation('relu'))\n\nmodel2.add(Dropout(0.25))\nmodel2.add(Dense(256))\nmodel2.add(Activation('relu'))\n\nmodel2.add(Dropout(0.25))\nmodel2.add(Dense(64))\nmodel2.add(Activation('relu')) \n\nmodel2.add(Dropout(0.25))\nmodel2.add(Dense(1, activation='sigmoid'))\nopt = tf.keras.optimizers.Adam(0.001)\nmodel2.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model2.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"STEP_SIZE_TRAIN=train_generator.n//train_generator.batch_size\nSTEP_SIZE_VALID=valid_generator.n//valid_generator.batch_size\n\nhistory2 = model2.fit_generator(generator=train_generator,\n                    steps_per_epoch=STEP_SIZE_TRAIN,\n                    validation_data=valid_generator,\n                    validation_steps=STEP_SIZE_VALID,\n                    epochs=30, verbose=1\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I will now take a look at the plots of the accuracy of the training vs validation sets to see which model architecture appears to be the best.  I will then move forward with further hyperparameter tuning.","metadata":{}},{"cell_type":"code","source":"plt.plot(history.history['accuracy'])\nplt.plot(history.history['val_accuracy'])\nplt.title('model loss')\nplt.ylabel('loss')\nplt.xlabel('epoch')\nplt.legend(['train', 'val'], loc='upper left')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(history2.history['accuracy'])\nplt.plot(history2.history['val_accuracy'])\nplt.title('model2 accuracy')\nplt.ylabel('accuracy')\nplt.xlabel('epoch')\nplt.legend(['train', 'val'], loc='upper left')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So, we see the second model with more convolutional layers seems to be slightly better on the validation set.  We are able to reach about 90% accuracy after 10-15 epochs.  Using this model going forward, I will compare a different optimization method and finally different activations in order to tune the hyperparameters.","metadata":{}},{"cell_type":"code","source":"#next model with 3 sets of 5 convolutional layers\nmodel3 = Sequential()\nmodel3.add(Conv2D(32, (3, 3), padding='same',input_shape=(96,96,3)))\nmodel3.add(Activation('relu'))\nmodel3.add(Conv2D(32, (3, 3)))\nmodel3.add(Activation('relu'))\nmodel3.add(Conv2D(32, (3, 3)))\nmodel3.add(Activation('relu'))\nmodel3.add(Conv2D(32, (3, 3)))\nmodel3.add(Activation('relu'))\nmodel3.add(Conv2D(32, (3, 3)))\nmodel3.add(Activation('relu'))\nmodel3.add(MaxPooling2D(pool_size=(2, 2)))\nmodel3.add(BatchNormalization())\n\nmodel3.add(Conv2D(64, (3, 3)))\nmodel3.add(Activation('relu'))\nmodel3.add(Conv2D(64, (3, 3)))\nmodel3.add(Activation('relu'))\nmodel3.add(Conv2D(64, (3, 3)))\nmodel3.add(Activation('relu'))\nmodel3.add(Conv2D(64, (3, 3)))\nmodel3.add(Activation('relu'))\nmodel3.add(Conv2D(64, (3, 3)))\nmodel3.add(Activation('relu'))\nmodel3.add(MaxPooling2D(pool_size=(2, 2)))\nmodel3.add(BatchNormalization())\n\nmodel3.add(Conv2D(128, (3, 3)))\nmodel3.add(Activation('relu'))\nmodel3.add(Conv2D(128, (3, 3)))\nmodel3.add(Activation('relu'))\nmodel3.add(Conv2D(128, (3, 3)))\nmodel3.add(Activation('relu'))\nmodel3.add(Conv2D(128, (3, 3)))\nmodel3.add(Activation('relu'))\nmodel3.add(Conv2D(128, (3, 3)))\nmodel3.add(Activation('relu'))\nmodel3.add(MaxPooling2D(pool_size=(2, 2)))\nmodel3.add(BatchNormalization())\n          \nmodel3.add(Flatten())\nmodel3.add(Dropout(0.25))\nmodel3.add(Dense(512))\nmodel3.add(Activation('relu'))\n\nmodel3.add(Dropout(0.25))\nmodel3.add(Dense(256))\nmodel3.add(Activation('relu'))\n\nmodel3.add(Dropout(0.25))\nmodel3.add(Dense(64))\nmodel3.add(Activation('relu')) \n\nmodel3.add(Dropout(0.25))\nmodel3.add(Dense(1, activation='sigmoid'))\nopt = tf.keras.optimizers.RMSprop(0.001)\nmodel3.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model3.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"STEP_SIZE_TRAIN=train_generator.n//train_generator.batch_size\nSTEP_SIZE_VALID=valid_generator.n//valid_generator.batch_size\n\nhistory3 = model3.fit_generator(generator=train_generator,\n                    steps_per_epoch=STEP_SIZE_TRAIN,\n                    validation_data=valid_generator,\n                    validation_steps=STEP_SIZE_VALID,\n                    epochs=30, verbose=1\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(history3.history['accuracy'])\nplt.plot(history3.history['val_accuracy'])\nplt.title('model3 accuracy')\nplt.ylabel('accuracy')\nplt.xlabel('epoch')\nplt.legend(['train', 'val'], loc='upper left')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So, we see the model with RMSprop as the optimizer does not seem to be as stable at least in the validation set and the accuracy still appears to be improving even after 30 epochs whereas the model utilizing Adam optimizer had converged by that point.  I will continue with the model utilizing Adam optimization, comparing the activation function (relu vs prelu) before determining what the final model will be.","metadata":{}},{"cell_type":"code","source":"#next model with 3 sets of 5 convolutional layers, using prelu activations\nmodel4 = Sequential()\nmodel4.add(Conv2D(32, (3, 3), padding='same',input_shape=(96,96,3)))\nmodel4.add(PReLU(alpha_initializer=Constant(value=0.25)))\nmodel4.add(Conv2D(32, (3, 3)))\nmodel4.add(PReLU(alpha_initializer=Constant(value=0.25)))\nmodel4.add(Conv2D(32, (3, 3)))\nmodel4.add(PReLU(alpha_initializer=Constant(value=0.25)))\nmodel4.add(Conv2D(32, (3, 3)))\nmodel4.add(PReLU(alpha_initializer=Constant(value=0.25)))\nmodel4.add(Conv2D(32, (3, 3)))\nmodel4.add(PReLU(alpha_initializer=Constant(value=0.25)))\nmodel4.add(MaxPooling2D(pool_size=(2, 2)))\nmodel4.add(BatchNormalization())\n\nmodel4.add(Conv2D(64, (3, 3)))\nmodel4.add(PReLU(alpha_initializer=Constant(value=0.25)))\nmodel4.add(Conv2D(64, (3, 3)))\nmodel4.add(PReLU(alpha_initializer=Constant(value=0.25)))\nmodel4.add(Conv2D(64, (3, 3)))\nmodel4.add(PReLU(alpha_initializer=Constant(value=0.25)))\nmodel4.add(Conv2D(64, (3, 3)))\nmodel4.add(PReLU(alpha_initializer=Constant(value=0.25)))\nmodel4.add(Conv2D(64, (3, 3)))\nmodel4.add(PReLU(alpha_initializer=Constant(value=0.25)))\nmodel4.add(MaxPooling2D(pool_size=(2, 2)))\nmodel4.add(BatchNormalization())\n\nmodel4.add(Conv2D(128, (3, 3)))\nmodel4.add(PReLU(alpha_initializer=Constant(value=0.25)))\nmodel4.add(Conv2D(128, (3, 3)))\nmodel4.add(PReLU(alpha_initializer=Constant(value=0.25)))\nmodel4.add(Conv2D(128, (3, 3)))\nmodel4.add(PReLU(alpha_initializer=Constant(value=0.25)))\nmodel4.add(Conv2D(128, (3, 3)))\nmodel4.add(PReLU(alpha_initializer=Constant(value=0.25)))\nmodel4.add(Conv2D(128, (3, 3)))\nmodel4.add(PReLU(alpha_initializer=Constant(value=0.25)))\nmodel4.add(MaxPooling2D(pool_size=(2, 2)))\nmodel4.add(BatchNormalization())\n          \nmodel4.add(Flatten())\nmodel4.add(Dropout(0.25))\nmodel4.add(Dense(512))\nmodel4.add(PReLU(alpha_initializer=Constant(value=0.25)))\n\nmodel4.add(Dropout(0.25))\nmodel4.add(Dense(256))\nmodel4.add(PReLU(alpha_initializer=Constant(value=0.25)))\n\nmodel4.add(Dropout(0.25))\nmodel4.add(Dense(64))\nmodel4.add(PReLU(alpha_initializer=Constant(value=0.25)))\n\nmodel4.add(Dropout(0.25))\nmodel4.add(Dense(1, activation='sigmoid'))\nopt = tf.keras.optimizers.RMSprop(0.001)\nmodel4.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model4.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"STEP_SIZE_TRAIN=train_generator.n//train_generator.batch_size\nSTEP_SIZE_VALID=valid_generator.n//valid_generator.batch_size\n\nhistory4 = model4.fit_generator(generator=train_generator,\n                    steps_per_epoch=STEP_SIZE_TRAIN,\n                    validation_data=valid_generator,\n                    validation_steps=STEP_SIZE_VALID,\n                    epochs=30, verbose=1\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(history4.history['accuracy'])\nplt.plot(history4.history['val_accuracy'])\nplt.title('model3 accuracy')\nplt.ylabel('accuracy')\nplt.xlabel('epoch')\nplt.legend(['train', 'val'], loc='upper left')\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So, we see with PReLU activation, the model converged faster (reaching nearly 100% accuracy on the training set within 30 epochs) and the validation set accuracy was much more stable.  This makes sense as the ReLU activation suffers from vanishing gradients and can essentially \"turn off\" nodes which may actually be the nodes we need for different instances.  From the images, we know that not every image is perfectly centered so the important \"nodes\" in our model could really occur anywhere and we shouldnt look to turn them completely off if at all possible.","metadata":{}},{"cell_type":"code","source":"test_set = os.listdir('../input/histopathologic-cancer-detection/test/')\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df = pd.DataFrame(test_set)\ntest_df.columns = ['id']\ntest_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_datagen=ImageDataGenerator(rescale=1/255)\n\ntest_generator=test_datagen.flow_from_dataframe(dataframe=test_df,directory=\"../input/histopathologic-cancer-detection/test/\",\n                x_col=\"id\",batch_size=64,seed=1234,shuffle=False,\n                class_mode=None,target_size=(96,96))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"STEP_SIZE_TEST=test_generator.n/2\n\npreds = model4.predict_generator(generator=test_generator,steps=STEP_SIZE_TEST, verbose = 1)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predictions = []\n\nfor pred in preds:\n    if pred >= 0.5:\n        predictions.append(1)\n    else:\n        predictions.append(0)\n        \npredictions[:10]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = test_df.copy()\nsubmission['id']=submission['id'].str[:-4]\nsubmission['label']=predictions\nsubmission.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv',index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}