{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction to Deep Learning Week 3\nThis notebook was forked from Mattison Hineline and Edited by Joseph Hardin\n###  Brief description of the problem and data (5 pts) \n\nIn week 3 of the Introduction to Deep Learning, we were asked to use a Convolutional Neural Network to work on the Cancer Detection challenge provided by Kaggle.  The goal of this challenge is to *\"identify metastatic cancer in small image patches taken from larger digital pathology scans\"*.  The benefits of this are obvious.  Pathologist are highly trained individuals and ,like all people, are prone to mistakes.  An algorithim that can be used in tandem, or even instead of in extremely specific circumstances (i.e. low impact undergraduate research projects), could increase throughput and decrease cost for positive diagnosis of metastatic cancer.\n\n\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"code","source":"#import libraries\n\n#general libraries \nimport numpy as np \nimport pandas as pd \nimport os\nimport random\nfrom sklearn.utils import shuffle\nimport shutil\n\n#visualizations\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport plotly.express as px\nimport matplotlib.patches as patches\n\n# work with images\nfrom skimage.transform import rotate\nfrom skimage import io\nimport cv2 as cv\n\n# model development\nfrom sklearn.model_selection import train_test_split\nimport tensorflow as tf\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.layers import RandomFlip, RandomZoom, RandomRotation\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D, AveragePooling2D\nfrom tensorflow.keras.layers import Dense, Flatten, Dropout\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import BatchNormalization\nfrom tensorflow.keras.optimizers import Adam\n\nimport warnings\nwarnings.simplefilter(\"ignore\", category=DeprecationWarning)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:20:44.588263Z","iopub.execute_input":"2022-11-10T12:20:44.588696Z","iopub.status.idle":"2022-11-10T12:20:53.105159Z","shell.execute_reply.started":"2022-11-10T12:20:44.588591Z","shell.execute_reply":"2022-11-10T12:20:53.104212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.__version__","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:20:53.107060Z","iopub.execute_input":"2022-11-10T12:20:53.107913Z","iopub.status.idle":"2022-11-10T12:20:53.119504Z","shell.execute_reply.started":"2022-11-10T12:20:53.107871Z","shell.execute_reply":"2022-11-10T12:20:53.118371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#get files\ntest_path = '../input/histopathologic-cancer-detection/test/'\ntrain_path = '../input/histopathologic-cancer-detection/train/'\nsample_submission = pd.read_csv('../input/histopathologic-cancer-detection/sample_submission.csv')\ntrain_data = pd.read_csv('../input/histopathologic-cancer-detection/train_labels.csv')","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:20:53.121336Z","iopub.execute_input":"2022-11-10T12:20:53.122057Z","iopub.status.idle":"2022-11-10T12:20:53.792609Z","shell.execute_reply.started":"2022-11-10T12:20:53.122013Z","shell.execute_reply":"2022-11-10T12:20:53.791657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# declare constants for reproduciblity\nRANDOM_STATE = 369","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:20:53.794025Z","iopub.execute_input":"2022-11-10T12:20:53.794370Z","iopub.status.idle":"2022-11-10T12:20:53.800559Z","shell.execute_reply.started":"2022-11-10T12:20:53.794334Z","shell.execute_reply":"2022-11-10T12:20:53.799256Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Brief Description of the Problem and Data\n\nEach file in the data given is a .tif file.  The pictures are 96x96x3 (Length x Width x Channel) with each pixel lying in the range of 0-255.  This presents a challenge, as the reading of .tif files is not standard in the keras framework (unlike .png or .jpeg for example).  Our training dataset consist of 220,025 images that are labeled either 1 or 0.  1 meanss there is a cancerous cell.\nThis data contains thousands of small images where the 96x96 pixel images with 3 channels, each with an identifying label and id. We have two datasets, a training and testing set already split for us. The training set contains 220,025 unique images and the test set contains about 57,500. To use these images in a machine learning model, we are also given an identifying dataframe with two columns: 'id' which is the unique image ID correpsonding to the training directory, and 'label' which tells us the classification category. Each label is either a 0 or 1, depending whether the image is non-cancerous (0) or cancerous (1). In the competition description, we find that if at least one pixel of an image is identified as cancerous then the whole image is therefore marked with a 1, otherwise it is 0. It is important to note that we do not have any missing values in this data which will make preprocessing more efficient.","metadata":{}},{"cell_type":"code","source":"# have a look at the format of the data\ntrain_data.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:20:53.805490Z","iopub.execute_input":"2022-11-10T12:20:53.806010Z","iopub.status.idle":"2022-11-10T12:20:53.822292Z","shell.execute_reply.started":"2022-11-10T12:20:53.805975Z","shell.execute_reply":"2022-11-10T12:20:53.821417Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# take a look at the data further\ntrain_data.describe()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:20:53.825100Z","iopub.execute_input":"2022-11-10T12:20:53.825354Z","iopub.status.idle":"2022-11-10T12:20:53.855806Z","shell.execute_reply.started":"2022-11-10T12:20:53.825330Z","shell.execute_reply":"2022-11-10T12:20:53.854683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# check information, data types, and for missing data\ntrain_data.info()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:20:53.857640Z","iopub.execute_input":"2022-11-10T12:20:53.858021Z","iopub.status.idle":"2022-11-10T12:20:53.884385Z","shell.execute_reply.started":"2022-11-10T12:20:53.857980Z","shell.execute_reply":"2022-11-10T12:20:53.883268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Exploratory Data Analysis (EDA) — Inspect, Visualize and Clean the Data (15 pts)\n\n\n\nAs can be seen below, the data is roughly balanced and there do not appear to be any missing labels.  ","metadata":{}},{"cell_type":"code","source":"#create histogram\nprint(pd.DataFrame(data={'Label Counts': train_data['label'].value_counts()}))\nsns.countplot(x=train_data['label'], palette='colorblind').set(title='Label Counts Histogram');","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T12:20:53.885918Z","iopub.execute_input":"2022-11-10T12:20:53.886327Z","iopub.status.idle":"2022-11-10T12:20:54.111878Z","shell.execute_reply.started":"2022-11-10T12:20:53.886291Z","shell.execute_reply":"2022-11-10T12:20:54.110952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#create pie chart\nfig = px.pie(train_data, \n             values = train_data['label'].value_counts().values, \n             names = train_data['label'].unique())\nfig.update_layout(\n    title={\n        'text': \"Label Percentage Pie Chart\",\n        'y':.99,\n        'x':0.5,\n        'xanchor': 'center',\n        'yanchor': 'top'})\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T12:20:54.114811Z","iopub.execute_input":"2022-11-10T12:20:54.115091Z","iopub.status.idle":"2022-11-10T12:20:54.879208Z","shell.execute_reply.started":"2022-11-10T12:20:54.115060Z","shell.execute_reply":"2022-11-10T12:20:54.877498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Below we can see some example images from the training data. As someone who is not familiar at looking at cancer cells and images, it would be very challenging for me to classify these. However, we mitigate this by putting the correct label for each image is below it. Additionally, we were told that each image has the (potentially) cancerous cells centered in each 32x32 pixel image, so we have drawn a box around this area as a focal point. ","metadata":{}},{"cell_type":"code","source":"#visualize a few images\nfig, ax = plt.subplots(5, 5, figsize=(15, 15))\nfor i, axis in enumerate(ax.flat):\n    file = str(train_path + train_data.id[i] + '.tif')\n    image = io.imread(file)\n    axis.imshow(image)\n    box = patches.Rectangle((32,32),32,32, linewidth=2, edgecolor='r',facecolor='none', linestyle='-')\n    axis.add_patch(box)\n    axis.set(xticks=[], yticks=[], xlabel = train_data.label[i]);\n    #cv2.waitKey(0)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T12:20:54.880819Z","iopub.execute_input":"2022-11-10T12:20:54.881236Z","iopub.status.idle":"2022-11-10T12:20:56.321233Z","shell.execute_reply.started":"2022-11-10T12:20:54.881195Z","shell.execute_reply":"2022-11-10T12:20:56.320002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Preprocessing Techniques**\n\n\nWhen working with training data, first we want to handle any missing data. From the source description and our findings above, we can see that we have no missing data in the training set. Next, we are told that the part of the image which is going to be of interest is the center pixels (red box above). Right now the images are 96x96x3 and we know that the \"3\" is the RGB channel. For now, we will leave this as well. Lastly, there are many augmentations you can perform on images to produce better results and facilitate model learning, but to start we will not touch these either for simplicity. \n\nWhat we will do with the image data is **shuffle** the data so that the model doesn't learn based on the image ordering/pattern of input, which could potentially have consequences in the model training. We will also **split** the data into training and validation set to improve model development. During training we will also **normalize** the pixels by dividing by 255.0, which should help data processing and model training.","metadata":{}},{"cell_type":"markdown","source":"# 3. DModel Architecture (25 pts)\n\nFor this model we will be using Keras' library to run a convolutional neural network (CNN). The first model we will run without tuning any hyperparameters within the model and use that as our baseline. Then, we will run a second model, tuning hyperparameters such as learning rate, batch normalization, regularization, filter size, stride, activation layers, etc. \n\nOur CNN model will have a network such that there are two convolutional layers then a MaxPool layer, and we repeat this *n* number of times. Specifically, we will create a fairly simple model with two (n=2) of these clusters. In other words, our model will be input --> Conv2D --> Conv2D --> MaxPool --> Conv2D --> Conv2D --> MaxPool --> Flatten --> Output with sigmoid activation.\n\n**First model:**\n1. Normalize images pre-training (image/255)\n2. Output layer activation (sigmoid)\n\n**Second model contains all the first model parameters, but we also add:**\n1. Dropout (0.1)\n2. Batch Normalization\n3. Optimization (Adam)\n4. Learning rate (0.0001) \n5. Hidden layer activations (ReLU)\n\nBefore we start with the models, let's describe what each of the parameters will do to the model. In both models we will use 1 and 2, numbers 3 to 5 will be used in the second and final models. \n1. **Normalize images** : this will take the pixels and divide each pixel by 255 to normalize the data and have values between 0-1.\n2. **Output layer activation** : we will use a sigmoid activation function on the output layer since we are working with binary data\n3. **Dropout** : we will set dropout at 0.1 which will randonly select some weights and set them to equal 0 which regularizes the model because it is using a smaller number of weights for each training run. \n4. **Optimization** : we will use adaptive moment estimation (Adam) for optimizing the model which essentially mimics momentum for gradient adn gradient-squared. \n5. **Learning rate**: we will set our learning rate to 0.0001 which will assist in the gradient descent such that as the model learns, the speed of learning decreases so that it is less likely to overstep the (hopefully) global minimum. \n6. **Hidden layer activations**: we will use rectified linear regression (ReLU) as our hidden layer activation function which will help the model to converge better, prevent saturation, and provide less need for computation power. \n\nIn addition, we will use fairly large **batch sizes**, set at 256 to help reduce variance. Finally we wil train our two models with 10 **epochs**. We will use accuracy and the ROC-AUC curve to measure model performance, as well as binary cross-entropy as our loss function.","metadata":{}},{"cell_type":"code","source":"# set model constants\nBATCH_SIZE = 256","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:20:56.322525Z","iopub.execute_input":"2022-11-10T12:20:56.322937Z","iopub.status.idle":"2022-11-10T12:20:56.327733Z","shell.execute_reply.started":"2022-11-10T12:20:56.322903Z","shell.execute_reply":"2022-11-10T12:20:56.326907Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# prepare data for training\ndef append_tif(string):\n    return string+\".tif\"\n\ntrain_data[\"id\"] = train_data[\"id\"].apply(append_tif)\ntrain_data['label'] = train_data['label'].astype(str)\n\n# randomly shuffle training data\ntrain_data = shuffle(train_data, random_state=RANDOM_STATE)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:20:56.329055Z","iopub.execute_input":"2022-11-10T12:20:56.330156Z","iopub.status.idle":"2022-11-10T12:20:56.565756Z","shell.execute_reply.started":"2022-11-10T12:20:56.330122Z","shell.execute_reply":"2022-11-10T12:20:56.564768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# modify training data by normalizing it \n# and split data into training and validation sets\ndatagen = ImageDataGenerator(rescale=1./255.,\n                            validation_split=0.15)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:20:56.567255Z","iopub.execute_input":"2022-11-10T12:20:56.567618Z","iopub.status.idle":"2022-11-10T12:20:56.573712Z","shell.execute_reply.started":"2022-11-10T12:20:56.567583Z","shell.execute_reply":"2022-11-10T12:20:56.572682Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# generate training data\ntrain_generator = datagen.flow_from_dataframe(\n    dataframe=train_data,\n    directory=train_path,\n    x_col=\"id\",\n    y_col=\"label\",\n    subset=\"training\",\n    batch_size=BATCH_SIZE,\n    seed=RANDOM_STATE,\n    class_mode=\"binary\",\n    target_size=(64,64))        # original image = (96, 96) ","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:20:56.578971Z","iopub.execute_input":"2022-11-10T12:20:56.579225Z","iopub.status.idle":"2022-11-10T12:29:17.277599Z","shell.execute_reply.started":"2022-11-10T12:20:56.579201Z","shell.execute_reply":"2022-11-10T12:29:17.276671Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# generate validation data\nvalid_generator = datagen.flow_from_dataframe(\n    dataframe=train_data,\n    directory=train_path,\n    x_col=\"id\",\n    y_col=\"label\",\n    subset=\"validation\",\n    batch_size=BATCH_SIZE,\n    seed=RANDOM_STATE,\n    class_mode=\"binary\",\n    target_size=(64,64))       # original image = (96, 96) ","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:29:17.278936Z","iopub.execute_input":"2022-11-10T12:29:17.279776Z","iopub.status.idle":"2022-11-10T12:31:03.650256Z","shell.execute_reply.started":"2022-11-10T12:29:17.279738Z","shell.execute_reply":"2022-11-10T12:31:03.649225Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Setup GPU accelerator - configure Strategy. Assume TPU...if not set default for GPU/CPU\ntpu = None\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.TPUStrategy(tpu)\nexcept ValueError:\n    strategy = tf.distribute.get_strategy()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:31:03.651760Z","iopub.execute_input":"2022-11-10T12:31:03.652370Z","iopub.status.idle":"2022-11-10T12:31:03.673114Z","shell.execute_reply.started":"2022-11-10T12:31:03.652331Z","shell.execute_reply":"2022-11-10T12:31:03.671913Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Start First model building and training**\n\nHere we build our first model. This model is fairly simple with a handful of layers, void of activations except the last layer. The last layer we use a sigmoid activation function since our data is binary (0-1). We can see below how the model is built and how many parameters we will need to train for this model. The next cell below we can see the model training per epoch. ","metadata":{}},{"cell_type":"code","source":"# set ROC AUC as metric\nROC_1 = tf.keras.metrics.AUC()\n\n# use GPU\nwith strategy.scope():\n    \n    #create model\n    model_one = Sequential()\n    \n    model_one.add(Conv2D(filters=16, kernel_size=(3,3)))\n    model_one.add(Conv2D(filters=16, kernel_size=(3,3)))\n    model_one.add(MaxPooling2D(pool_size=(2,2)))\n\n    model_one.add(Conv2D(filters=32, kernel_size=(3,3)))\n    model_one.add(Conv2D(filters=32, kernel_size=(3,3)))\n    model_one.add(AveragePooling2D(pool_size=(2,2)))\n\n    model_one.add(Flatten())\n    model_one.add(Dense(1, activation='sigmoid'))\n    \n    #build model by input size\n    model_one.build(input_shape=(BATCH_SIZE, 64, 64, 3))        # original image = (96, 96, 3) \n    \n    #compile\n    model_one.compile(loss='binary_crossentropy', metrics=['accuracy', ROC_1])\n    \n#quick look at model\nmodel_one.summary()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T12:31:03.674491Z","iopub.execute_input":"2022-11-10T12:31:03.675147Z","iopub.status.idle":"2022-11-10T12:31:06.316361Z","shell.execute_reply.started":"2022-11-10T12:31:03.675112Z","shell.execute_reply":"2022-11-10T12:31:06.315314Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"EPOCHS = 10\n\n# train the model\nhistory_model_one = model_one.fit_generator(\n                        train_generator,\n                        epochs = EPOCHS,\n                        validation_data = valid_generator)","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2022-11-10T12:31:06.317843Z","iopub.execute_input":"2022-11-10T12:31:06.318172Z","iopub.status.idle":"2022-11-10T13:33:48.903759Z","shell.execute_reply.started":"2022-11-10T12:31:06.318138Z","shell.execute_reply":"2022-11-10T13:33:48.902545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now that we have trained the model, we can take at look at how it did graphically with the training data. Below we can see the accuracy and loss with regards to the validation and training set. ","metadata":{}},{"cell_type":"code","source":"# plot model accuracy per epoch \nplt.plot(history_model_one.history['accuracy'])\nplt.plot(history_model_one.history['val_accuracy'])\nplt.title('Model One Accuracy per Epoch')\nplt.ylabel('accuracy')\nplt.xlabel('epoch')\nplt.legend(['train', 'validate'], loc='upper left')\nplt.show();\n\n# plot model loss per epoch\nplt.plot(history_model_one.history['loss'])\nplt.plot(history_model_one.history['val_loss'])\nplt.title('Model One Loss per Epoch')\nplt.ylabel('loss')\nplt.xlabel('epoch')\nplt.legend(['train', 'validate'], loc='upper left')\nplt.show();\n\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T13:33:48.908215Z","iopub.execute_input":"2022-11-10T13:33:48.909202Z","iopub.status.idle":"2022-11-10T13:33:49.292375Z","shell.execute_reply.started":"2022-11-10T13:33:48.909162Z","shell.execute_reply":"2022-11-10T13:33:49.291703Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Start second model building and training**\n\nOur second model incorporates hyperparameter tuning. We can see we are using ReLU activation functions for hidden layers, dropout between clusters, and batch normalization. In addition, we have added another hidden layer before the flatten and output layers. Lastly, we are using Adam optimizer with a low learning rate. ","metadata":{}},{"cell_type":"code","source":"# build second model like first but with hyperparameters and optimizer(s)\nROC_2 = tf.keras.metrics.AUC()\n\nwith strategy.scope():\n    \n    #create model\n    model_two = Sequential()\n    \n    model_two.add(Conv2D(filters=16, kernel_size=(3,3), activation='relu', ))\n    model_two.add(Conv2D(filters=16, kernel_size=(3,3), activation='relu'))\n    model_two.add(MaxPooling2D(pool_size=(2,2)))\n    model_two.add(Dropout(0.1))\n    \n    model_two.add(BatchNormalization())\n    model_two.add(Conv2D(filters=32, kernel_size=(3,3), activation='relu'))\n    model_two.add(Conv2D(filters=32, kernel_size=(3,3), activation='relu'))\n    model_two.add(AveragePooling2D(pool_size=(2,2)))\n    model_two.add(Dropout(0.1))\n    \n    model_two.add(BatchNormalization())\n    model_two.add(Conv2D(filters=32, kernel_size=(3,3), activation='relu'))\n    model_two.add(Flatten())\n    model_two.add(Dense(1, activation='sigmoid'))\n    \n    #build model by input size\n    model_two.build(input_shape=(BATCH_SIZE, 64, 64, 3))       # original image = (96, 96, 3) \n    \n    #compile\n    adam_optimizer = Adam(learning_rate=0.0001)\n    model_two.compile(loss='binary_crossentropy', metrics=['accuracy', ROC_2], optimizer=adam_optimizer)\n\n#quick look at model\nmodel_two.summary()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T13:33:49.293505Z","iopub.execute_input":"2022-11-10T13:33:49.294449Z","iopub.status.idle":"2022-11-10T13:33:49.389079Z","shell.execute_reply.started":"2022-11-10T13:33:49.294412Z","shell.execute_reply":"2022-11-10T13:33:49.387690Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"EPOCHS = 10\n\n# train model\nhistory_model_two = model_two.fit_generator(\n                        train_generator,\n                        epochs = EPOCHS,\n                        validation_data = valid_generator)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T13:33:49.390315Z","iopub.execute_input":"2022-11-10T13:33:49.390709Z","iopub.status.idle":"2022-11-10T14:22:55.471028Z","shell.execute_reply.started":"2022-11-10T13:33:49.390671Z","shell.execute_reply":"2022-11-10T14:22:55.469876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# graph loss\nplt.plot(np.arange(1,len(history_model_two.history['accuracy']) + 1), history_model_two.history['accuracy'])\nplt.plot(np.arange(1,len(history_model_two.history['val_accuracy']) + 1),history_model_two.history['val_accuracy'])\nplt.title('Model Two Accuracy')\nplt.ylabel('accuracy')\nplt.xlabel('epoch')\nplt.legend(['train', 'validate'], loc='upper left')\nplt.show();\n\nplt.plot(np.arange(1,len(history_model_two.history['loss']) + 1),history_model_two.history['loss'])\nplt.plot(np.arange(1,len(history_model_two.history['val_loss']) + 1),history_model_two.history['val_loss'])\nplt.title('Model Two Loss')\nplt.ylabel('loss')\nplt.xlabel('epoch')\nplt.legend(['train', 'validate'], loc='upper left')\nplt.show();\n\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T15:11:41.757001Z","iopub.execute_input":"2022-11-10T15:11:41.757494Z","iopub.status.idle":"2022-11-10T15:11:42.190384Z","shell.execute_reply.started":"2022-11-10T15:11:41.757442Z","shell.execute_reply":"2022-11-10T15:11:42.189435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"EPOCHS = 8\n\n# train model\nFinal_model = model_two.fit_generator(\n                        train_generator,\n                        epochs = EPOCHS,\n                        validation_data = valid_generator)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T14:32:30.666007Z","iopub.execute_input":"2022-11-10T14:32:30.666362Z","iopub.status.idle":"2022-11-10T15:10:44.519565Z","shell.execute_reply.started":"2022-11-10T14:32:30.666332Z","shell.execute_reply":"2022-11-10T15:10:44.518591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# graph loss\nplt.plot(np.arange(1,len(Final_model.history['accuracy']) + 1), Final_model.history['accuracy'])\nplt.plot(np.arange(1,len(Final_model.history['val_accuracy']) + 1),Final_model.history['val_accuracy'])\nplt.title('Final Model Accuracy')\nplt.ylabel('accuracy')\nplt.xlabel('epoch')\nplt.legend(['train', 'validate'], loc='upper left')\nplt.show();\n\nplt.plot(np.arange(1,len(Final_model.history['loss']) + 1),Final_model.history['loss'])\nplt.plot(np.arange(1,len(Final_model.history['val_loss']) + 1),Final_model.history['val_loss'])\nplt.title('Final Model Loss')\nplt.ylabel('loss')\nplt.xlabel('epoch')\nplt.legend(['train', 'validate'], loc='upper left')\nplt.show();\n","metadata":{"execution":{"iopub.status.busy":"2022-11-10T15:14:11.192214Z","iopub.execute_input":"2022-11-10T15:14:11.192585Z","iopub.status.idle":"2022-11-10T15:14:11.608233Z","shell.execute_reply.started":"2022-11-10T15:14:11.192553Z","shell.execute_reply":"2022-11-10T15:14:11.607290Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#double check what you're aiming the submission data set to look like\nsample_submission.head()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T15:15:02.301254Z","iopub.execute_input":"2022-11-10T15:15:02.301611Z","iopub.status.idle":"2022-11-10T15:15:02.311210Z","shell.execute_reply.started":"2022-11-10T15:15:02.301580Z","shell.execute_reply":"2022-11-10T15:15:02.310085Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#create a dataframe to run the predictions\ntest_df = pd.DataFrame({'id':os.listdir(test_path)})\ntest_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T15:14:55.755364Z","iopub.execute_input":"2022-11-10T15:14:55.756107Z","iopub.status.idle":"2022-11-10T15:14:56.612542Z","shell.execute_reply.started":"2022-11-10T15:14:55.756064Z","shell.execute_reply":"2022-11-10T15:14:56.611638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# prepare test data (in same way as train data)\ndatagen_test = ImageDataGenerator(rescale=1./255.)\n\ntest_generator = datagen_test.flow_from_dataframe(\n    dataframe=test_df,\n    directory=test_path,\n    x_col='id', \n    y_col=None,\n    target_size=(64,64),         # original image = (96, 96) \n    batch_size=1,\n    shuffle=False,\n    class_mode=None)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T15:15:09.363668Z","iopub.execute_input":"2022-11-10T15:15:09.364041Z","iopub.status.idle":"2022-11-10T15:17:14.212899Z","shell.execute_reply.started":"2022-11-10T15:15:09.364010Z","shell.execute_reply":"2022-11-10T15:17:14.210971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#run model to find predictions\n\n# predictions = model_one.predict(test_generator, verbose=1)\npredictions = model_two.predict(test_generator, verbose=1)\n","metadata":{"execution":{"iopub.status.busy":"2022-11-10T15:37:53.923506Z","iopub.execute_input":"2022-11-10T15:37:53.923919Z","iopub.status.idle":"2022-11-10T15:43:32.112303Z","shell.execute_reply.started":"2022-11-10T15:37:53.923888Z","shell.execute_reply":"2022-11-10T15:43:32.111168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#create submission dataframe\npredictions = np.transpose(predictions)[0]\nsubmission_df = pd.DataFrame()\nsubmission_df['id'] = test_df['id'].apply(lambda x: x.split('.')[0])\nsubmission_df['label'] = list(map(lambda x: 0 if x < 0.5 else 1, predictions))\nsubmission_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T15:45:11.723681Z","iopub.execute_input":"2022-11-10T15:45:11.724090Z","iopub.status.idle":"2022-11-10T15:45:11.859331Z","shell.execute_reply.started":"2022-11-10T15:45:11.724057Z","shell.execute_reply":"2022-11-10T15:45:11.858366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#view test prediction counts\nsubmission_df['label'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T15:45:15.342840Z","iopub.execute_input":"2022-11-10T15:45:15.343228Z","iopub.status.idle":"2022-11-10T15:45:15.355661Z","shell.execute_reply.started":"2022-11-10T15:45:15.343196Z","shell.execute_reply":"2022-11-10T15:45:15.354514Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#plot test predictions\nsns.countplot(data=submission_df, x='label').set(title='Predicted Labels for Test Set');","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T15:45:25.847433Z","iopub.execute_input":"2022-11-10T15:45:25.848130Z","iopub.status.idle":"2022-11-10T15:45:26.039740Z","shell.execute_reply.started":"2022-11-10T15:45:25.848091Z","shell.execute_reply":"2022-11-10T15:45:26.038558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#convert to csv to submit to competition\nsubmission_df.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T15:45:29.487597Z","iopub.execute_input":"2022-11-10T15:45:29.488901Z","iopub.status.idle":"2022-11-10T15:45:29.572368Z","shell.execute_reply.started":"2022-11-10T15:45:29.488860Z","shell.execute_reply":"2022-11-10T15:45:29.571484Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Results and Analysis\n\nWe can see the benefits of increasing epochs and paramater tuning when comparing model 1 to model 2. .  Each epoch took approximately 5 minutes in model 1 except for the first epoch.  The first epoch is longer (about 4x longer) because keras does lazy execution and is building the model to be used in all epochs during the 1st epoch, not when the model strategy is laid out. One thing that is unclear to me is what happened with the final model.  The intention was to cut down model 2 to only 8 epochs because that has the highest validation accuracy and was close to the training accuracy.  The seperation of the validation and training metrics seen in epochs 9 and 10 made me slightly worried about overfitting.  However, by using the the same model I apparently just increased the amount of epochs for model 2 from 10 to 18.  This link goes into more detail https://github.com/keras-team/keras/issues/4446  ","metadata":{}},{"cell_type":"markdown","source":"# 5. Conclusion\n\nOur first model was simple with no hyperparameter tuning. The second model incorporated much more tuning had a lower learning rate, and  rand for 2.5x more epochs. I would like to have the freedom to have the amount of epochs be a dependent variable based off the delta in loss functions between epochs.  However, due to time constratints simply plotting and eyballing had to suffice  ","metadata":{}}]}