{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Final Project: Flower Classification\n#### By Harsh Deshpande\n\n## Premise\n\nThe premise of this project is fairly simple: Given a picture of a flower, classify it appropriately. I got my data from a Flower Classification playground competition I was originally planning to compete in, but after some thought and technical difficulties, I decided to not to compete after all. \n\nI instead decided to make a web app that uses an appropriately trained model. In the web app, one can upload a picture of a flower and then display what type of flower it is. The data in the notebook below shows all the steps I did to produce a desirable model.\n\nFor this project, since I was dealing with very large images, I decided to utilize the use of the Tensor Processing Unit (TPU). A TPU is a piece of hardware like the CPU and GPU. It has far more arithmetic logic units than a GPU or CPU, but they are not general purpose and can only work on Tensors (multi-dimensional matrices). The TPU sped up my work greatly and was able to train Neural Network models quite quickly.\n\nNOTE: If you try to run this notebook locally, it WON'T work. Some data import libraries used as well as code that establishes connections to a clould TPU can only work inside of a Kaggle notebook environment. If you want to run this, feel free to fork this notebook and make sure you have selected a TPU to be the Accelerator in the notebook settings.\n\nHere are some important links that I referenced:\n\nThe competition where the data came from: https://www.kaggle.com/c/flower-classification-with-tpus\n\nA starter code notebook for the competition: https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu/ \n\n(NOTE: I used some of the code in starter code notebook for file decoding, TPU connections, and other global variables)\n\n## Getting Started"},{"metadata":{"trusted":true},"cell_type":"code","source":"import math, re, os\nimport tensorflow as tf\nimport numpy as np\nimport pandas as pd\nfrom matplotlib import pyplot as plt\nfrom kaggle_datasets import KaggleDatasets #Specific to Kaggle notebook environment\nfrom tensorflow import keras","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(tf.__version__)\nprint(np.__version__)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The code below connects to a Google Cloud TPU. This code was taken from the starter notebook up above as there really is no other way to connect"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection. No parameters necessary if TPU_NAME environment variable is set. On Kaggle this is always the case.\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Importing and Managing Data\n\nBelow I have set up a `files` dictonary that stores filenames for all the test, train, and validation data for each of the different image sizes"},{"metadata":{"trusted":true},"cell_type":"code","source":"files = {'192': {'train': [], 'val': [], 'test': []}, '224':{'train': [], 'val': [], 'test': []}, '331': {'train': [], 'val': [], 'test': []}, '512': {'train': [], 'val': [], 'test': []}}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"try:\n    data_loc = KaggleDatasets().get_gcs_path() #Built-in GCP Bucket in Kaggle for Data\nexcept:\n    data_loc = '/kaggle/input/flower-classification-with-tpus'\n    \n!gsutil ls $data_loc\n\nfor k in files.keys():\n    subdir = f'/tfrecords-jpeg-{k}x{k}'\n    loc = data_loc+subdir\n    #print(loc)\n    files[k]['train'] = tf.io.gfile.glob(loc + '/train/*.tfrec')\n    files[k]['test'] = tf.io.gfile.glob(loc + '/test/*.tfrec')\n    files[k]['val'] = tf.io.gfile.glob(loc + '/val/*.tfrec')\n        \nprint(files['192'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The Kaggle notebook environment stores input data into a Google Cloud Bucket. So in order to open up the files, I need to make a `KaggleDatasets().get_gcs_path()` call which gets the directory of where the bucket is stored, and only then can I get input data. I then store all the input data files"},{"metadata":{"trusted":true},"cell_type":"code","source":"IMAGE_SIZE = [192, 192]\n\nCLASSES = ['pink primrose',    'hard-leaved pocket orchid', 'canterbury bells', 'sweet pea',     'wild geranium',     'tiger lily',           'moon orchid',              'bird of paradise', 'monkshood',        'globe thistle',         # 00 - 09\n           'snapdragon',       \"colt's foot\",               'king protea',      'spear thistle', 'yellow iris',       'globe-flower',         'purple coneflower',        'peruvian lily',    'balloon flower',   'giant white arum lily', # 10 - 19\n           'fire lily',        'pincushion flower',         'fritillary',       'red ginger',    'grape hyacinth',    'corn poppy',           'prince of wales feathers', 'stemless gentian', 'artichoke',        'sweet william',         # 20 - 29\n           'carnation',        'garden phlox',              'love in the mist', 'cosmos',        'alpine sea holly',  'ruby-lipped cattleya', 'cape flower',              'great masterwort', 'siam tulip',       'lenten rose',           # 30 - 39\n           'barberton daisy',  'daffodil',                  'sword lily',       'poinsettia',    'bolero deep blue',  'wallflower',           'marigold',                 'buttercup',        'daisy',            'common dandelion',      # 40 - 49\n           'petunia',          'wild pansy',                'primula',          'sunflower',     'lilac hibiscus',    'bishop of llandaff',   'gaura',                    'geranium',         'orange dahlia',    'pink-yellow dahlia',    # 50 - 59\n           'cautleya spicata', 'japanese anemone',          'black-eyed susan', 'silverbush',    'californian poppy', 'osteospermum',         'spring crocus',            'iris',             'windflower',       'tree poppy',            # 60 - 69\n           'gazania',          'azalea',                    'water lily',       'rose',          'thorn apple',       'morning glory',        'passion flower',           'lotus',            'toad lily',        'anthurium',             # 70 - 79\n           'frangipani',       'clematis',                  'hibiscus',         'columbine',     'desert-rose',       'tree mallow',          'magnolia',                 'cyclamen ',        'watercress',       'canna lily',            # 80 - 89\n           'hippeastrum ',     'bee balm',                  'pink quill',       'foxglove',      'bougainvillea',     'camellia',             'mallow',                   'mexican petunia',  'bromelia',         'blanket flower',        # 90 - 99\n           'trumpet creeper',  'blackberry lily',           'common tulip',     'wild rose']\n\ndef readTrainVal(ex):\n    LABELED_TFREC_FORMAT_TRVAL = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"class\": tf.io.FixedLenFeature([], tf.int64)  # shape [] means single element\n    }\n    ex = tf.io.parse_single_example(ex, LABELED_TFREC_FORMAT_TRVAL)\n    image_data = ex['image']\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0  # convert image to floats in [0, 1] range\n    image = tf.reshape(image, [*IMAGE_SIZE, 3]) \n    label = tf.cast(ex['class'], tf.int32)\n    return image, label # returns a dataset of (image, label) pairs\n\ndef readTst(ex):\n    LABELED_TFREC_FORMAT_TEST = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"id\": tf.io.FixedLenFeature([], tf.string)\n    }\n    ex = tf.io.parse_single_example(ex, LABELED_TFREC_FORMAT_TEST)\n    image_data = ex['image']\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0  # convert image to floats in [0, 1] range\n    image = tf.reshape(image, [*IMAGE_SIZE, 3]) \n    return image, ex['id'] # returns a dataset of (image, id) pairs\n\nds_test = tf.data.TFRecordDataset(files['192']['train'][0])\nfor d_rec in ds_test.take(1):\n    i,l = readTrainVal(d_rec)\n    plt.imshow(i)\n    print(CLASSES[l])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The functions above help to split up the data in the `.tfrec` files. The file layout of the `.tfrec` files was similar to that of a JSON format, except of using a `{key:value}` format, the `.tfrec` file used a format more akin to `{key{value}}`. In the training and validation data, there were three keys that constituted an image object: the photo id, the label of the photo (which corresponds to its class) and the actual encoded bytes of the photo. The test dataset did not have the photo labels. First, template dictionaries were made. In the test and validation files, ids were not really necessary, so the dictionary only contained photo bytes and labels. For the testing data, an id was necessary if I was going to compete in the competition (which I eventually ended up not doing), so the photo bytes and the id was included. The dictionaries were used to parse the appropriate parts of the data and retrive values of the specified keys. The image was decoded and reshaped into a $w \\cdot h \\cdot 3$ shape. The values were then divided by 255 so that the data was normalized between 0 and 1. The labels and ids were retrieved as well. The labels were included in the test data and the ids were included in the train data. \n\nThe classes array was given from the intro notebook linked as I decided to utilize this feature"},{"metadata":{"trusted":true},"cell_type":"code","source":"ds_test_2 = tf.data.TFRecordDataset(files['224']['train'][0])\nIMAGE_SIZE = [224, 224]\nfor d_rec in ds_test_2.take(1):\n    i,l = readTrainVal(d_rec)\n    plt.imshow(i)\n    print(CLASSES[l])\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ds_test_3 = tf.data.TFRecordDataset(files['331']['train'][0])\nIMAGE_SIZE = [331, 331]\nfor d_rec in ds_test_3.take(1):\n    i,l = readTrainVal(d_rec)\n    plt.imshow(i)\n    print(CLASSES[l])\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ds_test_4 = tf.data.TFRecordDataset(files['512']['train'][0])\nIMAGE_SIZE = [512, 512]\nfor d_rec in ds_test_4.take(1):\n    i,l = readTrainVal(d_rec)\n    plt.imshow(i)\n    print(CLASSES[l])\n    \nIMAGE_SIZE = [192, 192]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As seen above, the test, train, and validation data pictures are the same across all three sizes. The only thing differing is the aforementioned image size. For the interest of time and TPU availability, I will only be working with the 192x192 images"},{"metadata":{"trusted":true},"cell_type":"code","source":"num_img = {'test': 462, 'train': 798, 'val': 232}\nAUTO = tf.data.experimental.AUTOTUNE\n\ndef training_dataset(f=None, batch_factor=2):\n    d_train = tf.data.TFRecordDataset(files[str(IMAGE_SIZE[0])]['train'])\n    d_train = d_train.map(readTrainVal)\n    if f != None:\n        d_train = d_train.map(f)\n    b_size =  num_img['train'] // batch_factor\n    d_train = d_train.shuffle(2048)\n    d_train = d_train.batch(b_size)\n    d_train = d_train.prefetch(AUTO)\n    return d_train\n\ndef validation_dataset(batch_factor=2):\n    d_val = tf.data.TFRecordDataset(files[str(IMAGE_SIZE[0])]['val'])\n    d_val = d_val.map(readTrainVal)\n    b_size = num_img['val'] // batch_factor\n    d_val = d_val.shuffle(2048)\n    d_val = d_val.batch(b_size)\n    d_val = d_val.prefetch(AUTO)\n    return d_val\n\ndef testing_dataset(batch_factor=2):\n    d_tst = tf.data.TFRecordDataset(files[str(IMAGE_SIZE[0])]['test'])\n    d_tst = d_tst.map(readTst)\n    b_size =  num_img['test'] // batch_factor\n    d_tst = d_tst.shuffle(2048)\n    d_tst = d_tst.batch(b_size)\n    d_tst = d_tst.prefetch(AUTO)\n    return d_tst\n    \n    \ntr = training_dataset()\nval = validation_dataset()\ntst = testing_dataset()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The methods above extract all the test, train, and validation data for the 192x192 images. Due to the large amount of images and the large sizes of them, Batches need to be made. The `num_img` dictionary declared above has the number of images inside one file of the test, train, and val folders of data. The layout of the `.tfrec` filenames are `[number]-[image_size]x[image_size]-[# images].tfrec`. The `num_img` takes from the `[# images]` part of the file name. I decided to make the batch size 1/2 the number of images in a single test/train/val file"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Training data shapes:\")\nfor image, label in tr.take(-1):\n    print(image.numpy().shape, label.numpy().shape)\nprint(\"Training data label examples:\", label.numpy()[:10])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As seen above, there are 32 batches for the training dataset. Each line of `(399, 192, 192, 3) (399,)` indicates one batch. Now we can start building or model. For this first model, I will build a simple neural network with one convolutional layer to check if the images are being properly processed"},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.layers import Input, Dense, Dropout, Conv2D, MaxPooling2D, Flatten\nfrom tensorflow.keras.models import Model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Demo model - will need to run convolutional layers on tpus\n\nwith strategy.scope(): #This is for determining if the model runs on the TPU\n    mod_conv = keras.Sequential()\n    mod_conv.add(Conv2D(32, kernel_size=3, padding='same' ,activation='relu', input_shape=(IMAGE_SIZE[0], IMAGE_SIZE[1], 3)))\n    mod_conv.add(MaxPooling2D(pool_size=(2, 2)))\n    mod_conv.add(Flatten())\n    mod_conv.add(Dense(512, activation='relu'))\n    mod_conv.add(Dense(104,activation='softmax'))\n    o = keras.optimizers.SGD(learning_rate = 0.05, momentum=0.3)\n    mod_conv.compile(optimizer=o, loss='sparse_categorical_crossentropy', metrics=['accuracy'])\nmod_conv.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mod_conv.fit(tr, epochs=5, validation_data=val) #Too much to run on cpu","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As seen above, a neural network was trained successfully above. A loss function of sparse categorical entropy was used because the labels were not given in one-hot form. The accuracy metric also seems to be the best here due to the classification nature of the problem. As you see, given that the model only ran for 5 epochs and the batch sizes were relatively large, there was a low accuracy rate for both the training and validation data. In order to increase this accuracy, we will have to increase the number of epochs, the number of layers, augment the training data with different image transformations, and perhaps incorporate some pre-trained models from keras into this project as well. After this, we can test the accuracy by submitting results to the competition and exporting our model"},{"metadata":{},"cell_type":"markdown","source":"## Image Augmentation"},{"metadata":{"trusted":true},"cell_type":"code","source":"augs = []\nfor d_rec in ds_test.take(1):\n    i, l = readTrainVal(d_rec)\n    plt.imshow(i)\n    augs.append(tf.image.random_brightness(i, 0.8))\n    augs.append(tf.image.random_contrast(i, 0.2, 0.8))\n    augs.append(tf.image.random_flip_left_right(i))\n    augs.append(tf.image.random_flip_up_down(i))\n    augs.append(tf.image.random_hue(i, 0.3))\n    augs.append(tf.image.random_jpeg_quality(i, 30, 95))\n    augs.append(tf.image.random_saturation(i, 0.1, 0.8))\n    augs.append(tf.image.central_crop(i, central_fraction=np.random.rand()))\n    augs.append(tf.image.rot90(i, k=np.random.randint(4)))\n\nfor i in augs:\n    fig, ax = plt.subplots()\n    plt.imshow(i)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The first image of the images above shows the original. The next 9 images show different transformations on it. The data augmentation function will randomly incorporate some of these 9 augments on some of the images in the training set to help prevent overfitting"},{"metadata":{"trusted":true},"cell_type":"code","source":"def augment(img, lab):\n    ch = np.random.rand()\n    if ch <= 0.6:\n        return img, lab #keep ~60% of images in batch same\n    n_apply = np.random.randint(9)\n    make_trans = np.random.randint(9, size=n_apply)\n    for m in make_trans:\n        if m == 0:\n            img = tf.image.random_brightness(img, 0.8)\n        elif m == 1:\n            img = tf.image.random_contrast(img, 0.2, 0.8)\n        elif m == 2:\n            img = tf.image.random_flip_left_right(img)\n        elif m == 3:\n            img = tf.image.random_flip_up_down(img)\n        elif m == 4:\n            img = tf.image.random_hue(img, 0.3)\n        elif m == 5: \n            img = tf.image.random_jpeg_quality(img, 30, 95)\n        elif m == 6:\n            img = tf.image.random_saturation(img, 0.1, 0.8)\n        elif m == 7:\n            img = tf.image.central_crop(img, central_fraction=np.random.rand())\n        else:\n            img = tf.image.rot90(img, k=np.random.randint(4))\n    return img, lab\n\ntr = training_dataset(f=augment)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for d_rec in ds_test.take(1):\n    i, l = readTrainVal(d_rec)\n    i, l = augment(i,l)\n    plt.imshow(i)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The code above applies a random number of the 9 transformations above onto about 40% of the images. The sample image above shows one of the possible results of applying the augment function to an image. Training using this new dataset on the previous neural network model, we get:"},{"metadata":{"trusted":true},"cell_type":"code","source":"mod_conv.fit(tr, epochs=5, validation_data=val)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The accuracy for both training and validation are still very low as predicted due to the low number of epochs and layers. It is slightly increased because I am fitting some data on a previously trained model. I should also note that I do not plan on augmenting the validation images as I want to keep a check of classifying real-life photos instead of possible altered and unrealistic photos that the training set should now contain"},{"metadata":{},"cell_type":"markdown","source":"## Building the Neural Network"},{"metadata":{},"cell_type":"markdown","source":"### Creating Convolutional Layers"},{"metadata":{"trusted":true},"cell_type":"code","source":"for image, label in tr.take(1): #Get first batch of images\n    for i in range(10): #take 10 images from first batch\n        fig, ax = plt.subplots()\n        plt.title(\"train\")\n        plt.imshow(image[i])\n        \nfor image, label in val.take(1): \n    for i in range(10): \n        fig, ax = plt.subplots()\n        plt.title(\"val\")\n        plt.imshow(image[i])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As seen above, much of the flower images are not of the actual flowers themselves, but of the background. Due to this, I believe that the first convolutional layer should have a fairly large size. From the images above, a 25x25 filter size is good. However, there is more differentiating details of the flower when you move toward its middle. This is why I think stride size should be small throughout. The next convolutional layers should have a smaller filter size and stride. First I will try to buld a model with three convolutional layers and a basic dense layer. I will also increase the epoch size to about 30."},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope(): #This is for determining if the model runs on the TPU\n    mod_conv_2 = keras.Sequential()\n    mod_conv_2.add(Conv2D(32, kernel_size=25, strides=3,padding='same' ,activation='relu', input_shape=(IMAGE_SIZE[0], IMAGE_SIZE[1], 3)))\n    mod_conv_2.add(MaxPooling2D(pool_size=(6, 6))) \n    mod_conv_2.add(Conv2D(64, kernel_size=11,padding='same' ,activation='relu'))\n    mod_conv_2.add(MaxPooling2D(pool_size=(2, 2))) \n    mod_conv_2.add(Conv2D(128, kernel_size=3 ,padding='same' ,activation='relu'))\n    mod_conv_2.add(MaxPooling2D(pool_size=(2, 2))) \n    mod_conv_2.add(Flatten())\n    mod_conv_2.add(Dense(512, activation='relu'))\n    mod_conv_2.add(Dense(104,activation='softmax'))\n    o = keras.optimizers.SGD(learning_rate = 0.05, momentum=0.3)\n    mod_conv_2.compile(optimizer=o, loss='sparse_categorical_crossentropy', metrics=['accuracy'])\nmod_conv_2.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = mod_conv_2.fit(tr, epochs=30, validation_data=val)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plotLoss(ep, h):\n    fig, ax = plt.subplots()\n    plt.plot(range(ep), h.history['loss'], label='training loss')\n    plt.plot(range(ep), h.history['val_loss'], label='validation loss')\n    plt.legend()\n    plt.xlabel(\"Epoch\")\n    plt.ylabel(\"Loss\")\n    \ndef plotAcc(ep, h):\n    fig, ax = plt.subplots()\n    plt.plot(range(ep), h.history['accuracy'], label='accuracy')\n    plt.plot(range(ep), h.history['val_accuracy'], label='validation accuracy')\n    plt.legend()\n    plt.xlabel(\"Epoch\")\n    plt.ylabel(\"Accuracy\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plotLoss(30, history)\nplotAcc(30, history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As seen above, while training accuracy and loss becomes better, the validation accuracy peaks and loss reaches its lowest point at around 20 epochs. This means that our current model is overfitting"},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope(): #This is for determining if the model runs on the TPU\n    mod_conv_3 = keras.Sequential()\n    mod_conv_3.add(Conv2D(32, kernel_size=11,padding='same' ,activation='relu', input_shape=(IMAGE_SIZE[0], IMAGE_SIZE[1], 3)))\n    mod_conv_3.add(MaxPooling2D(pool_size=(2, 2))) \n    mod_conv_3.add(Conv2D(64, kernel_size=3 ,padding='same' ,activation='relu'))\n    mod_conv_3.add(MaxPooling2D(pool_size=(2, 2))) \n    mod_conv_3.add(Flatten())\n    mod_conv_3.add(Dense(512, activation='relu'))\n    mod_conv_3.add(Dense(104,activation='softmax'))\n    o = keras.optimizers.SGD(learning_rate = 0.05, momentum=0.3)\n    mod_conv_3.compile(optimizer=o, loss='sparse_categorical_crossentropy', metrics=['accuracy'])\nmod_conv_3.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The model above removes the top layer and decreases the filter size, strides, and number of pooling layers. However, the size of each pooling layer increases. Overfitting may have been possible due to the large size of the filters which led to liberal generalizations in the pooling layer"},{"metadata":{"trusted":true},"cell_type":"code","source":"history = mod_conv_3.fit(tr, epochs=30, validation_data=val)\nplotLoss(30, history)\nplotAcc(30, history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The new model has an even more egregious overfitting problem. While the neural network is able to get to near a 100% in accuracy with the training dataset, the validation accuracy flattens out at about 25%. I also noticed that the number of nodes in the hidden layer (512) are a lot less than in the input (147456), so I will try to add more dense layers that are larger. One other aspect is to increase the number of filters produced, which may also help to boost validation scores as well. While validation scores are still very low, the model above did do a much better job with classifying on the training data, so it is possible that the convolutional layers need not change"},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope(): #This is for determining if the model runs on the TPU\n    mod_conv_4 = keras.Sequential()\n    mod_conv_4.add(Conv2D(32, kernel_size=11,padding='same' ,activation='relu', input_shape=(IMAGE_SIZE[0], IMAGE_SIZE[1], 3)))\n    mod_conv_4.add(Conv2D(32, kernel_size=11 ,activation='relu', input_shape=(IMAGE_SIZE[0], IMAGE_SIZE[1], 3)))\n    mod_conv_4.add(MaxPooling2D(pool_size=(2, 2))) \n    mod_conv_4.add(Conv2D(64, kernel_size=3 ,padding='same' ,activation='relu'))\n    mod_conv_4.add(Conv2D(64, kernel_size=3,activation='relu'))\n    mod_conv_4.add(MaxPooling2D(pool_size=(2, 2))) \n    mod_conv_4.add(Flatten())\n    mod_conv_4.add(Dense(2048, activation='relu'))\n    mod_conv_4.add(Dense(104,activation='softmax'))\n    o = keras.optimizers.SGD(learning_rate = 0.05, momentum=0.3)\n    mod_conv_4.compile(optimizer=o, loss='sparse_categorical_crossentropy', metrics=['accuracy'])\nmod_conv_4.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = mod_conv_4.fit(tr, epochs=30, validation_data=val)\nplotLoss(30, history)\nplotAcc(30, history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The results are much worse for this neural network than the previous one. The previous neural network produced the best results despite its overfitting. However, I also notice that another version of accuracy - sparse categorical accuracy - which I will try on with the previous two models\n\nI should also mention that I could not make the first Dense layer as large as the output of the Flatten layer, as that kept on throwing size errors. I made the Dense layer as large as I could without breaking the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plotSparse(ep, h):\n    fig, ax = plt.subplots()\n    plt.plot(range(ep), h.history['sparse_categorical_accuracy'], label='accuracy')\n    plt.plot(range(ep), h.history['val_sparse_categorical_accuracy'], label='validation accuracy')\n    plt.legend()\n    plt.xlabel(\"Epoch\")\n    plt.ylabel(\"Accuracy\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    mod_conv_5 = keras.models.clone_model(mod_conv_3)\n    o = keras.optimizers.SGD(learning_rate = 0.05, momentum=0.3)\n    mod_conv_5.compile(optimizer=o, loss='sparse_categorical_crossentropy', metrics=['sparse_categorical_accuracy'])\nmod_conv_5.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = mod_conv_5.fit(tr, epochs=30, validation_data=val)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plotLoss(30, history)\nplotSparse(30, history)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    mod_conv_6 = keras.models.clone_model(mod_conv_4)\n    o = keras.optimizers.SGD(learning_rate = 0.05, momentum=0.3)\n    mod_conv_6.compile(optimizer=o, loss='sparse_categorical_crossentropy', metrics=['sparse_categorical_accuracy'])\nmod_conv_6.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = mod_conv_6.fit(tr, epochs=30, validation_data=val)\nplotLoss(30, history)\nplotSparse(30, history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As seen above, there is no difference with the new metric for the duplicate 3rd model. For the duplicate 4th model though, while it generally stays the same for most of it, it does start to improve towards the end especially for loss. We can try to run this for 20 more epochs"},{"metadata":{"trusted":true},"cell_type":"code","source":"history = mod_conv_6.fit(tr, epochs=20, validation_data=val)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plotLoss(20, history)\nplotSparse(20, history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As seen above, the validation sparse categorical accuracy does continue to improve but very very slowly, so it is not very efficient to train this. None of the convolutional neural networks from scratch are working particularly well, which is why I plan on implementing a pre-trained neural network into the model"},{"metadata":{},"cell_type":"markdown","source":"### Using Pre-trained Models\n\n\nAfter doing some research, I believe that the best Neural Network Archtitectures to use would be ResNet and Google's InceptionNet, as they seem to be the most detailed and least erroneous. Both will be accessing the ImageNet database. After looking around on the ImageNet website, they do seem to have a detailed database of flowers"},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    pt = keras.applications.resnet.ResNet50(weights='imagenet', input_shape=[*IMAGE_SIZE, 3], include_top=False)\n    mod_pt_1 = keras.Sequential()\n    mod_pt_1.add(pt)\n    mod_pt_1.add(keras.layers.GlobalAveragePooling2D()) #using this to get average of large datasets\n    mod_pt_1.add(Dense(2048, activation='relu'))\n    mod_pt_1.add(Dense(104,activation='softmax'))\n    o = keras.optimizers.SGD(learning_rate = 0.05, momentum=0.3)\n    mod_pt_1.compile(optimizer=o, loss='sparse_categorical_crossentropy', metrics=['sparse_categorical_accuracy'])\nmod_pt_1.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = mod_pt_1.fit(tr, epochs=30, validation_data=val)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plotLoss(30, history)\nplotSparse(30, history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As seen above, while there is still some overfitting, this pretrained model does a much better job at classifying validation data and therefore overfits less than any of the made from scratch convolutional models. "},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    pt = keras.applications.inception_v3.InceptionV3(weights='imagenet', input_shape=[*IMAGE_SIZE, 3], include_top=False)\n    mod_pt_2 = keras.Sequential()\n    mod_pt_2.add(pt)\n    mod_pt_2.add(keras.layers.GlobalAveragePooling2D()) #using this to get average of large datasets\n    mod_pt_2.add(Dense(2048, activation='relu'))\n    mod_pt_2.add(Dense(104,activation='softmax'))\n    o = keras.optimizers.SGD(learning_rate = 0.05, momentum=0.3)\n    mod_pt_2.compile(optimizer=o, loss='sparse_categorical_crossentropy', metrics=['sparse_categorical_accuracy'])\nmod_pt_2.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = mod_pt_2.fit(tr, epochs=30, validation_data=val)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plotLoss(30, history)\nplotSparse(30, history)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The InceptionNet optimizer is by far the most effective model made yet. While there is still a slight overfitting concern, the loss and accuracy scores for both training and validation are the closest to each other out of all the models. It does not make more sense to train for more epochs on this model as the curves flatten out"},{"metadata":{},"cell_type":"markdown","source":"### Optimizers\n\nso far I have been using a custom SGD optimizer with a `learning_rate = 0.05` and `momentum = 0.3`. From previous experiences throughout the semester, I have learned that too high of a learning rate and/or momentum is generally very faulty, but I can still re-test them again. Aditionally, I will be testing Adagrad and Adam optimizers. Since the InceptionNet model gave the best result of the models, I will be reusing that"},{"metadata":{"trusted":true},"cell_type":"code","source":"opt = [keras.optimizers.SGD(momentum=0.7), keras.optimizers.SGD(learning_rate = 0.7), 'adagrad', 'adam']\nhist = []\nfor o in opt:\n    with strategy.scope():\n        pt = keras.applications.inception_v3.InceptionV3(weights='imagenet', input_shape=[*IMAGE_SIZE, 3], include_top=False)\n        m = keras.Sequential()\n        m.add(pt)\n        m.add(keras.layers.GlobalAveragePooling2D()) #using this to get average of large datasets\n        m.add(Dense(2048, activation='relu'))\n        m.add(Dense(104,activation='softmax'))\n        m.compile(optimizer=o, loss='sparse_categorical_crossentropy', metrics=['sparse_categorical_accuracy'])\n    history = m.fit(tr, epochs=30, validation_data=val)\n    print('~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~')\n    hist.append(history)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(len(hist))\nfor i,h in enumerate(hist):\n    if i == 0:\n        print('High Momentum')\n    elif i == 1:\n        print('High learning rate')\n    elif i == 2:\n        print('Adagrad')\n    else:\n        print('Adam')\n    plotLoss(30, h)\n    plotSparse(30, h)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the result above, it seems like the custom optimzer used throughout the notebook and adagrad are very similar to each other, but it seems like adagrad narrowly edges out as the best optimizer with a slightly higher accuracy rate"},{"metadata":{},"cell_type":"markdown","source":"### Other Tweaks\n\n#### Batch Sizing\n\nAs mentioned earlier, I said that batch sizes being too large could've been a problem, so I will try to reduce them now and test those out"},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    pt = keras.applications.inception_v3.InceptionV3(weights='imagenet', input_shape=[*IMAGE_SIZE, 3], include_top=False)\n    best_model = keras.Sequential()\n    best_model.add(pt)\n    best_model.add(keras.layers.GlobalAveragePooling2D()) #using this to get average of large datasets\n    best_model.add(Dense(2048, activation='relu'))\n    best_model.add(Dense(104,activation='softmax'))\n    best_model.compile(optimizer='adagrad', loss='sparse_categorical_crossentropy', metrics=['sparse_categorical_accuracy'])\n\nbest_model.summary()\ntr_2 = training_dataset(batch_factor=3)\nval_2 = validation_dataset(batch_factor=3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bt_1 = keras.models.clone_model(best_model)\nbt_1.compile(optimizer='adagrad', loss='sparse_categorical_crossentropy', metrics=['sparse_categorical_accuracy'])\nh = bt_1.fit(tr_2, epochs=20, validation_data=val_2)\nplotLoss(20, h)\nplotSparse(20, h)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I ended the training for this model early because it was taking too long and the accuracy was not significantly improving with lower batch size. Sticking to a batch size of 32 seems to be the most optimal at this point"},{"metadata":{},"cell_type":"markdown","source":"#### Dropout Layers\n\nIn order to make sure that there is no overfitting, I will add a small dropout layer. Since there are some overfitting concerns, I will have to make the dropout a bit high. This should yield a lower accuracy score for both the validation and train data"},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    pt = keras.applications.inception_v3.InceptionV3(weights='imagenet', input_shape=[*IMAGE_SIZE, 3], include_top=False)\n    final_model = keras.Sequential()\n    final_model.add(pt)\n    final_model.add(keras.layers.GlobalAveragePooling2D()) #using this to get average of large datasets\n    final_model.add(Dropout(0.3))\n    final_model.add(Dense(2048, activation='relu'))\n    final_model.add(Dense(104,activation='softmax'))\n    final_model.compile(optimizer='adagrad', loss='sparse_categorical_crossentropy', metrics=['sparse_categorical_accuracy'])\n\nfinal_model.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"h = final_model.fit(tr, epochs=30, validation_data=val)\nplotLoss(30, h)\nplotSparse(30, h)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Overall, the Dropuout layer didn't affect accuracy or loss all that much"},{"metadata":{},"cell_type":"markdown","source":"## Conclusions"},{"metadata":{"trusted":true},"cell_type":"code","source":"def export(model):\n    pic_batch = tst.map(lambda image, idnum: image)\n    p = model.predict(pic_batch)\n    pred = np.argmax(p, axis=-1)\n    i = 0\n    for f in pic_batch.unbatch().take(10):\n        fig, ax = plt.subplots()\n        plt.title(CLASSES[pred[i]])\n        plt.imshow(f)\n        i += 1\n\nexport(final_model)\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As seen above with some of the test data, the predictions have ended up not working so well. Nonetheless this was a very complex project with a lot of fitering required - many times the main features in the flower pic was not even the flower itself, which probably led to some inaccurate results. It's also clear that overfitting was prevalant in this model, and more bias towards validation data may have resulted with more and more runs. But given circumstances and time constraints, this model is probably the most accurate model so far - I will export and use this model in the web app."},{"metadata":{"trusted":true},"cell_type":"code","source":"final_model.save('final_model.h5')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.10"}},"nbformat":4,"nbformat_minor":4}