{"cells":[{"metadata":{},"cell_type":"markdown","source":"**[Deep Learning Home Page](https://www.kaggle.com/learn/deep-learning)**\n\n---\n"},{"metadata":{},"cell_type":"markdown","source":"## Tom Segal's \"Petal to the Metal\" competition entry - https://www.kaggle.com/c/tpu-getting-started ## \n\nfeatures include:\n\n2d colorful image identification using:\n- TPUs\n- oversampling\n- augmentation\n- pretraining\n- warmup layers\n- variable learning function\n- early stopping\n\nthis notebook is a result of researching and borrowing of content from other notebooks. I learned a lot creating it and I acknowledge the support of this great community for my learning experience.\n\nI specifically learned a lot by using these notebooks:\n\nhttps://www.kaggle.com/yihdarshieh/detailed-guide-to-custom-training-with-tpus\n\nhttps://www.kaggle.com/philculliton/a-simple-petals-tf-2-2-notebook\n\nhttps://www.kaggle.com/dimitreoliveira/flower-classification-with-tpus-eda-and-baseline\n\nand of course the provided documentation from Kaggle:\n\nhttps://www.kaggle.com/ryanholbrook/create-your-first-submission\n\nhttps://www.kaggle.com/c/tpu-getting-started\n\nhttps://www.kaggle.com/c/tpu-getting-started/data\n\nhttps://www.kaggle.com/ryanholbrook/petal-helper\n\n\nmore useful resources:\n\nthe available pre-trained models https://keras.io/api/applications/ of which I used Xception, VGG16 and DenseNet201\n\nhttps://www.kaggle.com/timoboz/data-science-cheat-sheets\n\n\n\n"},{"metadata":{},"cell_type":"markdown","source":"## import ##"},{"metadata":{"trusted":true},"cell_type":"code","source":"# there is a library of helper functions for this challenge, petal_helper, which was not used. One can import it using:\n# from petal_helper import *\n# documentation at https://www.kaggle.com/ryanholbrook/petal-helper\n\nfrom kaggle_datasets import KaggleDatasets # used to retrieve the dataset\nimport tensorflow as tf\nimport re\nimport math\nimport numpy as np\nimport os\nimport seaborn as sns\nfrom collections import Counter\nfrom matplotlib import pyplot as plt\nfrom matplotlib import gridspec\nimport itertools\nfrom sklearn.metrics import f1_score, precision_score, recall_score,confusion_matrix\nprint(\"the version of Tensorflow currently used is \" + tf.__version__)\nfrom tensorflow.keras.callbacks import ModelCheckpoint,EarlyStopping,LearningRateScheduler\nfrom tensorflow.keras import optimizers","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"define the randomness"},{"metadata":{"trusted":true},"cell_type":"code","source":"seed = 0 # the random seed to be used. a fixed seed helps comparing results in a more systematic manner.\nnp.random.seed(seed)\ntf.random.set_seed(seed)\nos.environ['PYTHONHASHSEED'] = str(seed)\nos.environ['TF_DETERMINISTIC_OPS'] = '1'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"a list of the flower classes, copy-pasted from the petal-helper library."},{"metadata":{"trusted":true},"cell_type":"code","source":"CLASSES = ['pink primrose',    'hard-leaved pocket orchid', 'canterbury bells', 'sweet pea',     'wild geranium',     'tiger lily',           'moon orchid',              'bird of paradise', 'monkshood',        'globe thistle',         # 00 - 09\n           'snapdragon',       \"colt's foot\",               'king protea',      'spear thistle', 'yellow iris',       'globe-flower',         'purple coneflower',        'peruvian lily',    'balloon flower',   'giant white arum lily', # 10 - 19\n           'fire lily',        'pincushion flower',         'fritillary',       'red ginger',    'grape hyacinth',    'corn poppy',           'prince of wales feathers', 'stemless gentian', 'artichoke',        'sweet william',         # 20 - 29\n           'carnation',        'garden phlox',              'love in the mist', 'cosmos',        'alpine sea holly',  'ruby-lipped cattleya', 'cape flower',              'great masterwort', 'siam tulip',       'lenten rose',           # 30 - 39\n           'barberton daisy',  'daffodil',                  'sword lily',       'poinsettia',    'bolero deep blue',  'wallflower',           'marigold',                 'buttercup',        'daisy',            'common dandelion',      # 40 - 49\n           'petunia',          'wild pansy',                'primula',          'sunflower',     'lilac hibiscus',    'bishop of llandaff',   'gaura',                    'geranium',         'orange dahlia',    'pink-yellow dahlia',    # 50 - 59\n           'cautleya spicata', 'japanese anemone',          'black-eyed susan', 'silverbush',    'californian poppy', 'osteospermum',         'spring crocus',            'iris',             'windflower',       'tree poppy',            # 60 - 69\n           'gazania',          'azalea',                    'water lily',       'rose',          'thorn apple',       'morning glory',        'passion flower',           'lotus',            'toad lily',        'anthurium',             # 70 - 79\n           'frangipani',       'clematis',                  'hibiscus',         'columbine',     'desert-rose',       'tree mallow',          'magnolia',                 'cyclamen ',        'watercress',       'canna lily',            # 80 - 89\n           'hippeastrum ',     'bee balm',                  'pink quill',       'foxglove',      'bougainvillea',     'camellia',             'mallow',                   'mexican petunia',  'bromelia',         'blanket flower',        # 90 - 99\n           'trumpet creeper',  'blackberry lily',           'common tulip',     'wild rose']                                                                                                                                               # 100 - 102","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## retrieve TPU ##"},{"metadata":{"trusted":true},"cell_type":"code","source":"# this code tries to retrieve a TPU and then sets TensorFlow to use the accelerator that we get,\n# so if we don't get a TPU it will get a GPU or a CPU and run Tensorflow on it\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver() \n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() \n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"we see that as an output we got \"Running on TPU\" so we did get a TPU to run Tensorflow on. Replicas 8 means we have 8 TPUs to work on."},{"metadata":{},"cell_type":"markdown","source":"get the path to the dataset. the data structure is shown here:\nhttps://www.kaggle.com/c/tpu-getting-started/data"},{"metadata":{"trusted":true},"cell_type":"code","source":"# note that we could have also used the get_gcs_path function from petal-helper\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('tpu-getting-started') \nprint(GCS_DS_PATH)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"select the data. the flower pictures come in 4 different sizes. here 192x192, the lowest size, was used, in order to have faster running times, and because the performance of the different strategies is expected to scale monotonously with the picture size, such that the strategy comparison written at the end remains valid."},{"metadata":{"trusted":true},"cell_type":"code","source":"GCS_PATH_SELECT = { # available image sizes\n    192: GCS_DS_PATH + '/tfrecords-jpeg-192x192',\n    224: GCS_DS_PATH + '/tfrecords-jpeg-224x224',\n    331: GCS_DS_PATH + '/tfrecords-jpeg-331x331',\n    512: GCS_DS_PATH + '/tfrecords-jpeg-512x512'\n}\n# Select the dataset containing the size we chose above\nIMAGE_SIZE = [192,192]\nHEIGHT = IMAGE_SIZE[0]\nWIDTH = IMAGE_SIZE[1]\nGCS_PATH = GCS_PATH_SELECT[IMAGE_SIZE[0]]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"helper functions for loading the datasets\n\nnext the datasets will be loaded. There are three, train, validation and test. They will be mapped according to read_labeled_tfrecord and read_unlabeled_tfrecord according to whether they are labeled (train, validation) or not (test). Both functions rely on decode_image to decode the images. \n\n\nThe TFRecord format is a container format frequently used in Tensorflow to group and shard data data files for optimal training performace.\nEach file contains the id, label the class of the sample, for training data) and img (the actual pixels in array form) information for many images."},{"metadata":{"trusted":true},"cell_type":"code","source":"# decode_image decodes image_data into a 3-channeled (RGB) JPEG, normalizes the values to a [0,1] float range\n# and reshapes it into the given size \ndef decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0  # convert image to floats in [0, 1] range\n    image = tf.reshape(image, [*IMAGE_SIZE, 3]) # explicit size needed for TPU. The \"*\" turns the *[192,192] to 192,192.\n    return image\n\n# read_labeled_tfrecord reads example and returns its splitted components image and label.\ndef read_labeled_tfrecord(example):\n    # define the expected format: image, a tf.string, and class, a tf.int64.\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"class\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n    }\n    # parse example\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    # decode the image and cast the label obtained from example and return them\n    image = decode_image(example['image'])\n    label = tf.cast(example['class'], tf.int32)\n    return image, label # returns a dataset of (image, label) pairs\n\n# read_unlabeled_tfrecord reads example and returns its splitted components image and id.\ndef read_unlabeled_tfrecord(example):\n    # define the expected format: image, a tf.string as before, but in addition id, a tf.string, instead of class.\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"id\": tf.io.FixedLenFeature([], tf.string),  # shape [] means single element\n        # class is missing, this competitions's challenge is to predict flower classes for the test dataset\n    }\n    # parse example\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    # decode the image and return both it and the id obtained from example and return them\n    image = decode_image(example['image'])\n    idnum = example['id']\n    return image, idnum # returns a dataset of image(s)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## load the data ##"},{"metadata":{"trusted":true},"cell_type":"code","source":"# loadData is a general function which will be used for loading all three datasets.\n# filePaths are the paths of the tfrecords to be loaded. labeled defines which mapping function to use\n# ordered determines whether to read at a random order, which then speeds up the process.\ndef loadData(filePaths , labeled = True , ordered = False):\n    # multiple files are read simultaneously at random order by default in order to speed things up in case not ordered.\n    if not ordered:\n        tf.data.Options().experimental_deterministic = False # read at random order\n    dataset = tf.data.TFRecordDataset(filePaths) # read from multiple files simulataneously\n    dataset = dataset.with_options(tf.data.Options()) # read data as it is streamed and not in its original order if set\n    # if labeled = True, return pairs of (image, label) otherwise (image, id)\n    dataset = dataset.map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord)\n    return dataset\n\n# dataAugment augments the data by receiving the images and their labels and performing a set of random modifications\n# on the images before returning them.\n# data augmentation was attempted but ultimately not used. \ndef dataAugment(image, label):\n    image = tf.image.random_flip_left_right(image, seed=seed) # randomly flip the image horizontally \n    image = tf.image.random_flip_up_down(image, seed=seed) # randomly flip the image vertically\n    #image = tf.image.random_saturation(image, 0,2, seed=seed) \n    #image = tf.image.random_saturation(image, lower=0.5, upper=1.5, seed=seed)\n    #image = tf.image.random_contrast(image, lower=.5, upper=1.5, seed=seed)\n    #image = tf.image.random_brightness(image, max_delta=0.8, seed=seed)\n    #image = tf.image.random_crop(image, size=[int(HEIGHT*.8), int(WIDTH*.8), 3], seed=seed)\n    return image, label\n\n\ndataSetTrainPathString = GCS_PATH + \"/train/*.tfrec\" # define the file paths of the training data\ndataSetTrainPaths = tf.io.gfile.glob(dataSetTrainPathString) # obtain all of these file paths\n# (tf.io.gfile.glob retrieves all the files that match the string in the folder)\ndataTrain = loadData(dataSetTrainPaths, labeled = True, ordered = False) # load the training data\n\n# augments the data by creating different versions of it with different modifications as explained above.\n# note that this was experimented with but not used.\n# dataTrain = dataTrain.map(dataAugment, num_parallel_calls = tf.data.experimental.AUTOTUNE)\n# dataTrain = dataTrain.repeat() # set the train data to repeat so it repeats for several epochs.\n# note that repeat is commented-out because setting the dataTrain to repeat at this stage will cause problems in the\n# data visualiation sections later on. So we turn it off for now and turn it on later.\n# for example setting it to repeat causes an infinite loop while counting the labels below.\n# the training data will be shuffled, such that the network will train on images in different order for each epoch.\n# it is done by using the following line. However we don't do this at this stage because it would interfere with analysing\n# the data beforehand.\n# dataTrain = dataTrain.shuffle(2048) # shuffle such that we will train on images in different order in each epoch\n# the training data will be batched in recommended batch sizes of 16 times the TPU units in order for the TPUs\n# to be able to handle them more efficiently. This will be 16 with TPU off and 16*8=128 with TPU on\nbatchSize = 16 * strategy.num_replicas_in_sync\ndataTrain = dataTrain.batch(batchSize)\n# prefatch the next batch while training to improve speed.\n# note that tf.data.experimental.AUTOTUNE means that the API will read from multiple files if they are available\ndataTrain = dataTrain.prefetch(tf.data.experimental.AUTOTUNE)\nprint(\"train data: \" , dataTrain) # display the training data shapes\n\n# load the validation data. Similar to the training data only that no augmentation was considered,\n# and theres no shuffling.\ndataSetValPathString = GCS_PATH + \"/val/*.tfrec\"\ndataSetValPaths = tf.io.gfile.glob(dataSetValPathString)\ndataVal = loadData(tf.io.gfile.glob(dataSetValPaths) , labeled = True , ordered = True)\n# it is not needed to repeat the val data as we don't train on them, but we would still like\n# to feed them in batches to make the TPU work at optimal speed\ndataVal = dataVal.batch(batchSize)\ndataVal = dataVal.cache() # caches the elements of the dataset in memory\ndataVal = dataVal.prefetch(tf.data.experimental.AUTOTUNE)\nprint(\"validation data: \" , dataVal)\n\n# load the test set.\n# note that this dataset is not augmented, shuffled, cached or labeled.\ndataSetTestPathString = GCS_PATH + \"/test/*.tfrec\"\ndataSetTestPaths = tf.io.gfile.glob(dataSetTestPathString)\ndataTest = loadData(dataSetTestPaths , labeled = False , ordered = True)\ndataTest = dataTest.batch(batchSize)\ndataTest = dataTest.prefetch(tf.data.experimental.AUTOTUNE)\nprint(\"test data: \" , dataTest)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"count the number of images in each dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"myStr = \"flowers00-230.tfrec\"\nprint(myStr)\nprint(str(re.compile(r\"-([0-9]*\\.)\").search(myStr).group(1)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# count_data_items counts the amount of items in filenames.\ndef count_data_items(filenames):\n    # due to the filename structure of i.e. flowers00-230.tfrec, the number is located in the file name: 230\n    # a re (regular expression) is used to locate the number:\n    # r\"something\" means \"something\" is handled in raw string notiation which means \"\\\" are not handled in any special way\n    # \"-\" means look for a \"-\", but don't consider it a part of the re as it is outside the round brackets ()\n    # [0-9] means any single digit\n    # [0-9]* means any number of single digits in a row\n    # \\. means look for \".\" (\"\\\" is used here as an escape character)\n    # this way one obtains i.e. flowers00-230.tfrec = 230 data items\n    # see https://docs.python.org/3/library/re.html\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n\n# calculate the number of images for each set. Note that the number of images for the train data is called\n# originalNumTrainImages and not numTrainImages as oversampling and augmentation was experimented with.\noriginalNumTrainImages = count_data_items(dataSetTrainPaths)\nnumValImages = count_data_items(dataSetValPaths)\nnumTestImages = count_data_items(dataSetTestPaths)\n\nprint (\"Number of training images: {}\".format(originalNumTrainImages))\nprint (\"Number of validation images: {}\".format(numValImages ))\nprint (\"Number of test images: {}\".format(numTestImages))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"the result: 12753 train, 3712 validation and 7382 test images."},{"metadata":{},"cell_type":"markdown","source":"label counting\n\nit is important to check whether the labels are evenly represented\n\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# getLabelCounting counts the number of appearances of each lebel in labeled_database.\n# tf.autograph.experimental.do_not_convert prevents tf from converting this function into a graph in order\n# to supress errors. This is ok because the function is not used as part of the input for the machine learning procedure.\n@tf.autograph.experimental.do_not_convert\ndef getLabelCounting(labeled_dataset):\n    c = Counter() # labels counter\n    labels = []\n    # retrieve the different batches from the labeled_dataset. For each one, append it to the labels array\n    # after filling up the labels array, concatenate it and update the counter with the labels.\n    for batch in labeled_dataset.map(lambda image, label: label, num_parallel_calls=tf.data.experimental.AUTOTUNE):\n        labels.append(batch)\n    labels = tf.concat(labels, axis=0).numpy()\n    c.update(labels)\n    return labels, c\n# if we call getLabelCounting using dataTrain while dataTrain is set to repeat we will get an infinite loop\n# but because it is commented-out above it is safe to execute this and we need to remember to set trainData\n# to repeat down below\n\ndataTrainLabelsIndex , dataTrainLabelsCount = getLabelCounting(dataTrain)\ndataValLabelsIndex , dataValLabelsCount = getLabelCounting(dataVal)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"plot the label distribution"},{"metadata":{"trusted":true},"cell_type":"code","source":"# plot_label_dist plots the label distributions dist_1 and dist_2 (named dist_label_1 and dist_label_2)\n# of the labels set labels\ndef plot_label_dist(labels, dist_1, dist_2, dist_label_1, dist_label_2, title=''):\n    \n    x = np.arange(len(labels)) # the horizontal axis, as in the label locations\n    width = 0.3 # the width of the bars\n\n    fig, ax = plt.subplots(figsize=(15, 4)) # height and width of the plots\n    # offset the plots from one another by width to make them more readable\n    rects1 = ax.bar(x - width / 2, dist_1, width, label=dist_label_1)\n    rects2 = ax.bar(x + width / 2, dist_2, width, label=dist_label_2)\n\n    ax.set_ylabel('portion in dataset') # vertical axis name\n    ax.set_title(title) # graph name\n    ax.set_xticks(x) # horizontal axis ticks\n    ax.set_xticklabels([str(x) if x % 5 in [0] else '' for x in range(len(labels))] ) # horizontal axis tick names\n    ax.legend() # add a legend at the top right\n    plt.show()\n    plt.close()\n    \nlabels = list(range(len(CLASSES))) # convert the classes into a list of numberss\ndistTrain = [dataTrainLabelsCount[x] / originalNumTrainImages for x in labels] # the label dist for the train data\ndistVal = [dataValLabelsCount[x] / numValImages for x in labels] #  # the label dist for the validation data\n\nhalf = len(labels) // 2  # plot in 2 separate plots\n\n# plot the first half\nplot_label_dist(\n    labels[:half],\n    distTrain[:half],\n    distVal[:half],\n    'Train',\n    'Valid',\n    title='Labels distribution of the Train and Validation Datasets: Labels 0-{}'.format(half - 1)\n)\n\n# plot the second half\nplot_label_dist(\n    labels[half:],\n    distTrain[half:],\n    distVal[half:],\n    'Train',\n    'Valid',    \n    title='Labels distribution of the Train and Validation Datasets: Labels {}-{}'.format(half, len(labels) - 1)\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"numerical representation:"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(dataTrainLabelsCount)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"we see that the labels are not distributed evenly. Some have 6% of the pictures while others have about a tenth of that. To balance that we could either create weights with larger weights for under-represented labels (\"class weights\"), or repeating the pictures of the under-represented labels (\"oversampling\".\nThese two options would have practically identical results, but seeing how we are working in batches, it is better to oversample as the change is more spread out across batches and thus provides a smoother gradient signal.\n\nwe see that lebel 67 appears 782 times while label 44 appears only 18 times\nnext we decide whats the minimal occurance of each class that we want to have. We could choose any number between 18 and 782. If we choose 18 we would not duplicate any images, and if we choose 782 we will duplicate the maximum amount of images. We don't need it to be completely balanced and maybe by doing so we also overdo it with the oversampling.\nSo instead we will choose 200.\n"},{"metadata":{},"cell_type":"markdown","source":"define the functions necessary for oversampling the train data."},{"metadata":{"trusted":true},"cell_type":"code","source":"# get_num_to_repeat_for_class calculates the number of times a training data with label label class_id\n# should be repeated to reach target_counting appearances total. Calling this function for all class ids will\n# result in a training datasetwhere  each class occurs more or less target_counting times.\n# \"more or less\" because the number returned has some randomness to it.\ndef get_num_to_repeat_for_class(class_id, target_counting):\n    # count how many train images we have of the class class_id\n    counting = dataTrainLabelsCount[class_id]\n    \n    # if the class already has at least target_counting appearances return 1 which means nothing needs to be done.\n    if counting >= target_counting:\n        return 1.0\n    # otherwise the ratio between the desired and existing appearances is the desired ratio\n    num_to_repeat = target_counting / counting\n    return num_to_repeat\n\n# get_nums_to_repeat calls get_num_to_repeat_for_class for every class in order to obtain the number of necessary\n# occurences for each image in every class in order for the entire dataset to have all of its labels being\n# more or less equally represented target_counting times\ndef get_nums_to_repeat(target_counting):\n    \"\"\"Compute a table that stores the results of `get_num_to_repeat_for_class`\n       for every class and for the given `target_counting`.\n    \n    Args:\n        target_counting: int, the targeted occurrence number.\n    \n    Returns:\n        table: A `tf.lookup.StaticHashTable`.\n        d: A dictionary storing the same information as `table`.\n    \"\"\"\n    \n    keys = range(len(CLASSES)) # numerical representation of the different classes/labels\n    # calculate the number of appearances each class needs to have in order to reach\n    #   a uniform distribution of target_counting\n    values = [get_num_to_repeat_for_class(x, target_counting) for x in keys]\n\n    keys_tensor = tf.constant(keys) # the label indices are saved in a keys tensor\n    vals_tensor = tf.constant(values) # the values for each label are saved in a values tensor\n    \n    # a static hash table of type tf.lookup.StaticHashTable is initialized with the keys and values \n    table_initializers = tf.lookup.KeyValueTensorInitializer(keys_tensor, vals_tensor)\n    \n    table = tf.lookup.StaticHashTable(table_initializers, default_value=0.0)\n    # the static hash table is stored into a dictionary \n    d = {k: v for k, v in zip(keys, values)}\n\n    return table, d\n\n# get_num_to_repeat_for_example uses the table calculated by get_nums_to_repeat, which contains inprecise, float\n# values, in order to calculate final, int values, which also contain an element of randomness\n# example is a training data example, a tuple of 2 tensors, image and label.\ndef get_num_to_repeat_for_example(example, table):\n    \"\"\"Compute the actual number of times a training example will repeat\n       in order to get a dataset where each class occur <approximately> \n       N times with N being a pre-defined number that is used for constructing\n       `table`.\n\n    Args:\n        example: A tuple of 2 tensors, which is a labeled training example and\n            represented as (image, label).\n                          \n        tabel: A tf.lookup.StaticHashTable, as obtained from `get_nums_to_repeat`.\n                          \n    Returns:\n        A tf.int64 scalar tensor, the number of times `example` will repeat.\n    \"\"\"\n    \n    image, label = example\n    num_to_repeat = table.lookup(label) # obtain the non-final repetition number for this label    \n    num_to_repeat_integral = tf.cast(int(num_to_repeat), tf.float32) # cast it into an int and save it separately\n    # With a probability `residue`, we allow `example` to repeat one more time.\n    residue = num_to_repeat - num_to_repeat_integral # calculate the difference between the float and the int\n    # randomly decide in which direction to round it\n    num_to_repeat = num_to_repeat_integral + tf.cast(tf.random.uniform(shape=()) <= residue, tf.float32)\n    return tf.cast(num_to_repeat, tf.int64)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"calculate the number of times each picture should be repeated for each label. Note that this is the exact number, as in, a float.\nI tried N=200 but saw the data set still looked very non-uniform so I went up to 400. I did not want to select too large of a number like 600 or 700 in order to not overdo with the oversampling."},{"metadata":{"trusted":true},"cell_type":"code","source":"numOfRepetitions = 400\n_, d = get_nums_to_repeat(numOfRepetitions) # retrieve the repetitions dictionary (don't save the static hash table)\nd = sorted(d.items(), key=lambda x: x[1], reverse=True) # sort in reverse order according to the value\n\nprint('how many repetitions for every label? (class id, num to repeat)\\n')\nfor x in d:\n    print(x)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## oversampling ##"},{"metadata":{"trusted":true},"cell_type":"code","source":"# get_oversampled_training_dataset uses loadData to load the training data set and then creates a modified version\n# of it dataset which is oversampled such that each class has approximately target_counting repetitions.\n# oversample determines whether to oversample or not. The oversampled data can be augmented using an augmentation\n# function augmentation_fn which is performed with probability probability.can be used to augment the oversampled data\n# using an augmentation function. the oversampled dataset can be set to repeat using repeat_dataset, to be ordered\n# using ordered and if not ordered is shuffled according to shuffle_buffer_size. It is also batched using batch_size.\ndef get_oversampled_training_dataset(\n        target_counting, batch_size, shuffle_buffer_size,\n        repeat_dataset=False, ordered=False,\n        oversample=True, augmentation_fn=None, probability=1.0\n    ):\n    \"\"\"\n    Construct an oversampled dataset in which each class occurs approximately\n    `target_counting` times.\n    \n    (Special) Args:\n    \n        target_counting: int, the target occurrence.\n        oversampe: bool, if to use oversampling. If `False`, no oversampliing and\n            the arguement `target_counting` has no effect.\n        augmentation_fn: A funtion used to map the dataset for data augmentation.\n        probability: float, the probability to perform the augmentation\n        \n    Returns:\n        A tf.data.Dataset.\n    \"\"\"\n    # retrieve the repetitions table and dictionary for the different classes\n    table, d = get_nums_to_repeat(target_counting)\n    \n    nb_examples = originalNumTrainImages # nb_examples initially holds the original number of training images\n    \n    dataset = loadData(dataSetTrainPaths, labeled=True, ordered=ordered) # load the training data set\n\n    if oversample:\n        # if oversampling, nb_examples holds an approximation of the number of images in the oversampled dataset\n        nb_examples = int(sum([dataTrainLabelsCount[k] *  v for k, v in d.items()]))\n        # map the different images in the dataset to repeat, each with its appropriate number of repetitions\n        dataset = dataset.flat_map(\n            lambda image, label: tf.data.Dataset.from_tensors((image, label)).repeat(get_num_to_repeat_for_example((image, label), table))\n        )\n        \n    if repeat_dataset: # if set to repeat, do so.\n        dataset = dataset.repeat()\n \n    # if not set to be ordered, shuffle it, either according to the given shuffle buffer size or according to the\n    # number of training images\n    if not ordered:\n        if not shuffle_buffer_size:\n            shuffle_buffer_size = nb_examples\n        dataset = dataset.shuffle(shuffle_buffer_size)\n    \n    dataset = dataset.batch(batch_size, drop_remainder=True) # separate the dataset into batches\n    \n    if augmentation_fn: # if an augmentation is given, augment the data with the given probability for each image\n        probability = tf.constant(probability, dtype=tf.float32)\n        dataset = dataset.map(\n            lambda images, labels: augmentation_fn(images, labels, probability=probability),\n            num_parallel_calls=tf.data.experimental.AUTOTUNE\n        )\n        \n    dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE) # prefetch the next batch while processing to impro\n    \n    return dataset, nb_examples\n\n# call get_oversampled_training_dataset and store the oversampled dataset in dataTrainOversampled\ndataTrainOversampled, _ = get_oversampled_training_dataset(target_counting=numOfRepetitions, batch_size=16, shuffle_buffer_size=1, repeat_dataset=False, ordered=True, oversample=True, augmentation_fn=None)\n\n\n_, oversampled_train_counter = getLabelCounting(dataTrainOversampled) # count the number of appearances of each label\nNUM_OVERSAMPLED_TRAINING_DATA = sum(oversampled_train_counter.values()) # sum them to arrive at the total dataset size\nprint('Oversampled training dataset:\\ntraining images: {}\\n'.format(NUM_OVERSAMPLED_TRAINING_DATA)) # print it\n\nprint(\"labels in the oversampled training dataset, sorted by occurrence: pairs of (label_id, label_counting)\\n\")\nprint(oversampled_train_counter.most_common())\n\nprint('\\n' + 'averaged number of occurrences: ', np.array(list(oversampled_train_counter.values())).mean())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"we now have 42912 instead of 12753 training images, about 3-4 times as much.\n\nas a result the new oversampled data has on average 412 pictures per label. Note that due to a random factor not all labels have at least 400 but rather a number around that.\n\ncomparison to what we had before:"},{"metadata":{"trusted":true},"cell_type":"code","source":"dist_train_oversampled = np.array([oversampled_train_counter[x] for x in labels]) / sum(oversampled_train_counter.values())\n\nhalf = len(labels) // 2\nplot_label_dist(\n    labels[:half], distTrain[:half], dist_train_oversampled[:half],'original','oversampled',\n    title='Labels distribution in the original and the oversampled train datasets: Labels 0-{}'.format(half - 1)\n)\n\nplot_label_dist(\n    labels[half:],distTrain[half:],dist_train_oversampled[half:],'original','oversampled',    \n    title='Label distribution in the original and the oversampled train datasets: Labels {}-{}'.format(half, len(labels) - 1)\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"the dataset is now significantly more balanced. At first I chose numOfRepetitions=200 and show that there were still large differences, so I changed it to numOfRepetitions=400.\nNote that we only changed the train dataset, not the validation dataset, so that now theres a big difference between their distributions.\n\n## visualize the oversampled dataset ##"},{"metadata":{},"cell_type":"markdown","source":"visualization functions"},{"metadata":{"trusted":true},"cell_type":"code","source":"def batch_to_numpy_images_and_labels(data):\n    images, labels = data\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    if numpy_labels.dtype == object: # binary string in this case, these are image ID strings\n        numpy_labels = [None for _ in enumerate(numpy_images)]\n    # If no labels, only image IDs, return None for labels (this is the case for test data)\n    return numpy_images, numpy_labels\n\ndef display_one_flower(image, title, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    if len(title) > 0:\n        plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n    return (subplot[0], subplot[1], subplot[2]+1)\n\ndef display_batch_of_images(databatch, predictions=None):\n    \"\"\"This will work with:\n    display_batch_of_images(images)\n    display_batch_of_images(images, predictions)\n    display_batch_of_images((images, labels))\n    display_batch_of_images((images, labels), predictions)\n    \"\"\"\n    # data\n    images, labels = batch_to_numpy_images_and_labels(databatch)\n    if labels is None:\n        labels = [None for _ in enumerate(images)]\n        # auto-squaring: this will drop data that does not fit into square or square-ish rectangle\n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n        \n    # size and spacing\n    FIGSIZE = 13.0\n    SPACING = 0.1\n    subplot=(rows,cols,1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n    else:\n        plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n    \n    # display\n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        title = '' if label is None else CLASSES[label]\n        correct = True\n        if predictions is not None:\n            title, correct = title_from_label_and_target(predictions[i], label)\n        dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n        subplot = display_one_flower(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n    \n    #layout\n    plt.tight_layout()\n    if label is None and predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dataTrainOversampledBatch = next(iter(dataTrainOversampled.unbatch().batch(25)))\ndisplay_batch_of_images(dataTrainOversampledBatch)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"note that the duplicate images all appear one after the other. It is therefore also important to shuffle the oversampled data. It is also advantageous to augment it."},{"metadata":{"trusted":true},"cell_type":"code","source":"# augment the oversampled data, set it to repeat and shuffle it\ndataTrainOversampledModified = dataTrainOversampled.map(dataAugment, num_parallel_calls = tf.data.experimental.AUTOTUNE)\ndataTrainOversampledModified = dataTrainOversampledModified.repeat() # set the train data to repeat so it repeats for several epochs                                                              \ndataTrainOversampledModified = dataTrainOversampledModified.shuffle(2048) # shuffle 2048 such that the network will train on images in differenet order\n# batch and prefetch it\ndataTrainOversampledModified = dataTrainOversampledModified.batch(batchSize)\ndataTrainOversampledModified = dataTrainOversampledModified.prefetch(tf.data.experimental.AUTOTUNE)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## visualize the validation dataset ##"},{"metadata":{"trusted":true},"cell_type":"code","source":"display_batch_of_images(next(iter(dataVal.unbatch().batch(9))))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## visualize the test dataset ##"},{"metadata":{"trusted":true},"cell_type":"code","source":"display_batch_of_images(next(iter(dataTest.unbatch().batch(9))))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## define model ##\nnext the model will be defined.\n\nIn this project I tried VGG16, Xception and DenseNet201, all are pretrained models of imagenet."},{"metadata":{"trusted":true},"cell_type":"code","source":"# we use warmup epochs, which are a few epochs in the beginning with especially high learning rates in order to\n# help the model to settle in around the global minimum area\n\nWARMUP_LEARNING_RATE = 1e-4 * strategy.num_replicas_in_sync\noptimizer = optimizers.Adam(lr=WARMUP_LEARNING_RATE)\nmetric_list = ['sparse_categorical_accuracy']\n\nwith strategy.scope():\n    # the model used is a pretrained model using the weights of imagenet\n    #pretrained_model = tf.keras.applications.VGG16(\n    #pretrained_model = tf.keras.applications.Xception(\n    pretrained_model = tf.keras.applications.DenseNet201(\n        weights='imagenet',\n        # we set include_top to False because we are adding our own top.\n        include_top=False ,\n        input_shape=[*IMAGE_SIZE, 3] # the input shape is set to 192,192,3\n    )\n    pretrained_model.trainable = False # setting pretrained_model.trainable to False means that we use transfer learning.\n    \n    model = tf.keras.Sequential([\n        # To a base pretrained on ImageNet to extract features from images...\n        pretrained_model,\n        # ... attach a new head to act as a classifier, a GlobalAveragePooling2D and a dense layer\n        # with a size fitting the output size, 104.\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n    # use the adam model optimizer with lr=warmUpLearningRate, loss sparse_categorical_crossentropy and metric sparse_categorical_accuracy\n    #model.compile(\n    #    optimizer='adam',\n    #    loss = 'sparse_categorical_crossentropy',\n    #    metrics=['sparse_categorical_accuracy'],\n    model.compile(\n        optimizer= optimizer,\n        loss = 'sparse_categorical_crossentropy',\n        metrics= metric_list,\n    )\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## train model ##"},{"metadata":{"trusted":true},"cell_type":"code","source":"# before training the model we will pre-train it for a small number of epochs - warmup\nSTEPS_PER_EPOCH = originalNumTrainImages // batchSize\nWARMUP_EPOCHS = 3\ndataTrain = dataTrain.repeat() # don't forget to set dataTrain to repeat like we planned to\nwarmup_history = model.fit(x=dataTrain, \n                           steps_per_epoch=STEPS_PER_EPOCH, \n                           validation_data=dataVal,\n                           epochs=WARMUP_EPOCHS, \n                           verbose=2).history","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"define a learning rate function lrfn which has a linearly increasing learning rate for the first 3 epochs and an exponentially decreasing one for the following epochs."},{"metadata":{"trusted":true},"cell_type":"code","source":"LEARNING_RATE = 3e-5 * strategy.num_replicas_in_sync\nLR_START = 0.00000001\nLR_MIN = 0.000001\nLR_MAX = LEARNING_RATE\nLR_RAMPUP_EPOCHS = 3\nLR_SUSTAIN_EPOCHS = 0\nLR_EXP_DECAY = .8\nEPOCHS = 12 # testing showed that the losses reach a minimum around 12 epochs, after that we get over-fitting\n\ndef lrfn(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n    return lr\n    \nrng = [i for i in range(EPOCHS)]\ny = [lrfn(x) for x in rng]\n\nsns.set(style=\"whitegrid\")\nfig, ax = plt.subplots(figsize=(20, 6))\nplt.plot(rng, y)\nprint(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\".format(y[0], max(y), y[-1]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## fine tuning the layers ##"},{"metadata":{"trusted":true},"cell_type":"code","source":"model_path = 'DenseNet201_%sx%s.h5' % (HEIGHT, WIDTH)\nES_PATIENCE = 6\n\n# unfreeze all layers\nfor layer in model.layers:\n    layer.trainable = True\n\ncheckpoint = ModelCheckpoint(model_path, monitor='val_loss', mode='min', save_best_only=True)\n# early stopping is a regularization which we use to avoid overfitting. we set the number of iterations allowed\n# to run before the learner begins over-fitting to the training data at the cost of the validation (or later, test) data.\nes = EarlyStopping(monitor='val_loss', mode='min', patience=ES_PATIENCE, \n                   restore_best_weights=True, verbose=1) # \nlr_callback = LearningRateScheduler(lrfn, verbose=1) # set the learning rate according to the defined function\n\ncallback_list = [checkpoint, es, lr_callback]\n\noptimizer = optimizers.Adam(lr=LEARNING_RATE)\nmodel.compile(optimizer=optimizer, loss='sparse_categorical_crossentropy', metrics=metric_list)\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## train the model ##"},{"metadata":{"trusted":true},"cell_type":"code","source":"STEPS_PER_EPOCH = NUM_OVERSAMPLED_TRAINING_DATA // batchSize # used when using the oversampled data\nhistory = model.fit(\n    dataTrain,\n    #dataTrainOversampled, # used when using the oversampled data\n    validation_data=dataVal,\n    epochs=EPOCHS,\n    steps_per_epoch=STEPS_PER_EPOCH,\n).history # the \".history\" at the end is needed to make the history object subscriptale so we can plot it later","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Plot the training results ##"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_metrics(history, metric_list):\n    fig, axes = plt.subplots(len(metric_list), 1, sharex='col', figsize=(24, 12))\n    axes = axes.flatten()\n    \n    for index, metric in enumerate(metric_list):\n        axes[index].plot(history[metric], label='Train %s' % metric)\n        axes[index].plot(history['val_%s' % metric], label='Validation %s' % metric)\n        axes[index].legend(loc='best', fontsize=16)\n        axes[index].set_title(metric)\n\n    plt.xlabel('Epochs', fontsize=16)\n    sns.despine()\n    plt.show()\n\nplot_metrics(history, metric_list=['loss', 'sparse_categorical_accuracy'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## plot the confusion matrix ##\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"def display_confusion_matrix(cmat, score, precision, recall):\n    plt.figure(figsize=(15,15))\n    ax = plt.gca()\n    ax.matshow(cmat, cmap='Reds')\n    ax.set_xticks(range(len(CLASSES)))\n    ax.set_xticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_xticklabels(), rotation=45, ha=\"left\", rotation_mode=\"anchor\")\n    ax.set_yticks(range(len(CLASSES)))\n    ax.set_yticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_yticklabels(), rotation=45, ha=\"right\", rotation_mode=\"anchor\")\n    titlestring = \"\"\n    if score is not None:\n        titlestring += 'f1 = {:.3f} '.format(score)\n    if precision is not None:\n        titlestring += '\\nprecision = {:.3f} '.format(precision)\n    if recall is not None:\n        titlestring += '\\nrecall = {:.3f} '.format(recall)\n    if len(titlestring) > 0:\n        ax.text(101, 1, titlestring, fontdict={'fontsize': 18, 'horizontalalignment':'right', 'verticalalignment':'top', 'color':'#804040'})\n    plt.show()\n    \n# cmdataset = get_validation_dataset(ordered=True)\ncmdataset = dataVal\nimages_ds = cmdataset.map(lambda image, label: image)\nlabels_ds = cmdataset.map(lambda image, label: label).unbatch()\n\ncm_correct_labels = next(iter(labels_ds.batch(numValImages))).numpy()\ncm_probabilities = model.predict(images_ds)\ncm_predictions = np.argmax(cm_probabilities, axis=-1)\n\nlabels = range(len(CLASSES))\ncmat = confusion_matrix(\n    cm_correct_labels,\n    cm_predictions,\n    labels=labels,\n)\ncmat = (cmat.T / cmat.sum(axis=1)).T # normalize","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"score = f1_score(\n    cm_correct_labels,\n    cm_predictions,\n    labels=labels,\n    average='macro',\n)\nprecision = precision_score(\n    cm_correct_labels,\n    cm_predictions,\n    labels=labels,\n    average='macro',\n)\nrecall = recall_score(\n    cm_correct_labels,\n    cm_predictions,\n    labels=labels,\n    average='macro',\n)\ndisplay_confusion_matrix(cmat, score, precision, recall)\nprint(\"f1 = \" , round(score,3) , \"precision = \", round(precision,3), \" recall = \" , round(recall,3))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# results:\n\n1.\nepochs 12, resolution 192px-192px, model VGG16, no data augumentation\n12-13s per epoch (except for the first one 34s)\nsparse_categorical_accuracy: 0.5260 - loss: 2.0724 - val_sparse_categorical_accuracy: 0.5062 - val_loss: 2.1567\nf1 = 0.396 precision = 0.543 recall = 0.361\nremarks: poor results\n\n2.\nepochs 12, resolution 192px-192px, model Xception, no data augumentation\n12s per epoch (except for the first one 37s)\nsparse_categorical_accuracy: 0.8933 - loss: 0.4813 - val_sparse_categorical_accuracy: 0.7398 - val_loss: 1.0002\nf1 = 0.733 precision = 0.794 recall = 0.706\nremarks: Xception performs much better than VGG16\n\n3.\nepochs 12, resolution 192px-192px, model VGG16, data augumentation - up/down left/right flips, saturation [0,2], crop 1/8\n11-13s per epoch (except for the first one 25s)\nsparse_categorical_accuracy: 0.4658 - loss: 2.2505 - val_sparse_categorical_accuracy: 0.4300 - val_loss: 2.4010\nf1 =  0.34 precision =  0.461  recall =  0.353\nremarks: augumenting the data the way we did seemed to have made things worse\n\n4.\nepochs 12, resolution 192px-192px, model Xception, data augumentation - up/down left/right flips, saturation [0,2], crop 1/8\n11-12s per epoch (except for the first one 26s)\nsparse_categorical_accuracy: 0.7537 - loss: 0.9390 - val_sparse_categorical_accuracy: 0.7239 - val_loss: 1.0965, f1 =  0.704 precision =  0.782  recall =  0.671\nremarks: augumenting the data the way we did seemed to have made things worse, also for Xception\n\n5.\nepochs 30, resolution 192px-192px, model Xception, data augumentation - up/down left/right flips, saturation [0,2], crop 1/8\n11-12s per epoch (except for the first one 26s)\nsparse_categorical_accuracy: 0.8236 - loss: 0.6586 - val_sparse_categorical_accuracy: 0.7357 - val_loss: 0.9989, f1 =  0.72 precision =  0.784  recall =  0.694\nremarks: increasing the epochs from 12 to 30 did not help\n\n\n\n"},{"metadata":{},"cell_type":"markdown","source":"\n6.\nepochs 120, resolution 192px-192px, model VGG16, no data augumentation 11-12s per epoch (except for the first one 26s)\nsparse_categorical_accuracy: 0.8164 - loss: 0.7671 - val_sparse_categorical_accuracy: 0.6897 - val_loss: 1.2356\n\n7.\nepochs 120, resolution 192px-192px, model XCeption, no data augumentation 12s per epoch\nsparse_categorical_accuracy: 0.9999 - loss: 0.0072 - val_sparse_categorical_accuracy: 0.7395 - val_loss: 1.4572\nremarks: train data accuracy and losses greatly improved and validation accuracy and losses slightly improved. Actually the validation accuracy and losses reached some optimum around 10-15 epochs and started degrading from there - overfitting. This shows that we only want 12 epochs.\n\n8.\nepochs 12, resolution 192px-192px, model DenseNet201, no data augumentation 12s per epoch\nsparse_categorical_accuracy: 0.9523 - loss: 0.2620 - val_sparse_categorical_accuracy: 0.8295 - val_loss: 0.6651\nremarks: so far the best model, better than XCeption\n\n9.\n3 warmup epochs with higher learning rate, epochs 12, resolution 192px-192px, model DenseNet201, no data augumentation 12s per epoch\n20s 200ms/step - sparse_categorical_accuracy: 0.9938 - loss: 0.0298 - val_sparse_categorical_accuracy: 0.9003 - val_loss: 0.4495 f1 =  0.923 precision =  0.935  recall =  0.917\n\n10. same but re-run from a different session\nloss: 7.3095e-04 - sparse_categorical_accuracy: 1.0000 - val_loss: 0.3543 - val_sparse_categorical_accuracy: 0.9186 , f1 =  0.914 precision =  0.931  recall =  0.906\n\n\n\n\nsignificantly better"},{"metadata":{},"cell_type":"markdown","source":"Look at examples from the dataset, with true and predicted classes."},{"metadata":{"trusted":true},"cell_type":"code","source":"# dataset = get_validation_dataset()\ndataset = dataVal\ndataset = dataset.unbatch().batch(20)\nbatch = iter(dataset)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"images, labels = next(batch)\nprobabilities = model.predict(images)\npredictions = np.argmax(probabilities, axis=-1)\ndisplay_batch_of_images((images, labels), predictions)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Test Predictions ##\n\nCreate predictions to submit to the competition."},{"metadata":{"trusted":true},"cell_type":"code","source":"#test_ds = get_test_dataset(ordered=True)\ntest_ds = dataTest\nprint('Computing predictions...')\ntest_images_ds = test_ds.map(lambda image, idnum: image)\nprobabilities = model.predict(test_images_ds)\npredictions = np.argmax(probabilities, axis=-1)\nprint(predictions)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Generating submission.csv file...')\n\n# Get image ids from test set and convert to integers\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(numTestImages))).numpy().astype('U')\n\n# Write the submission file\nnp.savetxt(\n    'submission.csv',\n    np.rec.fromarrays([test_ids, predictions]),\n    fmt=['%s', '%d'],\n    delimiter=',',\n    header='id,label',\n    comments='',\n)\n\n# Look at the first few predictions\n!head submission.csv","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Summary #\n\nIn this project I learned about and worked with multiple topics:\n- TPUs\n- oversampling\n- augmentation\n- pretraining\n- warmup layers\n- variable learning function\n- early stopping\n\nspecifically, I tried 3 different models with combinations of different augmentations, no augmentations, oversampling, no oversampling, pre-training and no pre-training.\n\nThe winning strategy was:\n9. 3 warmup epochs with higher learning rate, epochs 12, resolution 192px-192px, model DenseNet201, no data augumentation 12s per epoch\n20s 200ms/step - sparse_categorical_accuracy: 0.9938 - loss: 0.0298 - val_sparse_categorical_accuracy: 0.9003 - val_loss: 0.4495 f1 =  0.923 precision =  0.935  recall =  0.917"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}