{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Deep Learning Project 1: Petals to the Metal - Flower Classification on TPUs\n==================================================\n\n#### By: Rafael Baez & Issac Rivas ####\n\nhttps://www.kaggle.com/competitions/tpu-getting-started/overview\n\n\n\n### The goal of this project is to build a machine learning model that can identify the type of flowers in a data set. (For simplicity we will only be training this model on 104 diffrent flower types.) ","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nfrom kaggle_datasets import KaggleDatasets\nfrom sklearn.metrics import f1_score, precision_score, recall_score, confusion_matrix\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom tensorflow import keras\nfrom IPython.display import Image\nimport pandas as pd\nimport math\nimport os\nimport re","metadata":{"execution":{"iopub.status.busy":"2022-11-03T05:44:39.257124Z","iopub.execute_input":"2022-11-03T05:44:39.257435Z","iopub.status.idle":"2022-11-03T05:44:39.263511Z","shell.execute_reply.started":"2022-11-03T05:44:39.257407Z","shell.execute_reply":"2022-11-03T05:44:39.262682Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# What is a TPU(Tensor Processing Units)?\n\n### A TPU is a hardware accelerator that specialize in deep learning tasks. https://www.kaggle.com/docs/tpu\n#### To go fast on a TPU, increase the batch size. The rule of thumb is to use batches of 128 elements per core (ex: batch size of 128*8=1024 for a TPU with 8 cores). At this size, the 128x128 hardware matrix multipliers of the TPU are most likely to be kept busy. You start seeing interesting speedups from a batch size of 8 per core though. In the sample above, the batch size is scaled with the core count through this line of code: `BATCH_SIZE = 16 * tpu_strategy.num_replicas_in_sync`\n\n## \n### Before Running this code we recommend you enable a TPU if you are able too, as it will be used to create a distribution strategy in a later part of the code.","metadata":{}},{"cell_type":"code","source":"isTPU = False\n# Detect hardware, return appropriate distribution strategy\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection. No parameters necessary if TPU_NAME environment variable is set. On Kaggle this is always the case.\n    print('Running on TPU ', tpu.master())\n    isTPU = True\nexcept ValueError:\n    print('Not running on TPU ')\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","metadata":{"execution":{"iopub.status.busy":"2022-11-03T05:19:26.624082Z","iopub.execute_input":"2022-11-03T05:19:26.624335Z","iopub.status.idle":"2022-11-03T05:19:32.712113Z","shell.execute_reply.started":"2022-11-03T05:19:26.624307Z","shell.execute_reply":"2022-11-03T05:19:32.711147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### With TPU's your data must be added into a Google Cloud Storage Bucket. https://cloud.google.com/storage/\n### This code acesses kaggles public GCS bucket by using its path, `tpu-getting-started`.","metadata":{}},{"cell_type":"code","source":"# Define the batch size. This will be 16 with TPU off and 128 (=16*8) with TPU on\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\nprint(\"BATCH_SIZE:\",BATCH_SIZE)\n\nEPOCHS = 30\n\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('tpu-getting-started')\nprint(GCS_DS_PATH)","metadata":{"execution":{"iopub.status.busy":"2022-11-03T05:19:34.592992Z","iopub.execute_input":"2022-11-03T05:19:34.593267Z","iopub.status.idle":"2022-11-03T05:19:34.916120Z","shell.execute_reply.started":"2022-11-03T05:19:34.593239Z","shell.execute_reply":"2022-11-03T05:19:34.915114Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Here is code from a sample kaggle notebook for creating data sets from our Google Cloud Storage Buckets\n##### https://www.kaggle.com/code/philculliton/a-simple-petals-tf-2-2-notebook/notebook","metadata":{}},{"cell_type":"code","source":"IMAGE_SIZE = [192,192]\n\nGCS_PATH = GCS_DS_PATH + '/tfrecords-jpeg-192x192'\nAUTO = tf.data.experimental.AUTOTUNE # Allows for data to be read from multiple files at once and determines how many files at a time automatically. \n\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/*.tfrec') \n\nCLASSES = ['pink primrose',    'hard-leaved pocket orchid', 'canterbury bells', 'sweet pea',     'wild geranium',     'tiger lily',           'moon orchid',              'bird of paradise', 'monkshood',        'globe thistle',         \n           'snapdragon',       \"colt's foot\",               'king protea',      'spear thistle', 'yellow iris',       'globe-flower',         'purple coneflower',        'peruvian lily',    'balloon flower',   'giant white arum lily',\n           'fire lily',        'pincushion flower',         'fritillary',       'red ginger',    'grape hyacinth',    'corn poppy',           'prince of wales feathers', 'stemless gentian', 'artichoke',        'sweet william',\n           'carnation',        'garden phlox',              'love in the mist', 'cosmos',        'alpine sea holly',  'ruby-lipped cattleya', 'cape flower',              'great masterwort', 'siam tulip',       'lenten rose',\n           'barberton daisy',  'daffodil',                  'sword lily',       'poinsettia',    'bolero deep blue',  'wallflower',           'marigold',                 'buttercup',        'daisy',            'common dandelion',\n           'petunia',          'wild pansy',                'primula',          'sunflower',     'lilac hibiscus',    'bishop of llandaff',   'gaura',                    'geranium',         'orange dahlia',    'pink-yellow dahlia',\n           'cautleya spicata', 'japanese anemone',          'black-eyed susan', 'silverbush',    'californian poppy', 'osteospermum',         'spring crocus',            'iris',             'windflower',       'tree poppy',\n           'gazania',          'azalea',                    'water lily',       'rose',          'thorn apple',       'morning glory',        'passion flower',           'lotus',            'toad lily',        'anthurium',\n           'frangipani',       'clematis',                  'hibiscus',         'columbine',     'desert-rose',       'tree mallow',          'magnolia',                 'cyclamen ',        'watercress',       'canna lily',\n           'hippeastrum ',     'bee balm',                  'pink quill',       'foxglove',      'bougainvillea',     'camellia',             'mallow',                   'mexican petunia',  'bromelia',         'blanket flower',\n           'trumpet creeper',  'blackberry lily',           'common tulip',     'wild rose']\n\ndef decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0  # convert image to floats in [0, 1] range\n    image = tf.reshape(image, [*IMAGE_SIZE, 3]) # explicit size needed for TPU\n    return image\n\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"class\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = tf.cast(example['class'], tf.int32)\n    return image, label # returns a dataset of (image, label) pairs\n\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"id\": tf.io.FixedLenFeature([], tf.string),  # shape [] means single element\n        # class is missing, this competitions's challenge is to predict flower classes for the test dataset\n    }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['id']\n    return image, idnum # returns a dataset of image(s)\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    # Read from TFRecords. For optimal performance, reading from multiple files at once and\n    # disregarding data order. Order does not matter since we will be shuffling the data anyway.\n\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order, increase speed\n\n    dataset = tf.data.TFRecordDataset(filenames) # automatically interleaves reads from multiple files\n    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord)\n    # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n    return dataset\n\ndef get_training_dataset():\n    dataset = load_dataset(tf.io.gfile.glob(GCS_DS_PATH + '/tfrecords-jpeg-192x192/train/*.tfrec'), labeled=True)\n    dataset = dataset.repeat()\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO)\n    return dataset\n\ndef get_validation_dataset():\n    dataset = load_dataset(tf.io.gfile.glob(GCS_DS_PATH + '/tfrecords-jpeg-192x192/val/*.tfrec'), labeled=True, ordered=False)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTO)\n    return dataset\n\ndef get_test_dataset(ordered=False):\n    dataset = load_dataset(tf.io.gfile.glob(GCS_DS_PATH + '/tfrecords-jpeg-192x192/test/*.tfrec'), labeled=False, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO)\n    return dataset\n\ntraining_dataset = get_training_dataset()\nvalidation_dataset = get_validation_dataset()","metadata":{"execution":{"iopub.status.busy":"2022-11-03T05:19:37.755121Z","iopub.execute_input":"2022-11-03T05:19:37.755407Z","iopub.status.idle":"2022-11-03T05:19:38.134110Z","shell.execute_reply.started":"2022-11-03T05:19:37.755370Z","shell.execute_reply":"2022-11-03T05:19:38.133213Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def count_data_items(filenames):\n    # the number of data items is written in the name of the .tfrec files, i.e. flowers00-230.tfrec = 230 data items\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_VALIDATION_IMAGES = count_data_items(VALIDATION_FILENAMES)\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\nprint('Dataset: {} training images, {} validation images, {} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))","metadata":{"execution":{"iopub.status.busy":"2022-11-03T05:19:42.972873Z","iopub.execute_input":"2022-11-03T05:19:42.973175Z","iopub.status.idle":"2022-11-03T05:19:42.979975Z","shell.execute_reply.started":"2022-11-03T05:19:42.973149Z","shell.execute_reply":"2022-11-03T05:19:42.979131Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Train = get_training_dataset()\nVal = get_validation_dataset()\nTest = get_test_dataset()\ninput_shape = []\n\nprint(\"Training:\", Train)\nfor image, label in Train.take(1):\n    print(image.numpy().shape, label.numpy().shape)\n    input_shape = image.numpy().shape[1:]\nprint(\"Training data example:\", label.numpy())\n\nprint(\"Test:\", Test)\nfor image, idnum in Test.take(1):\n    print(image.numpy().shape, idnum.numpy().shape)\nprint(\"Test data IDs:\", idnum.numpy().astype('U')) # idnum is a unique ID given to the image","metadata":{"execution":{"iopub.status.busy":"2022-11-03T05:19:44.322784Z","iopub.execute_input":"2022-11-03T05:19:44.323087Z","iopub.status.idle":"2022-11-03T05:19:48.452997Z","shell.execute_reply.started":"2022-11-03T05:19:44.323057Z","shell.execute_reply":"2022-11-03T05:19:48.452192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from matplotlib import pyplot as plt\n\ndef batch_to_numpy_images_and_labels(data):\n    images, labels = data\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    if numpy_labels.dtype == object: # binary string in this case,these are image ID strings\n        numpy_labels = [None for _ in enumerate(numpy_images)]\n        # If no labels, only image IDs, return None for labels (this is the case for test data)\n    return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n    if correct_label is None:\n        return CLASSES[label], True\n    correct = (label == correct_label)\n    return \"{} [{}{}{}]\".format(CLASSES[label], \n                                'OK' if correct else 'NO', \n                                u\"\\u2192\" if not correct else '',\n                                CLASSES[correct_label] if not correct else ''), correct\n\ndef display_one_flower(image, title, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    if len(title) > 0:\n        plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n    return (subplot[0], subplot[1], subplot[2]+1)\n\ndef display_batch_of_images(databatch, predictions=None, display_mismatches_only=False):\n    # data\n    images, labels = batch_to_numpy_images_and_labels(databatch)\n    if labels is None:\n        labels = [None for _ in enumerate(images)]\n        \n    # auto-squaring: this will drop data that does not fit into square or square-ish rectangle\n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n        \n    # size and spacing\n    FIGSIZE = 13.0\n    SPACING = 0.1\n    subplot=(rows,cols,1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n    else:\n        plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n    \n    # display\n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        title = '' if label is None else CLASSES[label]\n        correct = True\n        if predictions is not None:\n            title, correct = title_from_label_and_target(predictions[i], label)\n        dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n        if display_mismatches_only:\n            if predictions[i] != label:\n                subplot = display_one_flower(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n        else:        \n            subplot = display_one_flower(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n    \n    #layout\n    plt.tight_layout()\n    if label is None and predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp = iter(Train.unbatch().batch(20))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### This can be re-run to get a new batch set of flowers","metadata":{}},{"cell_type":"code","source":"next_temp = next(temp)\ndisplay_batch_of_images(next_temp)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for images, labels in Train.take(1):  # only take first element of dataset\n    X_Train = images.numpy()\n    Y_Train = labels.numpy()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"X_Train Example\", X_Train[0])\nprint(\"Y_Train Example\",Y_Train[0])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### For our baseline we decided to set all predictions to the most common flower with in our case was flower number 67","metadata":{}},{"cell_type":"code","source":"test_dataset = get_test_dataset(ordered = True)\npredictions = np.ones(7382)*67\n\n# test_ids_ds = test_dataset.map(lambda image, idnum: idnum).unbatch()\n# test_ids = next(iter(test_ids_ds.batch(7382))).numpy().astype('U')\n\n# np.savetxt(\n#     'submission.csv',\n#     np.rec.fromarrays([test_ids, predictions]),\n#     fmt=['%s', '%d'],\n#     delimiter=',',\n#     header='id,label',\n#     comments='',\n# )\n\n# # Look at the first few predictions\n# !head submission.csv","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### This gave us a kaggle score of ","metadata":{}},{"cell_type":"code","source":"Image(\"../input/images/nice.PNG\")","metadata":{"execution":{"iopub.status.busy":"2022-11-03T05:51:30.844549Z","iopub.execute_input":"2022-11-03T05:51:30.844824Z","iopub.status.idle":"2022-11-03T05:51:30.856704Z","shell.execute_reply.started":"2022-11-03T05:51:30.844793Z","shell.execute_reply":"2022-11-03T05:51:30.855403Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model 1: CNN ","metadata":{}},{"cell_type":"code","source":"def build_model(input_shape, learning_rate=0.01):\n  \n    tf.keras.backend.clear_session()\n    np.random.seed(0)\n    tf.random.set_seed(0)\n\n    model = keras.Sequential()\n\n    # 32x32x3\n    model.add(keras.layers.Conv2D(filters=192, kernel_size=(3, 3), padding='same',\n                  input_shape=input_shape))\n\n    # 32x32x32\n    model.add(keras.layers.Activation('relu'))\n\n    model.add(keras.layers.Conv2D(filters=192, kernel_size=(3, 3)))\n\n    # 30x30x32\n\n    model.add(keras.layers.Activation('relu'))\n    model.add(keras.layers.MaxPooling2D(pool_size=(2, 2)))\n\n    # 15X15x32\n\n    model.add(keras.layers.Conv2D(filters=96, kernel_size=(3, 3), padding='same'))\n    model.add(keras.layers.Activation('relu'))\n\n    # 15X15x64\n\n    model.add(keras.layers.Conv2D(filters=96, kernel_size=(3, 3)))\n\n    # 13X13x64\n\n    model.add(keras.layers.Activation('relu'))\n    model.add(keras.layers.MaxPooling2D(pool_size=(2, 2)))\n\n    model.add(keras.layers.Conv2D(filters=48, kernel_size=(3, 3), padding='same'))\n    model.add(keras.layers.Activation('relu'))\n\n    model.add(keras.layers.Conv2D(filters=48, kernel_size=(3, 3)))\n\n    model.add(keras.layers.Activation('relu'))\n    model.add(keras.layers.MaxPooling2D(pool_size=(2, 2)))\n\n    model.add(keras.layers.Flatten())\n\n    model.add(keras.layers.Dense(units=2048))\n    model.add(keras.layers.Activation('relu'))\n    model.add(keras.layers.Dense(units=1024))\n    model.add(keras.layers.Activation('relu'))\n    model.add(keras.layers.Dense(units=512))\n    \n    model.add(keras.layers.Activation('relu'))\n\n    model.add(keras.layers.Dense(units=len(CLASSES)))\n    model.add(keras.layers.Activation('softmax'))\n    optimizer = tf.keras.optimizers.SGD(learning_rate=learning_rate)\n    # We need to choose an optimizer. We'll use SGD, which is actually mini-batch\n    # SGD. We can specify the batch size to use for training later.\n    # Finally, we compile the model. This finalizes the graph for training.\n    model.compile(loss=keras.losses.sparse_categorical_crossentropy,\n              optimizer=optimizer,\n              metrics=['accuracy'])\n\n\n    return model","metadata":{"execution":{"iopub.status.busy":"2022-11-03T05:28:05.226874Z","iopub.execute_input":"2022-11-03T05:28:05.227357Z","iopub.status.idle":"2022-11-03T05:28:05.235074Z","shell.execute_reply.started":"2022-11-03T05:28:05.227325Z","shell.execute_reply":"2022-11-03T05:28:05.233975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = build_model(input_shape=input_shape, learning_rate=0.01)\nmodel.summary()\ntf.keras.utils.plot_model(model, show_shapes=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tensorflow_datasets as tfds\ntraining_iter = tfds.as_numpy(Train)\nraw_imagesTr = []\nraw_labelsTr = []\nfor e in training_iter:\n    raw_imagesTr.append(e[0])\n    raw_labelsTr.append(e[1])\nX_train = np.array(raw_imagesTr)\nY_train = np.array(raw_labelsTr)\nprint(X_train.shape)\nprint(Y_train.shape)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = build_model(input_shape=X_Train[0].shape)\n\n# Fit the model.\nhistory = model.fit(\n    x = X_train,\n    y = Y_train,\n    epochs=EPOCHS,\n    batch_size=BATCH_SIZE,\n    validation_split=0.1,\n    verbose=1\n    )","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(pd.DataFrame(history.history))\n\n# summarize history for accuracy\nplt.plot(history.history['accuracy'])\nplt.plot(history.history['val_accuracy'])\nplt.title('model accuracy')\nplt.ylabel('accuracy')\nplt.xlabel('epoch')\nplt.legend(['train', 'test'], loc='upper left')\nplt.show()\n# summarize history for loss\nplt.plot(history.history['loss'])\nplt.plot(history.history['val_loss'])\nplt.title('model loss')\nplt.ylabel('loss')\nplt.xlabel('epoch')\nplt.legend(['train', 'test'], loc='upper left')\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model 2: VGG Model","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.layers import Conv2D, MaxPool2D, Dense, Flatten, Dropout\n\ndef plot_results(all_history, sloss = \"sparse_categorical_crossentropy\", smetric = \"sparse_categorical_accuracy\"):\n    loss, val_loss, MSE, val_MSE = [], [], [], []\n    for history in all_history:\n        loss += history.history[sloss]\n        val_loss += history.history[f'val_{sloss}']\n        MSE += history.history[smetric]\n        val_MSE += history.history[f'val_{smetric}']\n\n    fig, ax = plt.subplots()\n    ax.plot(loss,label = 'train')\n    ax.plot(val_loss,label = 'test')\n    ax.set_title(sloss)\n    ax.legend(loc='upper right')\n\n    fig, ax = plt.subplots()\n    ax.plot(MSE,label = 'train')\n    ax.plot(val_MSE,label = 'test')\n    ax.set_title(smetric)\n    ax.legend(loc='lower right')\n    return\n  \ndef build_VGG_model(input_shape = (192,192,3), VGG = 4, start_units = 3, kernel = 8):\n    tf.keras.backend.clear_session\n    tf.random.set_seed(0)\n    model = tf.keras.models.Sequential()\n    model.add(Conv2D(filters = 2**(start_units), kernel_size = kernel, padding='same',\n                             input_shape=input_shape))\n    for i in range(VGG):\n        model.add(Conv2D(filters = 2**(i+start_units), kernel_size=kernel, padding='same',activation='relu'))\n        model.add(Conv2D(filters = 2**(i+start_units), kernel_size=kernel, padding='same',activation='relu'))\n        model.add(Dropout(0.1))\n        model.add(MaxPool2D(pool_size=2, strides=2, padding = 'same'))\n    # Dense Layers\n    model.add(Flatten())\n    model.add(Dense(units = 1024, activation = 'relu'))\n    model.add(Dense(units = 512, activation = 'relu'))\n    model.add(Dense(units = 104, activation = 'softmax'))\n    return model\nVGGmodel = build_VGG_model(VGG=3, start_units = 2, kernel = 4);\nVGGmodel.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"opt = tf.keras.optimizers.Nadam()\nVGGmodel.compile(optimizer = opt, \n                 loss = 'sparse_categorical_crossentropy', \n                 metrics='sparse_categorical_accuracy')\nall_history = []\nhistory = VGGmodel.fit(\n    Train,\n    epochs = 10,\n    steps_per_epoch = NUM_TRAINING_IMAGES // BATCH_SIZE,\n    batch_size=BATCH_SIZE,\n    validation_data = Val\n)\nall_history.append(history)\nplot_results(all_history, sloss = 'loss')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Kaggle Score:","metadata":{}},{"cell_type":"code","source":"Image(\"../input/images/prettyGood.PNG\")","metadata":{"execution":{"iopub.status.busy":"2022-11-03T05:51:56.084373Z","iopub.execute_input":"2022-11-03T05:51:56.084600Z","iopub.status.idle":"2022-11-03T05:51:56.093513Z","shell.execute_reply.started":"2022-11-03T05:51:56.084576Z","shell.execute_reply":"2022-11-03T05:51:56.092635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model 3: Transfer Learning\n\n###Here we are using a pretrained model `VGG16` with `imagenet` as weights. This will allow our model to ","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    img_adjust_layer = tf.keras.layers.Lambda(lambda data: tf.keras.applications.vgg16.preprocess_input(tf.cast(data, tf.float32)), input_shape=[192,192, 3])\n    pretrained_model = tf.keras.applications.VGG16(weights='imagenet', include_top=False)\n\n    pretrained_model.trainable = False # False = transfer learning, True = fine-tuning\n\n    modelT = tf.keras.Sequential([\n        #img_adjust_layer,\n        pretrained_model,\n\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n","metadata":{"execution":{"iopub.status.busy":"2022-11-03T05:21:05.552478Z","iopub.execute_input":"2022-11-03T05:21:05.553225Z","iopub.status.idle":"2022-11-03T05:21:08.048882Z","shell.execute_reply.started":"2022-11-03T05:21:05.553195Z","shell.execute_reply":"2022-11-03T05:21:08.047738Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"modelT.compile(\n    optimizer='adam',\n    loss = 'sparse_categorical_crossentropy',\n    metrics=['sparse_categorical_accuracy']\n)\n\nmodelT.summary()\ntf.keras.utils.plot_model(modelT, show_shapes=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = modelT.fit(\n            Train,\n            validation_data=Val,\n            epochs=EPOCHS,\n            steps_per_epoch= NUM_TRAINING_IMAGES // BATCH_SIZE,\n            validation_steps= -(-NUM_VALIDATION_IMAGES // BATCH_SIZE) \n            )\n\ndisplay(pd.DataFrame(history.history))\n","metadata":{"execution":{"iopub.status.busy":"2022-11-03T05:21:15.196566Z","iopub.execute_input":"2022-11-03T05:21:15.196864Z","iopub.status.idle":"2022-11-03T05:27:03.018105Z","shell.execute_reply.started":"2022-11-03T05:21:15.196837Z","shell.execute_reply":"2022-11-03T05:27:03.016697Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_history = []\nall_history.append(history)\nplot_results(all_history, sloss = 'loss')","metadata":{"execution":{"iopub.status.busy":"2022-11-03T05:28:12.761638Z","iopub.execute_input":"2022-11-03T05:28:12.762632Z","iopub.status.idle":"2022-11-03T05:28:13.147093Z","shell.execute_reply.started":"2022-11-03T05:28:12.762586Z","shell.execute_reply":"2022-11-03T05:28:13.146180Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_ds = get_test_dataset(ordered=True)\ntest_images_ds = test_ds.map(lambda image, idnum: image)\nprobabilities = modelT.predict(test_images_ds, steps=-(-NUM_TEST_IMAGES // BATCH_SIZE))\npredictions = np.argmax(probabilities, axis=-1)\nprint(predictions)","metadata":{"execution":{"iopub.status.busy":"2022-11-03T05:28:25.674424Z","iopub.execute_input":"2022-11-03T05:28:25.674698Z","iopub.status.idle":"2022-11-03T05:28:46.634531Z","shell.execute_reply.started":"2022-11-03T05:28:25.674666Z","shell.execute_reply":"2022-11-03T05:28:46.633754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Generating submission.csv file...')\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U') \nnp.savetxt('submission.csv', \n           np.rec.fromarrays([test_ids, predictions]), \n           fmt=['%s', '%d'], \n           delimiter=',', \n           header='id,label', \n           comments='')","metadata":{"execution":{"iopub.status.busy":"2022-11-03T05:29:05.146126Z","iopub.execute_input":"2022-11-03T05:29:05.146413Z","iopub.status.idle":"2022-11-03T05:29:11.229363Z","shell.execute_reply.started":"2022-11-03T05:29:05.146386Z","shell.execute_reply":"2022-11-03T05:29:11.228244Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Submission=pd.read_csv('submission.csv')\nSubmission.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-03T05:29:14.402126Z","iopub.execute_input":"2022-11-03T05:29:14.402385Z","iopub.status.idle":"2022-11-03T05:29:14.418320Z","shell.execute_reply.started":"2022-11-03T05:29:14.402358Z","shell.execute_reply":"2022-11-03T05:29:14.417523Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Kaggle Score:","metadata":{}},{"cell_type":"code","source":"Image(\"../input/images/image.png\")","metadata":{"execution":{"iopub.status.busy":"2022-11-03T05:52:23.084865Z","iopub.execute_input":"2022-11-03T05:52:23.085180Z","iopub.status.idle":"2022-11-03T05:52:23.097678Z","shell.execute_reply.started":"2022-11-03T05:52:23.085154Z","shell.execute_reply":"2022-11-03T05:52:23.096563Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Error Analysis\n\n### While it is not the most sophisticated baseline, predicting the most common class is a quick way of creating an extremely simple baseline. When submitting to the Kaggle Competition it performed as expected, earning a score of 0.0011. We also tried a simple model with a few CNN layers and it got []. The last model created by us uses the VGG3 neural network architecture and it was likely limited by the amount of parameters that we are able to create and train, but it did earn a score of 0.33489. The last model we tried was a pretrained VGG16 model and it earned a respectable score of [].","metadata":{}},{"cell_type":"markdown","source":"## Conclusion \n\n### We learned a little bit about how to use Tensor Processing Units (TPU), as well as gained more experience about using tensorflow datasets to avoid loading all of the data at once, since it would often crash any runtime session if we did. Regarding the models, we learned how difficult it is to apply neural networks to a more practical problem, with optimistic conditions. A large part of the images were easily digestible by the networks and we had a fair amount of data. But even with these ideal conditions, we struggled to get good results. Our VGG3 network would often overfit and could only predict the final test images with 33% accuracy, while a pretrained network reached up to 60%. This goes to show that even with pretrained networks, they have to be properly adapted to the problem to be used to their potential. Given a little bit more time, we could try to augment the data and produce more data for the network, or we could look for either better architectures or more sophisticated trained models.","metadata":{}}]}