{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import math, re, os\nimport tensorflow as tf\nimport numpy as np\nfrom matplotlib import pyplot as plt\nfrom kaggle_datasets import KaggleDatasets\nfrom sklearn.metrics import f1_score, precision_score, recall_score, confusion_matrix\nprint(\"Tensorflow version \" + tf.__version__)\nAUTO = tf.data.experimental.AUTOTUNE","metadata":{"execution":{"iopub.status.busy":"2022-03-02T05:59:57.959069Z","iopub.execute_input":"2022-03-02T05:59:57.959490Z","iopub.status.idle":"2022-03-02T06:00:04.253149Z","shell.execute_reply.started":"2022-03-02T05:59:57.959389Z","shell.execute_reply":"2022-03-02T06:00:04.251999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# What are TPU's? \nTPU's are holy grail of computers for any Machine Learning Practitioners! A tensor processing unit (TPU) is an AI accelerator application-specific integrated circuit (ASIC) developed by Google specifically for neural network machine learning. TPUs are hardware accelerators specialized in deep learning tasks. In this code lab, you will see how to use them with Keras and Tensorflow 2. Cloud TPUs are available in a base configuration with 8 cores and also in larger configurations called \"TPU pods\" of up to 2048 cores. The extra hardware can be used to accelerate training by increasing the training batch size.","metadata":{}},{"cell_type":"markdown","source":"# Why TPUs?\n\nModern GPUs are organized around programmable \"cores\", a very flexible architecture that allows them to handle a variety of tasks such as 3D rendering, deep learning, physical simulations, etc.. TPUs on the other hand pair a classic vector processor with a dedicated matrix multiply unit and excel at any task where large matrix multiplications dominate, such as neural networks.","metadata":{}},{"cell_type":"markdown","source":"# Distribution Strategy\nA TPU has eight different cores and each of these cores acts as its own accelerator. (A TPU is sort of like having eight GPUs in one machine.) We tell TensorFlow how to make use of all these cores at once through a distribution strategy. Run the following cell to create the distribution strategy that we'll later apply to our model.","metadata":{}},{"cell_type":"markdown","source":"# What TPUClusterResolver() does?\nTPUs are network-connected accelerators, and you must first locate them on the network. In TPUStrategy, the main objective is to contain the necessary distributed training code that will work on TPUs with their 8 compute cores. Whenever you use the TPUStrategy by instantiating your model in the scope of the strategy, this creates the model on the TPU. The model size is constrained by the TPU RAM only, not by the amount of memory available on the VM running your Python code. Model creation and model training use the usual Keras APIs. Further, read about [TPUClusterResolver()](https://www.tensorflow.org/api_docs/python/tf/distribute/cluster_resolver/ClusterResolver) and [Kaggle TPU Doc](https://www.kaggle.com/docs/tpu).\n","metadata":{}},{"cell_type":"markdown","source":"You then instantiate a TPUStrategy. This object contains the necessary distributed training code that will work on TPUs with their 8 compute cores (see hardware section below).\n\nFinally, you use the TPUStrategy by instantiating your model in the scope of the strategy. This creates the model on the TPU. Model size is constrained by the TPU RAM only, not by the amount of memory available on the VM running your Python code. Model creation and model training use the usual Keras APIs.\nWe’ll use the distribution strategy when we create our neural network model. Then, TensorFlow will distribute the training among the eight TPU cores by creating eight different replicas of the model, one for each core.","metadata":{}},{"cell_type":"code","source":"#Detect hardware, return appropriate distribution strategy\n\ntry:\n    #TPU detection, no parameters necessary if TPU_NAME environment variable is set\n    #On kaggle this is always the case\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver() \n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() \n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","metadata":{"execution":{"iopub.status.busy":"2022-03-02T06:00:04.257954Z","iopub.execute_input":"2022-03-02T06:00:04.258207Z","iopub.status.idle":"2022-03-02T06:00:10.275459Z","shell.execute_reply.started":"2022-03-02T06:00:04.258179Z","shell.execute_reply":"2022-03-02T06:00:10.274560Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# NEW on TPU in TensorFlow 24: shorter cross-compatible TPU/GPU/multi-GPU/cluster-GPU detection code\n\ntry: # detect TPUs\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver.connect() # TPU detection\n    strategy = tf.distribute.TPUStrategy(tpu)\nexcept ValueError: # detect GPUs\n    strategy = tf.distribute.MirroredStrategy() # for GPU or multi-GPU machines\n    #strategy = tf.distribute.get_strategy() # default strategy that works on CPU and single GPU\n    #strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy() # for clusters of multi-GPU machines\n\nprint(\"Number of accelerators: \", strategy.num_replicas_in_sync)","metadata":{"execution":{"iopub.status.busy":"2022-03-02T06:00:10.280274Z","iopub.execute_input":"2022-03-02T06:00:10.280539Z","iopub.status.idle":"2022-03-02T06:00:17.979987Z","shell.execute_reply.started":"2022-03-02T06:00:10.280511Z","shell.execute_reply":"2022-03-02T06:00:17.979081Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Loading the Competition Data**\n\n**Get GCS Path**\n\nWhen used with TPUs, datasets need to be stored in a [Google Cloud Storage bucket](https://cloud.google.com/storage/). You can use data from any public GCS bucket by giving its path just like you would data from '/kaggle/input'. The following will retrieve the GCS path for this competition's dataset.","metadata":{}},{"cell_type":"markdown","source":"# Load Data\nWhen used with TPUs, datasets are often serialized into TFRecords. This is a format convenient for distributing data to each of the TPUs cores. We've hidden the cell that reads the TFRecords for our dataset since the process is a bit long. You could come back to it later for some guidance on using your own datasets with TPUs.","metadata":{}},{"cell_type":"code","source":"from kaggle_datasets import KaggleDatasets\n\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('tpu-getting-started')\nprint(GCS_DS_PATH) # what do gcs paths look like?\n#PATH2 = KaggleDatasets().get_gcs_path('tf-flower-photo-tfrec')\n#print(PATH2)\n\nIMAGE_SIZE = [512, 512] # At this size, a GPU will run out of memory. Use the TPU.\n                        # For GPU training, please select 224 x 224 px image size.\nEPOCHS = 35\n# Define the batch size. This will be 16 with TPU off and 128 (=16*8) with TPU on\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync\n# To make TPU faster, increase the Batch Size\n\nGCS_PATH_SELECT = { # available image sizes\n    192: GCS_DS_PATH + '/tfrecords-jpeg-192x192',\n    224: GCS_DS_PATH + '/tfrecords-jpeg-224x224',\n    331: GCS_DS_PATH + '/tfrecords-jpeg-331x331',\n    512: GCS_DS_PATH + '/tfrecords-jpeg-512x512'\n}\nGCS_PATH = GCS_PATH_SELECT[IMAGE_SIZE[0]]\n\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/*.tfrec') # predictions on this dataset should be submitted for the competition\n\nCLASSES = ['pink primrose',    'hard-leaved pocket orchid', 'canterbury bells', 'sweet pea',     'wild geranium',     'tiger lily',           'moon orchid',              'bird of paradise', 'monkshood',        'globe thistle',         # 00 - 09\n           'snapdragon',       \"colt's foot\",               'king protea',      'spear thistle', 'yellow iris',       'globe-flower',         'purple coneflower',        'peruvian lily',    'balloon flower',   'giant white arum lily', # 10 - 19\n           'fire lily',        'pincushion flower',         'fritillary',       'red ginger',    'grape hyacinth',    'corn poppy',           'prince of wales feathers', 'stemless gentian', 'artichoke',        'sweet william',         # 20 - 29\n           'carnation',        'garden phlox',              'love in the mist', 'cosmos',        'alpine sea holly',  'ruby-lipped cattleya', 'cape flower',              'great masterwort', 'siam tulip',       'lenten rose',           # 30 - 39\n           'barberton daisy',  'daffodil',                  'sword lily',       'poinsettia',    'bolero deep blue',  'wallflower',           'marigold',                 'buttercup',        'daisy',            'common dandelion',      # 40 - 49\n           'petunia',          'wild pansy',                'primula',          'sunflower',     'lilac hibiscus',    'bishop of llandaff',   'gaura',                    'geranium',         'orange dahlia',    'pink-yellow dahlia',    # 50 - 59\n           'cautleya spicata', 'japanese anemone',          'black-eyed susan', 'silverbush',    'californian poppy', 'osteospermum',         'spring crocus',            'iris',             'windflower',       'tree poppy',            # 60 - 69\n           'gazania',          'azalea',                    'water lily',       'rose',          'thorn apple',       'morning glory',        'passion flower',           'lotus',            'toad lily',        'anthurium',             # 70 - 79\n           'frangipani',       'clematis',                  'hibiscus',         'columbine',     'desert-rose',       'tree mallow',          'magnolia',                 'cyclamen ',        'watercress',       'canna lily',            # 80 - 89\n           'hippeastrum ',     'bee balm',                  'pink quill',       'foxglove',      'bougainvillea',     'camellia',             'mallow',                   'mexican petunia',  'bromelia',         'blanket flower',        # 90 - 99\n           'trumpet creeper',  'blackberry lily',           'common tulip',     'wild rose']                                                                                                                                               # 100 - 102","metadata":{"execution":{"iopub.status.busy":"2022-03-02T06:00:17.981510Z","iopub.execute_input":"2022-03-02T06:00:17.981772Z","iopub.status.idle":"2022-03-02T06:00:18.708607Z","shell.execute_reply.started":"2022-03-02T06:00:17.981741Z","shell.execute_reply":"2022-03-02T06:00:18.707795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Visualization utilities\ndata -> pixels, nothing of much interest for the machine learning practitioner in this section.","metadata":{}},{"cell_type":"code","source":"# numpy and matplotlib defaults\nnp.set_printoptions(threshold=15, linewidth=80)\n\ndef batch_to_numpy_images_and_labels(data):\n    images, labels = data\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    if numpy_labels.dtype == object: # binary string in this case, these are image ID strings\n        numpy_labels = [None for _ in enumerate(numpy_images)]\n    # If no labels, only image IDs, return None for labels (this is the case for test data)\n    return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n    if correct_label is None:\n        return CLASSES[label], True\n    correct = (label == correct_label)\n    return \"{} [{}{}{}]\".format(CLASSES[label], 'OK' if correct else 'NO', u\"\\u2192\" if not correct else '',\n                                CLASSES[correct_label] if not correct else ''), correct\n\ndef display_one_flower(image, title, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    if len(title) > 0:\n        plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))\n    return (subplot[0], subplot[1], subplot[2]+1)\n    \ndef display_batch_of_images(databatch, predictions=None):\n    \"\"\"This will work with:\n    display_batch_of_images(images)\n    display_batch_of_images(images, predictions)\n    display_batch_of_images((images, labels))\n    display_batch_of_images((images, labels), predictions)\n    \"\"\"\n    # data\n    images, labels = batch_to_numpy_images_and_labels(databatch)\n    if labels is None:\n        labels = [None for _ in enumerate(images)]\n        \n    # auto-squaring: this will drop data that does not fit into square or square-ish rectangle\n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n        \n    # size and spacing\n    FIGSIZE = 13.0\n    SPACING = 0.1\n    subplot=(rows,cols,1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))\n    else:\n        plt.figure(figsize=(FIGSIZE/rows*cols,FIGSIZE))\n    \n    # display\n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        title = '' if label is None else CLASSES[label]\n        correct = True\n        if predictions is not None:\n            title, correct = title_from_label_and_target(predictions[i], label)\n        dynamic_titlesize = FIGSIZE*SPACING/max(rows,cols)*40+3 # magic formula tested to work from 1x1 to 10x10 images\n        subplot = display_one_flower(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n    \n    #layout\n    plt.tight_layout()\n    if label is None and predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()\n\ndef display_confusion_matrix(cmat, score, precision, recall):\n    plt.figure(figsize=(15,15))\n    ax = plt.gca()\n    ax.matshow(cmat, cmap='Reds')\n    ax.set_xticks(range(len(CLASSES)))\n    ax.set_xticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_xticklabels(), rotation=45, ha=\"left\", rotation_mode=\"anchor\")\n    ax.set_yticks(range(len(CLASSES)))\n    ax.set_yticklabels(CLASSES, fontdict={'fontsize': 7})\n    plt.setp(ax.get_yticklabels(), rotation=45, ha=\"right\", rotation_mode=\"anchor\")\n    titlestring = \"\"\n    if score is not None:\n        titlestring += 'f1 = {:.3f} '.format(score)\n    if precision is not None:\n        titlestring += '\\nprecision = {:.3f} '.format(precision)\n    if recall is not None:\n        titlestring += '\\nrecall = {:.3f} '.format(recall)\n    if len(titlestring) > 0:\n        ax.text(101, 1, titlestring, fontdict={'fontsize': 18, 'horizontalalignment':'right', 'verticalalignment':'top', 'color':'#804040'})\n    plt.show()\n    \ndef display_training_curves(training, validation, title, subplot):\n    if subplot%10==1: # set up the subplots on the first call\n        plt.subplots(figsize=(10,10), facecolor='#F0F0F0')\n        plt.tight_layout()\n    ax = plt.subplot(subplot)\n    ax.set_facecolor('#F8F8F8')\n    ax.plot(training)\n    ax.plot(validation)\n    ax.set_title('model '+ title)\n    ax.set_ylabel(title)\n    #ax.set_ylim(0.28,1.05)\n    ax.set_xlabel('epoch')\n    ax.legend(['train', 'valid.'])","metadata":{"execution":{"iopub.status.busy":"2022-03-02T06:00:18.710265Z","iopub.execute_input":"2022-03-02T06:00:18.710772Z","iopub.status.idle":"2022-03-02T06:00:18.742585Z","shell.execute_reply.started":"2022-03-02T06:00:18.710717Z","shell.execute_reply":"2022-03-02T06:00:18.741640Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"num_parallel_reads=AUTO is used to read multiple files automatically.\nexperimental_deterministic = False, we used “experimental_deterministic” to maintain the order of the data. Here, we disable the enforcement order to shuffle the data anyway.\n","metadata":{}},{"cell_type":"code","source":"def decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)  # image format uint8 [0,255]\n    image = tf.reshape(image, [*IMAGE_SIZE, 3]) # explicit size needed for TPU\n    return image\n\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"class\": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = tf.cast(example['class'], tf.int32)\n    return image, label # returns a dataset of (image, label) pairs\n\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"id\": tf.io.FixedLenFeature([], tf.string),  # shape [] means single element\n        # class is missing, this competitions's challenge is to predict flower classes for the test dataset\n    }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['id']\n    return image, idnum # returns a dataset of image(s)\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    # Read from TFRecords. For optimal performance, reading from multiple files at once and\n    # disregarding data order. Order does not matter since we will be shuffling the data anyway.\n\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order, increase speed\n\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) # automatically interleaves reads from multiple files\n    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord, num_parallel_calls=AUTO)\n    # returns a dataset of (image, label) pairs if labeled=True or (image, id) pairs if labeled=False\n    return dataset\n\ndef data_augment(image, label):\n    # data augmentation. Thanks to the dataset.prefetch(AUTO) statement in the next function (below),\n    # this happens essentially for free on TPU. Data pipeline code is executed on the \"CPU\" part\n    # of the TPU while the TPU itself is computing gradients.\n    image = tf.image.random_flip_left_right(image)\n    #image = tf.image.random_saturation(image, 0, 2)\n    return image, label   \n\n\ndef get_training_dataset():\n    dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\n    dataset = dataset.map(data_augment, num_parallel_calls=AUTO)\n    dataset = dataset.repeat() # the training dataset must repeat for several epochs\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_validation_dataset(ordered=False):\n    dataset = load_dataset(VALIDATION_FILENAMES, labeled=True, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef get_test_dataset(ordered=False):\n    dataset = load_dataset(TEST_FILENAMES, labeled=False, ordered=ordered)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset\n\ndef count_data_items(filenames):\n    # the number of data items is written in the name of the .tfrec files, i.e. flowers00-230.tfrec = 230 data items\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n\nNUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_VALIDATION_IMAGES = count_data_items(VALIDATION_FILENAMES)\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\nVALIDATION_STEPS = -(-NUM_VALIDATION_IMAGES // BATCH_SIZE) # The \"-(-//)\" trick rounds up instead of down :-)\nTEST_STEPS = -(-NUM_TEST_IMAGES // BATCH_SIZE)             # The \"-(-//)\" trick rounds up instead of down :-)\nprint('Dataset: {} training images, {} validation images, {} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))\nprint(\"Validation Steps:\", VALIDATION_STEPS)\nprint(\"Test Steps:\", TEST_STEPS)","metadata":{"execution":{"iopub.status.busy":"2022-03-02T06:00:18.745343Z","iopub.execute_input":"2022-03-02T06:00:18.745595Z","iopub.status.idle":"2022-03-02T06:00:18.769839Z","shell.execute_reply.started":"2022-03-02T06:00:18.745566Z","shell.execute_reply":"2022-03-02T06:00:18.768920Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# data dump\nprint(\"Training data shapes:\")\nfor image, label in get_training_dataset().take(3):\n    print(image.numpy().shape, label.numpy().shape)\nprint(\"Training data label examples:\", label.numpy())\nprint(\"Validation data shapes:\")\nfor image, label in get_validation_dataset().take(3):\n    print(image.numpy().shape, label.numpy().shape)\nprint(\"Validation data label examples:\", label.numpy())\nprint(\"Test data shapes:\")\nfor image, idnum in get_test_dataset().take(3):\n    print(image.numpy().shape, idnum.numpy().shape)\nprint(\"Test data IDs:\", idnum.numpy().astype('U')) # U=unicode string","metadata":{"execution":{"iopub.status.busy":"2022-03-02T06:00:18.771241Z","iopub.execute_input":"2022-03-02T06:00:18.771524Z","iopub.status.idle":"2022-03-02T06:00:24.630578Z","shell.execute_reply.started":"2022-03-02T06:00:18.771493Z","shell.execute_reply":"2022-03-02T06:00:24.629449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"These datasets are tf.data.Dataset objects. You can think about a dataset in TensorFlow as a stream of data records. The training and validation sets are streams of (image, label) pairs.","metadata":{}},{"cell_type":"code","source":"ds_train = get_training_dataset()\nprint(\"Training:\", ds_train)","metadata":{"execution":{"iopub.status.busy":"2022-03-02T06:00:24.633268Z","iopub.execute_input":"2022-03-02T06:00:24.633537Z","iopub.status.idle":"2022-03-02T06:00:24.679696Z","shell.execute_reply.started":"2022-03-02T06:00:24.633508Z","shell.execute_reply":"2022-03-02T06:00:24.678846Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Peek at training data\ntraining_dataset = get_training_dataset()\ntraining_dataset = training_dataset.unbatch().batch(20)\ntrain_batch = iter(training_dataset)","metadata":{"execution":{"iopub.status.busy":"2022-03-02T06:00:24.681121Z","iopub.execute_input":"2022-03-02T06:00:24.681349Z","iopub.status.idle":"2022-03-02T06:00:24.734784Z","shell.execute_reply.started":"2022-03-02T06:00:24.681312Z","shell.execute_reply":"2022-03-02T06:00:24.734070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# run this cell again for next set of images\ndisplay_batch_of_images(next(train_batch))","metadata":{"execution":{"iopub.status.busy":"2022-03-02T06:00:24.735868Z","iopub.execute_input":"2022-03-02T06:00:24.736502Z","iopub.status.idle":"2022-03-02T06:00:27.266757Z","shell.execute_reply.started":"2022-03-02T06:00:24.736466Z","shell.execute_reply":"2022-03-02T06:00:27.266036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# peer at test data\ntest_dataset = get_test_dataset()\ntest_dataset = test_dataset.unbatch().batch(20)\ntest_batch = iter(test_dataset)\ndisplay_batch_of_images(next(test_batch))","metadata":{"execution":{"iopub.status.busy":"2022-03-02T06:00:27.267615Z","iopub.execute_input":"2022-03-02T06:00:27.267859Z","iopub.status.idle":"2022-03-02T06:00:29.584085Z","shell.execute_reply.started":"2022-03-02T06:00:27.267831Z","shell.execute_reply":"2022-03-02T06:00:29.583132Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model","metadata":{}},{"cell_type":"code","source":"\"\"\"\"\nwith strategy.scope():\n    #img_adjust_layer = tf.keras.layers.Lambda(lambda data: tf.keras.applications.xception.preprocess_input(tf.cast(data, tf.float32)), input_shape=[*IMAGE_SIZE, 3])\n    #pretrained_model = tf.keras.applications.Xception(weights='imagenet', include_top=False)\n    \n    img_adjust_layer = tf.keras.layers.Lambda(lambda data: tf.keras.applications.vgg16.preprocess_input(tf.cast(data, tf.float32)), input_shape=[*IMAGE_SIZE, 3])\n    pretrained_model = tf.keras.applications.VGG16(weights='imagenet', include_top=False)\n    \n    pretrained_model.trainable = False # False = transfer learning, True = fine-tuning\n    \n    model = tf.keras.Sequential([\n        img_adjust_layer,\n        pretrained_model,\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation='softmax')\n    ])\n        \nmodel.compile(\n    optimizer='adam',\n    loss = 'sparse_categorical_crossentropy',\n    metrics=['sparse_categorical_accuracy'],\n    # NEW on TPU in TensorFlow 24: sending multiple batches to the TPU at once saves communications\n    # overheads and allows the XLA compiler to unroll the loop on TPU and optimize hardware utilization.\n    steps_per_execution=16\n)\nmodel.summary()\n\"\"\"","metadata":{"execution":{"iopub.status.busy":"2022-03-02T06:00:29.585306Z","iopub.execute_input":"2022-03-02T06:00:29.585576Z","iopub.status.idle":"2022-03-02T06:00:29.592081Z","shell.execute_reply.started":"2022-03-02T06:00:29.585547Z","shell.execute_reply":"2022-03-02T06:00:29.591476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Learning Rate Schedule for Fine Tuning #\n# Learning rate schedule for TPU, GPU and CPU.\n# Using an LR ramp up because fine-tuning a pre-trained model.\n# Starting with a high LR would break the pre-trained weights.\n\nLR_START = 0.00001\nLR_MAX = 0.00005 * strategy.num_replicas_in_sync\nLR_MIN = 0.00001\nLR_RAMPUP_EPOCHS = 5\nLR_SUSTAIN_EPOCHS = 0\nLR_EXP_DECAY = .8\n\ndef lrfn(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = LR_START + (epoch * (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS)\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n    return lr\n    \nlr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True)\nrng = [i for i in range(EPOCHS)]\ny = [lrfn(x) for x in rng]\nplt.plot(rng, y)\nprint(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\".format(y[0], max(y), y[-1]))","metadata":{"execution":{"iopub.status.busy":"2022-03-02T06:00:29.593070Z","iopub.execute_input":"2022-03-02T06:00:29.593495Z","iopub.status.idle":"2022-03-02T06:00:29.833283Z","shell.execute_reply.started":"2022-03-02T06:00:29.593464Z","shell.execute_reply":"2022-03-02T06:00:29.832387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install efficientnet\nimport efficientnet.tfkeras as efficientnet","metadata":{"execution":{"iopub.status.busy":"2022-03-02T06:00:29.834845Z","iopub.execute_input":"2022-03-02T06:00:29.835368Z","iopub.status.idle":"2022-03-02T06:00:40.353523Z","shell.execute_reply.started":"2022-03-02T06:00:29.835307Z","shell.execute_reply":"2022-03-02T06:00:40.352606Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n    pretrained_model = efficientnet.EfficientNetB7(weights = 'noisy-student', include_top = False, input_shape = [*IMAGE_SIZE, 3])\n    pretrained_model.trainable = True\n\n    model = tf.keras.Sequential([\n        pretrained_model,\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation = 'softmax')\n    ])\n\nmodel.compile(optimizer = 'adam', loss = 'sparse_categorical_crossentropy',\n              metrics = ['sparse_categorical_accuracy'])\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2022-03-02T06:00:40.356531Z","iopub.execute_input":"2022-03-02T06:00:40.356805Z","iopub.status.idle":"2022-03-02T06:01:29.867772Z","shell.execute_reply.started":"2022-03-02T06:00:40.356775Z","shell.execute_reply":"2022-03-02T06:01:29.866785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"early_stop = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=20, restore_best_weights = True)\n\nhistory = model.fit(get_training_dataset(), \n                    steps_per_epoch=STEPS_PER_EPOCH, \n                    epochs=EPOCHS,\n                    validation_data=get_validation_dataset(), \n                    validation_steps=VALIDATION_STEPS,\n                    callbacks=[lr_callback, early_stop],)","metadata":{"execution":{"iopub.status.busy":"2022-03-02T06:01:29.869282Z","iopub.execute_input":"2022-03-02T06:01:29.870069Z","iopub.status.idle":"2022-03-02T06:57:21.039571Z","shell.execute_reply.started":"2022-03-02T06:01:29.870031Z","shell.execute_reply":"2022-03-02T06:57:21.038534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display_training_curves(history.history['loss'], history.history['val_loss'], 'loss', 211)\ndisplay_training_curves(history.history['sparse_categorical_accuracy'], history.history['val_sparse_categorical_accuracy'], 'accuracy', 212)","metadata":{"execution":{"iopub.status.busy":"2022-03-02T06:57:21.041954Z","iopub.execute_input":"2022-03-02T06:57:21.042484Z","iopub.status.idle":"2022-03-02T06:57:21.588368Z","shell.execute_reply.started":"2022-03-02T06:57:21.042430Z","shell.execute_reply":"2022-03-02T06:57:21.587455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Confusion Matrix","metadata":{}},{"cell_type":"code","source":"cmdataset = get_validation_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and labels, order matters.\nimages_ds = cmdataset.map(lambda image, label: image)\nlabels_ds = cmdataset.map(lambda image, label: label).unbatch()\ncm_correct_labels = next(iter(labels_ds.batch(NUM_VALIDATION_IMAGES))).numpy() # get everything as one batch\ncm_probabilities = model.predict(images_ds, steps=VALIDATION_STEPS)\ncm_predictions = np.argmax(cm_probabilities, axis=-1)\nprint(\"Correct   labels: \", cm_correct_labels.shape, cm_correct_labels)\nprint(\"Predicted labels: \", cm_predictions.shape, cm_predictions)","metadata":{"execution":{"iopub.status.busy":"2022-03-02T06:57:21.589641Z","iopub.execute_input":"2022-03-02T06:57:21.589888Z","iopub.status.idle":"2022-03-02T06:57:43.952181Z","shell.execute_reply.started":"2022-03-02T06:57:21.589862Z","shell.execute_reply":"2022-03-02T06:57:43.951314Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cmat = confusion_matrix(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)))\nscore = f1_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\nprecision = precision_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\nrecall = recall_score(cm_correct_labels, cm_predictions, labels=range(len(CLASSES)), average='macro')\ncmat = (cmat.T / cmat.sum(axis=1)).T # normalized\ndisplay_confusion_matrix(cmat, score, precision, recall)\nprint('f1 score: {:.3f}, precision: {:.3f}, recall: {:.3f}'.format(score, precision, recall))","metadata":{"execution":{"iopub.status.busy":"2022-03-02T06:57:43.953693Z","iopub.execute_input":"2022-03-02T06:57:43.953938Z","iopub.status.idle":"2022-03-02T06:57:49.008537Z","shell.execute_reply.started":"2022-03-02T06:57:43.953912Z","shell.execute_reply":"2022-03-02T06:57:49.007513Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Predictions","metadata":{}},{"cell_type":"code","source":"test_ds = get_test_dataset(ordered=True) # since we are splitting the dataset and iterating separately on images and ids, order matters.\n\nprint('Computing predictions...')\ntest_images_ds = test_ds.map(lambda image, idnum: image)\nprobabilities = model.predict(test_images_ds, steps=TEST_STEPS)\npredictions = np.argmax(probabilities, axis=-1)\nprint(predictions)\n\nprint('Generating submission.csv file...')\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U') # all in one batch\nnp.savetxt('submission.csv', np.rec.fromarrays([test_ids, predictions]), fmt=['%s', '%d'], delimiter=',', header='id,label', comments='')\nnp.savetxt('submission2.csv', np.rec.fromarrays([test_ids, predictions]), fmt=['%s', '%d'], delimiter=',', header='id,label', comments='')\nprint('File generated.')\n#!head submission.csv","metadata":{"execution":{"iopub.status.busy":"2022-03-02T06:57:49.009923Z","iopub.execute_input":"2022-03-02T06:57:49.010182Z","iopub.status.idle":"2022-03-02T06:58:10.544264Z","shell.execute_reply.started":"2022-03-02T06:57:49.010152Z","shell.execute_reply":"2022-03-02T06:58:10.543167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# # # Visual Validation","metadata":{}},{"cell_type":"code","source":"dataset = get_validation_dataset()\ndataset = dataset.unbatch().batch(20)\nbatch = iter(dataset)","metadata":{"execution":{"iopub.status.busy":"2022-03-02T06:58:10.546094Z","iopub.execute_input":"2022-03-02T06:58:10.546460Z","iopub.status.idle":"2022-03-02T06:58:10.587524Z","shell.execute_reply.started":"2022-03-02T06:58:10.546417Z","shell.execute_reply":"2022-03-02T06:58:10.586555Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# run this cell again for next set of images\nimages, labels = next(batch)\nprobabilities = model.predict(tf.cast(images, tf.float32))\npredictions = np.argmax(probabilities, axis=-1)\ndisplay_batch_of_images((images, labels), predictions)","metadata":{"execution":{"iopub.status.busy":"2022-03-02T06:58:10.588779Z","iopub.execute_input":"2022-03-02T06:58:10.589022Z","iopub.status.idle":"2022-03-02T06:58:30.707126Z","shell.execute_reply.started":"2022-03-02T06:58:10.588994Z","shell.execute_reply":"2022-03-02T06:58:30.705824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}