{"cells":[{"metadata":{},"cell_type":"markdown","source":"## TPU(Tensor Processing Unit)\n\nTPUs are hardware accelerators specialized in deep learning taks. The extra hardware can be used to accelerate training by increasing the training batch size.\n\nTPUs pair a classic vector processor with a dedicated matrix multiply unit and excel at any task where large matrix multiplications dominate, such as neural networks.\n\n### The hardware\n#### MXU and VPU\n\nA TPU v2 core is made of a Matrix Multiply Unit (MXU) which runs matrix multiplications and a Vector Processing Unit (VPU) for all other tasks such as activations, softmax, etc. The VPU handles float32 and int32 computations. The MXU on the other hand operates in a mixed precision 16-32 bit floating point format.\n\n### Mixed precision floating point and bfloat16\n\nThe MXU computes matrix multiplications using bfloat16 inputs and float32 outputs. Intermediate accumulations are performed in float32 precision. Google introduced the bfloat16 format in TPUs. bfloat16 is a truncated float32 with exactly the same exponent bits and range as float32. This, added to the fact that TPUs compute matrix multiplications in mixed precision with bfloat16 inputs but float32 outputs, means that, typically, no code changes are necessary to benefit from the performance gains of reduced precision."},{"metadata":{},"cell_type":"markdown","source":"### The software\n\n#### Large batch size training\n\nThe ideal batch size for TPUs is 128 data items per TPU core but the hardware can already show good utilization from 8 data items per TPU core. Remember that one Cloud TPU has 8 cores.\n\nIn this notebook, we will be using the Keras API. In Keras, the batch you specify is the global batch size for the entire TPU. Our batches will automatically be split in 8 and ran on the 8 cores of the TPU."},{"metadata":{},"cell_type":"markdown","source":"## Imports"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"from kaggle_datasets import KaggleDatasets\nimport numpy as np\nimport tensorflow as tf\nimport re\nfrom tensorflow import keras\n\nprint(f'Tensorflow version: {tf.__version__}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Distribution Strategy"},{"metadata":{"trusted":true},"cell_type":"code","source":"# TPU detection\ntry:\n    # TPU detection. No parameters necessary if TPU_NAME environment variable is set.\n    # On Kaggle this is always the case.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\n# TPUStrategy for distributing training\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse: # default strategy that works on CPU and single GPU\n    strategy = tf.distribute.get_strategy()\n\nprint('Replicas ',strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In this code snippet :\n\n- **TPUStrategyResolver()** finds the TPU on the network.\n- **TPUStrategy** is the part that implements the distribution and the 'All-Reduce' gradient synchronization algorithm.\n- The strategy is applied through a scope. The model must be defined within the strategy scope().\n- The **tpu_model.fit** function expects a tf.data.Dataset object for input for TPU training."},{"metadata":{"trusted":true},"cell_type":"code","source":"AUTO = tf.data.experimental.AUTOTUNE\n\n# CONFIGURATIONS\nIMAGE_SIZE =  [192, 192]\nEPOCHS = 20\nFOLDS = 3\nBATCH_SIZE = 16 * strategy.num_replicas_in_sync","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Loading Data\n\n**Get GCS Path**\n* When used with TPUs, datasets must be stored in a [Google Cloud Storage Buckets](https://cloud.google.com/storage).\n\nWe can use data from any public GCS bucket by giving its path just like we would data from '/kaggle/input'. The following will retrieve the GCS path for this competition's dataset."},{"metadata":{},"cell_type":"markdown","source":"# Data Directories"},{"metadata":{"trusted":true},"cell_type":"code","source":"GCS_DS_PATH = KaggleDatasets().get_gcs_path()\nprint(GCS_DS_PATH)\n\nGCS_PATH_SELECT = { # Image Sizes\n    192: GCS_DS_PATH + '/tfrecords-jpeg-192x192',\n    224: GCS_DS_PATH + '/tfrecords-jpeg-224x224',\n    331: GCS_DS_PATH + '/tfrecords-jpeg-331x331',\n    512: GCS_DS_PATH + '/tfrecords-jpeg-512x512'\n}\n\nGCS_PATH = GCS_PATH_SELECT[IMAGE_SIZE[0]]\nprint(GCS_PATH)\n\nTRAINING_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/train/*.tfrec')\nVALIDATION_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/val/*.tfrec')\nTEST_FILENAMES = tf.io.gfile.glob(GCS_PATH + '/test/*.tfrec') # predictions on this dataset should be submitted for the competition","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# CLASSES"},{"metadata":{"trusted":true},"cell_type":"code","source":"# 0 to 103\nCLASSES = [\n    \"pink primrose\",\n    \"hard-leaved pocket orchid\",\n    \"canterbury bells\",\n    \"sweet pea\",\n    \"wild geranium\",\n    \"tiger lily\",\n    \"moon orchid\",\n    \"bird of paradise\",\n    \"monkshood\",\n    \"globe thistle\",\n    \"snapdragon\",\n    \"colt's foot\",\n    \"king protea\",\n    \"spear thistle\",\n    \"yellow iris\",\n    \"globe-flower\",\n    \"purple coneflower\",\n    \"peruvian lily\",\n    \"balloon flower\",\n    \"giant white arum lily\",\n    \"fire lily\",\n    \"pincushion flower\",\n    \"fritillary\",\n    \"red ginger\",\n    \"grape hyacinth\",\n    \"corn poppy\",\n    \"prince of wales feathers\",\n    \"stemless gentian\",\n    \"artichoke\",\n    \"sweet william\",\n    \"carnation\",\n    \"garden phlox\",\n    \"love in the mist\",\n    \"cosmos\",\n    \"alpine sea holly\",\n    \"ruby-lipped cattleya\",\n    \"cape flower\",\n    \"great masterwort\",\n    \"siam tulip\",\n    \"lenten rose\",\n    \"barberton daisy\",\n    \"daffodil\",\n    \"sword lily\",\n    \"poinsettia\",\n    \"bolero deep blue\",\n    \"wallflower\",\n    \"marigold\",\n    \"buttercup\",\n    \"daisy\",\n    \"common dandelion\",\n    \"petunia\",\n    \"wild pansy\",\n    \"primula\",\n    \"sunflower\",\n    \"lilac hibiscus\",\n    \"bishop of llandaff\",\n    \"gaura\",\n    \"geranium\",\n    \"orange dahlia\",\n    \"pink-yellow dahlia\",\n    \"cautleya spicata\",\n    \"japanese anemone\",\n    \"black-eyed susan\",\n    \"silverbush\",\n    \"californian poppy\",\n    \"osteospermum\",\n    \"spring crocus\",\n    \"iris\",\n    \"windflower\",\n    \"tree poppy\",\n    \"gazania\",\n    \"azalea\",\n    \"water lily\",\n    \"rose\",\n    \"thorn apple\",\n    \"morning glory\",\n    \"passion flower\",\n    \"lotus\",\n    \"toad lily\",\n    \"anthurium\",\n    \"frangipani\",\n    \"clematis\",\n    \"hibiscus\",\n    \"columbine\",\n    \"desert-rose\",\n    \"tree mallow\",\n    \"magnolia\",\n    \"cyclamen \",\n    \"watercress\",\n    \"canna lily\",\n    \"hippeastrum \",\n    \"bee balm\",\n    \"pink quill\",\n    \"foxglove\",\n    \"bougainvillea\",\n    \"camellia\",\n    \"mallow\",\n    \"mexican petunia\",\n    \"bromelia\",\n    \"blanket flower\",\n    \"trumpet creeper\",\n    \"blackberry lily\",\n    \"common tulip\",\n    \"wild rose\",\n]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# DATASET FUNCTIONS"},{"metadata":{"trusted":true},"cell_type":"code","source":"def decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels = 3)\n    image = tf.cast(image, tf.float32) / 255.0\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        'image':tf.io.FixedLenFeature([], tf.string),\n        'class':tf.io.FixedLenFeature([], tf.int64),\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = tf.cast(example['class'], tf.int32)\n    return image, label","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_unlabeled_tfrecord(test_example):\n    UNLABELED_TFREC_FORMAT = {\n        'image':tf.io.FixedLenFeature([], tf.string),\n        'id':tf.io.FixedLenFeature([], tf.string),\n    }\n    example = tf.io.parse_single_example(test_example,UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['id']\n    return image, idnum","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def data_augment(image, label):\n    # data augmentation. Thanks to the dataset.prefetch(AUTO) statement in the next function (below),\n    # this happens essentially for free on TPU. Data pipeline code is executed on the \"CPU\" part\n    # of the TPU while the TPU itself is computing gradients.\n    image = tf.image.random_flip_left_right(image)\n    return image, label ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_dataset(filenames, labeled = True, ordered = False):\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False\n        # disable order, increase speed\n    # automatically interleaves reads from multiple files\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads = AUTO)\n    \n    dataset = dataset.with_options(ignore_order)\n    \n    dataset = dataset.map(read_labeled_tfrecord if labeled else read_unlabeled_tfrecord, num_parallel_calls = AUTO)\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_training_dataset():\n    dataset = load_dataset(TRAINING_FILENAMES, labeled = True, ordered = False)\n    dataset = dataset.map(data_augment, num_parallel_calls = AUTO)\n    dataset = dataset.repeat()\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_validation_dataset():\n    dataset = load_dataset(VALIDATION_FILENAMES, labeled = True, ordered = False)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.cache()\n    dataset = dataset.prefetch(AUTO) # prefetch next batch while training (autotune prefetch buffer size)\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_test_dataset():\n    dataset = load_dataset(TEST_FILENAMES, labeled = False, ordered = True)\n    dataset = dataset.batch(BATCH_SIZE)\n    # prefetch next batch while training (autotune prefetch buffer size)\n    dataset = dataset.prefetch(AUTO)\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def count_data_items(filenames):\n    # the number of data items is written in the name of the .tfrec files, \n    # i.e. flowers00-230.tfrec = 230 data items\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"NUM_TRAINING_IMAGES = count_data_items(TRAINING_FILENAMES)\nNUM_VALIDATION_IMAGES = count_data_items(VALIDATION_FILENAMES)\nNUM_TEST_IMAGES = count_data_items(TEST_FILENAMES)\nSTEPS_PER_EPOCH = NUM_TRAINING_IMAGES // BATCH_SIZE\nprint('Dataset: {} training images, {} validation images, {} unlabeled test images'.format(NUM_TRAINING_IMAGES, NUM_VALIDATION_IMAGES, NUM_TEST_IMAGES))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"training_dataset = get_training_dataset()\nvalidation_dataset = get_validation_dataset()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualizing dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Iterate over first batch (128 images)\nfor img, label in training_dataset.take(1):\n#     Get first 16 images\n    data = [img[0:16,:,:,:].numpy(),label[0:16].numpy()]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data[0].shape, data[1].shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nrows = 4\ncols = 4\nfig = plt.figure(figsize  = (18, 10))\nfor index in range(1, rows * cols + 1):\n    ax = fig.add_subplot(rows, cols, index)\n    img = data[0][index -1]\n    label = data[1][index - 1]\n    ax.axis('off')\n    plt.imshow(img)\n    plt.title(CLASSES[label])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Custom Learning Rate Scheduler"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Learning rate schedule for TPU, GPU and CPU.\n# Using an LR ramp up because fine-tuning a pre-trained model.\n# Starting with a high LR would break the pre-trained weights.\nLR_START = 0.00001\nLR_MAX = 0.00005 * strategy.num_replicas_in_sync\nLR_MIN = 0.00001\nLR_RAMPUP_EPOCHS = 5\nLR_SUSTAIN_EPOCHS = 0\nLR_EXP_DECAY = 0.8\n\n#  (0, LR_START) <-> (LR_RAMPUP_EPOCHS, LR_MAX)\n\ndef lrfun(epoch):\n    if epoch < LR_RAMPUP_EPOCHS:\n        # Equation of a straight line\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS*epoch + LR_START\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    else:\n        lr = (LR_MAX - LR_MIN)*LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n    return lr","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfun, verbose = True)\n\nrng = [i for i in range(25 if  EPOCHS < 25 else EPOCHS)]\ny = [lrfun(x) for x in rng]\nplt.plot(rng, y)\nprint(\"Learning rate schedule: {:.3g} to {:.3g} to {:.3g}\".format(y[0], max(y), y[-1]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Using Pretrained Model VGG16"},{"metadata":{"trusted":true},"cell_type":"code","source":"# with strategy.scope():\n#     pretrained_model = tf.keras.applications.VGG16(weights = 'imagenet', include_top = False, input_shape = [*IMAGE_SIZE, 3])\n#     pretrained_model.trainable = False\n    \n#     model = tf.keras.Sequential([\n#         pretrained_model,\n#         tf.keras.layers.GlobalAveragePooling2D(),\n#         tf.keras.layers.Dense(len(CLASSES), activation = 'softmax', dtype = 'float32')\n#     ])\n    \n# model.compile(\n#     optimizer = 'adam',\n#     loss = 'sparse_categorical_crossentropy',\n#     metrics = ['sparse_categorical_accuracy'])\n\n# historical = model.fit(\n#     training_dataset,\n#     steps_per_epoch = STEPS_PER_EPOCH,\n#     epochs = EPOCHS,\n#     callbacks = [lr_callback],\n#     validation_data = validation_dataset)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Using Pretrained Model VGG19"},{"metadata":{"trusted":true},"cell_type":"code","source":"# with strategy.scope():\n#     pretrained_model = tf.keras.applications.VGG19(weights = 'imagenet', include_top = False, input_shape = [*IMAGE_SIZE, 3])\n#     pretrained_model.trainable = False\n    \n#     model = tf.keras.Sequential([\n#         pretrained_model,\n#         tf.keras.layers.GlobalAveragePooling2D(),\n#         tf.keras.layers.Dense(len(CLASSES), activation = 'softmax', dtype = 'float32')\n#     ])\n    \n# model.compile(\n#     optimizer = 'adam',\n#     loss = 'sparse_categorical_crossentropy',\n#     metrics = ['sparse_categorical_accuracy'])\n\n# historical = model.fit(\n#     training_dataset,\n#     steps_per_epoch = STEPS_PER_EPOCH,\n#     epochs = EPOCHS,\n#     callbacks = [lr_callback],\n#     validation_data = validation_dataset)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Using Pretrained model DenseNet201"},{"metadata":{"trusted":true},"cell_type":"code","source":"# with strategy.scope():\n#     pretrained_model = tf.keras.applications.DenseNet201(weights = 'imagenet', include_top = False, input_shape = [*IMAGE_SIZE, 3])\n#     pretrained_model.trainable = False\n    \n#     model = tf.keras.Sequential([\n#         pretrained_model,\n#         tf.keras.layers.GlobalAveragePooling2D(),\n#         tf.keras.layers.Dense(len(CLASSES), activation = 'softmax', dtype = 'float32')\n#     ])\n    \n# model.compile(\n#     optimizer = 'adam',\n#     loss = 'sparse_categorical_crossentropy',\n#     metrics = ['sparse_categorical_accuracy'])\n\n# historical = model.fit(\n#     training_dataset,\n#     steps_per_epoch = STEPS_PER_EPOCH,\n#     epochs = EPOCHS,\n#     callbacks = [lr_callback],\n#     validation_data = validation_dataset)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Using Trainable DenseNet201"},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    pretrained_model = tf.keras.applications.DenseNet201(weights = 'imagenet', include_top = False, input_shape = [*IMAGE_SIZE, 3])\n    pretrained_model.trainable = True\n    \n    model = tf.keras.Sequential([\n        pretrained_model,\n        tf.keras.layers.GlobalAveragePooling2D(),\n        tf.keras.layers.Dense(len(CLASSES), activation = 'softmax', dtype = 'float32')\n    ])\n    \nmodel.compile(\n    optimizer = 'adam',\n    loss = 'sparse_categorical_crossentropy',\n    metrics = ['sparse_categorical_accuracy'])\n\nhistorical = model.fit(\n    training_dataset,\n    steps_per_epoch = STEPS_PER_EPOCH,\n    epochs = EPOCHS,\n    callbacks = [lr_callback],\n    validation_data = validation_dataset)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Plotting the results"},{"metadata":{"trusted":true},"cell_type":"code","source":"training_loss = historical.history['loss']\ntraining_sparse_categorical_accuracy = historical.history['sparse_categorical_accuracy']\n\nvalidation_loss = historical.history['val_loss']\nvalidation_sparse_categorical_accuracy = historical.history['val_sparse_categorical_accuracy']\n\nepochs = np.arange(EPOCHS)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.subplots(1,2)\n\nplt.subplot(1, 2, 1)\nplt.plot(epochs, training_loss,label = 'Training Loss')\nplt.plot(epochs, validation_loss, label = 'Validation Loss')\nplt.xlabel('Epochs')\nplt.legend()\n\nplt.subplot(1, 2, 2)\nplt.plot(epochs, training_sparse_categorical_accuracy,label = 'Training Accuracy')\nplt.plot(epochs, validation_sparse_categorical_accuracy, label = 'Validation Accuracy')\nplt.xlabel('Epochs')\nplt.legend()\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Confusion Matrix and Validation Score"},{"metadata":{"trusted":true},"cell_type":"code","source":"# TODO","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Computing Predictions on testset"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_dataset = get_test_dataset()\ntest_images = test_dataset.map(lambda image, idnum: image)\nprob = model.predict(test_images)\npred = np.argmax(prob, axis = -1)\nprint(pred)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_ids_ds = test_dataset.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(NUM_TEST_IMAGES))).numpy().astype('U') # all in one batch\nnp.savetxt('submission.csv', np.rec.fromarrays([test_ids, pred]), fmt=['%s', '%d'], delimiter=',', header='id,label', comments='')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}