{"cells":[{"metadata":{},"cell_type":"markdown","source":"# A Beginner's TPU Kernel\n\nThis is a beginner-friendly kernel for those who are beginners to Kaggle like me, that are:\n\n  *  Are not really familiar with TensorFlow - the code is made as readable as possible with a lot of comments.\n  *  Are new to TPU - step-by-step example of how to execute your code on TPU\n  \nThis notebook does not contain any fancy visualizations, analysis, or ensembling techniques. Just a bare minimum single model training pipeline that will yield good results (97% accuracy or more).","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true,"collapsed":true},"cell_type":"code","source":"!pip install -q efficientnet\n\nimport os\nimport re\nimport time\nimport numpy as np\nimport tensorflow as tf\nimport tensorflow.keras.backend as K\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport math\nfrom kaggle_datasets import KaggleDatasets\nimport kernel_tensorflow_utils as ktu\nimport efficientnet.tfkeras as efficientnet\n\nktu.HardwareInfo.print_tf_version()\nsns.set()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Configure TPU\n\nHere, we resolve TPU delegate. If found, we connect to the cluster and initialize the TPU for use. Then, we create a distribution strategy (since we will have not one, but 8 TPU cores in the instance, each with 8GB of in-unit memory). You can find more resources in following links:\n\n* [Tensorflow quick guide on using TPU](https://www.tensorflow.org/guide/tpu#improving_performance_by_multiple_steps_within_tffunction)\n* [Kaggle TPU Documentation](https://www.kaggle.com/docs/tpu)\n* [Kaggle discussions on training optimization](https://www.kaggle.com/c/flower-classification-with-tpus/discussion/135443)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection. No parameters necessary if TPU_NAME environment variable is set. On Kaggle this is always the case.\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Importing Data\n\nFirst, we define the paths to our data. The dataset contains different directories for different image sizes, ranging from 192, 224, 331, to 512. We will only use the size 224, as it is consistent with input sizes of models pre-trained on ImageNet. We will use two sets of data:\n\n* [`tpu-getting-started`](https://www.kaggle.com/c/tpu-getting-started/data) - the dataset of this competition\n* [`tf-flower-photo-tfrec`](https://www.kaggle.com/kirillblinov/tf-flower-photo-tfrec) - the dataset from previous similar competition (+55k images) with the same structure","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Let's use 224x224 by default\nimage_size = (224, 224)\n\n# Data dirs\ndata_gcs = KaggleDatasets().get_gcs_path('tpu-getting-started')\next_gcs = KaggleDatasets().get_gcs_path('tf-flower-photo-tfrec')\n\n# Subdirs by image size\ndata_dir_by_size = {\n    (512, 512): '/tfrecords-jpeg-512x512',\n    (331, 331): '/tfrecords-jpeg-331x331',\n    (224, 224): '/tfrecords-jpeg-224x224',\n    (192, 192): '/tfrecords-jpeg-192x192'\n}\nsubdir = data_dir_by_size[image_size]\n\n# Paths to data files\ntrain_file_names = tf.io.gfile.glob(data_gcs + subdir + '/train' + '/*.tfrec')\nval_file_names = tf.io.gfile.glob(data_gcs + subdir + '/val' + '/*.tfrec')\ntest_file_names = tf.io.gfile.glob(data_gcs + subdir + '/test' + '/*.tfrec')\n\n# Extending the dataset with additional data\nimagenet_files = tf.io.gfile.glob(ext_gcs + '/imagenet' + subdir + '/*.tfrec')\ninaturelist_files = tf.io.gfile.glob(ext_gcs + '/inaturalist' + subdir + '/*.tfrec')\nopenimage_files = tf.io.gfile.glob(ext_gcs + '/openimage' + subdir + '/*.tfrec')\noxford_files = tf.io.gfile.glob(ext_gcs + '/oxford_102' + subdir + '/*.tfrec')\ntensorflow_files = tf.io.gfile.glob(ext_gcs + '/tf_flowers' + subdir + '/*.tfrec')\n\ntrain_file_names = train_file_names + imagenet_files + inaturelist_files + \\\n    openimage_files + oxford_files + tensorflow_files","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Classification Categories in our Dataset\nflower_categories = [\n    'pink primrose',     'hard-leaved pocket orchid', 'canterbury bells',\n    'sweet pea',         'wild geranium',             'tiger lily',\n    'moon orchid',       'bird of paradise',          'monkshood',\n    'globe thistle',     'snapdragon',                'colt\\'s foot',\n    'king protea',       'spear thistle',             'yellow iris',\n    'globe-flower',      'purple coneflower',         'peruvian lily',\n    'balloon flower',    'giant white arum lily',     'fire lily',\n    'pincushion flower', 'fritillary',                'red ginger',\n    'grape hyacinth',    'corn poppy',                'prince of wales feathers',\n    'stemless gentian',  'artichoke',                 'sweet william',\n    'carnation',         'garden phlox',              'love in the mist',\n    'cosmos',            'alpine sea holly',          'ruby-lipped cattleya',\n    'cape flower',       'great masterwort',          'siam tulip',\n    'lenten rose',       'barberton daisy',           'daffodil',\n    'sword lily',        'poinsettia',                'bolero deep blue',\n    'wallflower',        'marigold',                  'buttercup',\n    'daisy',             'common dandelion',          'petunia',\n    'wild pansy',        'primula',                   'sunflower',\n    'lilac hibiscus',    'bishop of llandaff',        'gaura',\n    'geranium',          'orange dahlia',             'pink-yellow dahlia',\n    'cautleya spicata',  'japanese anemone',          'black-eyed susan',\n    'silverbush',        'californian poppy',         'osteospermum',\n    'spring crocus',     'iris',                      'windflower',\n    'tree poppy',        'gazania',                   'azalea',\n    'water lily',        'rose',                      'thorn apple',\n    'morning glory',     'passion flower',            'lotus',\n    'toad lily',         'anthurium',                 'frangipani',\n    'clematis',          'hibiscus',                  'columbine',\n    'desert-rose',       'tree mallow',               'magnolia',\n    'cyclamen ',         'watercress',                'canna lily',\n    'hippeastrum ',      'bee balm',                  'pink quill',\n    'foxglove',          'bougainvillea',             'camellia',\n    'mallow',            'mexican petunia',           'bromelia',\n    'blanket flower',    'trumpet creeper',           'blackberry lily',\n    'common tulip',      'wild rose'\n]\n\nprint('Number of flower categories:', len(flower_categories))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, we just need to form a dataset out of the `*.tfrec` files. Below, we define some helper functions to combine them into a single memory map and parse the structure of our dataset into a more standard `(image, label)` format for classification. Please refer to [Tensorflow Data API](https://www.tensorflow.org/guide/data_performance) and [TFRecordDataset Documentation](https://www.tensorflow.org/api_docs/python/tf/data/TFRecordDataset) for more information.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def decode_image(image_data):\n    \"\"\"Decodes JPEG data and return a normalized image.\n    \n    WARNING: you may need a different normalization if you\n    use VGG-like networks.\n    \"\"\"\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0\n    image = tf.reshape(image, [*image_size, 3])\n    return image\n\n\ndef read_labeled_tfrecord(example):\n    \"\"\"\n    Converts a single record in labeled dataset (i.e. train and validation\n    sets) to the more convenient format (image, label)\n    \"\"\"\n    example = tf.io.parse_single_example(\n        serialized=example,\n        features={\n            'image': tf.io.FixedLenFeature([], tf.string),\n            'class': tf.io.FixedLenFeature([], tf.int64),\n        }\n    )\n    \n    image = decode_image(example['image'])\n    label = tf.cast(example['class'], tf.int32)\n    return image, label\n\n\ndef read_unlabeled_tfrecord(example):\n    \"\"\"\n    Converts a single record in labeled dataset (i.e. test\n    set) to the more convenient format (image, id)\n    \"\"\"\n    example = tf.io.parse_single_example(\n        serialized=example,\n        features={\n            'image': tf.io.FixedLenFeature([], tf.string),\n            'id': tf.io.FixedLenFeature([], tf.string),\n        }\n    )\n    \n    image = decode_image(example['image'])\n    idnum = example['id']\n    return image, idnum\n\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    \"\"\"\n    Given a list of `*.tfrec` file names, converts them into a `tf.data.Dataset`\n    object that yields elements of format (image, label) or (image, id)\n    \n    # Arguments\n        filenames: list of paths to `*.tfrec` files\n        labeled: if True, the resulting dataset will yield data in format\n            (image, label). Otherwise it will yield in format (image, id)\n        ordered: whether to shuffle the dataset (not desirable for test/val)\n        \n    # Returns\n        a `tf.data.Dataset` object that holds memory map to `*.tfrec` files\n    \"\"\"\n    ignore_order = tf.data.Options()\n    ignore_order.experimental_deterministic = ordered\n    \n    dataset = tf.data.TFRecordDataset(\n        filenames, num_parallel_reads=tf.data.experimental.AUTOTUNE)\n    dataset = dataset.with_options(ignore_order)\n    dataset = dataset.map(\n        read_labeled_tfrecord if labeled else read_unlabeled_tfrecord,\n        num_parallel_calls=tf.data.experimental.AUTOTUNE)\n    return dataset\n\n\ndef count_data_items(filenames):\n    \"\"\"\n    There's no way to obtain explicitly the number of elements in each dataset\n    (see: https://stackoverflow.com/questions/40472139/), but we can infer that\n    from file names, i.e. flowers00-230.tfrec = 230 data items\n    \"\"\"\n    return np.sum([\n        int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1))\n        for filename in filenames])\n\n\n# Import Datasets\ntrain_dataset = load_dataset(train_file_names, labeled=True, ordered=False)\nval_dataset = load_dataset(val_file_names, labeled=True, ordered=False)\ntest_dataset = load_dataset(test_file_names, labeled=False, ordered=True)\n\n# Calculate number of items\nnum_training_samples = count_data_items(train_file_names)\nnum_validation_samples = count_data_items(val_file_names)\nnum_testing_samples = count_data_items(test_file_names)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print('Train samples:', num_training_samples, end=', ')\nprint('Val samples:', num_validation_samples, end=', ')\nprint('Test samples:', num_testing_samples)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Below are all available categories in the given dataset (we display a sample from each of 104 categories). As expected from flowers, they are quite symmetric, which is good - we can apply augmentations like flipping, shearing, and rotating. Since the images were taken in nature, the exposure varies, so we can use brightness augmentations also.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"rows, cols = 13, 8\nplt.figure(figsize=(2.2 * cols, rows * 2))\n\nimage_dict = dict([(key, None) for key in range(len(flower_categories))])\n\ndef is_not_full_yet():\n    for key, value in image_dict.items():\n        if value is None:\n            return True\n    return False\n\nfor i, (image, label) in enumerate(train_dataset.as_numpy_iterator()):\n    if i >= num_training_samples:\n        break\n\n    if label in image_dict and image_dict[label] is None:\n        plt.subplot(rows, cols, label + 1)\n        plt.axis('off')\n        plt.imshow(image)\n        plt.title(flower_categories[label])\n        image_dict[label] = True\n    \n    if not is_not_full_yet():\n        break","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data augmentation methods in GPU/TPU\n\nThe definition of our transformation object `transformer` is self explanatory. The `augmentation` method will be applied later by our training dataset (`tf.data.Dataset` object).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"transformer = ktu.TFImageTransform(\n    image_size,\n    horizontal_flip=True,\n    vertical_flip=True,\n    brightness_adjustments=True,\n    rotation_range=15.,\n    shear_range=5.,\n    zoom_range=(0.1, 0.1),\n    shift_range=(16., 16.)\n)\n\ndef augmentation(image, label):\n    return transformer.transform(image), label","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Below are examples of our data augmentation algorithm. There are several data augmentation methods being used here. These methods can be divided into \"Fast\" and \"Slow\" categories:\n\n1. Fast Augmentation Methods:\n   - Random horizontal/vertical flip\n   - Random brightness correction (up to certain gamma)\n2. Slow Augmentation Methods:\n   - Random Rotation (to the left or to the right, up to a certain degree)\n   - Random shear (to the left or to the right, up to a certain degree)\n   - Random zooming (in or out, horizontal or vertical, up to a certain value)\n   - Random shifting (up to a certain value)","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"rows, cols = 3, 6\nplt.figure(figsize=(2.9 * cols, rows * 2.9))\n\nfor row, element in zip(range(rows), train_dataset):\n    one_element = tf.data.Dataset.from_tensors(element).repeat()\n    for col, (image, _) in zip(range(cols), one_element.map(augmentation).as_numpy_iterator()):\n        plt.subplot(rows, cols, row * cols + col + 1)\n        plt.axis('off')\n        plt.imshow(image)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prepare, Build Model, and Train","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Here we define how the data will be feeded to our model during training:\n\n* For training dataset, we will apply random augmentations on each image, shuffle it, group into batches, and pre-fetch the data to our TPU (so that we don't have to wait for the data to be loaded each time). [`tf.data.experimental.AUTOTUNE`](https://www.tensorflow.org/guide/data_performance) will try to apply the most optimal fetching strategies, depending on our hardware configuration.\n* For validation and testing datasets, we don't augment or shuffle (obviously), but employ the same prefetch method. Also, since our validation set is small, we will try to cache it.\n\nAlso, we define some [Keras Callbacks](https://keras.io/api/callbacks/) to improve our training:\n\n* `early_stopping` - performs early stopping, i.e. halts the training if the validation loss has not improved in several epochs \n* `save_checkpoints` - just save intermediate models, so that we don't lose good models\n* `lr_callback` - we will schedule our learning rate in such a way, that is suitable to fine-tuning from a pre-trained network","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Training Parameters\nbatch_size = 16 * strategy.num_replicas_in_sync\nepochs = 25\n\n# Prepare dataflow configuration\nbatched_train_dataset = train_dataset.map(augmentation)\nbatched_train_dataset = batched_train_dataset.repeat()\nbatched_train_dataset = batched_train_dataset.shuffle(buffer_size=2048)\nbatched_train_dataset = batched_train_dataset.batch(batch_size)\nbatched_train_dataset = batched_train_dataset.prefetch(tf.data.experimental.AUTOTUNE)\n\n# Prepare validation configuration\nbatched_val_dataset = val_dataset.batch(batch_size)\nbatched_val_dataset = batched_val_dataset.cache()\nbatched_val_dataset = batched_val_dataset.prefetch(tf.data.experimental.AUTOTUNE)\n\n# Prepare testing configuration\nbatched_test_dataset = test_dataset.batch(batch_size)\nbatched_test_dataset = batched_test_dataset.prefetch(tf.data.experimental.AUTOTUNE)\n\n# Early Stopping callback - in case if we hit a plateau\nearly_stopping = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=3)\n\n# Saving checkpoins every 4 epochs\nsave_checkpoints = tf.keras.callbacks.ModelCheckpoint(\n    '/kaggle/working/weights.{epoch:02d}-{val_loss:.2f}.hdf5',\n    save_freq='epoch', period=4)\n\n# Learning Rate Scheduler for fine-tuning jobs (first increase lr, then decrease)\nlr_callback = ktu.LRSchedulers.FineTuningLR(\n    lr_start=1e-5, lr_max=5e-5 * strategy.num_replicas_in_sync, lr_min=1e-5,\n    lr_rampup_epochs=5, lr_sustain_epochs=0, lr_exp_decay=0.8, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The `ktu.LRSchedulers.FineTuningLR` starts from a small learning rate (to preserve delicate pre-trained weights on Imagenet). Then, it gradually increases the learning rate and drops to minimum again at the end. The graph of the learning rate against batch number is shown below (you can see that lr changes only in the end of each epoch):","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(8, 5))\nlr_callback.visualize(steps_per_epoch=num_training_samples // batch_size, epochs=epochs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we build our model, based on some pre-trained ImageNet feature extractor, and fine tune it for the new dataset. In this case, I chose [EfficientNetB7](https://github.com/qubvel/efficientnet), since it is shown by other competitors that this networks performs better on this competition than ResNets, DenseNets, VGGs, and other networks. Please notice that I am using `weights='noisy-student'` here - it obtained in [this paper earlier this year](https://arxiv.org/pdf/1911.04252.pdf) which gives better results on ImageNet than the original one.\n\nTraining is quite straightforward. For more information please refer to the [Keras Models API documentation](https://keras.io/api/models/).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_model():\n    with strategy.scope():\n        feature_extractor= efficientnet.EfficientNetB7(\n            weights='noisy-student', include_top=False, input_shape=[*image_size, 3])\n        feature_extractor.trainable = True\n        \n        model = tf.keras.Sequential([\n            feature_extractor,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            tf.keras.layers.Dense(len(flower_categories), activation='softmax', dtype='float32')\n        ])\n        \n    model.compile(\n        optimizer='adam',\n        loss = 'sparse_categorical_crossentropy',\n        metrics=['sparse_categorical_accuracy'])\n\n    return model\n\n\n# Build model\nmodel = get_model()\nmodel.summary()\n\n# Train model\nhistory = model.fit(\n    batched_train_dataset,\n    steps_per_epoch=num_training_samples // batch_size,\n    epochs=epochs,\n    callbacks=[lr_callback, early_stopping, save_checkpoints],\n    validation_data=batched_val_dataset,\n    verbose=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Generating submission","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Generate submission file\nprint('Generating Submission file...', end=' ')\ntest_images_dataset = batched_test_dataset.map(lambda image, idnum: image)\npredictions = np.argmax(model.predict(test_images_dataset), axis=-1)\ntest_ids_dataset = batched_test_dataset.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_dataset.batch(num_testing_samples))).numpy().astype('U') # all in one batch\nnp.savetxt('submission.csv', np.rec.fromarrays([test_ids, predictions]), fmt=['%s', '%d'], delimiter=',', header='id,label', comments='')\nprint('Done!')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}