{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Intro\nWelcome to the [Petals to the Metal - Flower Classification on TPU](https://www.kaggle.com/c/tpu-getting-started).\n![](https://storage.googleapis.com/kaggle-competitions/kaggle/21154/logos/header.png)\n\nThe goal of this notebook is to give a short tutorial for the usage of TFRecords. We don't focus on optimization of the prediction model.\n\nFor a more general tutorial we recommend [this notebook](https://www.kaggle.com/drcapa/tutorial-tfrecords-create-and-read).\n\n<span style=\"color: royalblue;\">Please vote the notebook up if it helps you. Thank you. </span>"},{"metadata":{},"cell_type":"markdown","source":"# Motivation\nTFRecord files (.tfrec) are based on a binary format for storing sequences of values. The TFRecord format was developed by TensorFlow. The motivation of the development is to use Tensor Processing Units (TPUs) to accelerate the applications of machine learning applications.\n\nTo use the advantages of TPU you have to switch on your notebook:\n1. Klick on the notebook seetings (right upper corner of the notebook).\n2. Klick on \"Accelerator\".\n3. Choose TPU v3-8.\n![](https://i.ibb.co/mHFPHpN/setting.png)"},{"metadata":{},"cell_type":"markdown","source":"# Libraries"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport matplotlib.pyplot as plt\nimport re\nimport json\nfrom collections import Counter\n\nfrom sklearn.model_selection import train_test_split\n\nimport tensorflow as tf\nfrom functools import partial\nfrom kaggle_datasets import KaggleDatasets\nprint(\"Tensorflow version \" + tf.__version__)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Set Up"},{"metadata":{"trusted":true},"cell_type":"code","source":"try:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print(\"Device:\", tpu.master())\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept:\n    strategy = tf.distribute.get_strategy()\nprint(\"Number of replicas:\", strategy.num_replicas_in_sync)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Path"},{"metadata":{"trusted":true},"cell_type":"code","source":"path = '/kaggle/input/tpu-getting-started/'\nos.listdir(path)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are 4 folders with each 3 subfolders train, val and test:"},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir(path+'tfrecords-jpeg-192x192/')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Create the GCS path:"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"path_gcs = KaggleDatasets().get_gcs_path('tpu-getting-started')\nprint(path_gcs) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Parameter"},{"metadata":{"trusted":true},"cell_type":"code","source":"AUTOTUNE = tf.data.experimental.AUTOTUNE\nBATCH_SIZE = 16*strategy.num_replicas_in_sync\nPIXEL = 512\nIMAGE_SIZE = [PIXEL, PIXEL]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"samp_subm = pd.read_csv(path+'sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"samp_subm.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_filenames = tf.io.gfile.glob(path_gcs+'/tfrecords-jpeg-'+str(PIXEL)+'x'+str(PIXEL)+'/train/*.tfrec')\nval_filenames = tf.io.gfile.glob(path_gcs+'/tfrecords-jpeg-'+str(PIXEL)+'x'+str(PIXEL)+'/val/*.tfrec')\ntest_filenames = tf.io.gfile.glob(path_gcs+'/tfrecords-jpeg-'+str(PIXEL)+'x'+str(PIXEL)+'/test/*.tfrec')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Functions"},{"metadata":{"trusted":true},"cell_type":"code","source":"def number_of_files(filenames):\n    \"\"\" Evaluate the number on files \"\"\"\n    \n    num = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(num)\n\n\ndef decode_image(image):\n    image = tf.image.decode_jpeg(image, channels=3)\n    image = tf.image.resize(image, [*IMAGE_SIZE])\n    image = tf.cast(image, tf.float32)/255.\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image\n\n\ndef read_tfrecord(example, labeled):\n    tfrecord_format = {\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"class\": tf.io.FixedLenFeature([], tf.int64)\n    } if labeled else {\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"id\": tf.io.FixedLenFeature([], tf.string)\n    }\n    example = tf.io.parse_single_example(example, tfrecord_format)\n    image = decode_image(example['image'])\n    if labeled:\n        label = tf.cast(example['class'], tf.int32)\n        return image, label\n    idnum = example['id']\n    return image, idnum\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False  # disable order, increase speed\n    dataset = tf.data.TFRecordDataset(\n        filenames\n    )  # automatically interleaves reads from multiple files\n    dataset = dataset.with_options(\n        ignore_order\n    )  # uses data as soon as it streams in, rather than in its original order\n    dataset = dataset.map(\n        partial(read_tfrecord, labeled=labeled), num_parallel_calls=AUTOTUNE\n    )\n    # returns a dataset of (image, label) pairs if labeled=True or just images if labeled=False\n    return dataset\n\n\ndef get_train_dataset(filenames, labeled=True, ordered=False):\n    dataset = load_dataset(filenames, labeled=labeled, ordered=ordered)\n    dataset = dataset.repeat()\n    dataset = dataset.shuffle(2020)\n    dataset = dataset.prefetch(buffer_size=AUTOTUNE)\n    dataset = dataset.batch(BATCH_SIZE)\n    return dataset\n\ndef get_val_dataset(filenames, labeled=True, ordered=False):\n    dataset = load_dataset(filenames, labeled=labeled, ordered=ordered)\n    dataset = dataset.prefetch(buffer_size=AUTOTUNE)\n    dataset = dataset.batch(BATCH_SIZE)\n    return dataset\n\ndef get_test_dataset(filenames, labeled=False, ordered=True):\n    dataset = load_dataset(filenames, labeled=labeled, ordered=ordered)\n    dataset = dataset.prefetch(buffer_size=AUTOTUNE)\n    dataset = dataset.batch(BATCH_SIZE)\n    return dataset\n\ndef show_batch(image_batch, label_batch):\n    \"\"\" Plot 25 images of a batch \"\"\"\n    \n    plt.figure(figsize=(20, 20))\n    for n in range(25):\n        ax = plt.subplot(5, 5, n + 1)\n        plt.imshow(image_batch[n])\n        plt.title(str(label_batch[n].numpy()))\n        plt.axis(\"off\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Overview"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Number samples of submission file:', len(samp_subm))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Number of train tfrec files:', len(train_filenames))\nprint('Number of val tfrec files:', len(val_filenames))\nprint('Number of test tfrec files:', len(test_filenames))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Number Files train:', number_of_files(train_filenames))\nprint('Number Files train:', number_of_files(val_filenames))\nprint('Number Files test:', number_of_files(test_filenames))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"First we have to extract the features keys. To see the feature keys we have to execute the following code.\n\nThere are the 3 feature keys class, id and image for this dataset:\n![](https://i.ibb.co/C6gDn4h/features-flowers.png)"},{"metadata":{"trusted":true},"cell_type":"code","source":"raw_dataset = tf.data.TFRecordDataset(train_filenames)\nfor raw_record in raw_dataset.take(1):\n    example = tf.train.Example()\n    example.ParseFromString(raw_record.numpy())\n    #print(example.features)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Get Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset = get_train_dataset(train_filenames)\nval_dataset = get_val_dataset(val_filenames)\ntest_dataset = get_test_dataset(test_filenames)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train_dataset)\nprint(val_dataset)\nprint(test_dataset)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Show Data"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"image_batch, label_batch = next(iter(train_dataset))\nshow_batch(image_batch, label_batch)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for image, idnum in test_dataset.take(1):\n    print(image.numpy().shape, idnum.numpy().shape)\n    print(idnum.numpy().astype('U')[0:10])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model\n[Keras Applications](https://www.tensorflow.org/api_docs/python/tf/keras/applications)"},{"metadata":{"trusted":true},"cell_type":"code","source":"learning_rate = 1e-4","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We use the sparse_categorical_accuracy metric. So we have not to encode the 104 target labels."},{"metadata":{"trusted":true},"cell_type":"code","source":"def make_model():\n    base_model = tf.keras.applications.ResNet50(weights = 'imagenet', \n                                                include_top = False,\n                                                input_shape = [*IMAGE_SIZE, 3])\n    base_model.trainable = True\n    model = tf.keras.Sequential([\n            base_model,\n            tf.keras.layers.GlobalAveragePooling2D(),\n            tf.keras.layers.Dense(104, activation='softmax',dtype='float32')])\n    model.compile(\n        optimizer=tf.keras.optimizers.Adam(learning_rate=learning_rate),\n                                           loss=\"sparse_categorical_crossentropy\",\n                                           metrics=['sparse_categorical_accuracy']\n    )\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    model = make_model()\n\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit(train_dataset,\n                    epochs=10,\n                    validation_data = val_dataset,\n                    steps_per_epoch = number_of_files(train_filenames)//BATCH_SIZE)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Analyse Training"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axs = plt.subplots(1, 2, figsize=(20, 6))\nfig.subplots_adjust(hspace = .2, wspace=.2)\naxs = axs.ravel()\nloss = history.history['loss']\nloss_val = history.history['val_loss']\nepochs = range(1, len(loss)+1)\naxs[0].plot(epochs, loss, 'bo', label='loss_train')\naxs[0].plot(epochs, loss_val, 'ro', label='loss_val')\naxs[0].set_title('Value of the loss function')\naxs[0].set_xlabel('epochs')\naxs[0].set_ylabel('value of the loss function')\naxs[0].legend()\naxs[0].grid()\nacc = history.history['sparse_categorical_accuracy']\nacc_val = history.history['val_sparse_categorical_accuracy']\naxs[1].plot(epochs, acc, 'bo', label='accuracy_train')\naxs[1].plot(epochs, acc_val, 'ro', label='accuracy_val')\naxs[1].set_title('Accuracy')\naxs[1].set_xlabel('Epochs')\naxs[1].set_ylabel('Value of accuracy')\naxs[1].legend()\naxs[1].grid()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Predict Test Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"def to_float32(image, idnum):\n    return tf.cast(image, tf.float32), idnum\n\ntest_dataset = test_dataset.map(to_float32)\ntest_images = test_dataset.map(lambda image, idnum: image)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred_propa = model.predict(test_images, verbose=1)\npreds = np.argmax(pred_propa, axis=-1)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"output = pd.DataFrame(columns=['id', 'label'])\ncounter = 0\nlenght = BATCH_SIZE\nfor image, idnum in test_dataset:\n    ids = idnum.numpy().astype('U')\n    labels = preds[counter*lenght:counter*lenght+len(ids)]\n    temp = pd.DataFrame({'id': ids, 'label': preds[0:len(ids)]})\n    output = pd.concat([output, temp])\n    lenght = len(ids)\n    counter += 1\n    \noutput.index=output['id']\noutput = output.loc[samp_subm['id']]\noutput.index=range(len(output))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Analyse Predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(1, 1, figsize=(16, 4))\nx = output['label'].value_counts().keys()\ny = output['label'].value_counts().values\nax.bar(x, y)\nplt.title('Distribution of prediction labels')\nplt.grid()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Write Output"},{"metadata":{"trusted":true},"cell_type":"code","source":"output.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}