{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# Creating TFRecords for [House of Blocks Competition](http://https://www.kaggle.com/c/applications-of-deep-learning-wustl-fall-2020/overview)\n\nThis notebook:\n- collects the images and meta-data from the competition dataset.\n- performs some basic image manipulation (creating two possible image sizes 192x192 and 331x331)\n- writes both the train and test data to TFRecord files for efficient use on Kaggle TPUs.\n\nThe resulting TFRecords can be found in the public dataset at https://www.kaggle.com/jesseallardice/tfrecords-for-adl-wustl-fall-2020\n\nMost of the code is adapted from:\n- building the TFRecords https://www.kaggle.com/cdeotte/how-to-create-tfrecords"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import math, re, os\nimport cv2\nimport tensorflow as tf\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom kaggle_datasets import KaggleDatasets\nfrom sklearn.metrics import f1_score, precision_score, recall_score, confusion_matrix\nfrom google.cloud import storage\nprint(f\"Tensor Flow version: {tf.__version__}\")\nAUTO = tf.data.experimental.AUTOTUNE","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Extract Meta-Files"},{"metadata":{"trusted":true},"cell_type":"code","source":"PATH = \"/kaggle/input/applications-of-deep-learning-wustl-fall-2020/final-kaggle-data/\"\n\nPATH_TRAIN = os.path.join(PATH, \"train.csv\")\nPATH_TEST = os.path.join(PATH, \"test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# load the meta data\ndf_train = pd.read_csv(PATH_TRAIN)\ndf_test = pd.read_csv(PATH_TEST)\n\n# remove image 1300\nremoval_mask = df_train[\"id\"] != 1300 # corrupt image\ndf_train = df_train[removal_mask]\n\ndf_train[\"filename\"] = df_train[\"id\"].astype(str) + \".png\"\ndf_train[\"stable\"] = df_train[\"stable\"].astype(int)\n\ndf_test[\"filename\"] = df_test[\"id\"].astype(str) + \".png\"\n\ndf_train.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"IMGS = df_train[\"filename\"].to_list()\nlen(IMGS)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Some helper functions for serialising the dataset\n\nFunctions to serialise each type of feature (string/image, float and int)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def _bytes_feature(value):\n    \"\"\"Returns a bytes_list from a string / byte.\"\"\"\n    if isinstance(value, type(tf.constant(0))):\n        value = value.numpy() # BytesList won't unpack a string from an EagerTensor.\n    return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))\n\ndef _float_feature(value):\n    \"\"\"Returns a float_list from a float / double.\"\"\"\n    return tf.train.Feature(float_list=tf.train.FloatList(value=[value]))\n\ndef _int64_feature(value):\n    \"\"\"Returns an int64_list from a bool / enum / int / uint.\"\"\"\n    return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Function to serialise a train example"},{"metadata":{"trusted":true},"cell_type":"code","source":"def serialize_train_example(feature0, feature1, feature2, feature3):\n    feature = {\n        'image': _bytes_feature(feature0),\n        'id': _int64_feature(feature1),\n        'filename': _bytes_feature(feature2),\n        'stable': _int64_feature(feature3)\n    }\n    example_proto = tf.train.Example(features=tf.train.Features(feature=feature))\n    return example_proto.SerializeToString()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"List of desired output sizes.\nThe output size must be square for use with the TPUs."},{"metadata":{"trusted":true},"cell_type":"code","source":"dims = [ # output image sizes\n    (192, 192),\n    (331, 331),\n    ]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prepear the TFRecords files"},{"metadata":{"trusted":true},"cell_type":"code","source":"os.mkdir('./Train')\n\nfor dim in dims:\n    SIZE = 2071\n    OUT_PATH = './Train/%ix%i'%(dim[0],dim[1])\n    os.mkdir(OUT_PATH)\n    CT = len(IMGS) // SIZE + int(len(IMGS) % SIZE != 0)\n    for j in range(CT):\n        print()\n        print('Writing TFRecord %i of %i...'%(j,CT))\n        CT2 = min(SIZE, len(IMGS) - j * SIZE)\n        with tf.io.TFRecordWriter(OUT_PATH + '/train%.2i-%i.tfrec'%(j,CT2)) as writer:\n            for k in range(CT2):\n                img = cv2.imread(PATH + IMGS[SIZE * j + k])\n                img = cv2.cvtColor(img, cv2.COLOR_RGB2BGR) # fix incorrect colors\n                # reshape image to square\n                img = cv2.resize(img, dim, interpolation=cv2.INTER_AREA)           \n                img = cv2.imencode('.jpg', img, (cv2.IMWRITE_JPEG_QUALITY, 94))[1].tostring() # maybe change to '.png'?\n                filename = IMGS[SIZE * j + k] # .split('.')[0]\n                row = df_train[df_train.filename == filename]\n                example = serialize_train_example(\n                    img,\n                    row.id.values[0],\n                    str.encode(filename),\n                    row.stable.values[0]\n                )\n                writer.write(example)\n                if k % 100 == 0:\n                    print(k, ',  ',end='')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Verify TFRecords\nBy displaying the TFRecords"},{"metadata":{"trusted":true},"cell_type":"code","source":"# numpy and matplotlib defaults\nnp.set_printoptions(threshold=15, linewidth=80)\nCLASSES = [0,1]\n\ndef batch_to_numpy_images_and_labels(data):\n    images, labels = data\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    #if numpy_labels.dtype == object: # binary string in this case, these are image ID strings\n    #    numpy_labels = [None for _ in enumerate(numpy_images)]\n    # If no labels, only image IDs, return None for labels (this is the case for test data)\n    return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n    if correct_label is None:\n        return CLASSES[label], True\n    correct = (label == correct_label)\n    return \"{} [{}{}{}]\".format(\n        CLASSES[label], \n        'OK' if correct else 'NO', \n        u\"\\u2192\" if not correct else '',\n        CLASSES[correct_label] if not correct else ''\n    ), correct\n\ndef display_one_image(image, title, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    if len(title) > 0:\n        plt.title(\n            title, \n            fontsize=int(titlesize) if not red else int(titlesize/1.2), \n            color='red' if red else 'black', \n            fontdict={'verticalalignment':'center'}, \n            pad=int(titlesize/1.5)\n        )\n    return (subplot[0], subplot[1], subplot[2]+1)\n\ndef display_batch_of_images(databatch, predictions=None):\n    \"\"\"This will work with:\n    display_batch_of_images(images)\n    display_batch_of_images(images, predictions)\n    display_batch_of_images((images, labels))\n    display_batch_of_images((images, labels), predictions)\n    \"\"\"\n    # data\n    images, labels = batch_to_numpy_images_and_labels(databatch)\n    if labels is None:\n        labels = [None for _ in enumerate(images)]\n    # auto-squaring: this will drop data that doesnt fit into a square subplot\n    rows = int(math.sqrt(len(images)))\n    cols = len(images)//rows\n    # size and spacing\n    FIGSIZE = 13.0\n    SPACING = 0.1\n    subplot = (rows,cols,1)\n    if rows < cols:\n        plt.figure(figsize = (FIGSIZE, FIGSIZE / cols * rows))\n    else:\n        plt.figure(figsize = (FIGSIZE / cols * rows, FIGSIZE))\n    #display\n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        title = label\n        correct = True\n        if predictions is not None:\n            title, correct = title_from_label_and_target(predictions[i], label)\n        dynamic_titlesize = FIGSIZE * SPACING / max(rows, cols) * 40 + 3\n        subplot = display_one_image(image, title, subplot, not correct, titlesize=dynamic_titlesize)\n    #layout\n    plt.tight_layout()\n    if label is None and predictions is None:\n        plt.subplots_adjust(wspace=0, hspace=0)\n    else:\n        plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def decode_image(image_data):\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    image = tf.cast(image, tf.float32) / 255.0 # converts image to float in [0,1]\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image\n\ndef read_labeled_tfrecord(example):\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring\n        \"filename\": tf.io.FixedLenFeature([], tf.string), # shape [] means single element\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example[\"image\"])\n    label = example[\"filename\"]\n    return image, label\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    \"\"\"Read from TFRecords. For optimal performance, reading from multiple \n    files at once and disregarding data order. Order does not matter since \n    we will be shuffling the data anyway\"\"\"\n    \n    ignore_order = tf.data.Options()\n    if not ordered:\n        ignore_order.experimental_deterministic = False # disable order\n    \n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO)\n    dataset = dataset.with_options(ignore_order)\n    dataset = dataset.map(read_labeled_tfrecord)\n    return dataset\n\ndef get_training_dataset():\n    dataset = load_dataset(TRAINING_FILENAMES, labeled=True)\n    dataset = dataset.repeat()\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(AUTO)\n    return dataset\n\ndef count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Initialise variables\nHEIGHT, WIDTH = dims[0] # look at the first image size\nIMAGE_SIZE = [HEIGHT, WIDTH]\nBATCH_SIZE = 32\nAUTO = tf.data.experimental.AUTOTUNE\nOUT_PATH = './Train/%ix%i'%(HEIGHT, WIDTH)\nTRAINING_FILENAMES = tf.io.gfile.glob(OUT_PATH + '/train*.tfrec')\nprint('There are %i train images'%count_data_items(TRAINING_FILENAMES))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Display train images\ntraining_dataset = get_training_dataset()\ntraining_dataset = training_dataset.unbatch().batch(20)\ntrain_batch = iter(training_dataset)\ndisplay_batch_of_images(next(train_batch))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# Creat Test TFRecords"},{"metadata":{"trusted":true},"cell_type":"code","source":"TEST_IMGS = df_test[\"filename\"].to_list()\nlen(TEST_IMGS)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def serialize_test_example(feature0, feature1, feature2):\n    feature = {\n        'image': _bytes_feature(feature0),\n        'id': _int64_feature(feature1),\n        'filename': _bytes_feature(feature2),\n    }\n    example_proto = tf.train.Example(features=tf.train.Features(feature=feature))\n    return example_proto.SerializeToString()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"os.mkdir('./Test')\n\nfor dim in dims:\n    SIZE = 2071\n    OUT_PATH = './Test/%ix%i'%(dim[0],dim[1])\n    os.mkdir(OUT_PATH)\n    CT = len(TEST_IMGS) // SIZE + int(len(TEST_IMGS) % SIZE != 0)\n    for j in range(CT):\n        print()\n        print('Writing TFRecord %i of %i...'%(j,CT))\n        CT2 = min(SIZE, len(TEST_IMGS) - j * SIZE)\n        with tf.io.TFRecordWriter(OUT_PATH + '/test%.2i-%i.tfrec'%(j,CT2)) as writer:\n            for k in range(CT2):\n                img = cv2.imread(PATH + TEST_IMGS[SIZE * j + k])\n                img = cv2.cvtColor(img, cv2.COLOR_RGB2BGR) # fix incorrect colors\n                # reshape image to square\n                img = cv2.resize(img, dim, interpolation=cv2.INTER_AREA)           \n                img = cv2.imencode('.jpg', img, (cv2.IMWRITE_JPEG_QUALITY, 94))[1].tostring() # maybe change to '.png'?\n                filename = TEST_IMGS[SIZE * j + k] # .split('.')[0]\n                row = df_test[df_test.filename == filename]\n                example = serialize_test_example(\n                    img,\n                    row.id.values[0],\n                    str.encode(filename)\n                )\n                writer.write(example)\n                if k % 100 == 0:\n                    print(k, ',  ',end='')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# Last steps and Conclusion\n\nFrom here I downloaded the TFRecords and uploaded them to a new Kaggle dataset (a public dataset so that TPU usage would be free). Ideally instead of creating a new dataset it would of be nice to load the TFRecords into a GCS bucket (possibably through a GCP account) co-located with the TPU and run the training all in the one notebook. Saddly, I couldnt actually get the GCS bucketing to work (I tried flowing this https://www.kaggle.com/paultimothymooney/how-to-move-data-from-kaggle-to-gcs-and-back but got stuck with GCP account that wasnt cooperating). I would be really keen to know if anyone has had some luck with this, as it seems like the workflow expected in more professional contexts. However, I happy with the GCP free way implemented here as its a little more accessible."},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}