{"cells":[{"metadata":{},"cell_type":"markdown","source":"# ***Disclaimer:*** \nHello Kagglers! I am a Solution Architect with the Google Cloud Platform. I am a coach for this competition, the focus of my contributions is on helping users to leverage GCP components (GCS, TPUs, BigQueryetc..) in order to solve large problems. My ideas and contributions represent my own opinion, and are not representative of an official recommendation by Google. Also, I try to develop notebooks quickly in order to help users early in competitions. There may be better ways to solving particular problems, I welcome comments and suggestions. Use my contributions at your own risk, I don't garantee that they will help on winning any competition, but I am hoping to learn by collaborating with everyone."},{"metadata":{},"cell_type":"markdown","source":"# Objective:\nThe objective of this notebook is to demonstrate how to build a TFRecord Dataset designed to be used with a TPU Accelerator. In previous notebooks (see list below) I have build a TFRecord dataset using 1 512x512 tile per file. This resulted in thousands of files which severely hurt the TPU performance. It turns out that the recommended TFRecord file size for feeding TPUs is about 100M. So, I have re-packated the previously built TFRecord Dataset packing 256 512x512 tiles per record, which amounts to about 170M per TFRecord file. In this notebook I group tiles by type in 3 datasets:\n1) Tiles with Gloms\n2) Tiles with NoGloms\n3) Tiles with both Gloms and NoGloms\n\nThe datasets are stored in the following public dataset (hubmap-large-records):\n[https://www.kaggle.com/marcosnovaes/hubmap-large-records](https://www.kaggle.com/marcosnovaes/hubmap-large-records)\n\nThe 3 datasets consist of several file parts. I illustrate how to use that dataset with a TPU enabled Keras model in this Notebook:\n[https://www.kaggle.com/marcosnovaes/hubmap-unet-keras-model-fit-with-tpu/](https://www.kaggle.com/marcosnovaes/hubmap-unet-keras-model-fit-with-tpu/)\n\nPrevious Notebooks in this competition:\n\nhttps://www.kaggle.com/marcosnovaes/hubmap-3-unet-models-with-keras-cpu-gpu/: Investigates three implementations of the Unet model\n\nhttps://www.kaggle.com/marcosnovaes/hubmap-read-data-and-build-tfrecords/: Demonstrates how the TFRecord Dataset was built\n\nhttps://www.kaggle.com/marcosnovaes/hubmap-looking-at-tfrecords/: Explains how to read the data using the TFRecord Dataset\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport sys\nimport random\nimport warnings\n\nimport numpy as np\nimport pandas as pd\n\nimport matplotlib.pyplot as plt\n\nfrom tqdm import tqdm\n\nimport tensorflow as tf","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Utilities serialize data into a TFRecord\ndef _bytes_feature(value):\n  \"\"\"Returns a bytes_list from a string / byte.\"\"\"\n  if isinstance(value, type(tf.constant(0))):\n    value = value.numpy() # BytesList won't unpack a string from an EagerTensor.\n  return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))\n\ndef _float_feature(value):\n  \"\"\"Returns a float_list from a float / double.\"\"\"\n  return tf.train.Feature(float_list=tf.train.FloatList(value=[value]))\n\ndef _int64_feature(value):\n  \"\"\"Returns an int64_list from a bool / enum / int / uint.\"\"\"\n  return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def image_example(image, mask):\n    image_shape = image.shape\n    \n    img_bytes = image.tostring()\n\n    mask_bytes = mask.tostring()\n    \n    feature = {\n        'height': _int64_feature(image_shape[0]),\n        'width': _int64_feature(image_shape[1]),\n        'num_channels': _int64_feature(image_shape[2]),\n        'img_bytes': _bytes_feature(img_bytes),\n        'mask' : _bytes_feature(mask_bytes),\n    }\n    return tf.train.Example(features=tf.train.Features(feature=feature))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_tfrecord( image, mask,output_path):\n    opts = tf.io.TFRecordOptions(compression_type=\"GZIP\")\n    with tf.io.TFRecordWriter(output_path, opts) as writer:\n        tf_example = image_example(image, mask)\n        writer.write(tf_example.SerializeToString())\n    writer.close()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create a dictionary for reading. \nimage_feature_description = {\n    'img_index': tf.io.FixedLenFeature([], tf.int64),\n    'height': tf.io.FixedLenFeature([], tf.int64),\n    'width': tf.io.FixedLenFeature([], tf.int64),\n    'num_channels': tf.io.FixedLenFeature([], tf.int64),\n    'img_bytes': tf.io.FixedLenFeature([], tf.string),\n    'mask': tf.io.FixedLenFeature([], tf.string),\n    'tile_id': tf.io.FixedLenFeature([], tf.int64),\n    'tile_col_pos': tf.io.FixedLenFeature([], tf.int64),\n    'tile_row_pos': tf.io.FixedLenFeature([], tf.int64),\n}\n\ndef _parse_image_and_masks_function(example_proto):\n    single_example = tf.io.parse_single_example(example_proto, image_feature_description)\n    img_height = single_example['height']\n    img_width = single_example['width']\n    num_channels = single_example['num_channels']\n    \n    img_bytes =  tf.io.decode_raw(single_example['img_bytes'],out_type='uint8')\n    \n    #dynamic shape\n    #img_array = tf.reshape( img_bytes, (img_height, img_width, num_channels))\n    \n    #fixed shape\n    img_array = tf.reshape( img_bytes, (512, 512, 3))\n    \n    mask_bytes =  tf.io.decode_raw(single_example['mask'],out_type='bool')\n\n    mask = tf.reshape(mask_bytes, (512,512))\n    \n    #normalize images array and cast image and mask to float32\n    #img_array = tf.cast(img_array, tf.float32) / 255.0\n    #mask = tf.cast(mask, tf.float32)\n    return img_array, mask\n\ndef read_dataset(storage_file_path):\n    encoded_image_dataset = tf.data.TFRecordDataset(storage_file_path, compression_type=\"GZIP\")\n    parsed_image_dataset = encoded_image_dataset.map(_parse_image_and_masks_function)\n    return parsed_image_dataset","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"!ls /kaggle/input","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls /kaggle/input/hubmap-tfrecord-512/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_tiles_csv = '/kaggle/input/hubmap-tfrecord-512/train_all_tiles.csv'\ntest_tiles_csv = '/kaggle/input/hubmap-tfrecord-512/test_all_tiles.csv'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# build a dataset of all images tiles from the train set that have gloms in them\n#for csv_file in file_list:\ntrain_tiles_df = pd.read_csv(train_tiles_csv)\ntrain_gloms_df = train_tiles_df.loc[train_tiles_df[\"mask_density\"]  > 0]\ntrain_gloms_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_gloms_df.__len__()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_cropped_df = train_tiles_df.loc[train_tiles_df[\"lowband_density\"]  > 1000]\ntrain_cropped_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_cropped_df.__len__()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"no_gloms_df = train_cropped_df.loc[train_cropped_df[\"mask_density\"]  == 0]\nno_gloms_df.__len__()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Shuffle all the dataframes. This helps for training later."},{"metadata":{"trusted":true},"cell_type":"code","source":"#shuffle the data frames\n#train_gloms_df = train_gloms_df.sample(frac=1)\n#train_cropped_df = train_cropped_df.sample(frac=1)\n#no_gloms_df = no_gloms_df.sample(frac=1)\ntrain_gloms_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# read selected ranges of the datasets into train and validation datasets. Use about 10% for validation\n\ntrain_glom_files = train_gloms_df[0:3000]['local_path']\nvalidation_glom_files = train_gloms_df[3001:]['local_path']\n\ntrain_cropped_files = train_cropped_df[0:13500]['local_path']\nvalidation_cropped_files = train_cropped_df[13500:]['local_path']\n\ntrain_no_glom_files = no_gloms_df[0:11000]['local_path']\nvalidation_no_glom_files = no_gloms_df[11000:]['local_path']\n\ntrain_glom_dataset = read_dataset(train_glom_files)\nvalidation_glom_dataset = read_dataset(validation_glom_files)\n\ntrain_cropped_dataset = read_dataset(train_cropped_files)\nvalidation_cropped_dataset = read_dataset(validation_cropped_files)\n\ntrain_no_glom_dataset = read_dataset(train_no_glom_files)\nvalidation_no_glom_dataset = read_dataset(validation_no_glom_files)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def write_dataset( dataset, records_per_part, prefix):\n    opts = tf.io.TFRecordOptions(compression_type=\"GZIP\")\n    part_num = 0\n    num_records = 0\n    output_path = prefix+'_part{}.tfrecords'.format(part_num)\n    writer = tf.io.TFRecordWriter(output_path, opts)\n    \n    for image, mask in dataset.as_numpy_iterator(): \n            tf_example = image_example(image, mask)\n            writer.write(tf_example.SerializeToString())\n            num_records += 1   \n            if(num_records == records_per_part - 1):\n                # close current file and open new one\n                print(\"wrote part #{}\".format(part_num))\n                writer.close()\n                part_num += 1\n                output_path = prefix+'_part{}.tfrecords'.format(part_num)\n                writer = tf.io.TFRecordWriter(output_path, opts)\n                num_records = 0\n    writer.close()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"writing train_glom_dataset\")\nwrite_dataset( train_glom_dataset, 256, '/kaggle/working/train_gloms')\nprint(\"writing validation_glom_dataset\")\nwrite_dataset( validation_glom_dataset, 256, '/kaggle/working/validation_gloms')\nprint(\"writing train_cropped_dataset\")\nwrite_dataset( train_cropped_dataset, 256, '/kaggle/working/train_cropped')\nprint(\"writing validation_cropped_dataset\")\nwrite_dataset( validation_cropped_dataset, 256, '/kaggle/working/validation_cropped')\nprint(\"writing train_no_glom_dataset\")\nwrite_dataset( train_no_glom_dataset, 256, '/kaggle/working/train_no_gloms')\nprint(\"writing validation_no_glom_dataset\")\nwrite_dataset( validation_no_glom_dataset, 256, '/kaggle/working/validation_no_gloms')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls -l /kaggle/working","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}