{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.model_selection import train_test_split\nfrom mpl_toolkits.axes_grid1 import ImageGrid\nimport tensorflow as tf","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-07-13T04:34:30.978095Z","iopub.execute_input":"2022-07-13T04:34:30.979138Z","iopub.status.idle":"2022-07-13T04:34:30.985498Z","shell.execute_reply.started":"2022-07-13T04:34:30.979083Z","shell.execute_reply":"2022-07-13T04:34:30.984230Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. What is a TFRecord\n\nTFRecord is the TensorFlow preferred file format.\n\nSince the new release of TensorFlow (2.0 and after), almost everything drastically changed in TensorFlow. \n\nIt became the trend again after being intimidated by PyTorch! \n\nEspecially its tf.data API can bring cakes for you!! \n\nIt helps to create the TensorFlow input pipeline very efficiently.\n\nWith all those great functionalities, one issue remains: Working with large data due to memory limitations!\n\nTo address that issue and many other setbacks, TensorFlow offers TFRecord format. \n\nTFRecord brings very important advantages:\n\n##### 1. TFRecord keeps serialized data in binary format, which allows efficient reading data. \n\n##### 2. It significantly affects the performance of the model.\n\n##### 3. It is optimized for TensorFlow. Simple! It is not a surprise as it is the TensorFlow recommended format.","metadata":{}},{"cell_type":"markdown","source":"# 2. Purpose of this notebook\nIn this notebook, we will learn:\n\n##### 1. How to write data into the TFRecord file format.\n##### 2. How to read a TFRecord file.","metadata":{}},{"cell_type":"markdown","source":"# 3. Writing Data to a TFRecord File\n\n- First we will read MNIST dataset (even though the data set is not huge to create any kind of memory issues, but we are using this dataset here for educational purpose)\n- Write MNIST data to TFRecords","metadata":{}},{"cell_type":"markdown","source":"#### 3.1 Load MNIST dataset","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(\"../input/digit-recognizer/train.csv\")\ndisplay(train.head())\n\ny = train.label\ntrain.drop('label',1,inplace=True)\n\nx_train, x_test, y_train, y_test = train_test_split(train, y, stratify=y, test_size=0.2, random_state=1)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T04:34:31.021357Z","iopub.execute_input":"2022-07-13T04:34:31.021822Z","iopub.status.idle":"2022-07-13T04:34:34.128414Z","shell.execute_reply.started":"2022-07-13T04:34:31.021785Z","shell.execute_reply":"2022-07-13T04:34:34.126893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 3.2 Data Preprocessing","metadata":{}},{"cell_type":"code","source":"# Preprocessing\nx_train = x_train / 255.0\nx_test = x_test / 255.0\n# Track the data type\ndataType = x_train.pixel0.dtype\nprint(f\"Data type: {dataType}\")\nlabelType = y_test.dtype\nprint(f\"Data type: {labelType}\")","metadata":{"execution":{"iopub.status.busy":"2022-07-13T04:34:34.131206Z","iopub.execute_input":"2022-07-13T04:34:34.131927Z","iopub.status.idle":"2022-07-13T04:34:34.214385Z","shell.execute_reply.started":"2022-07-13T04:34:34.131861Z","shell.execute_reply":"2022-07-13T04:34:34.212696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 3.3 visualize some examples","metadata":{}},{"cell_type":"code","source":"# visualize some examples\nim_list = []\nn_samples_to_show = 16\nc = 0\nfor i in range(n_samples_to_show):\n    img = x_train.iloc[i].values.reshape(28,28)\n    im_list.append(img)\n# Visualization\nfig = plt.figure(figsize=(4., 4.))\n# Ref: https://matplotlib.org/3.1.1/gallery/axes_grid1/simple_axesgrid.html\ngrid = ImageGrid(fig, 111,  # similar to subplot(111)\n                 nrows_ncols=(4, 4),  # creates 2x2 grid of axes\n                 axes_pad=0.1,  # pad between axes in inch.\n                 )\n# Show image grid\nfor ax, im in zip(grid, im_list):\n    # Iterating over the grid returns the Axes.\n    ax.imshow(im, 'gray')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-13T04:34:34.216626Z","iopub.execute_input":"2022-07-13T04:34:34.218270Z","iopub.status.idle":"2022-07-13T04:34:35.806382Z","shell.execute_reply.started":"2022-07-13T04:34:34.218205Z","shell.execute_reply":"2022-07-13T04:34:35.804813Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 3.4 Structure the data with tf.Example\n\nThe `tf.Example` is a Protocol Buffers (Protobuf), which is a method of serializing structured data.\n\nSince in a TFRecord file, we are going to store our data as binary strings (sequences of strings), we need to specify the structure of data before storing it. \n\nOtherwise, how do we know the original shape and characteristics of the data when trying to read and reconstruct it from the TFRecord file?\n\nTwo approaches can be used for this aim provided by TensorFlow:\n\n- tf.train.Example\n- tf.train.SequenceExample\n\nWe will use `tf.train.Example` for our experiments. \n\nFor using tf.train.Example, we should convert our data to compatible feature types. \n\nFor that, the `tf.train.Feature` protocol message should be used. \n\nThe tf.train.Feature supports three types of features: \n- tf.train.BytesList\n- tf.train.FloatList\n- tf.train.Int64List\n\nFor making features compatible, we use the following auxiliary functions.","metadata":{}},{"cell_type":"code","source":"# Convert values to compatible tf.Example types.\ndef _bytes_feature(value):\n    \"\"\"Returns a bytes_list from a string / byte.\"\"\"\n    if isinstance(value, type(tf.constant(0))):\n        value = value.numpy()  # BytesList won't unpack a string from an EagerTensor.\n    return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))\ndef _float_feature(value):\n    \"\"\"Returns a float_list from a float / double.\"\"\"\n    return tf.train.Feature(float_list=tf.train.FloatList(value=[value]))\ndef _int64_feature(value):\n    \"\"\"Returns an int64_list from a bool / enum / int / uint.\"\"\"\n    return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))","metadata":{"execution":{"iopub.status.busy":"2022-07-13T04:34:35.809267Z","iopub.execute_input":"2022-07-13T04:34:35.810050Z","iopub.status.idle":"2022-07-13T04:34:35.819230Z","shell.execute_reply.started":"2022-07-13T04:34:35.810004Z","shell.execute_reply":"2022-07-13T04:34:35.817184Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### Now we can set the feature dictionary and structure the data with tf.train.Example:","metadata":{}},{"cell_type":"code","source":"# Create the features dictionary.\ndef image_example(feature1, feature2, feature3):\n    feature = {\n        'image_raw': _bytes_feature(feature1.tobytes()),\n        'dimension': _int64_feature(feature2),\n        'label': _int64_feature(feature3),\n    }\n    return tf.train.Example(features=tf.train.Features(feature=feature))","metadata":{"execution":{"iopub.status.busy":"2022-07-13T04:34:35.821036Z","iopub.execute_input":"2022-07-13T04:34:35.822130Z","iopub.status.idle":"2022-07-13T04:34:35.832134Z","shell.execute_reply.started":"2022-07-13T04:34:35.822081Z","shell.execute_reply":"2022-07-13T04:34:35.830975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"###### There are two extremely important points in the above code:\n\n- We stored the dimension of the image in the structure since we need this information when reading the image so that we can reconstruct images in their original shape.\n\n- For image, we first convert it to bytes with `.tostring()` or `.tobytes()` function (both functions does the same and .tostring() is in fact an alias for .tobytes). Then, we feed it to _bytes_feature function. Remember that even Python strings should be converted to bytes before being fed to the `tf.train.BytesList`.\n\n- Remember, in the end, we used `tf.train.Features` as it is slightly different from tf.train.Feature (the former has one extra s letter in the end if you did not notice!!!). tf.train.Features it’s like the wrapper of named features and takes a dictionary (called feature) as its feature argument. The dictionary keys are the feature name with values of type `tf.train.Feature`.","metadata":{}},{"cell_type":"markdown","source":"#### 3.5. Write Records to TFRecord File\nNow, we set the structures and provided the functionalities for the final step of writing the data into the TFRecord file. \n\nNow we serialize and store the structured samples in the file as follows:","metadata":{}},{"cell_type":"code","source":"record_file = 'mnistTrain.tfrecords'\nn_samples = x_train.shape[0]\ndimension = x_train.shape[1]\nwith tf.io.TFRecordWriter(record_file) as writer:\n    for i in range(n_samples):\n        image = x_train.iloc[i].values\n        label = y_train.iloc[i]\n        tf_example = image_example(image, label, dimension)\n        writer.write(tf_example.SerializeToString())","metadata":{"execution":{"iopub.status.busy":"2022-07-13T04:34:35.833884Z","iopub.execute_input":"2022-07-13T04:34:35.834630Z","iopub.status.idle":"2022-07-13T04:34:44.969057Z","shell.execute_reply.started":"2022-07-13T04:34:35.834591Z","shell.execute_reply":"2022-07-13T04:34:44.967891Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The `tf.io.TFRecordWriter` function is used to write a string to the specified file. \n\nThe serialization is done with the `SerializeToString()` function. \n\nIn fact, we can serialize any proto message to a binary-string by operating the `SerializeToString()` method.","metadata":{}},{"cell_type":"markdown","source":"# 4. Reading from TFRecord\n\nNow it’s the time to read from a TFRecord file given the knowledge of the serialized structured samples.\n\n#### 4.1 Creating the Dataset\nWe use the TensorFlow `tf.data` API. \n\nThe Tensorflow tf.data API makes the process of creating the input pipeline very handy! \n\nFirst, we need to read the TFRecord file and create a dataset by `tf.data` function:","metadata":{}},{"cell_type":"code","source":"# Create the dataset object from tfrecord file(s)\ndataset = tf.data.TFRecordDataset(record_file, buffer_size=100)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T04:34:44.971108Z","iopub.execute_input":"2022-07-13T04:34:44.971692Z","iopub.status.idle":"2022-07-13T04:34:44.992874Z","shell.execute_reply.started":"2022-07-13T04:34:44.971637Z","shell.execute_reply":"2022-07-13T04:34:44.991322Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"With the `tf.data.TFRecordDataset` function, we can read multiple TFRecord files as well. \n\nThe argument `buffer_size` is very useful when we have memory restrictions. \n\nIt represents the number of bytes in the read buffer. \n\nIt is not mandatory though. \n\nEven if you do not set that, TensorFlow pick a reasonable number itself!","metadata":{}},{"cell_type":"markdown","source":"#### 4.2 Retrieving Records\nNow that we have the dataset, we can loop through the dataset to extract the records.\n\nBut we would also need functions to parse the records and decode the records","metadata":{}},{"cell_type":"code","source":"# parse the record\ndef parse_record(record):\n    name_to_features = {\n        'image_raw': tf.io.FixedLenFeature([], tf.string),\n        'dimension': tf.io.FixedLenFeature([], tf.int64),\n        'label': tf.io.FixedLenFeature([], tf.int64)\n    }\n    return tf.io.parse_single_example(record, name_to_features)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T04:34:44.995156Z","iopub.execute_input":"2022-07-13T04:34:44.995585Z","iopub.status.idle":"2022-07-13T04:34:45.002115Z","shell.execute_reply.started":"2022-07-13T04:34:44.995546Z","shell.execute_reply":"2022-07-13T04:34:45.001043Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Above function operates on each record.\n\nReturn the different features stored in the record structure based on the labels of the features. \n\nAbove we have `dimension`, `label`, `image_raw` features.\n\nAs we have `fix length features`, we use `tf.io.FixedLenFeature` function. \n\nIf you have `variable length features`, you can use `tf.io.FixedLenSequenceFeature` function. The first argument is the shape of the input data and we set it as `[]`.\n\nFinally, the `tf.io.parse_single_example` function is used to parse the serialized record given the predetermined structure.","metadata":{}},{"cell_type":"code","source":"# decode the record\ndef decode_record(record):\n    image = tf.io.decode_raw(\n        record['image_raw'], out_type=dataType, little_endian=True, fixed_length=None, name=None\n    )\n    label = record['label']\n    dimension = record['dimension']\n    image = tf.reshape(image, (28,28))\n    return (image, label)","metadata":{"execution":{"iopub.status.busy":"2022-07-13T04:34:45.003262Z","iopub.execute_input":"2022-07-13T04:34:45.004378Z","iopub.status.idle":"2022-07-13T04:34:45.016972Z","shell.execute_reply.started":"2022-07-13T04:34:45.004324Z","shell.execute_reply":"2022-07-13T04:34:45.015789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the above function we used the `tf.io.decode_raw` to revert back the image from string to its original form. \n\nConsider:\nWe used the `dataType` for the `out_type` argument in the `tf.io.decode_raw` function. \n\nIt is very important. If you do NOT use the same data type, you reconstruct something inconsistent!\n\nWe used `tf.reshape(image, (dimension, dimension))` as the tf.io.decode_raw have no idea about the shape! \n\nThe dimension information should have been saved when we were writing to TFRecords.","metadata":{}},{"cell_type":"markdown","source":"##### Now we are good to iterate over the dataset and reconstruct the images in their original form ","metadata":{}},{"cell_type":"code","source":"# now iterate throgh each record in the dataset\nfor record in dataset:\n    parsed_record = parse_record(record)\n    decoded_record = decode_record(parsed_record)\n    image, label = decoded_record\n    print(image.shape, label.shape)\n    break","metadata":{"execution":{"iopub.status.busy":"2022-07-13T04:34:45.020889Z","iopub.execute_input":"2022-07-13T04:34:45.022155Z","iopub.status.idle":"2022-07-13T04:34:45.056394Z","shell.execute_reply.started":"2022-07-13T04:34:45.022097Z","shell.execute_reply":"2022-07-13T04:34:45.055041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 4.3 Sanity Check - To ensure we have correctly reconstructed the images, we will visualize few of them","metadata":{}},{"cell_type":"code","source":"# visualize the reconstruced images\nim_list = []\nn_samples_to_show = 16\nc = 0\nfor record in dataset:\n    c+=1\n    if c > n_samples_to_show:\n        break\n    parsed_record = parse_record(record)\n    decoded_record = decode_record(parsed_record)\n    image, label = decoded_record\n    im_list.append(image)\n# Visualization\nfig = plt.figure(figsize=(4., 4.))\n# Ref: https://matplotlib.org/3.1.1/gallery/axes_grid1/simple_axesgrid.html\ngrid = ImageGrid(fig, 111,  # similar to subplot(111)\n                 nrows_ncols=(4, 4),  # creates 2x2 grid of axes\n                 axes_pad=0.1,  # pad between axes in inch.\n                 )\n# Show image grid\nfor ax, im in zip(grid, im_list):\n    # Iterating over the grid returns the Axes.\n    ax.imshow(im, 'gray')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-13T04:34:45.058652Z","iopub.execute_input":"2022-07-13T04:34:45.059055Z","iopub.status.idle":"2022-07-13T04:34:47.340440Z","shell.execute_reply.started":"2022-07-13T04:34:45.059020Z","shell.execute_reply":"2022-07-13T04:34:47.338986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### We have successfully reproduced the images","metadata":{}},{"cell_type":"markdown","source":"# 5. Conclusion\nIn this notebook, we learnt what TFRecord file format is, what its advantages are, and how to work with it in TensorFlow. \n\nI provided a simple example of the MNIST data. \n\nThe same approach can be applied to the majority of the scenarios. \n\nIt’s usually better to convert your data to TFRecords first and then work with them. \n\nOnce you learn how to do it, I doubt it that you go back and work with other types of data formats!! Unless your data is too small!! \n\nDefinitely, the story does not finish here. \n\nFeel free to explore more and comment below if you think I missed anything, disagree with me, have any questions, or any other reason that I forgot to this about it! \n\n##### Thank you for your attention and reading so far.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}