{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Acknowledgement!\n\n### Adapted some codes from https://www.tensorflow.org/tutorials/load_data/tfrecord\n","metadata":{"papermill":{"duration":0.009909,"end_time":"2022-05-17T00:53:39.988582","exception":false,"start_time":"2022-05-17T00:53:39.978673","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Get gsbucket PATH!","metadata":{}},{"cell_type":"markdown","source":"### Using TFrecords often requires gsbucket address. You could get the gsbucket address by using kaggle API as below:","metadata":{}},{"cell_type":"code","source":"from kaggle_datasets import KaggleDatasets\nimport tensorflow as tf\nimport os\n\ngcsPATH = KaggleDatasets().get_gcs_path('tpu-getting-started')\ngcsPATH\n","metadata":{"papermill":{"duration":6.618493,"end_time":"2022-05-17T00:53:46.616225","exception":false,"start_time":"2022-05-17T00:53:39.997732","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-17T16:17:50.764507Z","iopub.execute_input":"2022-05-17T16:17:50.76522Z","iopub.status.idle":"2022-05-17T16:17:51.364992Z","shell.execute_reply.started":"2022-05-17T16:17:50.765181Z","shell.execute_reply":"2022-05-17T16:17:51.364196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# Get the list of tfrecs from PATH! tf.io.gfile.glob ","metadata":{"papermill":{"duration":0.008819,"end_time":"2022-05-17T00:53:46.634596","exception":false,"start_time":"2022-05-17T00:53:46.625777","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### Next, we will get list of tfrecord to use [TFRecordDataset in tf.data API](http://https://www.tensorflow.org/api_docs/python/tf/data/TFRecordDataset). Use [tf.io.gfile.glob](http://https://www.tensorflow.org/api_docs/python/tf/io/gfile/glob) to get files by querying with some general patterns. Accepted patterns are: \n\npattern: { term }\n\nterm:\n\n    '*': matches any sequence of non-'/' characters\n    '?': matches a single non-'/' character\n    '[' [ '^' ] { match-list } ']': matches any single character (not) on the list\n    c: matches character c where c != '*', '?', '\\\\', '['\n    '\\\\' c: matches character c\n\ncharacter range:\n\n    c: matches character c while c != '\\\\', '-', ']'\n    '\\\\' c: matches character c\n    lo '-' hi: matches character c for lo <= c <= hi\n\n","metadata":{}},{"cell_type":"code","source":"config = dict()\nconfig['tfrec_shape'] = 512\nls_tfrecs = tf.io.gfile.glob(gcsPATH+\"/*{}*/train/*tfrec\".format(config['tfrec_shape'])) \nls_tfrecs\n","metadata":{"papermill":{"duration":0.0301,"end_time":"2022-05-17T00:53:46.674504","exception":false,"start_time":"2022-05-17T00:53:46.644404","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-17T16:17:53.984225Z","iopub.execute_input":"2022-05-17T16:17:53.984509Z","iopub.status.idle":"2022-05-17T16:17:54.2583Z","shell.execute_reply.started":"2022-05-17T16:17:53.984479Z","shell.execute_reply":"2022-05-17T16:17:54.257367Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Show the format of TFREC! tf.train.Example() and ParseFromString()\n","metadata":{}},{"cell_type":"markdown","source":"### You need to know the feature desciption, essentially the key and the datatype to parse your TFREC to the dataset.\n### Look at the TFREC with tf.train.Example() and ParseFromString(), and accessing [protobuf message](http://https://developers.google.com/protocol-buffers/)\n###  Link to tutorial: [Protocol Buffer Basics: Python](http://https://developers.google.com/protocol-buffers/docs/pythontutorial)","metadata":{}},{"cell_type":"code","source":"ds_raw = tf.data.TFRecordDataset(ls_tfrecs)\n\nfor raw_record in ds_raw.take(1):\n    example = tf.train.Example()\n    example.ParseFromString(raw_record.numpy())\n\ntfrec_dtype = {}\n\nfor key, feature in example.features.feature.items():\n    kind = feature.WhichOneof('kind')\n    tfrec_dtype[key] = kind\n\ntfrec_dtype    ","metadata":{"papermill":{"duration":0.370912,"end_time":"2022-05-17T00:53:47.055285","exception":false,"start_time":"2022-05-17T00:53:46.684373","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-17T16:17:57.771973Z","iopub.execute_input":"2022-05-17T16:17:57.772244Z","iopub.status.idle":"2022-05-17T16:17:59.147061Z","shell.execute_reply.started":"2022-05-17T16:17:57.772218Z","shell.execute_reply":"2022-05-17T16:17:59.14632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Great! We now know all the info goes into feature description dictionary. Now define a function to get feature description:","metadata":{}},{"cell_type":"code","source":"def get_tfrec_format(dictionary_obj):\n    tfrec_format= dict()\n    for key, value in dictionary_obj.items():\n        if value == \"bytes_list\":\n            tf_dtype =  tf.string\n        elif value == \"int64_list\":\n            tf_dtype = tf.int64\n        tfrec_format[key] = tf.io.FixedLenFeature([], tf_dtype)   \n    return tfrec_format\n\nget_tfrec_format(tfrec_dtype)","metadata":{"execution":{"iopub.status.busy":"2022-05-17T16:18:00.298881Z","iopub.execute_input":"2022-05-17T16:18:00.299416Z","iopub.status.idle":"2022-05-17T16:18:00.308282Z","shell.execute_reply.started":"2022-05-17T16:18:00.299369Z","shell.execute_reply":"2022-05-17T16:18:00.307362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Parse the raw data using the above key - datatype info!\n\n### And parse the raw data with the above feature description:","metadata":{}},{"cell_type":"code","source":"ds_parsed = ds_raw.map(lambda raw: tf.io.parse_single_example(raw, get_tfrec_format(tfrec_dtype)))\nds_parsed","metadata":{"papermill":{"duration":0.176748,"end_time":"2022-05-17T00:53:47.243756","exception":false,"start_time":"2022-05-17T00:53:47.067008","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-17T16:18:02.883197Z","iopub.execute_input":"2022-05-17T16:18:02.884007Z","iopub.status.idle":"2022-05-17T16:18:03.028412Z","shell.execute_reply.started":"2022-05-17T16:18:02.883949Z","shell.execute_reply":"2022-05-17T16:18:03.027559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"batch_size = 64\niter_ds = iter(ds_parsed)","metadata":{"execution":{"iopub.status.busy":"2022-05-17T16:18:05.149648Z","iopub.execute_input":"2022-05-17T16:18:05.150374Z","iopub.status.idle":"2022-05-17T16:18:21.36069Z","shell.execute_reply.started":"2022-05-17T16:18:05.150339Z","shell.execute_reply":"2022-05-17T16:18:21.359763Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Decode bytes to jpeg and display sample image!","metadata":{}},{"cell_type":"markdown","source":"Now we want to show the data to see if everything is as expected:","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nfrom tqdm import tqdm\n\nn_col = 8\nn_row = batch_size // n_col\nFIGSIZE = (25,25)\n\nfigs, axs = plt.subplots(n_row, n_col, figsize= FIGSIZE)\nfor row in tqdm(range(n_row)):\n    for col in range(n_col):\n        next_item = next(iter_ds)\n        axs[row,col].set_xticks([])\n        axs[row,col].set_yticks([])\n        axs[row,col].imshow(tf.io.decode_jpeg(next_item['image']))\n        axs[row,col].title.set_text(\"id: {}, class: {}\".format(next_item['id'].numpy().decode('ascii'), next_item['class']))\n","metadata":{"papermill":{"duration":0.342039,"end_time":"2022-05-17T00:53:47.597063","exception":false,"start_time":"2022-05-17T00:53:47.255024","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-17T15:55:58.111719Z","iopub.execute_input":"2022-05-17T15:55:58.112028Z","iopub.status.idle":"2022-05-17T15:56:04.036753Z","shell.execute_reply.started":"2022-05-17T15:55:58.111995Z","shell.execute_reply":"2022-05-17T15:56:04.032578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Batch data and display sample! ","metadata":{}},{"cell_type":"code","source":"def get_labels(item, list_labels):\n    return [item[key] for key in list_labels]\n\ntr_ds = ds_parsed.map(lambda item: get_labels(item, ['class','id']))\n\nnext(iter(tr_ds.batch(32)))","metadata":{"papermill":{"duration":0.148568,"end_time":"2022-05-17T00:53:47.759821","exception":false,"start_time":"2022-05-17T00:53:47.611253","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-05-17T15:30:31.363174Z","iopub.execute_input":"2022-05-17T15:30:31.363889Z","iopub.status.idle":"2022-05-17T15:30:31.461755Z","shell.execute_reply.started":"2022-05-17T15:30:31.363837Z","shell.execute_reply":"2022-05-17T15:30:31.46088Z"},"trusted":true},"execution_count":null,"outputs":[]}]}