I just found out about this excellent (and short!) tutorial written by Martin that goes over the data loading process with TFRecords. I think this will definitely be useful if you want to know how to run models with your own image data on the TPUs.
", "messages": [ { "id": "742902", "postDate": "02/11/2020 16:28:10", "content": "I just found out about this excellent (and short!) tutorial written by Martin that goes over the data loading process with TFRecords. I think this will definitely be useful if you want to know how to run models with your own image data on the TPUs.
", "rawMarkdown": "I just found out about this excellent (and short!) [tutorial written by Martin](https://codelabs.developers.google.com/codelabs/keras-flowers-data/#0) that goes over the data loading process with TFRecords. I think this will definitely be useful if you want to know how to run models with your own image data on the TPUs.", "votes": null }, { "id": "743066", "postDate": "02/11/2020 18:58:08", "content": "Thanks!\nThe TFRecord format is just a container format. The only problem it solves here is the large number of files in the dataset coupled with GCS (Google Cloud Storage). As any internet-accessed storage, a GCS request has a roundtrip time similar to any HTTP request, measured in tenths of a second. For a dataset with 100,000 files, that does not work. On the other hand, GCS can sustain serious throughput if streaming from multiple files in parallel. Sharding your data across a reasonable number (10s or 100s) of reasonably large files (10s to 100s of MB) provides the best GCS throughput.
\n\nBy the way, the TFRecord tutorial is the first part of the Keras and modern convnets, on TPUs code lab. And spoiler alert if you were planning to go through it, the TPU sample model in the Kaggle TPU documentation is the end result and best model you obtain at the end of the code lab.
", "rawMarkdown": "Thanks!\nThe TFRecord format is just a container format. The only problem it solves here is the large number of files in the dataset coupled with GCS (Google Cloud Storage). As any internet-accessed storage, a GCS request has a roundtrip time similar to any HTTP request, measured in tenths of a second. For a dataset with 100,000 files, that does not work. On the other hand, GCS can sustain serious throughput if streaming from multiple files in parallel. Sharding your data across a reasonable number (10s or 100s) of reasonably large files (10s to 100s of MB) provides the best GCS throughput.\n\nBy the way, the TFRecord tutorial is the first part of the [Keras and modern convnets, on TPUs](https://codelabs.developers.google.com/codelabs/keras-flowers-tpu/#0) code lab. And spoiler alert if you were planning to go through it, the [TPU sample model](https://www.kaggle.com/mgornergoogle/five-flowers-with-keras-and-xception-on-tpu) in the [Kaggle TPU documentation](https://www.kaggle.com/docs/tpu) is the end result and best model you obtain at the end of the code lab.", "votes": null }, { "id": "743718", "postDate": "02/12/2020 08:25:33", "content": "I recall using this line made things faster:
\n\nimage, label, height, width = map(lambda x: x.numpy(), (image, label, height, width))
And then remove the numpy() calls in the for i loop.
Is there any tutorial to use from_tensor_slices and compressed images? I always get an error if I use tf.string and try to decode on the fly via tf.image.decode_image. It works with GPUs.
No .numpy() calls are needed in your data pipeline. Generally only for printing and debugging.
\n\nBoth the TPU generic sample and the flowers competition getting started notebook read compressed JPEG images from TFRecords. You can look at the code there:\nFive flowers with Keras and Xception on TPU\nGetting started with 100+ flowers on TPU
\n\nIf you want to know everything about tf.data.Dataset and TFRecords, I have a tutorial here:\nTPU-speed data pipelines: tf.data.Dataset and TFRecords
", "rawMarkdown": "No .numpy() calls are needed in your data pipeline. Generally only for printing and debugging.\n\nBoth the TPU generic sample and the flowers competition getting started notebook read compressed JPEG images from TFRecords. You can look at the code there:\n[Five flowers with Keras and Xception on TPU](https://www.kaggle.com/mgornergoogle/five-flowers-with-keras-and-xception-on-tpu)\n[Getting started with 100+ flowers on TPU](https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu/)\n\nIf you want to know everything about tf.data.Dataset and TFRecords, I have a tutorial here:\n[TPU-speed data pipelines: tf.data.Dataset and TFRecords](https://codelabs.developers.google.com/codelabs/keras-flowers-data/#0)", "votes": null }, { "id": "744380", "postDate": "02/12/2020 19:28:59", "content": "It seems to be needed for BytesList. This matches the documentation: BytesList won't unpack a string from an EagerTensor..
Ah ok, you were talking about writing TFRecords. I though the question was about reading them.
", "rawMarkdown": "Ah ok, you were talking about **writing** TFRecords. I though the question was about reading them.", "votes": null }, { "id": "744397", "postDate": "02/12/2020 19:47:00", "content": "Yes, I was running your notebook. It actually does both.
", "rawMarkdown": "Yes, I was running [your notebook](https://colab.research.google.com/github/GoogleCloudPlatform/training-data-analyst/blob/master/courses/fast-and-lean-data-science/03_Flower_pictures_to_TFRecords.ipynb#scrollTo=cDIfPMGCjqLO). It actually does both.", "votes": null }, { "id": "778712", "postDate": "03/18/2020 17:04:28", "content": "@xhlulu Thank you for sharing this tutorial
", "rawMarkdown": "xhlulu Thank you for sharing this tutorial", "votes": null }, { "id": "825871", "postDate": "04/29/2020 09:46:13", "content": "Thanks it is so useful !
", "rawMarkdown": "Thanks it is so useful !", "votes": null }, { "id": "936067", "postDate": "07/20/2020 00:15:43", "content": "@mgornergoogle \nIndeed great tutorial!
\n\nI would like to ask:
\n\n1) When converting images to TFRecords, do I need to turn the GPU on? Would it make the conversion faster?\n2) Do you know how can I view TPU \"MXU\" and \"Idle Time\" in Colab?
\n\nMany thanks!
", "rawMarkdown": "mgornergoogle \nIndeed great tutorial! \n\nI would like to ask:\n\n1) When converting images to TFRecords, do I need to turn the GPU on? Would it make the conversion faster?\n2) Do you know how can I view TPU \"MXU\" and \"Idle Time\" in Colab?\n\nMany thanks!", "votes": null }, { "id": "938708", "postDate": "07/21/2020 17:35:47", "content": "1) I don't believe my script uses the GPU but maybe.
\n2) You cannot
@xhlulu you are the best man
", "rawMarkdown": "xhlulu you are the best man", "votes": null }, { "id": "969853", "postDate": "08/14/2020 02:11:01", "content": "@mgornergoogle Great tutorials! I particularly like your tutorials where you explain how CNNs work.
", "rawMarkdown": "mgornergoogle Great tutorials! I particularly like your tutorials where you explain how CNNs work.", "votes": null }, { "id": "972299", "postDate": "08/16/2020 12:48:16", "content": "Thanks man!
", "rawMarkdown": "Thanks man!", "votes": null } ], "comments": [ { "id": 969853, "author_name": "cdeotte", "author_url": "", "post_date": "08/14/2020 02:11:01", "content": "@mgornergoogle Great tutorials! I particularly like your tutorials where you explain how CNNs work.
", "votes": null, "replies": [] }, { "id": 972299, "author_name": "delllectron", "author_url": "", "post_date": "08/16/2020 12:48:16", "content": "Thanks man!
", "votes": null, "replies": [] }, { "id": 743066, "author_name": "mgorner", "author_url": "", "post_date": "02/11/2020 18:58:08", "content": "Thanks!\nThe TFRecord format is just a container format. The only problem it solves here is the large number of files in the dataset coupled with GCS (Google Cloud Storage). As any internet-accessed storage, a GCS request has a roundtrip time similar to any HTTP request, measured in tenths of a second. For a dataset with 100,000 files, that does not work. On the other hand, GCS can sustain serious throughput if streaming from multiple files in parallel. Sharding your data across a reasonable number (10s or 100s) of reasonably large files (10s to 100s of MB) provides the best GCS throughput.
\n\nBy the way, the TFRecord tutorial is the first part of the Keras and modern convnets, on TPUs code lab. And spoiler alert if you were planning to go through it, the TPU sample model in the Kaggle TPU documentation is the end result and best model you obtain at the end of the code lab.
", "votes": null, "replies": [] }, { "id": 743718, "author_name": "seesee", "author_url": "", "post_date": "02/12/2020 08:25:33", "content": "I recall using this line made things faster:
\n\nimage, label, height, width = map(lambda x: x.numpy(), (image, label, height, width))
And then remove the numpy() calls in the for i loop.
Is there any tutorial to use from_tensor_slices and compressed images? I always get an error if I use tf.string and try to decode on the fly via tf.image.decode_image. It works with GPUs.
No .numpy() calls are needed in your data pipeline. Generally only for printing and debugging.
\n\nBoth the TPU generic sample and the flowers competition getting started notebook read compressed JPEG images from TFRecords. You can look at the code there:\nFive flowers with Keras and Xception on TPU\nGetting started with 100+ flowers on TPU
\n\nIf you want to know everything about tf.data.Dataset and TFRecords, I have a tutorial here:\nTPU-speed data pipelines: tf.data.Dataset and TFRecords
", "votes": null, "replies": [] }, { "id": 744380, "author_name": "seesee", "author_url": "", "post_date": "02/12/2020 19:28:59", "content": "It seems to be needed for BytesList. This matches the documentation: BytesList won't unpack a string from an EagerTensor..
Ah ok, you were talking about writing TFRecords. I though the question was about reading them.
", "votes": null, "replies": [] }, { "id": 744397, "author_name": "seesee", "author_url": "", "post_date": "02/12/2020 19:47:00", "content": "Yes, I was running your notebook. It actually does both.
", "votes": null, "replies": [] } ] }, { "id": 778712, "author_name": "usharengaraju", "author_url": "", "post_date": "03/18/2020 17:04:28", "content": "@xhlulu Thank you for sharing this tutorial
", "votes": null, "replies": [] }, { "id": 825871, "author_name": "salihsarp", "author_url": "", "post_date": "04/29/2020 09:46:13", "content": "Thanks it is so useful !
", "votes": null, "replies": [] }, { "id": 936067, "author_name": "chho1229", "author_url": "", "post_date": "07/20/2020 00:15:43", "content": "@mgornergoogle \nIndeed great tutorial!
\n\nI would like to ask:
\n\n1) When converting images to TFRecords, do I need to turn the GPU on? Would it make the conversion faster?\n2) Do you know how can I view TPU \"MXU\" and \"Idle Time\" in Colab?
\n\nMany thanks!
", "votes": null, "replies": [ { "id": 938708, "author_name": "mgorner", "author_url": "", "post_date": "07/21/2020 17:35:47", "content": "1) I don't believe my script uses the GPU but maybe.
\n2) You cannot
@xhlulu you are the best man
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "742902": "I just found out about this excellent (and short!) [tutorial written by Martin](https://codelabs.developers.google.com/codelabs/keras-flowers-data/#0) that goes over the data loading process with TFRecords. I think this will definitely be useful if you want to know how to run models with your own image data on the TPUs.", "743066": "Thanks!\nThe TFRecord format is just a container format. The only problem it solves here is the large number of files in the dataset coupled with GCS (Google Cloud Storage). As any internet-accessed storage, a GCS request has a roundtrip time similar to any HTTP request, measured in tenths of a second. For a dataset with 100,000 files, that does not work. On the other hand, GCS can sustain serious throughput if streaming from multiple files in parallel. Sharding your data across a reasonable number (10s or 100s) of reasonably large files (10s to 100s of MB) provides the best GCS throughput.\n\nBy the way, the TFRecord tutorial is the first part of the [Keras and modern convnets, on TPUs](https://codelabs.developers.google.com/codelabs/keras-flowers-tpu/#0) code lab. And spoiler alert if you were planning to go through it, the [TPU sample model](https://www.kaggle.com/mgornergoogle/five-flowers-with-keras-and-xception-on-tpu) in the [Kaggle TPU documentation](https://www.kaggle.com/docs/tpu) is the end result and best model you obtain at the end of the code lab.", "743718": "I recall using this line made things faster:\n\n`image, label, height, width = map(lambda x: x.numpy(), (image, label, height, width))`\n\nAnd then remove the `numpy()` calls in the `for i` loop.\n\nIs there any tutorial to use `from_tensor_slices` and compressed images? I always get an error if I use `tf.string` and try to decode on the fly via `tf.image.decode_image`. It works with GPUs.", "744305": "No .numpy() calls are needed in your data pipeline. Generally only for printing and debugging.\n\nBoth the TPU generic sample and the flowers competition getting started notebook read compressed JPEG images from TFRecords. You can look at the code there:\n[Five flowers with Keras and Xception on TPU](https://www.kaggle.com/mgornergoogle/five-flowers-with-keras-and-xception-on-tpu)\n[Getting started with 100+ flowers on TPU](https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu/)\n\nIf you want to know everything about tf.data.Dataset and TFRecords, I have a tutorial here:\n[TPU-speed data pipelines: tf.data.Dataset and TFRecords](https://codelabs.developers.google.com/codelabs/keras-flowers-data/#0)", "744380": "It seems to be needed for `BytesList`. This matches the documentation: [BytesList won't unpack a string from an EagerTensor.](https://www.tensorflow.org/tutorials/load_data/tfrecord).", "744389": "Ah ok, you were talking about **writing** TFRecords. I though the question was about reading them.", "744397": "Yes, I was running [your notebook](https://colab.research.google.com/github/GoogleCloudPlatform/training-data-analyst/blob/master/courses/fast-and-lean-data-science/03_Flower_pictures_to_TFRecords.ipynb#scrollTo=cDIfPMGCjqLO). It actually does both.", "778712": "xhlulu Thank you for sharing this tutorial", "825871": "Thanks it is so useful !", "936067": "mgornergoogle \nIndeed great tutorial! \n\nI would like to ask:\n\n1) When converting images to TFRecords, do I need to turn the GPU on? Would it make the conversion faster?\n2) Do you know how can I view TPU \"MXU\" and \"Idle Time\" in Colab?\n\nMany thanks!", "938708": "1) I don't believe my script uses the GPU but maybe.\n2) You cannot", "957005": "xhlulu you are the best man", "969853": "mgornergoogle Great tutorials! I particularly like your tutorials where you explain how CNNs work.", "972299": "Thanks man!" }, "source": "meta" }