Is it possible to use the standard Keras API with TPU? I tried giving ImageDataGenerator a GCS path but it does not work, I guess you need the TF gfile module to access them. And without optimized GCS streaming with co-located TPU I guess it is way too slow.
\n\nI tried all day to smack the TF Datasets API and its way more complex and less documented than Keras API.
\n\nI managed to upload the dataset to GCS, uses a mapping function that reads the format, builds a batched dataset, iterates the batches, ... But I always end up with some custom Python objects that I have no idea what to do with. I tried to dispay images from the batch and no success.
\n\nGoing from there I guess I could copy-paste items from the official demo kernel but it seems pointless if I have no idea what I am doing or why.
\n\nSo I keep wondering about what type of object do I have, what is a batch, does running the map function over the dataset actually load everything to memory (limiting data size heavily), and what else is happening.
\n\nThe official kernel is also very difficult to follow for me. It goes from one thing to another, mixing functions here and there, calling some TF functions I really have little idea about.
\n\nI would like to see a kernel clearly structured to do some basic operations.
\n\nExplaining different parts and actions smaller pieces at a time. Maybe even split it to several kernels if it makes it easier to read. I can mostly use standard Python for Keras in general but with the TF Dataset API it is all just about the specific objects and structures. Hard to find good docs or explanations.
\n\nI am a simple person, I need simple instructions :). So if anyone has the knowledge, that would be great, thanks..
", "messages": [ { "id": "747659", "postDate": "02/16/2020 17:48:23", "content": "Is it possible to use the standard Keras API with TPU? I tried giving ImageDataGenerator a GCS path but it does not work, I guess you need the TF gfile module to access them. And without optimized GCS streaming with co-located TPU I guess it is way too slow.
\n\nI tried all day to smack the TF Datasets API and its way more complex and less documented than Keras API.
\n\nI managed to upload the dataset to GCS, uses a mapping function that reads the format, builds a batched dataset, iterates the batches, ... But I always end up with some custom Python objects that I have no idea what to do with. I tried to dispay images from the batch and no success.
\n\nGoing from there I guess I could copy-paste items from the official demo kernel but it seems pointless if I have no idea what I am doing or why.
\n\nSo I keep wondering about what type of object do I have, what is a batch, does running the map function over the dataset actually load everything to memory (limiting data size heavily), and what else is happening.
\n\nThe official kernel is also very difficult to follow for me. It goes from one thing to another, mixing functions here and there, calling some TF functions I really have little idea about.
\n\nI would like to see a kernel clearly structured to do some basic operations.
\n\nExplaining different parts and actions smaller pieces at a time. Maybe even split it to several kernels if it makes it easier to read. I can mostly use standard Python for Keras in general but with the TF Dataset API it is all just about the specific objects and structures. Hard to find good docs or explanations.
\n\nI am a simple person, I need simple instructions :). So if anyone has the knowledge, that would be great, thanks..
", "rawMarkdown": "Is it possible to use the standard Keras API with TPU? I tried giving ImageDataGenerator a GCS path but it does not work, I guess you need the TF gfile module to access them. And without optimized GCS streaming with co-located TPU I guess it is way too slow.\n\nI tried all day to smack the TF Datasets API and its way more complex and less documented than Keras API.\n\nI managed to upload the dataset to GCS, uses a mapping function that reads the format, builds a batched dataset, iterates the batches, ... But I always end up with some custom Python objects that I have no idea what to do with. I tried to dispay images from the batch and no success. \n\nGoing from there I guess I could copy-paste items from the official demo kernel but it seems pointless if I have no idea what I am doing or why.\n\nSo I keep wondering about what type of object do I have, what is a batch, does running the map function over the dataset actually load everything to memory (limiting data size heavily), and what else is happening.\n\nThe official kernel is also very difficult to follow for me. It goes from one thing to another, mixing functions here and there, calling some TF functions I really have little idea about.\n\nI would like to see a kernel clearly structured to do some basic operations. \n\n- Convert an image set (pngs, jpegs, ...) to TF records. \n- Add other items to the record (label etc). Explain the protobuf structures generated.\n- Read the records back, decode all parts, explain them, show an image. \n- Feed data items to Keras one at a time, streamed in parallel, etc.\n- All the usual basic actions (everything I forget)\n\nExplaining different parts and actions smaller pieces at a time. Maybe even split it to several kernels if it makes it easier to read. I can mostly use standard Python for Keras in general but with the TF Dataset API it is all just about the specific objects and structures. Hard to find good docs or explanations.\n\nI am a simple person, I need simple instructions :). So if anyone has the knowledge, that would be great, thanks..", "votes": null }, { "id": "747768", "postDate": "02/16/2020 20:23:27", "content": "This is actually the plan for when I've time, hopefully soon, maybe this weekend. 😅
\n\nAll I did so far was to convert the .tfrec files back to JPEGs to work on the inverse problem that you seek.
\n\nI've published all JPEGs for all image sizes as a dataset which you can start playing with via this kernel.
", "rawMarkdown": "This is actually the plan for when I've time, hopefully soon, maybe this weekend. 😅\n\nAll I did so far was to convert the .tfrec files back to JPEGs to work on the inverse problem that you seek.\n\nI've published all JPEGs for all image sizes as a [dataset](https://www.kaggle.com/msheriey/104-flowers-garden-of-eden) which you can start playing with via [this kernel](https://www.kaggle.com/msheriey/104-flowers-garden-of-eden-starter-kernel).", "votes": null }, { "id": "748151", "postDate": "02/17/2020 07:38:28", "content": "What do you mean \"Keras API\"? Beginning with TensorFlow 2.0+, Keras is built into TensorFlow. For example, in TensorFlow 1.15-, you would import Keras separately and write instructions like keras.layers.Dense(), but in TF2.0+, you don't need to import Keras, you just import tensorflow and write tensorflow.keras.layers.Dense().
It happened to me the same way. That's why I tried a simple example like the Mnist Contest. Here is my kernel: https://www.kaggle.com/jugglingsnakeboarder/simplecnnmlpmnisttestingtpu
", "rawMarkdown": "It happened to me the same way. That's why I tried a simple example like the Mnist Contest. Here is my kernel: https://www.kaggle.com/jugglingsnakeboarder/simplecnnmlpmnisttestingtpu", "votes": null }, { "id": "748690", "postDate": "02/17/2020 20:59:05", "content": "Great job @jugglingsnakeboarder Thanks for sharing.
\n\nI don't think that technique works for large training data however. According to the API here
\n\n\n", "rawMarkdown": "Great job @jugglingsnakeboarder Thanks for sharing.\n\nI don't think that technique works for large training data however. According to the API [here][1]\n\n> For all but the simplest experimentation (using tf.data.Dataset.from_tensor_slices or other in-graph data) you will need to store all data files read by the Dataset in Google Cloud Storage (GCS) buckets.\n\n>For most use-cases, it is recommended to convert your data into TFRecord format and use a tf.data.TFRecordDataset to read it. See TFRecord and tf.Example tutorial for details on how to do this. This, however, is not a hard requirement and you can use other dataset readers (FixedLengthRecordDataset or TextLineDataset) if you prefer.\n\n[1]: https://www.tensorflow.org/guide/tpu", "votes": null }, { "id": "749625", "postDate": "02/18/2020 20:22:55", "content": "For all but the simplest experimentation (using tf.data.Dataset.from_tensor_slices or other in-graph data) you will need to store all data files read by the Dataset in Google Cloud Storage (GCS) buckets.
\n \nFor most use-cases, it is recommended to convert your data into TFRecord format and use a tf.data.TFRecordDataset to read it. See TFRecord and tf.Example tutorial for details on how to do this. This, however, is not a hard requirement and you can use other dataset readers (FixedLengthRecordDataset or TextLineDataset) if you prefer.
\n
Indeed, the goal of the tf.data.Dataset API is to handle out of memory datasets. Training on numpy arrays in memory works but defeats that purpose.\nI have an MNIST TPU sample here if you want to have a look: bit.ly/keras-TPU
", "rawMarkdown": "Indeed, the goal of the tf.data.Dataset API is to handle out of memory datasets. Training on numpy arrays in memory works but defeats that purpose.\nI have an MNIST TPU sample here if you want to have a look: [bit.ly/keras-TPU](https://colab.research.google.com/github/GoogleCloudPlatform/training-data-analyst/blob/master/courses/fast-and-lean-data-science/01_MNIST_TPU_Keras.ipynb)", "votes": null }, { "id": "749637", "postDate": "02/18/2020 20:29:33", "content": "I understand your frustration with ImageDataGenerator not working. There is a team working on it (i.e. reimplementing ImageDataGenerator on top of tf.data.Dataset with TF primitives so that it works as any other TF library).
\n\nIn the meantime, you have to use tf.data.Dataset.\nI have a good primer here: TPU-speed data pipelines: tf.data.Dataset and TFRecords\n
\n\nIt will answer some of the questions you are asking about writing TFRecords, their format and so on.
\n\nYou can do data augmentation using TF APIs in tf.image or tensorflow-addons.image. There is an example of data augmentation in the getting started notebook. Look for the data_augment function.
Many thanks for the quick response. I will be very happy to look at the example bit.ly/keras-TPU, because at the moment I am very excited to see everything I can learn about using TPUs. (During my studies I worked a lot with MatLab at NN and now -- after spending the last two AI winters raising my children -- I am fascinated by the progress in AI and try to catch up a lot).
", "rawMarkdown": "Many thanks for the quick response. I will be very happy to look at the example bit.ly/keras-TPU, because at the moment I am very excited to see everything I can learn about using TPUs. (During my studies I worked a lot with MatLab at NN and now -- after spending the last two AI winters raising my children -- I am fascinated by the progress in AI and try to catch up a lot).", "votes": null }, { "id": "1330065", "postDate": "05/31/2021 15:09:34", "content": "is the ImageDataGenerator supported by Kaggle TPU now?
", "rawMarkdown": "is the ImageDataGenerator supported by Kaggle TPU now?", "votes": null } ], "comments": [ { "id": 1330065, "author_name": "gkmeng", "author_url": "", "post_date": "05/31/2021 15:09:34", "content": "is the ImageDataGenerator supported by Kaggle TPU now?
", "votes": null, "replies": [] }, { "id": 747768, "author_name": "msheriey", "author_url": "", "post_date": "02/16/2020 20:23:27", "content": "This is actually the plan for when I've time, hopefully soon, maybe this weekend. 😅
\n\nAll I did so far was to convert the .tfrec files back to JPEGs to work on the inverse problem that you seek.
\n\nI've published all JPEGs for all image sizes as a dataset which you can start playing with via this kernel.
", "votes": null, "replies": [] }, { "id": 748151, "author_name": "cdeotte", "author_url": "", "post_date": "02/17/2020 07:38:28", "content": "What do you mean \"Keras API\"? Beginning with TensorFlow 2.0+, Keras is built into TensorFlow. For example, in TensorFlow 1.15-, you would import Keras separately and write instructions like keras.layers.Dense(), but in TF2.0+, you don't need to import Keras, you just import tensorflow and write tensorflow.keras.layers.Dense().
It happened to me the same way. That's why I tried a simple example like the Mnist Contest. Here is my kernel: https://www.kaggle.com/jugglingsnakeboarder/simplecnnmlpmnisttestingtpu
", "votes": null, "replies": [ { "id": 748690, "author_name": "cdeotte", "author_url": "", "post_date": "02/17/2020 20:59:05", "content": "Great job @jugglingsnakeboarder Thanks for sharing.
\n\nI don't think that technique works for large training data however. According to the API here
\n\n\n", "votes": null, "replies": [] }, { "id": 749625, "author_name": "mgorner", "author_url": "", "post_date": "02/18/2020 20:22:55", "content": "For all but the simplest experimentation (using tf.data.Dataset.from_tensor_slices or other in-graph data) you will need to store all data files read by the Dataset in Google Cloud Storage (GCS) buckets.
\n \nFor most use-cases, it is recommended to convert your data into TFRecord format and use a tf.data.TFRecordDataset to read it. See TFRecord and tf.Example tutorial for details on how to do this. This, however, is not a hard requirement and you can use other dataset readers (FixedLengthRecordDataset or TextLineDataset) if you prefer.
\n
Indeed, the goal of the tf.data.Dataset API is to handle out of memory datasets. Training on numpy arrays in memory works but defeats that purpose.\nI have an MNIST TPU sample here if you want to have a look: bit.ly/keras-TPU
", "votes": null, "replies": [] } ] }, { "id": 749637, "author_name": "mgorner", "author_url": "", "post_date": "02/18/2020 20:29:33", "content": "I understand your frustration with ImageDataGenerator not working. There is a team working on it (i.e. reimplementing ImageDataGenerator on top of tf.data.Dataset with TF primitives so that it works as any other TF library).
\n\nIn the meantime, you have to use tf.data.Dataset.\nI have a good primer here: TPU-speed data pipelines: tf.data.Dataset and TFRecords\n
\n\nIt will answer some of the questions you are asking about writing TFRecords, their format and so on.
\n\nYou can do data augmentation using TF APIs in tf.image or tensorflow-addons.image. There is an example of data augmentation in the getting started notebook. Look for the data_augment function.
Many thanks for the quick response. I will be very happy to look at the example bit.ly/keras-TPU, because at the moment I am very excited to see everything I can learn about using TPUs. (During my studies I worked a lot with MatLab at NN and now -- after spending the last two AI winters raising my children -- I am fascinated by the progress in AI and try to catch up a lot).
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "747659": "Is it possible to use the standard Keras API with TPU? I tried giving ImageDataGenerator a GCS path but it does not work, I guess you need the TF gfile module to access them. And without optimized GCS streaming with co-located TPU I guess it is way too slow.\n\nI tried all day to smack the TF Datasets API and its way more complex and less documented than Keras API.\n\nI managed to upload the dataset to GCS, uses a mapping function that reads the format, builds a batched dataset, iterates the batches, ... But I always end up with some custom Python objects that I have no idea what to do with. I tried to dispay images from the batch and no success. \n\nGoing from there I guess I could copy-paste items from the official demo kernel but it seems pointless if I have no idea what I am doing or why.\n\nSo I keep wondering about what type of object do I have, what is a batch, does running the map function over the dataset actually load everything to memory (limiting data size heavily), and what else is happening.\n\nThe official kernel is also very difficult to follow for me. It goes from one thing to another, mixing functions here and there, calling some TF functions I really have little idea about.\n\nI would like to see a kernel clearly structured to do some basic operations. \n\n- Convert an image set (pngs, jpegs, ...) to TF records. \n- Add other items to the record (label etc). Explain the protobuf structures generated.\n- Read the records back, decode all parts, explain them, show an image. \n- Feed data items to Keras one at a time, streamed in parallel, etc.\n- All the usual basic actions (everything I forget)\n\nExplaining different parts and actions smaller pieces at a time. Maybe even split it to several kernels if it makes it easier to read. I can mostly use standard Python for Keras in general but with the TF Dataset API it is all just about the specific objects and structures. Hard to find good docs or explanations.\n\nI am a simple person, I need simple instructions :). So if anyone has the knowledge, that would be great, thanks..", "747768": "This is actually the plan for when I've time, hopefully soon, maybe this weekend. 😅\n\nAll I did so far was to convert the .tfrec files back to JPEGs to work on the inverse problem that you seek.\n\nI've published all JPEGs for all image sizes as a [dataset](https://www.kaggle.com/msheriey/104-flowers-garden-of-eden) which you can start playing with via [this kernel](https://www.kaggle.com/msheriey/104-flowers-garden-of-eden-starter-kernel).", "748151": "What do you mean \"Keras API\"? Beginning with TensorFlow 2.0+, Keras is built into TensorFlow. For example, in TensorFlow 1.15-, you would import Keras separately and write instructions like `keras.layers.Dense()`, but in TF2.0+, you don't need to import Keras, you just import tensorflow and write `tensorflow.keras.layers.Dense()`.", "748522": "It happened to me the same way. That's why I tried a simple example like the Mnist Contest. Here is my kernel: https://www.kaggle.com/jugglingsnakeboarder/simplecnnmlpmnisttestingtpu", "748690": "Great job @jugglingsnakeboarder Thanks for sharing.\n\nI don't think that technique works for large training data however. According to the API [here][1]\n\n> For all but the simplest experimentation (using tf.data.Dataset.from_tensor_slices or other in-graph data) you will need to store all data files read by the Dataset in Google Cloud Storage (GCS) buckets.\n\n>For most use-cases, it is recommended to convert your data into TFRecord format and use a tf.data.TFRecordDataset to read it. See TFRecord and tf.Example tutorial for details on how to do this. This, however, is not a hard requirement and you can use other dataset readers (FixedLengthRecordDataset or TextLineDataset) if you prefer.\n\n[1]: https://www.tensorflow.org/guide/tpu", "749625": "Indeed, the goal of the tf.data.Dataset API is to handle out of memory datasets. Training on numpy arrays in memory works but defeats that purpose.\nI have an MNIST TPU sample here if you want to have a look: [bit.ly/keras-TPU](https://colab.research.google.com/github/GoogleCloudPlatform/training-data-analyst/blob/master/courses/fast-and-lean-data-science/01_MNIST_TPU_Keras.ipynb)", "749637": "I understand your frustration with ImageDataGenerator not working. There is a team working on it (i.e. reimplementing ImageDataGenerator on top of tf.data.Dataset with TF primitives so that it works as any other TF library).\n\nIn the meantime, you have to use tf.data.Dataset.\nI have a good primer here: [TPU-speed data pipelines: tf.data.Dataset and TFRecords\n](https://codelabs.developers.google.com/codelabs/keras-flowers-data/#0)\n\nIt will answer some of the questions you are asking about writing TFRecords, their format and so on.\n\nYou can do data augmentation using TF APIs in `tf.image` or `tensorflow-addons.image`. There is an example of data augmentation in the [getting started notebook](https://www.kaggle.com/mgornergoogle/getting-started-with-100-flowers-on-tpu/). Look for the `data_augment` function.", "750562": "Many thanks for the quick response. I will be very happy to look at the example bit.ly/keras-TPU, because at the moment I am very excited to see everything I can learn about using TPUs. (During my studies I worked a lot with MatLab at NN and now -- after spending the last two AI winters raising my children -- I am fascinated by the progress in AI and try to catch up a lot).", "1330065": "is the ImageDataGenerator supported by Kaggle TPU now?" }, "source": "meta" }