Please tell me if you know about this topic!
\n\nI (and some kagglers maybe) like pytorch-like dataloader, which make custom class by implementing torch.utils.data.Dataset and torch.utils.data.DataLoader, then one can check the input one by one or on the way of training.
In many TF notebooks, however, TFRecord and tf.Example are used. And this is bit difficult for me for the first look. (Here is good for understanding.)\nI found an api from_generator in tf.data.Dataset, which allows me to implement pytorch-like dataloader by defining generator.\nI write my custom dataloader (sorry for simplified example) to try this and use it for inference.
But now I don't clearly understand the advantage of using generator way or using TFRecords. In my opinion:
\n\nfrom_generator case\nDo you try both ways to feed data, or do you know other ways? Is there any other differences?
", "messages": [ { "id": "724810", "postDate": "01/21/2020 13:57:19", "content": "Please tell me if you know about this topic!
\n\nI (and some kagglers maybe) like pytorch-like dataloader, which make custom class by implementing torch.utils.data.Dataset and torch.utils.data.DataLoader, then one can check the input one by one or on the way of training.
In many TF notebooks, however, TFRecord and tf.Example are used. And this is bit difficult for me for the first look. (Here is good for understanding.)\nI found an api from_generator in tf.data.Dataset, which allows me to implement pytorch-like dataloader by defining generator.\nI write my custom dataloader (sorry for simplified example) to try this and use it for inference.
But now I don't clearly understand the advantage of using generator way or using TFRecords. In my opinion:
\n\nfrom_generator case\nDo you try both ways to feed data, or do you know other ways? Is there any other differences?
", "rawMarkdown": "Please tell me if you know about this topic!\n\nI (and some kagglers maybe) like pytorch-like dataloader, which make custom class by implementing `torch.utils.data.Dataset` and `torch.utils.data.DataLoader`, then one can check the input one by one or on the way of training.\n\nIn many TF notebooks, however, TFRecord and tf.Example are used. And this is bit difficult for me for the first look. ([Here](https://www.tensorflow.org/tutorials/load_data/tfrecord?hl=en) is good for understanding.)\nI found an api [`from_generator` in `tf.data.Dataset`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#from_generator), which allows me to implement pytorch-like dataloader by defining generator.\nI write [my custom dataloader (sorry for simplified example)](https://www.kaggle.com/kentaronakanishi/my-custom-dataloader-simplified) to try this and use it for inference.\n\nBut now I don't clearly understand the advantage of using generator way or using TFRecords. In my opinion:\n\n- using `from_generator` case\n - no need to pre-compile\n - able to use object-base dataloader\n - memory efficiency (since this is based on generator) (edited)\n - cannot shuffle dataset (edited)\n- using TFRecords case\n - need pre-compile \n - maybe faster if there are many loops\n - can shuffle \n\nDo you try both ways to feed data, or do you know other ways? Is there any other differences?", "votes": null }, { "id": "724820", "postDate": "01/21/2020 14:14:00", "content": "One other way would be using from_tensor_slices (https://www.tensorflow.org/api_docs/python/tf/data/Dataset#from_tensor_slices). The drawback is that it only works for small datasets (it says <1GB, I am not sure this is still true). I don't like writing encoders and decoders for TFRecords either :P
from_generator does not work on TPUs but TFRecords do. According to this issue generator-like inputs for the TPU may be in the works.
Yes, from_tensor_slices is another way but It's allowed in small dataset.\nOne more merit for from_generator is memory efficiency since it's based on generator.\nI didn't know about TPU compatibility, maybe that's why many notebooks use TFRecords. Thanks for information!
One other way would be using from_tensor_slices (https://www.tensorflow.org/api_docs/python/tf/data/Dataset#from_tensor_slices). The drawback is that it only works for small datasets (it says <1GB, I am not sure this is still true). I don't like writing encoders and decoders for TFRecords either :P
from_generator does not work on TPUs but TFRecords do. According to this issue generator-like inputs for the TPU may be in the works.
Yes, from_tensor_slices is another way but It's allowed in small dataset.\nOne more merit for from_generator is memory efficiency since it's based on generator.\nI didn't know about TPU compatibility, maybe that's why many notebooks use TFRecords. Thanks for information!