I'm wondering what efficient method people are using for loading the training data, since I'm having trouble with my first option, which is parallelized loading by using tfio.IODataset.from_parquet for each sample. I am now working with tf.py_function and reading with pandas, but that is slow. I know I could simply save the dataset after waiting for it to load once, but a faster method would be nice.
\nTrying to load a parquet file into a dataset with tfio silently crashes the kernel. The error seems to be the following:
\nterminate called after throwing an instance of 'parquet::ParquetException'
\nwhat(): Unexpected end of stream
I'm wondering what efficient method people are using for loading the training data, since I'm having trouble with my first option, which is parallelized loading by using tfio.IODataset.from_parquet for each sample. I am now working with tf.py_function and reading with pandas, but that is slow. I know I could simply save the dataset after waiting for it to load once, but a faster method would be nice.
\nTrying to load a parquet file into a dataset with tfio silently crashes the kernel. The error seems to be the following:
\nterminate called after throwing an instance of 'parquet::ParquetException'
\nwhat(): Unexpected end of stream
I have now published a tf.Dataset that quite significantly speeds up loading for me (700 it/sec vs. 60 it/sec originally, with no batching), along with a notebook on how to easily save the data yourself, which should make it easier for anyone to apply their own desired level of preprocessing before saving, if you want even more performance. I myself will probably be using a variation that is padded to a specific sequence length and pre-batched.
", "rawMarkdown": "I have now published a [tf.Dataset](https://www.kaggle.com/datasets/aapokossi/saved-tfdataset-of-google-isl-recognition-data) that quite significantly speeds up loading for me (700 it/sec vs. 60 it/sec originally, with no batching), along with [a notebook](https://www.kaggle.com/code/aapokossi/how-to-save-parquet-data-as-tf-dataset) on how to easily save the data yourself, which should make it easier for anyone to apply their own desired level of preprocessing before saving, if you want even more performance. I myself will probably be using a variation that is padded to a specific sequence length and pre-batched. ", "votes": 2 }, { "id": 2162219, "postDate": "2023-02-28T04:10:16.230Z", "content": "I'm quite new to TF and Kaggle competitions, but taking reference some of the codes shared (shoutout to @lonnieqin), I believe the competition has specified a method they will use to load the data, available in the Evaluation page. I'm guessing we can adopt the same method to load the data, then incorporate any further processing into your model pipeline.
\n ():\n data_columns = [, , ]\n data = pd.read_parquet(pq_path, columns=data_columns)\n n_frames = ((data) / ROWS_PER_FRAME)\n data = data.values.reshape(n_frames, ROWS_PER_FRAME, (data_columns))\n data.astype(np.float32)\nThis only loads a single sample from disk. What I'm asking for is how to incorporate this into a Tf Dataset object to automatically load batches from disk during training. There is standard code for standard data formats (e.g. images), but writing this from scratch requires in-depth understanding of Tensorflow.
\nBased on the leaderboard, I's assuming that at least some people must have managed to solve this …
", "rawMarkdown": "This only loads a single sample from disk. What I'm asking for is how to incorporate this into a Tf Dataset object to automatically load batches from disk during training. There is standard code for standard data formats (e.g. images), but writing this from scratch requires in-depth understanding of Tensorflow.\n\nBased on the leaderboard, I's assuming that at least some people must have managed to solve this ..." } ] }, { "id": 2161922, "postDate": "2023-02-27T20:36:11.827Z", "content": "I have the same question. I've been browsing through multiple explanations about creating a Tf Dataset from parquet files, but this is way above my level of understanding. When I try pre-padding the data for training as a time-series model, the data becomes WAY too large, e.g. window size of 30 -> 94477x30x543x3 numbers
\nActually, since this is really essential for proper training here, and since documentation for this is really lacking, I feel the Tensorflow development team should provide example code for this. Otherwise, anyone who is not a really advanced Tf software engineer is excluded from properly participating because of getting stuck on something I'm sure the organisers have already solved.
\nSo please, can someone provide example code for a (train and val) Dataset with batch prefetching that can be used for efficient training, so we can focus on the model design.
", "rawMarkdown": "I have the same question. I've been browsing through multiple explanations about creating a Tf Dataset from parquet files, but this is way above my level of understanding. When I try pre-padding the data for training as a time-series model, the data becomes WAY too large, e.g. window size of 30 -> 94477x30x543x3 numbers\n\nActually, since this is really essential for proper training here, and since documentation for this is really lacking, I feel the Tensorflow development team should provide example code for this. Otherwise, anyone who is not a really advanced Tf software engineer is excluded from properly participating because of getting stuck on something I'm sure the organisers have already solved.\n\nSo please, can someone provide example code for a (train and val) Dataset with batch prefetching that can be used for efficient training, so we can focus on the model design.", "replies": [ { "id": 2162762, "postDate": "2023-02-28T11:58:55.640Z", "content": "I'm not sure about your data dimension here.
\nThis is indeed pretty big though.
", "rawMarkdown": "I'm not sure about your data dimension here. \n\n* a frame is represented by a vector of size (543,3)\n* the maximum number of frames in a sequence is 537\n* Therefore if you padded to the maximum length, each example should be of shape: **(537,543,3)**\n\nThis is indeed pretty big though." } ] }, { "id": 2160746, "postDate": "2023-02-27T00:12:00.680Z", "content": "Haven’t tried it but maybe this:
\nhttps://www.tensorflow.org/io/api_docs/python/tfio/experimental/IODataset#from_parquet
It’s in experimental so maybe the issue your having has been fixed?
\nI’ll dig more into this tomorrow.
", "rawMarkdown": "Haven’t tried it but maybe this:\nhttps://www.tensorflow.org/io/api_docs/python/tfio/experimental/IODataset#from_parquet\n\nIt’s in experimental so maybe the issue your having has been fixed?\n\nI’ll dig more into this tomorrow.\n", "replies": [ { "id": 2162411, "postDate": "2023-02-28T07:24:05.443Z", "content": "Eager to hear whether you've made any progress ??
", "rawMarkdown": "Eager to hear whether you've made any progress ??\n", "replies": [ { "id": 2162720, "postDate": "2023-02-28T11:32:42.310Z", "content": "No, I have the same issue as you where it crashes when I try to load it.
\nI am in the process of making TFRecords for efficient data loading. I'll share the notebook shortly and you can use that to create the dataset that reflects what you want.
\nSome Random Notes:
\nI'll include everything I can in my notebook.
", "rawMarkdown": "No, I have the same issue as you where it crashes when I try to load it.\n\nI am in the process of making TFRecords for efficient data loading. I'll share the notebook shortly and you can use that to create the dataset that reflects what you want.\n\n**Some Random Notes:**\n* More than likely you need to do dimensionality reduction PRIOR TO (or inline with) dataset creation.\n* This is because as you posted, the raw arrays would be too large.\n* An alternative would be to scale the x,y,z (or discard the z) to be from 0-255 and the byte encode it as a string to save on space (just like you would an image)\n\nI'll include everything I can in my notebook.", "votes": 1 }, { "id": 2163027, "postDate": "2023-02-28T14:43:06.747Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2163029, "postDate": "2023-02-28T14:43:43.217Z", "content": "I prefer the tf.data aligned Dataset.save compared to TFRecords, but I like that we will now have multiple alternatives! You can see my top level comment for a saved tf.Dataset and notebook on how to save your own if you want to try with tf.Datasets without the intermediate step of TFRecords.
", "rawMarkdown": "I prefer the tf.data aligned Dataset.save compared to TFRecords, but I like that we will now have multiple alternatives! You can see my top level comment for a saved tf.Dataset and notebook on how to save your own if you want to try with tf.Datasets without the intermediate step of TFRecords." }, { "id": 2163421, "postDate": "2023-02-28T19:47:38.763Z", "content": "Dimensionality reduction: The face landmarks might be total overkill, if removing them or grouping/averaging them, then that gets the dataset down to a more reasonable size. At the risk of removing even a little bit of useful data, though.
\nIf taking the raw data without padding, it would be around:
\n9447740753[float32 size aka 4 bytes] ~= 3.5 GB.
", "rawMarkdown": "Dimensionality reduction: The face landmarks might be total overkill, if removing them or grouping/averaging them, then that gets the dataset down to a more reasonable size. At the risk of removing even a little bit of useful data, though.\n\nIf taking the raw data without padding, it would be around:\n\n94477*40*75*3*[float32 size aka 4 bytes] ~= 3.5 GB.", "votes": 1 }, { "id": 2175334, "postDate": "2023-03-09T19:31:21.503Z", "content": "Hi @aapokossi
\nThanks a lot for your notebook. I've been using it for a while, but now it seems that some of the operations I want to do do not work on a ragged batch. I'm afraid the Dataset stuff is beyond me (and very poorly documented): do you happen to know how to adapt it to generate padded batches (i.e., each batch padded to the length of the longes sample it it)?
\nAlso, do I understand correctly that your code pre-forms batches, i.e. the batches would always be the same (but in a random order) during training?
", "rawMarkdown": "Hi @aapokossi \n\nThanks a lot for your notebook. I've been using it for a while, but now it seems that some of the operations I want to do do not work on a ragged batch. I'm afraid the Dataset stuff is beyond me (and very poorly documented): do you happen to know how to adapt it to generate padded batches (i.e., each batch padded to the length of the longes sample it it)?\n\nAlso, do I understand correctly that your code pre-forms batches, i.e. the batches would always be the same (but in a random order) during training?\n\n\n" }, { "id": 2175491, "postDate": "2023-03-09T21:57:09.763Z", "content": "Yes, you are correct that the batches will always be the same if you don't specifically unbatch and then shuffle them. You can generate padded batches for example with the following transformation:
\nds = ds.map(lambda x, y: (x.to_tensor(), y)
\nRagged tensors have the to_tensor method which pads ragged dimensions to the longest dimension in the tensor (or optionally some static shape which may truncate the dim size).
\nAdditionally, tf.keras.layers.Masking() may or may not be useful when you are using padding, depending on your model architecture and if your layers support masking.
I have now published a tf.Dataset that quite significantly speeds up loading for me (700 it/sec vs. 60 it/sec originally, with no batching), along with a notebook on how to easily save the data yourself, which should make it easier for anyone to apply their own desired level of preprocessing before saving, if you want even more performance. I myself will probably be using a variation that is padded to a specific sequence length and pre-batched.
", "votes": 2, "replies": [] }, { "id": 2162219, "author_name": "Shawn", "author_url": "", "post_date": "2023-02-28T04:10:16.230000", "content": "I'm quite new to TF and Kaggle competitions, but taking reference some of the codes shared (shoutout to @lonnieqin), I believe the competition has specified a method they will use to load the data, available in the Evaluation page. I'm guessing we can adopt the same method to load the data, then incorporate any further processing into your model pipeline.
\n ():\n data_columns = [, , ]\n data = pd.read_parquet(pq_path, columns=data_columns)\n n_frames = ((data) / ROWS_PER_FRAME)\n data = data.values.reshape(n_frames, ROWS_PER_FRAME, (data_columns))\n data.astype(np.float32)\nThis only loads a single sample from disk. What I'm asking for is how to incorporate this into a Tf Dataset object to automatically load batches from disk during training. There is standard code for standard data formats (e.g. images), but writing this from scratch requires in-depth understanding of Tensorflow.
\nBased on the leaderboard, I's assuming that at least some people must have managed to solve this …
", "votes": 0, "replies": [] } ] }, { "id": 2161922, "author_name": "Wondering Alice", "author_url": "", "post_date": "2023-02-27T20:36:11.827000", "content": "I have the same question. I've been browsing through multiple explanations about creating a Tf Dataset from parquet files, but this is way above my level of understanding. When I try pre-padding the data for training as a time-series model, the data becomes WAY too large, e.g. window size of 30 -> 94477x30x543x3 numbers
\nActually, since this is really essential for proper training here, and since documentation for this is really lacking, I feel the Tensorflow development team should provide example code for this. Otherwise, anyone who is not a really advanced Tf software engineer is excluded from properly participating because of getting stuck on something I'm sure the organisers have already solved.
\nSo please, can someone provide example code for a (train and val) Dataset with batch prefetching that can be used for efficient training, so we can focus on the model design.
", "votes": 0, "replies": [ { "id": 2162762, "author_name": "Darien Schettler", "author_url": "", "post_date": "2023-02-28T11:58:55.640000", "content": "I'm not sure about your data dimension here.
\nThis is indeed pretty big though.
", "votes": 0, "replies": [] } ] }, { "id": 2160746, "author_name": "Darien Schettler", "author_url": "", "post_date": "2023-02-27T00:12:00.680000", "content": "Haven’t tried it but maybe this:
\nhttps://www.tensorflow.org/io/api_docs/python/tfio/experimental/IODataset#from_parquet
It’s in experimental so maybe the issue your having has been fixed?
\nI’ll dig more into this tomorrow.
", "votes": 0, "replies": [ { "id": 2162411, "author_name": "Wondering Alice", "author_url": "", "post_date": "2023-02-28T07:24:05.443000", "content": "Eager to hear whether you've made any progress ??
", "votes": 0, "replies": [ { "id": 2162720, "author_name": "Darien Schettler", "author_url": "", "post_date": "2023-02-28T11:32:42.310000", "content": "No, I have the same issue as you where it crashes when I try to load it.
\nI am in the process of making TFRecords for efficient data loading. I'll share the notebook shortly and you can use that to create the dataset that reflects what you want.
\nSome Random Notes:
\nI'll include everything I can in my notebook.
", "votes": 1, "replies": [] }, { "id": 2163027, "author_name": "", "author_url": "", "post_date": "2023-02-28T14:43:06.747000", "content": "", "votes": 0, "replies": [] }, { "id": 2163029, "author_name": "aapo kossi", "author_url": "", "post_date": "2023-02-28T14:43:43.217000", "content": "I prefer the tf.data aligned Dataset.save compared to TFRecords, but I like that we will now have multiple alternatives! You can see my top level comment for a saved tf.Dataset and notebook on how to save your own if you want to try with tf.Datasets without the intermediate step of TFRecords.
", "votes": 0, "replies": [] }, { "id": 2163421, "author_name": "Robert Hatch", "author_url": "", "post_date": "2023-02-28T19:47:38.763000", "content": "Dimensionality reduction: The face landmarks might be total overkill, if removing them or grouping/averaging them, then that gets the dataset down to a more reasonable size. At the risk of removing even a little bit of useful data, though.
\nIf taking the raw data without padding, it would be around:
\n9447740753[float32 size aka 4 bytes] ~= 3.5 GB.
", "votes": 1, "replies": [] }, { "id": 2175334, "author_name": "Wondering Alice", "author_url": "", "post_date": "2023-03-09T19:31:21.503000", "content": "Hi @aapokossi
\nThanks a lot for your notebook. I've been using it for a while, but now it seems that some of the operations I want to do do not work on a ragged batch. I'm afraid the Dataset stuff is beyond me (and very poorly documented): do you happen to know how to adapt it to generate padded batches (i.e., each batch padded to the length of the longes sample it it)?
\nAlso, do I understand correctly that your code pre-forms batches, i.e. the batches would always be the same (but in a random order) during training?
", "votes": 0, "replies": [] }, { "id": 2175491, "author_name": "aapo kossi", "author_url": "", "post_date": "2023-03-09T21:57:09.763000", "content": "Yes, you are correct that the batches will always be the same if you don't specifically unbatch and then shuffle them. You can generate padded batches for example with the following transformation:
\nds = ds.map(lambda x, y: (x.to_tensor(), y)
\nRagged tensors have the to_tensor method which pads ragged dimensions to the longest dimension in the tensor (or optionally some static shape which may truncate the dim size).
\nAdditionally, tf.keras.layers.Masking() may or may not be useful when you are using padding, depending on your model architecture and if your layers support masking.