As always appreciate all the great materials & resources available on Kaggle just wonder about the input format.
\n\nI'm not familiar with TFRecords and the tutorial seems to kind of skip over the input with a helper function. I know the point of this exercise is TPUs, but I don't like running this without any understanding of the data input and loading.
\n\nI've really struggled with the documentation for these - was looking for some simple examples of saving/loading things like 2d numpy arrays, and probably I'm just being thick but I've struggled to adapt the examples provided on Tensorflow docs and stackoverflow and get them working for other data sets, which I think is a good measure of understanding them.
", "messages": [ { "id": 927009, "postDate": "2020-07-13T06:36:18.627Z", "content": "As always appreciate all the great materials & resources available on Kaggle just wonder about the input format.
\n\nI'm not familiar with TFRecords and the tutorial seems to kind of skip over the input with a helper function. I know the point of this exercise is TPUs, but I don't like running this without any understanding of the data input and loading.
\n\nI've really struggled with the documentation for these - was looking for some simple examples of saving/loading things like 2d numpy arrays, and probably I'm just being thick but I've struggled to adapt the examples provided on Tensorflow docs and stackoverflow and get them working for other data sets, which I think is a good measure of understanding them.
", "rawMarkdown": "As always appreciate all the great materials & resources available on Kaggle just wonder about the input format.\n\nI'm not familiar with TFRecords and the tutorial seems to kind of skip over the input with a helper function. I know the point of this exercise is TPUs, but I don't like running this without any understanding of the data input and loading.\n\nI've really struggled with the documentation for these - was looking for some simple examples of saving/loading things like 2d numpy arrays, and probably I'm just being thick but I've struggled to adapt the examples provided on Tensorflow docs and stackoverflow and get them working for other data sets, which I think is a good measure of understanding them. \n\n", "votes": 4 }, { "id": 938411, "postDate": "2020-07-21T14:05:28.383Z", "content": "Hi Dave!
\nI wrote up a small tutorial on TFRecords here: TFRecords Basics. It doesn't go over everything about constructing datasets for TPUs, but hopefully it clarifies a few things. Let me know if you have any questions!
", "rawMarkdown": "Hi Dave!\n\nI wrote up a small tutorial on TFRecords here: [TFRecords Basics](https://www.kaggle.com/ryanholbrook/tfrecords-basics). It doesn't go over everything about constructing datasets for TPUs, but hopefully it clarifies a few things. Let me know if you have any questions!", "votes": 1, "replies": [ { "id": 939809, "postDate": "2020-07-22T13:27:40.957Z", "content": "Thank you, that is definitely appreciated. I think that for now I've made some progress and can manage some saving/reading from TFRecords but if I get really stuck again then I might post here! Am sure others will appreciate the help also. Thanks again.
", "rawMarkdown": "Thank you, that is definitely appreciated. I think that for now I've made some progress and can manage some saving/reading from TFRecords but if I get really stuck again then I might post here! Am sure others will appreciate the help also. Thanks again.", "votes": 1 } ] }, { "id": 933709, "postDate": "2020-07-17T23:58:18.960Z", "content": "I....kind of....got to grips with some of the Tensorflow TfRecords documentation. Thanks also to Chris Deotte's excellent notebook in this comp - going through the code there gave me some clues, though you need the TF docs as well I think. Maybe struggled more because I am 100% used to pandas, numpy, csv style data where you load a dataframe and can inspect first few rows, columns etc. Also am not sure that the documentation is very user friendly - there are some limited examples of actual usage but it's not that extensive, and it didn't seem all that easy to me to figure out the cause when you change or build on these examples and it inevitably 'breaks'.
\n\nI still don't really know what I'm doing but I did manage to write some numpy arrays (from 2d images) and labels to TfRecords, save and open again and run in a keras model in another notebook. It took me a day+ and left me with a headache to get to the save/load/use stage.
\n\nDoes anyone on the forum have any better explained (think it needs line by line!) / clearer examples to share? I can share the save/load operations I managed to put together with my understanding of how it works but as I'm a newbie I may get stuff wrong!
", "rawMarkdown": "I....kind of....got to grips with some of the Tensorflow TfRecords documentation. Thanks also to Chris Deotte's excellent notebook in this comp - going through the code there gave me some clues, though you need the TF docs as well I think. Maybe struggled more because I am 100% used to pandas, numpy, csv style data where you load a dataframe and can inspect first few rows, columns etc. Also am not sure that the documentation is very user friendly - there are some limited examples of actual usage but it's not that extensive, and it didn't seem all that easy to me to figure out the cause when you change or build on these examples and it inevitably 'breaks'. \n\nI still don't really know what I'm doing but I did manage to write some numpy arrays (from 2d images) and labels to TfRecords, save and open again and run in a keras model in another notebook. It took me a day+ and left me with a headache to get to the save/load/use stage.\n\nDoes anyone on the forum have any better explained (think it needs line by line!) / clearer examples to share? I can share the save/load operations I managed to put together with my understanding of how it works but as I'm a newbie I may get stuff wrong!", "votes": 1 }, { "id": 971096, "postDate": "2020-08-15T07:17:45.223Z", "content": "l have written this blog How to read tfrecords files in PyTorch !
\nPlease go through this and let me know if anyone has any questions , comments or, concerns regarding this. I would love to help them out.
Keras with tfrecords\nRefer to this and you will get an elaborative idea how to convert, load, and train the data.
", "rawMarkdown": "[Keras with tfrecords](https://medium.com/@moritzkrger/speeding-up-keras-with-tfrecord-datasets-5464f9836c36)\nRefer to this and you will get an elaborative idea how to convert, load, and train the data." }, { "id": 932569, "postDate": "2020-07-17T06:38:46.607Z", "content": "Hi. I am also facing the same issue. Can anyone help?
", "rawMarkdown": "Hi. I am also facing the same issue. Can anyone help?" } ], "comments": [ { "id": 938411, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2020-07-21T14:05:28.383000", "content": "Hi Dave!
\nI wrote up a small tutorial on TFRecords here: TFRecords Basics. It doesn't go over everything about constructing datasets for TPUs, but hopefully it clarifies a few things. Let me know if you have any questions!
", "votes": 1, "replies": [ { "id": 939809, "author_name": "Dave E", "author_url": "", "post_date": "2020-07-22T13:27:40.957000", "content": "Thank you, that is definitely appreciated. I think that for now I've made some progress and can manage some saving/reading from TFRecords but if I get really stuck again then I might post here! Am sure others will appreciate the help also. Thanks again.
", "votes": 1, "replies": [] } ] }, { "id": 933709, "author_name": "Dave E", "author_url": "", "post_date": "2020-07-17T23:58:18.960000", "content": "I....kind of....got to grips with some of the Tensorflow TfRecords documentation. Thanks also to Chris Deotte's excellent notebook in this comp - going through the code there gave me some clues, though you need the TF docs as well I think. Maybe struggled more because I am 100% used to pandas, numpy, csv style data where you load a dataframe and can inspect first few rows, columns etc. Also am not sure that the documentation is very user friendly - there are some limited examples of actual usage but it's not that extensive, and it didn't seem all that easy to me to figure out the cause when you change or build on these examples and it inevitably 'breaks'.
\n\nI still don't really know what I'm doing but I did manage to write some numpy arrays (from 2d images) and labels to TfRecords, save and open again and run in a keras model in another notebook. It took me a day+ and left me with a headache to get to the save/load/use stage.
\n\nDoes anyone on the forum have any better explained (think it needs line by line!) / clearer examples to share? I can share the save/load operations I managed to put together with my understanding of how it works but as I'm a newbie I may get stuff wrong!
", "votes": 1, "replies": [] }, { "id": 971096, "author_name": "IamSparky", "author_url": "", "post_date": "2020-08-15T07:17:45.223000", "content": "l have written this blog How to read tfrecords files in PyTorch !
\nPlease go through this and let me know if anyone has any questions , comments or, concerns regarding this. I would love to help them out.
Keras with tfrecords\nRefer to this and you will get an elaborative idea how to convert, load, and train the data.
", "votes": 0, "replies": [] }, { "id": 932569, "author_name": "Bhagyesh Agresar", "author_url": "", "post_date": "2020-07-17T06:38:46.607000", "content": "Hi. I am also facing the same issue. Can anyone help?
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "927009": "As always appreciate all the great materials & resources available on Kaggle just wonder about the input format.\n\nI'm not familiar with TFRecords and the tutorial seems to kind of skip over the input with a helper function. I know the point of this exercise is TPUs, but I don't like running this without any understanding of the data input and loading.\n\nI've really struggled with the documentation for these - was looking for some simple examples of saving/loading things like 2d numpy arrays, and probably I'm just being thick but I've struggled to adapt the examples provided on Tensorflow docs and stackoverflow and get them working for other data sets, which I think is a good measure of understanding them. \n\n", "938411": "Hi Dave!\n\nI wrote up a small tutorial on TFRecords here: [TFRecords Basics](https://www.kaggle.com/ryanholbrook/tfrecords-basics). It doesn't go over everything about constructing datasets for TPUs, but hopefully it clarifies a few things. Let me know if you have any questions!", "933709": "I....kind of....got to grips with some of the Tensorflow TfRecords documentation. Thanks also to Chris Deotte's excellent notebook in this comp - going through the code there gave me some clues, though you need the TF docs as well I think. Maybe struggled more because I am 100% used to pandas, numpy, csv style data where you load a dataframe and can inspect first few rows, columns etc. Also am not sure that the documentation is very user friendly - there are some limited examples of actual usage but it's not that extensive, and it didn't seem all that easy to me to figure out the cause when you change or build on these examples and it inevitably 'breaks'. \n\nI still don't really know what I'm doing but I did manage to write some numpy arrays (from 2d images) and labels to TfRecords, save and open again and run in a keras model in another notebook. It took me a day+ and left me with a headache to get to the save/load/use stage.\n\nDoes anyone on the forum have any better explained (think it needs line by line!) / clearer examples to share? I can share the save/load operations I managed to put together with my understanding of how it works but as I'm a newbie I may get stuff wrong!", "971096": "l have written this blog [How to read tfrecords files in PyTorch !](https://medium.com/analytics-vidhya/how-to-read-tfrecords-files-in-pytorch-72763786743f?source=friends_link&sk=9a55feea4d3fc7272da12a4a962db39d) \nPlease go through this and let me know if anyone has any questions , comments or, concerns regarding this. I would love to help them out.", "951990": "[Keras with tfrecords](https://medium.com/@moritzkrger/speeding-up-keras-with-tfrecord-datasets-5464f9836c36)\nRefer to this and you will get an elaborative idea how to convert, load, and train the data.", "932569": "Hi. I am also facing the same issue. Can anyone help?" } }