Hi, I knew TFRecord Dataset in this competition.\nIt is a new type of dataset module for me. It's image data, but stored as string. \nLooking at docs, it said it's good for serialization. The reason is they divided the data into 100 to 200MB?\nWhat are the advantages of TFRecord, unlike Pytorch's dataloader module?
", "messages": [ { "id": "919877", "postDate": "07/08/2020 06:46:14", "content": "Hi, I knew TFRecord Dataset in this competition.\nIt is a new type of dataset module for me. It's image data, but stored as string. \nLooking at docs, it said it's good for serialization. The reason is they divided the data into 100 to 200MB?\nWhat are the advantages of TFRecord, unlike Pytorch's dataloader module?
", "rawMarkdown": "Hi, I knew TFRecord Dataset in this competition.\nIt is a new type of dataset module for me. It's image data, but stored as string. \nLooking at docs, it said it's good for serialization. The reason is they divided the data into 100 to 200MB?\nWhat are the advantages of TFRecord, unlike Pytorch's dataloader module?", "votes": null }, { "id": "919940", "postDate": "07/08/2020 07:57:11", "content": "Hi, \nfrom my understanding, these are the benefits of it\n1. Reading each individual image in the large dataset is often a slow process. So, to make it fast we store many images as tensor Tf records and read from there. \n2. At a time, we can feed data from many TF records to TPU for training without worrying about the order. This will make the data feeding process fast and hence improve the overall run time.
\n\nThese are the obvious one I know. There might be other benefits too.
", "rawMarkdown": "Hi, \nfrom my understanding, these are the benefits of it\n1. Reading each individual image in the large dataset is often a slow process. So, to make it fast we store many images as tensor Tf records and read from there. \n2. At a time, we can feed data from many TF records to TPU for training without worrying about the order. This will make the data feeding process fast and hence improve the overall run time. \n\nThese are the obvious one I know. There might be other benefits too.", "votes": null }, { "id": "920205", "postDate": "07/08/2020 12:42:19", "content": "Thanks a lot! Now it's clear to me
", "rawMarkdown": "Thanks a lot! Now it's clear to me", "votes": null }, { "id": "920559", "postDate": "07/08/2020 17:09:48", "content": "its clear thanks @urvishp80
", "rawMarkdown": "its clear thanks @urvishp80", "votes": null }, { "id": "920571", "postDate": "07/08/2020 17:16:47", "content": "Happy to help.
", "rawMarkdown": "Happy to help.", "votes": null }, { "id": "920605", "postDate": "07/08/2020 17:50:25", "content": "100-200mb is the recommended size in the TFRecord article on Tensorflow. see https://www.tensorflow.org/tutorials/load_data/tfrecord#walkthrough_reading_and_writing_image_data
\n\nI believe any size greater than 100mb is fine. This is because it takes time for the TPU to make a request for a file.
\n\nFor example, lets say you have 10k image files. Each time the TPU needs a new image, it has to make a request for that image file so it is making 10k requests to get data. That can really slow you down because it has to wait for the request to be filled by google cloud storage.
", "rawMarkdown": "100-200mb is the recommended size in the TFRecord article on Tensorflow. see https://www.tensorflow.org/tutorials/load_data/tfrecord#walkthrough_reading_and_writing_image_data\n\nI believe any size greater than 100mb is fine. This is because it takes time for the TPU to make a request for a file. \n\nFor example, lets say you have 10k image files. Each time the TPU needs a new image, it has to make a request for that image file so it is making 10k requests to get data. That can really slow you down because it has to wait for the request to be filled by google cloud storage.", "votes": null }, { "id": "920667", "postDate": "07/08/2020 18:38:25", "content": "Thanks for the insights @hooong \nIt makes sense. We can increase performance by reducing the number of requests.
", "rawMarkdown": "Thanks for the insights @hooong \nIt makes sense. We can increase performance by reducing the number of requests.", "votes": null } ], "comments": [ { "id": 919940, "author_name": "urvishp80", "author_url": "", "post_date": "07/08/2020 07:57:11", "content": "Hi, \nfrom my understanding, these are the benefits of it\n1. Reading each individual image in the large dataset is often a slow process. So, to make it fast we store many images as tensor Tf records and read from there. \n2. At a time, we can feed data from many TF records to TPU for training without worrying about the order. This will make the data feeding process fast and hence improve the overall run time.
\n\nThese are the obvious one I know. There might be other benefits too.
", "votes": null, "replies": [ { "id": 920205, "author_name": "telljoy", "author_url": "", "post_date": "07/08/2020 12:42:19", "content": "Thanks a lot! Now it's clear to me
", "votes": null, "replies": [] }, { "id": 920559, "author_name": "vishnurapps", "author_url": "", "post_date": "07/08/2020 17:09:48", "content": "its clear thanks @urvishp80
", "votes": null, "replies": [] }, { "id": 920571, "author_name": "urvishp80", "author_url": "", "post_date": "07/08/2020 17:16:47", "content": "Happy to help.
", "votes": null, "replies": [] }, { "id": 920605, "author_name": "hooong", "author_url": "", "post_date": "07/08/2020 17:50:25", "content": "100-200mb is the recommended size in the TFRecord article on Tensorflow. see https://www.tensorflow.org/tutorials/load_data/tfrecord#walkthrough_reading_and_writing_image_data
\n\nI believe any size greater than 100mb is fine. This is because it takes time for the TPU to make a request for a file.
\n\nFor example, lets say you have 10k image files. Each time the TPU needs a new image, it has to make a request for that image file so it is making 10k requests to get data. That can really slow you down because it has to wait for the request to be filled by google cloud storage.
", "votes": null, "replies": [] }, { "id": 920667, "author_name": "vishnurapps", "author_url": "", "post_date": "07/08/2020 18:38:25", "content": "Thanks for the insights @hooong \nIt makes sense. We can increase performance by reducing the number of requests.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "919877": "Hi, I knew TFRecord Dataset in this competition.\nIt is a new type of dataset module for me. It's image data, but stored as string. \nLooking at docs, it said it's good for serialization. The reason is they divided the data into 100 to 200MB?\nWhat are the advantages of TFRecord, unlike Pytorch's dataloader module?", "919940": "Hi, \nfrom my understanding, these are the benefits of it\n1. Reading each individual image in the large dataset is often a slow process. So, to make it fast we store many images as tensor Tf records and read from there. \n2. At a time, we can feed data from many TF records to TPU for training without worrying about the order. This will make the data feeding process fast and hence improve the overall run time. \n\nThese are the obvious one I know. There might be other benefits too.", "920205": "Thanks a lot! Now it's clear to me", "920559": "its clear thanks @urvishp80", "920571": "Happy to help.", "920605": "100-200mb is the recommended size in the TFRecord article on Tensorflow. see https://www.tensorflow.org/tutorials/load_data/tfrecord#walkthrough_reading_and_writing_image_data\n\nI believe any size greater than 100mb is fine. This is because it takes time for the TPU to make a request for a file. \n\nFor example, lets say you have 10k image files. Each time the TPU needs a new image, it has to make a request for that image file so it is making 10k requests to get data. That can really slow you down because it has to wait for the request to be filled by google cloud storage.", "920667": "Thanks for the insights @hooong \nIt makes sense. We can increase performance by reducing the number of requests." }, "source": "meta" }