I've seen implementations where people train multiple epochs on one of the parquet files, then they continue training on the next file and so on. I think this could really spoil the training as what you're effectively doing is training a model on a smaller amount of data than available, and then doing further training with a different dataset and so on.
\n\nI was looking into ways of loading rows from a csv on the fly (in Python using yield to create a generator), and I have two questions, the first being most important\n1) I'm guessing this will be the bottleneck in the speed. Although one epoch takes a while for training, probably it takes longer to load all the data.\n2) If I'm wrong with 1, then is there a way to load parquet data on the fly?
Bonus) Stepping outside the box which I've constricted this question into, what are some approaches you've tried to mitigate the original problem I proposed?
", "messages": [ { "id": "736947", "postDate": "02/04/2020 18:22:29", "content": "I've seen implementations where people train multiple epochs on one of the parquet files, then they continue training on the next file and so on. I think this could really spoil the training as what you're effectively doing is training a model on a smaller amount of data than available, and then doing further training with a different dataset and so on.
\n\nI was looking into ways of loading rows from a csv on the fly (in Python using yield to create a generator), and I have two questions, the first being most important\n1) I'm guessing this will be the bottleneck in the speed. Although one epoch takes a while for training, probably it takes longer to load all the data.\n2) If I'm wrong with 1, then is there a way to load parquet data on the fly?
Bonus) Stepping outside the box which I've constricted this question into, what are some approaches you've tried to mitigate the original problem I proposed?
", "rawMarkdown": "I've seen implementations where people train multiple epochs on one of the parquet files, then they continue training on the next file and so on. I think this could really spoil the training as what you're effectively doing is training a model on a smaller amount of data than available, and then doing further training with a different dataset and so on.\n\nI was looking into ways of loading rows from a csv on the fly (in Python using `yield` to create a generator), and I have two questions, the first being most important\n1) I'm guessing this will be the bottleneck in the speed. Although one epoch takes a while for training, probably it takes longer to load all the data.\n2) If I'm wrong with 1, then is there a way to load parquet data on the fly?\n\nBonus) Stepping outside the box which I've constricted this question into, what are some approaches you've tried to mitigate the original problem I proposed?", "votes": null }, { "id": "737077", "postDate": "02/04/2020 22:12:48", "content": "A lot of people have preprocessed the data and saved in a format where it can all be loaded into memory or split it into individual image files for a smoother flow of data. There are several different approaches if you search in the current notebooks or datasets.
", "rawMarkdown": "A lot of people have preprocessed the data and saved in a format where it can all be loaded into memory or split it into individual image files for a smoother flow of data. There are several different approaches if you search in the current notebooks or datasets.", "votes": null }, { "id": "737395", "postDate": "02/05/2020 09:46:22", "content": "Valid query. It will give better results to load all the data at once and train the model. But this will pose memory issues maybe. Could TensorFlow datasets prefetch feature be used here to optimize this. Not sure.
", "rawMarkdown": "Valid query. It will give better results to load all the data at once and train the model. But this will pose memory issues maybe. Could TensorFlow datasets prefetch feature be used here to optimize this. Not sure.", "votes": null } ], "comments": [ { "id": 737077, "author_name": "nathanpw", "author_url": "", "post_date": "02/04/2020 22:12:48", "content": "A lot of people have preprocessed the data and saved in a format where it can all be loaded into memory or split it into individual image files for a smoother flow of data. There are several different approaches if you search in the current notebooks or datasets.
", "votes": null, "replies": [] }, { "id": 737395, "author_name": "anirbank", "author_url": "", "post_date": "02/05/2020 09:46:22", "content": "Valid query. It will give better results to load all the data at once and train the model. But this will pose memory issues maybe. Could TensorFlow datasets prefetch feature be used here to optimize this. Not sure.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "736947": "I've seen implementations where people train multiple epochs on one of the parquet files, then they continue training on the next file and so on. I think this could really spoil the training as what you're effectively doing is training a model on a smaller amount of data than available, and then doing further training with a different dataset and so on.\n\nI was looking into ways of loading rows from a csv on the fly (in Python using `yield` to create a generator), and I have two questions, the first being most important\n1) I'm guessing this will be the bottleneck in the speed. Although one epoch takes a while for training, probably it takes longer to load all the data.\n2) If I'm wrong with 1, then is there a way to load parquet data on the fly?\n\nBonus) Stepping outside the box which I've constricted this question into, what are some approaches you've tried to mitigate the original problem I proposed?", "737077": "A lot of people have preprocessed the data and saved in a format where it can all be loaded into memory or split it into individual image files for a smoother flow of data. There are several different approaches if you search in the current notebooks or datasets.", "737395": "Valid query. It will give better results to load all the data at once and train the model. But this will pose memory issues maybe. Could TensorFlow datasets prefetch feature be used here to optimize this. Not sure." }, "source": "meta" }