What's the best way to train a network if we can't fit the entire training set into memory?
\n\nSplit the training set into chunks, and retrain the network on each several times? Or train on each chunk for several epochs then move onto another one? Or is there a better way?
", "messages": [ { "id": "230325", "postDate": "10/11/2017 18:13:38", "content": "What's the best way to train a network if we can't fit the entire training set into memory?
\n\nSplit the training set into chunks, and retrain the network on each several times? Or train on each chunk for several epochs then move onto another one? Or is there a better way?
", "rawMarkdown": "What's the best way to train a network if we can't fit the entire training set into memory?\n\nSplit the training set into chunks, and retrain the network on each several times? Or train on each chunk for several epochs then move onto another one? Or is there a better way?", "votes": null }, { "id": "230328", "postDate": "10/11/2017 18:22:22", "content": "I assume you're using a GPU for the actual training. You can never fit the entire training set on GPU memory anyway, only small batches of between 32 and 256 images. So there isn't really a need to fit the entire training set on the CPU either.
\n\nYou can load images from disk to fill up a batch and send this batch to the GPU. As long as creating the batches is faster than the GPU pass, this is fine. (Note that loading images from a hard disk may actually be too slow, which is why it's recommended to use an SSD for storing your training data.)
", "rawMarkdown": "I assume you're using a GPU for the actual training. You can never fit the entire training set on GPU memory anyway, only small batches of between 32 and 256 images. So there isn't really a need to fit the entire training set on the CPU either. \n\nYou can load images from disk to fill up a batch and send this batch to the GPU. As long as creating the batches is faster than the GPU pass, this is fine. (Note that loading images from a hard disk may actually be too slow, which is why it's recommended to use an SSD for storing your training data.)", "votes": null }, { "id": "232370", "postDate": "10/17/2017 12:09:33", "content": "Take a look at this and this. The first one indexes the image file and feed the images in batches with random access directly from original BSON files. The second one, uses the same indexes of the first but create separate binary files for train and validation. This approach eliminates the need of random access, thus improving access time, with the drawback of using more disk space.
", "rawMarkdown": "Take a look at [this][1] and [this][2]. The first one indexes the image file and feed the images in batches with random access directly from original BSON files. The second one, uses the same indexes of the first but create separate binary files for train and validation. This approach eliminates the need of random access, thus improving access time, with the drawback of using more disk space. \n\n\n [1]: https://www.kaggle.com/humananalog/keras-generator-for-reading-directly-from-bson\n [2]: https://www.kaggle.com/aloisiodn/fast-thread-safe-keras-generator-from-bin-files", "votes": null } ], "comments": [ { "id": 230328, "author_name": "humananalog", "author_url": "", "post_date": "10/11/2017 18:22:22", "content": "I assume you're using a GPU for the actual training. You can never fit the entire training set on GPU memory anyway, only small batches of between 32 and 256 images. So there isn't really a need to fit the entire training set on the CPU either.
\n\nYou can load images from disk to fill up a batch and send this batch to the GPU. As long as creating the batches is faster than the GPU pass, this is fine. (Note that loading images from a hard disk may actually be too slow, which is why it's recommended to use an SSD for storing your training data.)
", "votes": null, "replies": [] }, { "id": 232370, "author_name": "aloisiodn", "author_url": "", "post_date": "10/17/2017 12:09:33", "content": "Take a look at this and this. The first one indexes the image file and feed the images in batches with random access directly from original BSON files. The second one, uses the same indexes of the first but create separate binary files for train and validation. This approach eliminates the need of random access, thus improving access time, with the drawback of using more disk space.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "230325": "What's the best way to train a network if we can't fit the entire training set into memory?\n\nSplit the training set into chunks, and retrain the network on each several times? Or train on each chunk for several epochs then move onto another one? Or is there a better way?", "230328": "I assume you're using a GPU for the actual training. You can never fit the entire training set on GPU memory anyway, only small batches of between 32 and 256 images. So there isn't really a need to fit the entire training set on the CPU either. \n\nYou can load images from disk to fill up a batch and send this batch to the GPU. As long as creating the batches is faster than the GPU pass, this is fine. (Note that loading images from a hard disk may actually be too slow, which is why it's recommended to use an SSD for storing your training data.)", "232370": "Take a look at [this][1] and [this][2]. The first one indexes the image file and feed the images in batches with random access directly from original BSON files. The second one, uses the same indexes of the first but create separate binary files for train and validation. This approach eliminates the need of random access, thus improving access time, with the drawback of using more disk space. \n\n\n [1]: https://www.kaggle.com/humananalog/keras-generator-for-reading-directly-from-bson\n [2]: https://www.kaggle.com/aloisiodn/fast-thread-safe-keras-generator-from-bin-files" }, "source": "meta" }