Hi All,\nI tried to fit the training dataset to my model and as expected it failed. \nI read somewhere that 'multiprocessing' helps in dealing with hugesets of data.\nAnyone tried it??
", "messages": [ { "id": "224759", "postDate": "09/27/2017 15:38:51", "content": "Hi All,\nI tried to fit the training dataset to my model and as expected it failed. \nI read somewhere that 'multiprocessing' helps in dealing with hugesets of data.\nAnyone tried it??
", "rawMarkdown": "Hi All,\nI tried to fit the training dataset to my model and as expected it failed. \nI read somewhere that 'multiprocessing' helps in dealing with hugesets of data.\nAnyone tried it??", "votes": null }, { "id": "224812", "postDate": "09/27/2017 18:09:38", "content": "What do you mean it failed? Did you get an out-of-memory error? The dataset is too large to load into memory at once, so you'll have to process it in smaller chunks.
", "rawMarkdown": "What do you mean it failed? Did you get an out-of-memory error? The dataset is too large to load into memory at once, so you'll have to process it in smaller chunks.", "votes": null }, { "id": "224977", "postDate": "09/28/2017 02:14:10", "content": "If tensorflow OOM error, reduce your batch size.\nFYI: you should use streaming method to read data
", "rawMarkdown": "If tensorflow OOM error, reduce your batch size.\nFYI: you should use streaming method to read data", "votes": null }, { "id": "225222", "postDate": "09/28/2017 14:51:11", "content": "Yes.It gave a OOM error. I understand that I have to break it down.
", "rawMarkdown": "Yes.It gave a OOM error. I understand that I have to break it down.", "votes": null }, { "id": "225608", "postDate": "09/29/2017 15:59:01", "content": "To be honest, I just asked myself this question... Should data in a batch be balanced? feeded into training in same proportion in relation to classes. Or random is enough? I guess imagenet paperes where made with purely random. \nThough there might be dependence, which is very time-consuming to explore.
", "rawMarkdown": "To be honest, I just asked myself this question... Should data in a batch be balanced? feeded into training in same proportion in relation to classes. Or random is enough? I guess imagenet paperes where made with purely random. \nThough there might be dependence, which is very time-consuming to explore.", "votes": null }, { "id": "225894", "postDate": "09/30/2017 09:12:27", "content": "In this case the batch will have to be bigger than 5000, because that's how many classes we have. A better solution is to shuffle your training data for each epoch. Tricky in this case.
", "rawMarkdown": "In this case the batch will have to be bigger than 5000, because that's how many classes we have. A better solution is to shuffle your training data for each epoch. Tricky in this case.", "votes": null }, { "id": "226338", "postDate": "10/01/2017 23:14:29", "content": "Just a quick question on this, I have reduced my batch size to smallest I can which is 1. I am also using a generator to stream from folder with images in it. I have been able to train my Keras NN but when I go to predict with it on the test set of images it keeps giving me OOM error. Any thoughts on why this might be, I have batch size at 1 and I would think that training is the more intensive process so if I can train I am not sure why I cannot predict. Thanks!
", "rawMarkdown": "Just a quick question on this, I have reduced my batch size to smallest I can which is 1. I am also using a generator to stream from folder with images in it. I have been able to train my Keras NN but when I go to predict with it on the test set of images it keeps giving me OOM error. Any thoughts on why this might be, I have batch size at 1 and I would think that training is the more intensive process so if I can train I am not sure why I cannot predict. Thanks!", "votes": null }, { "id": "226340", "postDate": "10/01/2017 23:18:37", "content": "I am using the Keras flow_from_directory with a batch size of 1 but I still end up getting OOM with tensorflow when trying to predict. I can train my model fine without any memory issues so I am not sure why predicting is causing trouble.
", "rawMarkdown": "I am using the Keras flow_from_directory with a batch size of 1 but I still end up getting OOM with tensorflow when trying to predict. I can train my model fine without any memory issues so I am not sure why predicting is causing trouble.", "votes": null }, { "id": "226425", "postDate": "10/02/2017 08:34:32", "content": "Sounds like a bug. Check your code!
", "rawMarkdown": "Sounds like a bug. Check your code!", "votes": null }, { "id": "226429", "postDate": "10/02/2017 08:46:11", "content": "Does TensorFlow give the OOM error or does Python? It's probably Python because you're trying to store more than 32 GB (or however much RAM you have) of data in working memory. Note that the test set has 1768182 products, times 5270 for the prediction vector, times 4 because it's floating point is ~35 GB of memory.
", "rawMarkdown": "Does TensorFlow give the OOM error or does Python? It's probably Python because you're trying to store more than 32 GB (or however much RAM you have) of data in working memory. Note that the test set has 1768182 products, times 5270 for the prediction vector, times 4 because it's floating point is ~35 GB of memory.", "votes": null } ], "comments": [ { "id": 224812, "author_name": "humananalog", "author_url": "", "post_date": "09/27/2017 18:09:38", "content": "What do you mean it failed? Did you get an out-of-memory error? The dataset is too large to load into memory at once, so you'll have to process it in smaller chunks.
", "votes": null, "replies": [ { "id": 226340, "author_name": "jasonbenner", "author_url": "", "post_date": "10/01/2017 23:18:37", "content": "I am using the Keras flow_from_directory with a batch size of 1 but I still end up getting OOM with tensorflow when trying to predict. I can train my model fine without any memory issues so I am not sure why predicting is causing trouble.
", "votes": null, "replies": [] } ] }, { "id": 224977, "author_name": "jeffzhan", "author_url": "", "post_date": "09/28/2017 02:14:10", "content": "If tensorflow OOM error, reduce your batch size.\nFYI: you should use streaming method to read data
", "votes": null, "replies": [ { "id": 226338, "author_name": "jasonbenner", "author_url": "", "post_date": "10/01/2017 23:14:29", "content": "Just a quick question on this, I have reduced my batch size to smallest I can which is 1. I am also using a generator to stream from folder with images in it. I have been able to train my Keras NN but when I go to predict with it on the test set of images it keeps giving me OOM error. Any thoughts on why this might be, I have batch size at 1 and I would think that training is the more intensive process so if I can train I am not sure why I cannot predict. Thanks!
", "votes": null, "replies": [] }, { "id": 226425, "author_name": "timjoseph", "author_url": "", "post_date": "10/02/2017 08:34:32", "content": "Sounds like a bug. Check your code!
", "votes": null, "replies": [] }, { "id": 226429, "author_name": "humananalog", "author_url": "", "post_date": "10/02/2017 08:46:11", "content": "Does TensorFlow give the OOM error or does Python? It's probably Python because you're trying to store more than 32 GB (or however much RAM you have) of data in working memory. Note that the test set has 1768182 products, times 5270 for the prediction vector, times 4 because it's floating point is ~35 GB of memory.
", "votes": null, "replies": [] } ] }, { "id": 225222, "author_name": "sasjack", "author_url": "", "post_date": "09/28/2017 14:51:11", "content": "Yes.It gave a OOM error. I understand that I have to break it down.
", "votes": null, "replies": [] }, { "id": 225608, "author_name": "", "author_url": "", "post_date": "09/29/2017 15:59:01", "content": "To be honest, I just asked myself this question... Should data in a batch be balanced? feeded into training in same proportion in relation to classes. Or random is enough? I guess imagenet paperes where made with purely random. \nThough there might be dependence, which is very time-consuming to explore.
", "votes": null, "replies": [ { "id": 225894, "author_name": "ezietsman", "author_url": "", "post_date": "09/30/2017 09:12:27", "content": "In this case the batch will have to be bigger than 5000, because that's how many classes we have. A better solution is to shuffle your training data for each epoch. Tricky in this case.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "224759": "Hi All,\nI tried to fit the training dataset to my model and as expected it failed. \nI read somewhere that 'multiprocessing' helps in dealing with hugesets of data.\nAnyone tried it??", "224812": "What do you mean it failed? Did you get an out-of-memory error? The dataset is too large to load into memory at once, so you'll have to process it in smaller chunks.", "224977": "If tensorflow OOM error, reduce your batch size.\nFYI: you should use streaming method to read data", "225222": "Yes.It gave a OOM error. I understand that I have to break it down.", "225608": "To be honest, I just asked myself this question... Should data in a batch be balanced? feeded into training in same proportion in relation to classes. Or random is enough? I guess imagenet paperes where made with purely random. \nThough there might be dependence, which is very time-consuming to explore.", "225894": "In this case the batch will have to be bigger than 5000, because that's how many classes we have. A better solution is to shuffle your training data for each epoch. Tricky in this case.", "226338": "Just a quick question on this, I have reduced my batch size to smallest I can which is 1. I am also using a generator to stream from folder with images in it. I have been able to train my Keras NN but when I go to predict with it on the test set of images it keeps giving me OOM error. Any thoughts on why this might be, I have batch size at 1 and I would think that training is the more intensive process so if I can train I am not sure why I cannot predict. Thanks!", "226340": "I am using the Keras flow_from_directory with a batch size of 1 but I still end up getting OOM with tensorflow when trying to predict. I can train my model fine without any memory issues so I am not sure why predicting is causing trouble.", "226425": "Sounds like a bug. Check your code!", "226429": "Does TensorFlow give the OOM error or does Python? It's probably Python because you're trying to store more than 32 GB (or however much RAM you have) of data in working memory. Note that the test set has 1768182 products, times 5270 for the prediction vector, times 4 because it's floating point is ~35 GB of memory." }, "source": "meta" }