I am training a Image classification model, and I am using Keras' fit_generator method, but it is literally taking too much time, is there any way I can reduce this training time somehow?
", "messages": [ { "id": "1272742", "postDate": "04/13/2021 18:08:59", "content": "I am training a Image classification model, and I am using Keras' fit_generator method, but it is literally taking too much time, is there any way I can reduce this training time somehow?
", "rawMarkdown": "I am training a Image classification model, and I am using Keras' fit_generator method, but it is literally taking too much time, is there any way I can reduce this training time somehow?", "votes": null }, { "id": "1274949", "postDate": "04/15/2021 18:35:02", "content": "Hi Debjyoti Banerjee,
\nThe 'fit_generator' method is deprecated and you can use the 'fit'.
\nMaybe it can be part of the solucion to speed things up to you.
\nRegards
Thanks for suggesting, but the reason why I am using fit_generator is because, if I use fit method, then it will load all the images on RAM, and may be some memory error may come up, but if you can suggest me some other method, then it would be really helpful
", "rawMarkdown": "Thanks for suggesting, but the reason why I am using fit_generator is because, if I use fit method, then it will load all the images on RAM, and may be some memory error may come up, but if you can suggest me some other method, then it would be really helpful", "votes": null }, { "id": "1275129", "postDate": "04/16/2021 01:12:32", "content": "I am also struggling to speed things up without pre-trained network, as a knowledge approach.
\nI tried 3 methods:
\n1 - dataset using 'images_from_directory', but it could not find the image files. I don't know why but I think is due to filenames starting with number. (If you know how to solve it, I do appreciate)
\n2 - dataset using map mode. Build a dataset with ( 'path-to-file', 'label'), then map 'path-to-file' to image array.
\n3 - using generator and fit method. (not fit_generator)
\nAll of them seems to be quite similar in time, even in Kaggle's GPUs.
\nFor datasets, I found this tutorial in Tensorflow website. Check the 'Configure the dataset for performance' topic.
\nUse cache and prefetch. The first epoch will last long. From the 2nd it may be faster. For me, shuffle seems to slow down.
\nI hope it helps
", "rawMarkdown": "I am also struggling to speed things up without pre-trained network, as a knowledge approach.\n\nI tried 3 methods:\n\n1 - dataset using 'images_from_directory', but it could not find the image files. I don't know why but I think is due to filenames starting with number. (If you know how to solve it, I do appreciate)\n\n2 - dataset using map mode. Build a dataset with ( 'path-to-file', 'label'), then map 'path-to-file' to image array.\n\n3 - using generator and fit method. (not fit_generator) \n\nAll of them seems to be quite similar in time, even in Kaggle's GPUs.\n\nFor datasets, I found [this tutorial ](https://www.tensorflow.org/tutorials/images/classification) in Tensorflow website. Check the 'Configure the dataset for performance' topic. \n\nUse cache and prefetch. The first epoch will last long. From the 2nd it may be faster. For me, shuffle seems to slow down.\n\nI hope it helps", "votes": null }, { "id": "1275432", "postDate": "04/16/2021 09:49:45", "content": "Try using pytorch, I guess there is no other option other than that, I am also going to use the same
", "rawMarkdown": "Try using pytorch, I guess there is no other option other than that, I am also going to use the same", "votes": null }, { "id": "1278297", "postDate": "04/19/2021 18:05:00", "content": "https://wrosinski.github.io/deep-learning-frameworks/
\nHere it is shown that models based on tensorflow are much faster compared to keras and pytorch.
Hi Debjyoti Banerjee,
\nThe 'fit_generator' method is deprecated and you can use the 'fit'.
\nMaybe it can be part of the solucion to speed things up to you.
\nRegards
Thanks for suggesting, but the reason why I am using fit_generator is because, if I use fit method, then it will load all the images on RAM, and may be some memory error may come up, but if you can suggest me some other method, then it would be really helpful
", "votes": null, "replies": [] }, { "id": 1275129, "author_name": "thalesgaluchi", "author_url": "", "post_date": "04/16/2021 01:12:32", "content": "I am also struggling to speed things up without pre-trained network, as a knowledge approach.
\nI tried 3 methods:
\n1 - dataset using 'images_from_directory', but it could not find the image files. I don't know why but I think is due to filenames starting with number. (If you know how to solve it, I do appreciate)
\n2 - dataset using map mode. Build a dataset with ( 'path-to-file', 'label'), then map 'path-to-file' to image array.
\n3 - using generator and fit method. (not fit_generator)
\nAll of them seems to be quite similar in time, even in Kaggle's GPUs.
\nFor datasets, I found this tutorial in Tensorflow website. Check the 'Configure the dataset for performance' topic.
\nUse cache and prefetch. The first epoch will last long. From the 2nd it may be faster. For me, shuffle seems to slow down.
\nI hope it helps
", "votes": null, "replies": [] }, { "id": 1275432, "author_name": "jl18pg052", "author_url": "", "post_date": "04/16/2021 09:49:45", "content": "Try using pytorch, I guess there is no other option other than that, I am also going to use the same
", "votes": null, "replies": [] } ] }, { "id": 1278297, "author_name": "jl18pg052", "author_url": "", "post_date": "04/19/2021 18:05:00", "content": "https://wrosinski.github.io/deep-learning-frameworks/
\nHere it is shown that models based on tensorflow are much faster compared to keras and pytorch.