Hi,
\n\nI need some suggestions on how to make good GPU utilization while working on OpenCV and Tensorflow (Keras). Being a newbie, I feel difficulty in reducing the time complexity along with output space limitation (like in my case, capturing multiple frames eats up the output space until and unless I make predictions and then delete the frame files). I read one shouldn't train on fly, but in my case, if I preprocess first and train later, then then it exceeds the available 1GB space due to massive amount of frames to be stored in the output folder. So now I click frames per video, preprocess and train, then delete those frames and move on to the next video. This really has low GPU utilization. In fact, when I ran a loop only for capturing frames using OpenCV, even that had poor GPU utilization. And then when I see people are also detecting faces before predicting but my model is taking most of the time in only capturing the frames, I don't understand how they managed to do that. Pls suggest ways to improve GPU utilization or any tutorials I can follow. Thanks a lot!
", "messages": [ { "id": "765440", "postDate": "03/06/2020 16:15:34", "content": "Hi,
\n\nI need some suggestions on how to make good GPU utilization while working on OpenCV and Tensorflow (Keras). Being a newbie, I feel difficulty in reducing the time complexity along with output space limitation (like in my case, capturing multiple frames eats up the output space until and unless I make predictions and then delete the frame files). I read one shouldn't train on fly, but in my case, if I preprocess first and train later, then then it exceeds the available 1GB space due to massive amount of frames to be stored in the output folder. So now I click frames per video, preprocess and train, then delete those frames and move on to the next video. This really has low GPU utilization. In fact, when I ran a loop only for capturing frames using OpenCV, even that had poor GPU utilization. And then when I see people are also detecting faces before predicting but my model is taking most of the time in only capturing the frames, I don't understand how they managed to do that. Pls suggest ways to improve GPU utilization or any tutorials I can follow. Thanks a lot!
", "rawMarkdown": "Hi,\n\nI need some suggestions on how to make good GPU utilization while working on OpenCV and Tensorflow (Keras). Being a newbie, I feel difficulty in reducing the time complexity along with output space limitation (like in my case, capturing multiple frames eats up the output space until and unless I make predictions and then delete the frame files). I read one shouldn't train on fly, but in my case, if I preprocess first and train later, then then it exceeds the available 1GB space due to massive amount of frames to be stored in the output folder. So now I click frames per video, preprocess and train, then delete those frames and move on to the next video. This really has low GPU utilization. In fact, when I ran a loop only for capturing frames using OpenCV, even that had poor GPU utilization. And then when I see people are also detecting faces before predicting but my model is taking most of the time in only capturing the frames, I don't understand how they managed to do that. Pls suggest ways to improve GPU utilization or any tutorials I can follow. Thanks a lot!", "votes": null }, { "id": "765466", "postDate": "03/06/2020 16:53:11", "content": "If you're talking about ur local GPU workstation, then I'm afraid 1 GB will run out too quick when you train ur model or even if you use a face detector such as MTCNN (it will be dead slow) on this large dataset. I don't think OpenCV will eat up your GPU space. Few suggestions are\n1) Try AWS or GCP. The third deadline is over to apply for free credits from AWS for this competition but u can still get some free credits when you sign up for AWS first time\n2) Make ur batch size lower \n3) See TensorFlow documentation for some optimization tips like Allowing dynamic growth, using sessions where needed etc.,
", "rawMarkdown": "If you're talking about ur local GPU workstation, then I'm afraid 1 GB will run out too quick when you train ur model or even if you use a face detector such as MTCNN (it will be dead slow) on this large dataset. I don't think OpenCV will eat up your GPU space. Few suggestions are\n1) Try AWS or GCP. The third deadline is over to apply for free credits from AWS for this competition but u can still get some free credits when you sign up for AWS first time\n2) Make ur batch size lower \n3) See TensorFlow documentation for some optimization tips like Allowing dynamic growth, using sessions where needed etc.,", "votes": null }, { "id": "765480", "postDate": "03/06/2020 17:04:46", "content": "Thank you so much for your response. By 1GB space, I meant the output storage space. Sorry for the confusion. I have edited my query.
", "rawMarkdown": "Thank you so much for your response. By 1GB space, I meant the output storage space. Sorry for the confusion. I have edited my query.", "votes": null }, { "id": "765489", "postDate": "03/06/2020 17:25:02", "content": "Sry but the memory aspect and output folder you were referring is still not clear. Where are you storing the output frames obtained from a video? The way almost all the teams are doing is to save all the frames (in required size and with required fps and I m sure you didn't mean you don't have 1 GB left in your hard drive) as a one time job and then train
", "rawMarkdown": "Sry but the memory aspect and output folder you were referring is still not clear. Where are you storing the output frames obtained from a video? The way almost all the teams are doing is to save all the frames (in required size and with required fps and I m sure you didn't mean you don't have 1 GB left in your hard drive) as a one time job and then train", "votes": null }, { "id": "765709", "postDate": "03/07/2020 01:38:53", "content": "the issues occurs to me for you reference:\n1.depart the preprocess and train steps in one train loop.if I firstly preprocess then train(loading video---cv2 to frames---detecting face---clustering face---train) in one loop,I find doubles GPU utilization(detecting face process and train proecess) and occurs GPU is not space to use
\n\nso I then extract the preprocess process, firstly extacting all frames from folders's videoes to png file type(detecting face,clustering faces ,save to png images), then start a process to train loop.
\n\nif you do not have enough storage paces to save png files and like me startly putting the preprocess and train process in one train loop,the GPU will soon be eated.
", "rawMarkdown": "the issues occurs to me for you reference:\n1.depart the preprocess and train steps in one train loop.if I firstly preprocess then train(loading video---cv2 to frames---detecting face---clustering face---train) in one loop,I find doubles GPU utilization(detecting face process and train proecess) and occurs GPU is not space to use \n\nso I then extract the preprocess process, firstly extacting all frames from folders's videoes to png file type(detecting face,clustering faces ,save to png images), then start a process to train loop.\n\nif you do not have enough storage paces to save png files and like me startly putting the preprocess and train process in one train loop,the GPU will soon be eated.", "votes": null } ], "comments": [ { "id": 765466, "author_name": "ramanareddyeceb", "author_url": "", "post_date": "03/06/2020 16:53:11", "content": "If you're talking about ur local GPU workstation, then I'm afraid 1 GB will run out too quick when you train ur model or even if you use a face detector such as MTCNN (it will be dead slow) on this large dataset. I don't think OpenCV will eat up your GPU space. Few suggestions are\n1) Try AWS or GCP. The third deadline is over to apply for free credits from AWS for this competition but u can still get some free credits when you sign up for AWS first time\n2) Make ur batch size lower \n3) See TensorFlow documentation for some optimization tips like Allowing dynamic growth, using sessions where needed etc.,
", "votes": null, "replies": [ { "id": 765480, "author_name": "saanikagupta", "author_url": "", "post_date": "03/06/2020 17:04:46", "content": "Thank you so much for your response. By 1GB space, I meant the output storage space. Sorry for the confusion. I have edited my query.
", "votes": null, "replies": [] }, { "id": 765489, "author_name": "ramanareddyeceb", "author_url": "", "post_date": "03/06/2020 17:25:02", "content": "Sry but the memory aspect and output folder you were referring is still not clear. Where are you storing the output frames obtained from a video? The way almost all the teams are doing is to save all the frames (in required size and with required fps and I m sure you didn't mean you don't have 1 GB left in your hard drive) as a one time job and then train
", "votes": null, "replies": [] } ] }, { "id": 765709, "author_name": "wangyoucao1234", "author_url": "", "post_date": "03/07/2020 01:38:53", "content": "the issues occurs to me for you reference:\n1.depart the preprocess and train steps in one train loop.if I firstly preprocess then train(loading video---cv2 to frames---detecting face---clustering face---train) in one loop,I find doubles GPU utilization(detecting face process and train proecess) and occurs GPU is not space to use
\n\nso I then extract the preprocess process, firstly extacting all frames from folders's videoes to png file type(detecting face,clustering faces ,save to png images), then start a process to train loop.
\n\nif you do not have enough storage paces to save png files and like me startly putting the preprocess and train process in one train loop,the GPU will soon be eated.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "765440": "Hi,\n\nI need some suggestions on how to make good GPU utilization while working on OpenCV and Tensorflow (Keras). Being a newbie, I feel difficulty in reducing the time complexity along with output space limitation (like in my case, capturing multiple frames eats up the output space until and unless I make predictions and then delete the frame files). I read one shouldn't train on fly, but in my case, if I preprocess first and train later, then then it exceeds the available 1GB space due to massive amount of frames to be stored in the output folder. So now I click frames per video, preprocess and train, then delete those frames and move on to the next video. This really has low GPU utilization. In fact, when I ran a loop only for capturing frames using OpenCV, even that had poor GPU utilization. And then when I see people are also detecting faces before predicting but my model is taking most of the time in only capturing the frames, I don't understand how they managed to do that. Pls suggest ways to improve GPU utilization or any tutorials I can follow. Thanks a lot!", "765466": "If you're talking about ur local GPU workstation, then I'm afraid 1 GB will run out too quick when you train ur model or even if you use a face detector such as MTCNN (it will be dead slow) on this large dataset. I don't think OpenCV will eat up your GPU space. Few suggestions are\n1) Try AWS or GCP. The third deadline is over to apply for free credits from AWS for this competition but u can still get some free credits when you sign up for AWS first time\n2) Make ur batch size lower \n3) See TensorFlow documentation for some optimization tips like Allowing dynamic growth, using sessions where needed etc.,", "765480": "Thank you so much for your response. By 1GB space, I meant the output storage space. Sorry for the confusion. I have edited my query.", "765489": "Sry but the memory aspect and output folder you were referring is still not clear. Where are you storing the output frames obtained from a video? The way almost all the teams are doing is to save all the frames (in required size and with required fps and I m sure you didn't mean you don't have 1 GB left in your hard drive) as a one time job and then train", "765709": "the issues occurs to me for you reference:\n1.depart the preprocess and train steps in one train loop.if I firstly preprocess then train(loading video---cv2 to frames---detecting face---clustering face---train) in one loop,I find doubles GPU utilization(detecting face process and train proecess) and occurs GPU is not space to use \n\nso I then extract the preprocess process, firstly extacting all frames from folders's videoes to png file type(detecting face,clustering faces ,save to png images), then start a process to train loop.\n\nif you do not have enough storage paces to save png files and like me startly putting the preprocess and train process in one train loop,the GPU will soon be eated." }, "source": "meta" }