I was running my code and it worked well at 10 am UTC+00 but at 4pm UTC+00 the same code did not work anymore and showed the following error message:
\n\nError while calling cudaGetLastError() in file /tmp/pip-install-hdz9ztcw/dlib/dlib/cuda/gpu_data.cpp:114. code: 2, reason: out of memory.
This error happens when using GPU to run input through CNN model however using GPU to train network works still.
", "messages": [ { "id": 771985, "postDate": "2020-03-14T22:06:24.330Z", "content": "If the code is the same, then maybe the inputs aren't the same, cause you are using some random when choosing the images or videos to process.\nOut of memory on dlib, is most likely due to the frame size (or batch size, if you are batching) being to big for the available GPU memory.
", "rawMarkdown": "If the code is the same, then maybe the inputs aren't the same, cause you are using some random when choosing the images or videos to process.\nOut of memory on dlib, is most likely due to the frame size (or batch size, if you are batching) being to big for the available GPU memory.", "replies": [ { "id": 772252, "postDate": "2020-03-15T08:29:38.503Z", "content": "Thank you Nuno for replying. Actually the input has also been the same and batch size is as well (size 1). I was testing my code with the test videos and then after it processed them (in 2 hours) I tested again and it didn't work.
\n\nBy reducing the image size by 8, the dlib worked every now and then but of course with that small image size it didn't recognize the faces.
It seems you use dlib cnn-based face detection model. I see2 ways to solve your issue:\n- split image into overlapping chunks and run face model through them. It would require additional cleanup if some face rectangles repeat. \n- use MTCNN as face detector. It had no problem running 4K stream on a single card.
", "rawMarkdown": "It seems you use dlib cnn-based face detection model. I see2 ways to solve your issue:\n- split image into overlapping chunks and run face model through them. It would require additional cleanup if some face rectangles repeat. \n- use MTCNN as face detector. It had no problem running 4K stream on a single card." }, { "id": 775043, "postDate": "2020-03-16T08:21:46.777Z", "content": "Thank you Vlad for replying. Yes I used cnn-based face detection. Splitting image into smaller crops would make quite a lot extra work to face detection. I also tried MTCNN but its accuracy is not as good as that dlib cnn detection. Actually, I also noticed that dlib cannot be used anyway because it is not installed to Kaggle servers and competition rules say that no additional libraries can be used in the final evaluation.
", "rawMarkdown": "Thank you Vlad for replying. Yes I used cnn-based face detection. Splitting image into smaller crops would make quite a lot extra work to face detection. I also tried MTCNN but its accuracy is not as good as that dlib cnn detection. Actually, I also noticed that dlib cannot be used anyway because it is not installed to Kaggle servers and competition rules say that no additional libraries can be used in the final evaluation." }, { "id": 775190, "postDate": "2020-03-16T11:49:45.410Z", "content": "Actually MTCNN is good for detection (used it in couple of my projects and it works really well). You may want to adjust threshold values to avoid false positives, and minimum face size in case it does not detect yet.
\n\nFrom other side, it seems you can use 1) pretrained model in custom dataset attached to the notebook 2) pre-downloaded python wheel, which you can install then py !pip install /path/to/package in a separate notebook cell.
Aa, ok, thank you for telling this one. I didn't think that uploading wheel file and installing it through the script could be used. This can be useful in other libraries as well.
", "rawMarkdown": "Aa, ok, thank you for telling this one. I didn't think that uploading wheel file and installing it through the script could be used. This can be useful in other libraries as well." } ] }, { "id": 771919, "postDate": "2020-03-14T19:51:30.717Z", "content": "I was running my code and it worked well at 10 am UTC+00 but at 4pm UTC+00 the same code did not work anymore and showed the following error message:
\n\nError while calling cudaGetLastError() in file /tmp/pip-install-hdz9ztcw/dlib/dlib/cuda/gpu_data.cpp:114. code: 2, reason: out of memory.
This error happens when using GPU to run input through CNN model however using GPU to train network works still.
", "rawMarkdown": "I was running my code and it worked well at 10 am UTC+00 but at 4pm UTC+00 the same code did not work anymore and showed the following error message:\n\n`Error while calling cudaGetLastError() in file /tmp/pip-install-hdz9ztcw/dlib/dlib/cuda/gpu_data.cpp:114. code: 2, reason: out of memory.`\n\nThis error happens when using GPU to run input through CNN model however using GPU to train network works still." }, { "id": 772204, "postDate": "2020-03-15T06:40:47.090Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 771985, "author_name": "Nuno Ferreira", "author_url": "", "post_date": "2020-03-14T22:06:24.330000", "content": "If the code is the same, then maybe the inputs aren't the same, cause you are using some random when choosing the images or videos to process.\nOut of memory on dlib, is most likely due to the frame size (or batch size, if you are batching) being to big for the available GPU memory.
", "votes": 0, "replies": [ { "id": 772252, "author_name": "pter", "author_url": "", "post_date": "2020-03-15T08:29:38.503000", "content": "Thank you Nuno for replying. Actually the input has also been the same and batch size is as well (size 1). I was testing my code with the test videos and then after it processed them (in 2 hours) I tested again and it didn't work.
\n\nBy reducing the image size by 8, the dlib worked every now and then but of course with that small image size it didn't recognize the faces.
It seems you use dlib cnn-based face detection model. I see2 ways to solve your issue:\n- split image into overlapping chunks and run face model through them. It would require additional cleanup if some face rectangles repeat. \n- use MTCNN as face detector. It had no problem running 4K stream on a single card.
", "votes": 0, "replies": [] }, { "id": 775043, "author_name": "pter", "author_url": "", "post_date": "2020-03-16T08:21:46.777000", "content": "Thank you Vlad for replying. Yes I used cnn-based face detection. Splitting image into smaller crops would make quite a lot extra work to face detection. I also tried MTCNN but its accuracy is not as good as that dlib cnn detection. Actually, I also noticed that dlib cannot be used anyway because it is not installed to Kaggle servers and competition rules say that no additional libraries can be used in the final evaluation.
", "votes": 0, "replies": [] }, { "id": 775190, "author_name": "Vlad Hrynko", "author_url": "", "post_date": "2020-03-16T11:49:45.410000", "content": "Actually MTCNN is good for detection (used it in couple of my projects and it works really well). You may want to adjust threshold values to avoid false positives, and minimum face size in case it does not detect yet.
\n\nFrom other side, it seems you can use 1) pretrained model in custom dataset attached to the notebook 2) pre-downloaded python wheel, which you can install then py !pip install /path/to/package in a separate notebook cell.
Aa, ok, thank you for telling this one. I didn't think that uploading wheel file and installing it through the script could be used. This can be useful in other libraries as well.
", "votes": 0, "replies": [] } ] }, { "id": 772204, "author_name": "", "author_url": "", "post_date": "2020-03-15T06:40:47.090000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "771985": "If the code is the same, then maybe the inputs aren't the same, cause you are using some random when choosing the images or videos to process.\nOut of memory on dlib, is most likely due to the frame size (or batch size, if you are batching) being to big for the available GPU memory.", "771919": "I was running my code and it worked well at 10 am UTC+00 but at 4pm UTC+00 the same code did not work anymore and showed the following error message:\n\n`Error while calling cudaGetLastError() in file /tmp/pip-install-hdz9ztcw/dlib/dlib/cuda/gpu_data.cpp:114. code: 2, reason: out of memory.`\n\nThis error happens when using GPU to run input through CNN model however using GPU to train network works still.", "772204": "" } }