Hi guys
\n\nIs anyone observing memory leaks in Kaggle's kernel? For me it always happens with OpenCV (VideoCapture) no matter what. I've already tried using function from public kernel and it didn't work either. Funny thing is the same code run fine on my local machine.
", "messages": [ { "id": "722661", "postDate": "01/18/2020 22:05:26", "content": "Hi guys
\n\nIs anyone observing memory leaks in Kaggle's kernel? For me it always happens with OpenCV (VideoCapture) no matter what. I've already tried using function from public kernel and it didn't work either. Funny thing is the same code run fine on my local machine.
", "rawMarkdown": "Hi guys\n\nIs anyone observing memory leaks in Kaggle's kernel? For me it always happens with OpenCV (VideoCapture) no matter what. I've already tried using function from public kernel and it didn't work either. Funny thing is the same code run fine on my local machine.", "votes": null }, { "id": "722683", "postDate": "01/18/2020 23:14:51", "content": "Yes, I had similar issues:\nhttps://www.kaggle.com/timesler/guide-to-mtcnn-in-facenet-pytorch#710202
\n\nAnd one time (or maybe two), I got out of memory with GPU on Kaggle for some reasons. Just running the same code again worked. Weird.
", "rawMarkdown": "Yes, I had similar issues:\nhttps://www.kaggle.com/timesler/guide-to-mtcnn-in-facenet-pytorch#710202\n\nAnd one time (or maybe two), I got out of memory with GPU on Kaggle for some reasons. Just running the same code again worked. Weird.", "votes": null }, { "id": "722996", "postDate": "01/19/2020 11:28:27", "content": "Thanks. Just out of curiosity, what is the intuition of using copy to fix memory leaks?
", "rawMarkdown": "Thanks. Just out of curiosity, what is the intuition of using copy to fix memory leaks?", "votes": null }, { "id": "723018", "postDate": "01/19/2020 11:55:36", "content": "I've enabled python tracemalloc to track the leak and memory increase was coming from each image (either CV2 or PIL) on the np.append(small_face). The full image was retained in memory instead of the small face. So I tried many things to free it and .copy() worked. It only happened with GPU, no leak with CPU. It's like image was retained by a GPU side effect. The root cause might be obvious for Pytorch/TF/GPU experts but I'm not expert.
I see, thanks. But for me it didn't work. I'm no expert in profilling memory, but I've tried tracemalloc and pympler and neither worked, this is really frustrating :/
", "rawMarkdown": "I see, thanks. But for me it didn't work. I'm no expert in profilling memory, but I've tried tracemalloc and pympler and neither worked, this is really frustrating :/", "votes": null }, { "id": "723196", "postDate": "01/19/2020 16:17:02", "content": "Not sure what you were doing, but if you do small_face = big_face[t:b, l:r, :] then you're making a slice into the original array and the system will need to keep the entire original array around. By doing .copy(), it only copies the pixels from the slice into a new array, and the original can be freed.
I seem to be having the same issue. I'm only using (VideoCapture) to extract one frame at a time to then extract features from it, but memory use keeps increasing drmatically even though I'm not saving the picture to any array.
", "rawMarkdown": "I seem to be having the same issue. I'm only using (VideoCapture) to extract one frame at a time to then extract features from it, but memory use keeps increasing drmatically even though I'm not saving the picture to any array.", "votes": null }, { "id": "725889", "postDate": "01/22/2020 15:23:51", "content": "Try submitting the kernel anyway. For me, the submission worked although the UI was clearly showing a memory leak during kernel's interactive mode
", "rawMarkdown": "Try submitting the kernel anyway. For me, the submission worked although the UI was clearly showing a memory leak during kernel's interactive mode", "votes": null }, { "id": "725930", "postDate": "01/22/2020 16:18:17", "content": "I just noticed the same thing happening in my kernel. Looping through all the videos reading a single frame per video will bump memory usage to 1.3 GB and it never gets reclaimed.
\n\nIn my inference kernel memory goes up to 15.9 GB -- eek! It does work for making a submission, but it looks like I've got some memory debugging to do. Some of this seems to be caused by OpenCV's VideoCapture, but it doesn't explain everything...
", "rawMarkdown": "I just noticed the same thing happening in my kernel. Looping through all the videos reading a single frame per video will bump memory usage to 1.3 GB and it never gets reclaimed. \n\nIn my inference kernel memory goes up to 15.9 GB -- eek! It does work for making a submission, but it looks like I've got some memory debugging to do. Some of this seems to be caused by OpenCV's VideoCapture, but it doesn't explain everything...", "votes": null }, { "id": "726577", "postDate": "01/23/2020 05:12:21", "content": "Similar Issue:\nJust added x=x/x.max() and ran out of memory. Even though it is not even close to 12 GB.
", "rawMarkdown": "Similar Issue:\nJust added x=x/x.max() and ran out of memory. Even though it is not even close to 12 GB.", "votes": null }, { "id": "726932", "postDate": "01/23/2020 10:18:41", "content": "For me, debugging memory was in vain. I just trusted that the problem were in the UI and it wouldn't occur in the private dataset. At least so far, I'm not having any problems in my submissions so it is probably an UI bug
", "rawMarkdown": "For me, debugging memory was in vain. I just trusted that the problem were in the UI and it wouldn't occur in the private dataset. At least so far, I'm not having any problems in my submissions so it is probably an UI bug", "votes": null }, { "id": "726953", "postDate": "01/23/2020 10:35:26", "content": "I'm thinking my issue is a PyTorch memory bug (?) with running on the CPU. The memory usage is stable when using the GPU.
", "rawMarkdown": "I'm thinking my issue is a PyTorch memory bug (?) with running on the CPU. The memory usage is stable when using the GPU.", "votes": null }, { "id": "726965", "postDate": "01/23/2020 10:44:14", "content": "Hmm, there is also a real memory real in pytorch if you use multiprocessing and have lists, tuples or ndarray (of object type) in the Dataset. Take a look https://github.com/pytorch/pytorch/issues/13246
", "rawMarkdown": "Hmm, there is also a real memory real in pytorch if you use multiprocessing and have lists, tuples or ndarray (of object type) in the Dataset. Take a look https://github.com/pytorch/pytorch/issues/13246", "votes": null }, { "id": "728529", "postDate": "01/24/2020 21:22:38", "content": "Do you always call the .release() function on VideoReader for each video? ...maybe this might be the reason of your leak
I do call .release(). 😄
A simple test that loads all test videos in a loop eats up about 1.5 GB of memory on Kaggle (that never gets returned) while on my local machine it's only about 100 MB. I've heard it mentioned in the forums here that OpenCV on Kaggle has a memory leak issue and it seems plausible.
\n\nAs for my more serious memory bug that eats up all RAM when using PyTorch, this appears to happen if you change the batch size when doing inference on the CPU. It took me 3 days to figure out that the varying batch size was the culprit here. (I made a bug report in case you're curious.)
", "rawMarkdown": "I do call `.release()`. 😄 \n\nA simple test that loads all test videos in a loop eats up about 1.5 GB of memory on Kaggle (that never gets returned) while on my local machine it's only about 100 MB. I've heard it mentioned in the forums here that OpenCV on Kaggle has a memory leak issue and it seems plausible.\n\nAs for my more serious memory bug that eats up all RAM when using PyTorch, this appears to happen if you change the batch size when doing inference on the CPU. It took me 3 days to figure out that the varying batch size was the culprit here. ([I made a bug report in case you're curious.](https://github.com/pytorch/pytorch/issues/32596))", "votes": null }, { "id": "728681", "postDate": "01/25/2020 04:48:49", "content": "Seems to be a problem with opencv itself: https://github.com/opencv/opencv/issues/5715#issuecomment-500116136\nHaven't figured out a way to fix it yet in python.
", "rawMarkdown": "Seems to be a problem with opencv itself: https://github.com/opencv/opencv/issues/5715#issuecomment-500116136\nHaven't figured out a way to fix it yet in python.", "votes": null }, { "id": "729185", "postDate": "01/25/2020 21:44:22", "content": "Yes, I tried deleting all of the local variables and calling garbage collection within the loops (including any release functionality), but upon returning from the function, the memory was still allocated. Capturing 11 frames apiece for 400 videos drove the memory up to ~16 GB.
", "rawMarkdown": "Yes, I tried deleting all of the local variables and calling garbage collection within the loops (including any release functionality), but upon returning from the function, the memory was still allocated. Capturing 11 frames apiece for 400 videos drove the memory up to ~16 GB.", "votes": null }, { "id": "729535", "postDate": "01/26/2020 10:27:50", "content": "\n\n\nCapturing 11 frames apiece for 400 videos drove the memory up to ~16 GB.
\n
That's much more severe than the memory leak I found with OpenCV... It sounds like something else is going wrong in your loop as well.
", "rawMarkdown": "> Capturing 11 frames apiece for 400 videos drove the memory up to ~16 GB.\n\nThat's much more severe than the memory leak I found with OpenCV... It sounds like something else is going wrong in your loop as well.", "votes": null }, { "id": "778337", "postDate": "03/18/2020 11:01:22", "content": "I have the same issue, how did any of you resolve this? if 400 videos drives memory to ~16GB how can you have enough memory for 4000 videos even if submission works or for training?
", "rawMarkdown": "I have the same issue, how did any of you resolve this? if 400 videos drives memory to ~16GB how can you have enough memory for 4000 videos even if submission works or for training?", "votes": null } ], "comments": [ { "id": 722683, "author_name": "mpware", "author_url": "", "post_date": "01/18/2020 23:14:51", "content": "Yes, I had similar issues:\nhttps://www.kaggle.com/timesler/guide-to-mtcnn-in-facenet-pytorch#710202
\n\nAnd one time (or maybe two), I got out of memory with GPU on Kaggle for some reasons. Just running the same code again worked. Weird.
", "votes": null, "replies": [ { "id": 722996, "author_name": "arc144", "author_url": "", "post_date": "01/19/2020 11:28:27", "content": "Thanks. Just out of curiosity, what is the intuition of using copy to fix memory leaks?
", "votes": null, "replies": [] }, { "id": 723018, "author_name": "mpware", "author_url": "", "post_date": "01/19/2020 11:55:36", "content": "I've enabled python tracemalloc to track the leak and memory increase was coming from each image (either CV2 or PIL) on the np.append(small_face). The full image was retained in memory instead of the small face. So I tried many things to free it and .copy() worked. It only happened with GPU, no leak with CPU. It's like image was retained by a GPU side effect. The root cause might be obvious for Pytorch/TF/GPU experts but I'm not expert.
I see, thanks. But for me it didn't work. I'm no expert in profilling memory, but I've tried tracemalloc and pympler and neither worked, this is really frustrating :/
", "votes": null, "replies": [] }, { "id": 723196, "author_name": "humananalog", "author_url": "", "post_date": "01/19/2020 16:17:02", "content": "Not sure what you were doing, but if you do small_face = big_face[t:b, l:r, :] then you're making a slice into the original array and the system will need to keep the entire original array around. By doing .copy(), it only copies the pixels from the slice into a new array, and the original can be freed.
I seem to be having the same issue. I'm only using (VideoCapture) to extract one frame at a time to then extract features from it, but memory use keeps increasing drmatically even though I'm not saving the picture to any array.
", "votes": null, "replies": [ { "id": 725889, "author_name": "arc144", "author_url": "", "post_date": "01/22/2020 15:23:51", "content": "Try submitting the kernel anyway. For me, the submission worked although the UI was clearly showing a memory leak during kernel's interactive mode
", "votes": null, "replies": [] }, { "id": 725930, "author_name": "humananalog", "author_url": "", "post_date": "01/22/2020 16:18:17", "content": "I just noticed the same thing happening in my kernel. Looping through all the videos reading a single frame per video will bump memory usage to 1.3 GB and it never gets reclaimed.
\n\nIn my inference kernel memory goes up to 15.9 GB -- eek! It does work for making a submission, but it looks like I've got some memory debugging to do. Some of this seems to be caused by OpenCV's VideoCapture, but it doesn't explain everything...
", "votes": null, "replies": [] }, { "id": 726932, "author_name": "arc144", "author_url": "", "post_date": "01/23/2020 10:18:41", "content": "For me, debugging memory was in vain. I just trusted that the problem were in the UI and it wouldn't occur in the private dataset. At least so far, I'm not having any problems in my submissions so it is probably an UI bug
", "votes": null, "replies": [] }, { "id": 726953, "author_name": "humananalog", "author_url": "", "post_date": "01/23/2020 10:35:26", "content": "I'm thinking my issue is a PyTorch memory bug (?) with running on the CPU. The memory usage is stable when using the GPU.
", "votes": null, "replies": [] }, { "id": 726965, "author_name": "arc144", "author_url": "", "post_date": "01/23/2020 10:44:14", "content": "Hmm, there is also a real memory real in pytorch if you use multiprocessing and have lists, tuples or ndarray (of object type) in the Dataset. Take a look https://github.com/pytorch/pytorch/issues/13246
", "votes": null, "replies": [] } ] }, { "id": 726577, "author_name": "unkownhihi", "author_url": "", "post_date": "01/23/2020 05:12:21", "content": "Similar Issue:\nJust added x=x/x.max() and ran out of memory. Even though it is not even close to 12 GB.
", "votes": null, "replies": [] }, { "id": 728529, "author_name": "dagnelies", "author_url": "", "post_date": "01/24/2020 21:22:38", "content": "Do you always call the .release() function on VideoReader for each video? ...maybe this might be the reason of your leak
I do call .release(). 😄
A simple test that loads all test videos in a loop eats up about 1.5 GB of memory on Kaggle (that never gets returned) while on my local machine it's only about 100 MB. I've heard it mentioned in the forums here that OpenCV on Kaggle has a memory leak issue and it seems plausible.
\n\nAs for my more serious memory bug that eats up all RAM when using PyTorch, this appears to happen if you change the batch size when doing inference on the CPU. It took me 3 days to figure out that the varying batch size was the culprit here. (I made a bug report in case you're curious.)
", "votes": null, "replies": [] } ] }, { "id": 728681, "author_name": "teknas", "author_url": "", "post_date": "01/25/2020 04:48:49", "content": "Seems to be a problem with opencv itself: https://github.com/opencv/opencv/issues/5715#issuecomment-500116136\nHaven't figured out a way to fix it yet in python.
", "votes": null, "replies": [] }, { "id": 729185, "author_name": "mconway", "author_url": "", "post_date": "01/25/2020 21:44:22", "content": "Yes, I tried deleting all of the local variables and calling garbage collection within the loops (including any release functionality), but upon returning from the function, the memory was still allocated. Capturing 11 frames apiece for 400 videos drove the memory up to ~16 GB.
", "votes": null, "replies": [ { "id": 729535, "author_name": "humananalog", "author_url": "", "post_date": "01/26/2020 10:27:50", "content": "\n\n\nCapturing 11 frames apiece for 400 videos drove the memory up to ~16 GB.
\n
That's much more severe than the memory leak I found with OpenCV... It sounds like something else is going wrong in your loop as well.
", "votes": null, "replies": [] } ] }, { "id": 778337, "author_name": "ozzydeeeplearner", "author_url": "", "post_date": "03/18/2020 11:01:22", "content": "I have the same issue, how did any of you resolve this? if 400 videos drives memory to ~16GB how can you have enough memory for 4000 videos even if submission works or for training?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "722661": "Hi guys\n\nIs anyone observing memory leaks in Kaggle's kernel? For me it always happens with OpenCV (VideoCapture) no matter what. I've already tried using function from public kernel and it didn't work either. Funny thing is the same code run fine on my local machine.", "722683": "Yes, I had similar issues:\nhttps://www.kaggle.com/timesler/guide-to-mtcnn-in-facenet-pytorch#710202\n\nAnd one time (or maybe two), I got out of memory with GPU on Kaggle for some reasons. Just running the same code again worked. Weird.", "722996": "Thanks. Just out of curiosity, what is the intuition of using copy to fix memory leaks?", "723018": "I've enabled python tracemalloc to track the leak and memory increase was coming from each image (either CV2 or PIL) on the `np.append(small_face)`. The full image was retained in memory instead of the small face. So I tried many things to free it and `.copy()` worked. It only happened with GPU, no leak with CPU. It's like image was retained by a GPU side effect. The root cause might be obvious for Pytorch/TF/GPU experts but I'm not expert.", "723095": "I see, thanks. But for me it didn't work. I'm no expert in profilling memory, but I've tried tracemalloc and pympler and neither worked, this is really frustrating :/", "723196": "Not sure what you were doing, but if you do `small_face = big_face[t:b, l:r, :]` then you're making a slice into the original array and the system will need to keep the entire original array around. By doing `.copy()`, it only copies the pixels from the slice into a new array, and the original can be freed.", "725869": "I seem to be having the same issue. I'm only using (VideoCapture) to extract one frame at a time to then extract features from it, but memory use keeps increasing drmatically even though I'm not saving the picture to any array.", "725889": "Try submitting the kernel anyway. For me, the submission worked although the UI was clearly showing a memory leak during kernel's interactive mode", "725930": "I just noticed the same thing happening in my kernel. Looping through all the videos reading a single frame per video will bump memory usage to 1.3 GB and it never gets reclaimed. \n\nIn my inference kernel memory goes up to 15.9 GB -- eek! It does work for making a submission, but it looks like I've got some memory debugging to do. Some of this seems to be caused by OpenCV's VideoCapture, but it doesn't explain everything...", "726577": "Similar Issue:\nJust added x=x/x.max() and ran out of memory. Even though it is not even close to 12 GB.", "726932": "For me, debugging memory was in vain. I just trusted that the problem were in the UI and it wouldn't occur in the private dataset. At least so far, I'm not having any problems in my submissions so it is probably an UI bug", "726953": "I'm thinking my issue is a PyTorch memory bug (?) with running on the CPU. The memory usage is stable when using the GPU.", "726965": "Hmm, there is also a real memory real in pytorch if you use multiprocessing and have lists, tuples or ndarray (of object type) in the Dataset. Take a look https://github.com/pytorch/pytorch/issues/13246", "728529": "Do you always call the `.release()` function on VideoReader for each video? ...maybe this might be the reason of your leak", "728562": "I do call `.release()`. 😄 \n\nA simple test that loads all test videos in a loop eats up about 1.5 GB of memory on Kaggle (that never gets returned) while on my local machine it's only about 100 MB. I've heard it mentioned in the forums here that OpenCV on Kaggle has a memory leak issue and it seems plausible.\n\nAs for my more serious memory bug that eats up all RAM when using PyTorch, this appears to happen if you change the batch size when doing inference on the CPU. It took me 3 days to figure out that the varying batch size was the culprit here. ([I made a bug report in case you're curious.](https://github.com/pytorch/pytorch/issues/32596))", "728681": "Seems to be a problem with opencv itself: https://github.com/opencv/opencv/issues/5715#issuecomment-500116136\nHaven't figured out a way to fix it yet in python.", "729185": "Yes, I tried deleting all of the local variables and calling garbage collection within the loops (including any release functionality), but upon returning from the function, the memory was still allocated. Capturing 11 frames apiece for 400 videos drove the memory up to ~16 GB.", "729535": "> Capturing 11 frames apiece for 400 videos drove the memory up to ~16 GB.\n\nThat's much more severe than the memory leak I found with OpenCV... It sounds like something else is going wrong in your loop as well.", "778337": "I have the same issue, how did any of you resolve this? if 400 videos drives memory to ~16GB how can you have enough memory for 4000 videos even if submission works or for training?" }, "source": "meta" }