![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F864684%2Fa7f6a78395137962153a5e3a29449edf%2Fmemory_leak.png?generation=1599936382798391&alt=media\")
https://www.kaggle.com/ryches/memory-leak
\nI don't know if others have run into this but I have been fighting against a memory leak. I set up a minimal example to just iterate through the data and do nothing else. It isnt storing anything or accumulating gradients or anything, but memory usage seems to continue to drift overtime until eventually it kills the kernel. Have others run into this? Am I just doing something wrong?
", "messages": [ { "id": 1008037, "postDate": "2020-09-12T18:00:02.507Z", "content": "https://www.kaggle.com/ryches/memory-leak
\nI don't know if others have run into this but I have been fighting against a memory leak. I set up a minimal example to just iterate through the data and do nothing else. It isnt storing anything or accumulating gradients or anything, but memory usage seems to continue to drift overtime until eventually it kills the kernel. Have others run into this? Am I just doing something wrong?
", "rawMarkdown": "https://www.kaggle.com/ryches/memory-leak\n\nI don't know if others have run into this but I have been fighting against a memory leak. I set up a minimal example to just iterate through the data and do nothing else. It isnt storing anything or accumulating gradients or anything, but memory usage seems to continue to drift overtime until eventually it kills the kernel. Have others run into this? Am I just doing something wrong?", "votes": 12 }, { "id": 1056316, "postDate": "2020-10-21T15:31:51.600Z", "content": "Did anyone find the source of the memory leak? This keeps popping up from time to time!
", "rawMarkdown": "Did anyone find the source of the memory leak? This keeps popping up from time to time!", "votes": 3 }, { "id": 1008104, "postDate": "2020-09-12T18:49:23.907Z", "content": "The issue is not with your code, I had the same problem. After a few hours, my opened chrome tabs started to crash.. (I have 128Gb of RAM)
\n
I am using the latest code (l5kit repo's master branch) and it seems fixed now.
", "rawMarkdown": "The issue is not with your code, I had the same problem. After a few hours, my opened chrome tabs started to crash.. (I have 128Gb of RAM)\n\n\n\nI am using the latest code (l5kit repo's master branch) and it seems fixed now.", "votes": 4, "replies": [ { "id": 1008482, "postDate": "2020-09-13T06:40:06.110Z", "content": "Hmm I just updated and updated my pytorch to 1.6 and still seem to be getting it
", "rawMarkdown": "Hmm I just updated and updated my pytorch to 1.6 and still seem to be getting it" }, { "id": 1009007, "postDate": "2020-09-13T15:24:34.970Z", "content": "also, I've been using the utility script workaround you configured. Do you know which version that is?
", "rawMarkdown": "also, I've been using the utility script workaround you configured. Do you know which version that is?" }, { "id": 1009017, "postDate": "2020-09-13T15:37:54.717Z", "content": "No idea. I think it is the latest released version (1.0.6). Try this:
\nimport l5kit\nprint(l5kit.__version__)\nAre there any solutions for this? We are still experiencing this problem with the l5kit 1.1.0. It uses all my available physical memory (~11GB) in like 20 seconds and stay at 150-200MB free all the time.
", "rawMarkdown": "Are there any solutions for this? We are still experiencing this problem with the l5kit 1.1.0. It uses all my available physical memory (~11GB) in like 20 seconds and stay at 150-200MB free all the time.", "votes": 1, "replies": [ { "id": 1082944, "postDate": "2020-11-18T12:14:46.570Z", "content": "Have you disabled caching? That worked for me.
", "rawMarkdown": "Have you disabled caching? That worked for me." }, { "id": 1083094, "postDate": "2020-11-18T15:38:10.380Z", "content": "I was never able to make this totally go away and ended up just engineering around this by creating a script that I just run repeatedly from terminal. It's a janky solution but I had to move on from probing dependency versions.
", "rawMarkdown": "I was never able to make this totally go away and ended up just engineering around this by creating a script that I just run repeatedly from terminal. It's a janky solution but I had to move on from probing dependency versions. " }, { "id": 1083194, "postDate": "2020-11-18T17:41:04.053Z", "content": "If you're using mixed precision, you can simply cast image to fp16 inside rasterizer. This is what helped me. This is pretty crude solution, but I was able to stabilize RAM usage.
", "rawMarkdown": "If you're using mixed precision, you can simply cast image to fp16 inside rasterizer. This is what helped me. This is pretty crude solution, but I was able to stabilize RAM usage." } ] }, { "id": 1011941, "postDate": "2020-09-15T19:13:35.937Z", "content": "I ran a data loader loop in a script (not within a notebook). It certainly eats up memory and CPU like crazy but seemed to stabilize. 4 workers stabilized at 6.4GB virtual, 3GB physical. It started at 4/2GB in first epoch and jumped to the stable value in epoch 2/3 and then stayed steady.
\nBumping up the workers to 16, memory usage was 9GB virt, btw 3-4GB phys after epoch 3 with larger jumps per epoch. I killed it but suspect there is a steady state, just larger. There is likely a relationship between num workers and the steady state memory use.
\nI feel the design as a while isn't optimal, each worker process zipping through the zarrs independently without co-ordination, but not sure how else you'd get something simple working without threading and tighter synchronization (needing C++).
", "rawMarkdown": "I ran a data loader loop in a script (not within a notebook). It certainly eats up memory and CPU like crazy but seemed to stabilize. 4 workers stabilized at 6.4GB virtual, 3GB physical. It started at 4/2GB in first epoch and jumped to the stable value in epoch 2/3 and then stayed steady. \n\nBumping up the workers to 16, memory usage was 9GB virt, btw 3-4GB phys after epoch 3 with larger jumps per epoch. I killed it but suspect there is a steady state, just larger. There is likely a relationship between num workers and the steady state memory use. \n\nI feel the design as a while isn't optimal, each worker process zipping through the zarrs independently without co-ordination, but not sure how else you'd get something simple working without threading and tighter synchronization (needing C++).", "votes": 1, "replies": [ { "id": 1011956, "postDate": "2020-09-15T19:35:20.073Z", "content": "Hmm, I am wondering if it is a package version issue. I believe I am on newest build but I will have to check on that.
", "rawMarkdown": "Hmm, I am wondering if it is a package version issue. I believe I am on newest build but I will have to check on that. " }, { "id": 1011958, "postDate": "2020-09-15T19:39:41.780Z", "content": "Was what you ran close to what I ran on the kaggle kernels or do you think there was maybe a code difference there that is worth investigating?
", "rawMarkdown": "Was what you ran close to what I ran on the kaggle kernels or do you think there was maybe a code difference there that is worth investigating?" }, { "id": 1011964, "postDate": "2020-09-15T19:42:58.953Z", "content": "Basically the same, just with the normal tqdm instead of notebook one. Latest l5kit.
", "rawMarkdown": "Basically the same, just with the normal tqdm instead of notebook one. Latest l5kit." }, { "id": 1011982, "postDate": "2020-09-15T20:03:41.107Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1008813, "postDate": "2020-09-13T12:14:49.333Z", "content": "I've been investigated this but It's quite difficult to pin this down to a specific component. It's not clear if this is:
\nWe use caching in the zarr, which may conflict with torch loaders (it's not clear whether the cache is entirely copied between workers).
\nSomething you can try is disabling cache for the zarr dataset by using:
\nChunkedDataset(\"path\").open(cached=False)
I'll take a deep look into this next week, as it's affecting multiple users
", "rawMarkdown": "I've been investigated this but It's quite difficult to pin this down to a specific component. It's not clear if this is:\n- a l5kit issue;\n- a torch issue;\n- a combination of both.\n\nWe use caching in the zarr, which may conflict with torch loaders (it's not clear whether the cache is entirely copied between workers).\n\nSomething you can try is disabling cache for the zarr dataset by using:\n\n`ChunkedDataset(\"path\").open(cached=False)`\n\nI'll take a deep look into this next week, as it's affecting multiple users", "votes": 1, "replies": [ { "id": 1009010, "postDate": "2020-09-13T15:26:00.930Z", "content": "Yes, it is a bit unclear where it is coming from exactly. I just tried with caching turned off and it still seems to accumulate. Across 30 minutes it gained an extra 3gb.
", "rawMarkdown": "Yes, it is a bit unclear where it is coming from exactly. I just tried with caching turned off and it still seems to accumulate. Across 30 minutes it gained an extra 3gb. ", "votes": 1 }, { "id": 1009294, "postDate": "2020-09-13T20:25:47.433Z", "content": "Thanks for reporting the issue in L5kit GitHub page @ryches, I'll report back there what I'll find!
", "rawMarkdown": "Thanks for reporting the issue in L5kit GitHub page @ryches, I'll report back there what I'll find!" } ] }, { "id": 1010197, "postDate": "2020-09-14T15:15:22.440Z", "content": "I may have found a work around. Change the max number of steps to 100 and correspondingly increase the number of epochs to make up for the same number of training examples. This dumps the memory after the first 100 iterations. This along with a batch size of 16 seems to work on the Kaggle kernel with GPU. However, the bottleneck is the rasterization, so it maybe better to just use CPU for this project.
", "rawMarkdown": "I may have found a work around. Change the max number of steps to 100 and correspondingly increase the number of epochs to make up for the same number of training examples. This dumps the memory after the first 100 iterations. This along with a batch size of 16 seems to work on the Kaggle kernel with GPU. However, the bottleneck is the rasterization, so it maybe better to just use CPU for this project. ", "replies": [ { "id": 1010425, "postDate": "2020-09-14T18:23:21.657Z", "content": "hmm so maybe it only starts accumulating after taking a certain number of steps? Is that what this is implying?
", "rawMarkdown": "hmm so maybe it only starts accumulating after taking a certain number of steps? Is that what this is implying?" }, { "id": 1010429, "postDate": "2020-09-14T18:28:59.933Z", "content": "@ryches For me, the leak was there when I only ran the loader/iterator. I mean without any deep learning (no move to cuda, no forward/backward, no optimizer, etc). I am not 100% sure why is fixed now, but I'll try to reproduce, maybe I'll find something useful.
", "rawMarkdown": "@ryches For me, the leak was there when I only ran the loader/iterator. I mean without any deep learning (no move to cuda, no forward/backward, no optimizer, etc). I am not 100% sure why is fixed now, but I'll try to reproduce, maybe I'll find something useful." }, { "id": 1010448, "postDate": "2020-09-14T18:44:02.890Z", "content": "Here is what I did. Put in a dummy generator that gets reassigned after the inner loop. It ran for over an hour but finally crashed because of memory issue. Previously it would not even run for 10 mins.
\n`for i in range(epochs):
\n def gen(dataloader):\n\n for batch in dataloader:\n\n yield batch\n train_gen = gen(train_dataloader)\n\n progress_bar = tqdm(range(cfg[\"train_params\"][\"max_num_steps\"]))\n for _ in progress_bar:\n try:\n data = next(train_gen)\n except StopIteration:\n tr_it = iter(train_dataloader)\n data = next(train_gen)`\nYeah I was able to narrow my issue down to just the dataloader as well. I was originally trying with my training and validation and thinking maybe my validation was accidentally accumulating gradients over time so I turned that off. Then I turned off training after to see if just data loading alone was causing it and that seems to be the case.
", "rawMarkdown": "Yeah I was able to narrow my issue down to just the dataloader as well. I was originally trying with my training and validation and thinking maybe my validation was accidentally accumulating gradients over time so I turned that off. Then I turned off training after to see if just data loading alone was causing it and that seems to be the case. " }, { "id": 1011413, "postDate": "2020-09-15T12:54:59.277Z", "content": "\n\nmaybe it only starts accumulating after taking a certain number of steps? Is that what this is implying?
\n
PyTorch dataloader workers are created in the iterator. So restarting iteration will kill all the previous workers and release any leaked memory.
\nRestarting will also clear the LRU caches in the zarr dataset which otherwise seem liable to cause issues just through correct operation. With the default config of 16 workers each with a maximum cache of 1Gb that's 16Gb just for caching. Can't see any checks around available memory.
You can completely disable cache or reduce the memory footprint by tuning the parameters of the open() method in the ChunkedDataset
For me, a disabling caching solved the problem. Memory usage tends to fluctuate more now, but it doesn't accumulate and the process is no longer killed by the OS.
\nI didn't notice any measureable performance drop due to disabling caching.
I also didnt see any appreciable change in performance. I haven't inspected closely, but it seems like caching in this case isn't super useful. Maybe default should be set to False instead of True
", "rawMarkdown": "I also didnt see any appreciable change in performance. I haven't inspected closely, but it seems like caching in this case isn't super useful. Maybe default should be set to False instead of True" }, { "id": 1020620, "postDate": "2020-09-21T10:00:12.197Z", "content": "Yes, the issue is that zarr caches the raw chunk and not the decompressed one (no idea why that's not an option). That means it still needs to run blosc on the chunk even if it's cached
", "rawMarkdown": "Yes, the issue is that zarr caches the raw chunk and not the decompressed one (no idea why that's not an option). That means it still needs to run blosc on the chunk even if it's cached" } ] }, { "id": 1008277, "postDate": "2020-09-13T00:21:03.053Z", "content": "I face the same issue when num_workers > 0 in the dataloader. I think it has to do with using dicts for loading the data. See here: https://github.com/pytorch/pytorch/issues/13246#issuecomment-445770039
I haven't tried it with the latest l5kit update yet though.
", "rawMarkdown": "I face the same issue when `num_workers > 0` in the dataloader. I think it has to do with using dicts for loading the data. See here: [https://github.com/pytorch/pytorch/issues/13246#issuecomment-445770039](url)\n\nI haven't tried it with the latest l5kit update yet though." }, { "id": 1008039, "postDate": "2020-09-12T18:00:24.023Z", "content": "This is for the most part just code pulled from the example notebook from l5kit that was provided to us
", "rawMarkdown": "This is for the most part just code pulled from the example notebook from l5kit that was provided to us" } ], "comments": [ { "id": 1056316, "author_name": "A_Elsheikh", "author_url": "", "post_date": "2020-10-21T15:31:51.600000", "content": "Did anyone find the source of the memory leak? This keeps popping up from time to time!
", "votes": 3, "replies": [] }, { "id": 1008104, "author_name": "Peter", "author_url": "", "post_date": "2020-09-12T18:49:23.907000", "content": "The issue is not with your code, I had the same problem. After a few hours, my opened chrome tabs started to crash.. (I have 128Gb of RAM)
\n
I am using the latest code (l5kit repo's master branch) and it seems fixed now.
", "votes": 4, "replies": [ { "id": 1008482, "author_name": "ryches", "author_url": "", "post_date": "2020-09-13T06:40:06.110000", "content": "Hmm I just updated and updated my pytorch to 1.6 and still seem to be getting it
", "votes": 0, "replies": [] }, { "id": 1009007, "author_name": "ryches", "author_url": "", "post_date": "2020-09-13T15:24:34.970000", "content": "also, I've been using the utility script workaround you configured. Do you know which version that is?
", "votes": 0, "replies": [] }, { "id": 1009017, "author_name": "Peter", "author_url": "", "post_date": "2020-09-13T15:37:54.717000", "content": "No idea. I think it is the latest released version (1.0.6). Try this:
\nimport l5kit\nprint(l5kit.__version__)\nAre there any solutions for this? We are still experiencing this problem with the l5kit 1.1.0. It uses all my available physical memory (~11GB) in like 20 seconds and stay at 150-200MB free all the time.
", "votes": 1, "replies": [ { "id": 1082944, "author_name": "Pascal Pfeiffer", "author_url": "", "post_date": "2020-11-18T12:14:46.570000", "content": "Have you disabled caching? That worked for me.
", "votes": 0, "replies": [] }, { "id": 1083094, "author_name": "ryches", "author_url": "", "post_date": "2020-11-18T15:38:10.380000", "content": "I was never able to make this totally go away and ended up just engineering around this by creating a script that I just run repeatedly from terminal. It's a janky solution but I had to move on from probing dependency versions.
", "votes": 0, "replies": [] }, { "id": 1083194, "author_name": "Eek The Cat", "author_url": "", "post_date": "2020-11-18T17:41:04.053000", "content": "If you're using mixed precision, you can simply cast image to fp16 inside rasterizer. This is what helped me. This is pretty crude solution, but I was able to stabilize RAM usage.
", "votes": 0, "replies": [] } ] }, { "id": 1011941, "author_name": "RossWightman", "author_url": "", "post_date": "2020-09-15T19:13:35.937000", "content": "I ran a data loader loop in a script (not within a notebook). It certainly eats up memory and CPU like crazy but seemed to stabilize. 4 workers stabilized at 6.4GB virtual, 3GB physical. It started at 4/2GB in first epoch and jumped to the stable value in epoch 2/3 and then stayed steady.
\nBumping up the workers to 16, memory usage was 9GB virt, btw 3-4GB phys after epoch 3 with larger jumps per epoch. I killed it but suspect there is a steady state, just larger. There is likely a relationship between num workers and the steady state memory use.
\nI feel the design as a while isn't optimal, each worker process zipping through the zarrs independently without co-ordination, but not sure how else you'd get something simple working without threading and tighter synchronization (needing C++).
", "votes": 1, "replies": [ { "id": 1011956, "author_name": "ryches", "author_url": "", "post_date": "2020-09-15T19:35:20.073000", "content": "Hmm, I am wondering if it is a package version issue. I believe I am on newest build but I will have to check on that.
", "votes": 0, "replies": [] }, { "id": 1011958, "author_name": "ryches", "author_url": "", "post_date": "2020-09-15T19:39:41.780000", "content": "Was what you ran close to what I ran on the kaggle kernels or do you think there was maybe a code difference there that is worth investigating?
", "votes": 0, "replies": [] }, { "id": 1011964, "author_name": "RossWightman", "author_url": "", "post_date": "2020-09-15T19:42:58.953000", "content": "Basically the same, just with the normal tqdm instead of notebook one. Latest l5kit.
", "votes": 0, "replies": [] }, { "id": 1011982, "author_name": "", "author_url": "", "post_date": "2020-09-15T20:03:41.107000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1008813, "author_name": "Luca Bergamini", "author_url": "", "post_date": "2020-09-13T12:14:49.333000", "content": "I've been investigated this but It's quite difficult to pin this down to a specific component. It's not clear if this is:
\nWe use caching in the zarr, which may conflict with torch loaders (it's not clear whether the cache is entirely copied between workers).
\nSomething you can try is disabling cache for the zarr dataset by using:
\nChunkedDataset(\"path\").open(cached=False)
I'll take a deep look into this next week, as it's affecting multiple users
", "votes": 1, "replies": [ { "id": 1009010, "author_name": "ryches", "author_url": "", "post_date": "2020-09-13T15:26:00.930000", "content": "Yes, it is a bit unclear where it is coming from exactly. I just tried with caching turned off and it still seems to accumulate. Across 30 minutes it gained an extra 3gb.
", "votes": 1, "replies": [] }, { "id": 1009294, "author_name": "Luca Bergamini", "author_url": "", "post_date": "2020-09-13T20:25:47.433000", "content": "Thanks for reporting the issue in L5kit GitHub page @ryches, I'll report back there what I'll find!
", "votes": 0, "replies": [] } ] }, { "id": 1010197, "author_name": "Srinivas Gopal Krishna", "author_url": "", "post_date": "2020-09-14T15:15:22.440000", "content": "I may have found a work around. Change the max number of steps to 100 and correspondingly increase the number of epochs to make up for the same number of training examples. This dumps the memory after the first 100 iterations. This along with a batch size of 16 seems to work on the Kaggle kernel with GPU. However, the bottleneck is the rasterization, so it maybe better to just use CPU for this project.
", "votes": 0, "replies": [ { "id": 1010425, "author_name": "ryches", "author_url": "", "post_date": "2020-09-14T18:23:21.657000", "content": "hmm so maybe it only starts accumulating after taking a certain number of steps? Is that what this is implying?
", "votes": 0, "replies": [] }, { "id": 1010429, "author_name": "Peter", "author_url": "", "post_date": "2020-09-14T18:28:59.933000", "content": "@ryches For me, the leak was there when I only ran the loader/iterator. I mean without any deep learning (no move to cuda, no forward/backward, no optimizer, etc). I am not 100% sure why is fixed now, but I'll try to reproduce, maybe I'll find something useful.
", "votes": 0, "replies": [] }, { "id": 1010448, "author_name": "Srinivas Gopal Krishna", "author_url": "", "post_date": "2020-09-14T18:44:02.890000", "content": "Here is what I did. Put in a dummy generator that gets reassigned after the inner loop. It ran for over an hour but finally crashed because of memory issue. Previously it would not even run for 10 mins.
\n`for i in range(epochs):
\n def gen(dataloader):\n\n for batch in dataloader:\n\n yield batch\n train_gen = gen(train_dataloader)\n\n progress_bar = tqdm(range(cfg[\"train_params\"][\"max_num_steps\"]))\n for _ in progress_bar:\n try:\n data = next(train_gen)\n except StopIteration:\n tr_it = iter(train_dataloader)\n data = next(train_gen)`\nYeah I was able to narrow my issue down to just the dataloader as well. I was originally trying with my training and validation and thinking maybe my validation was accidentally accumulating gradients over time so I turned that off. Then I turned off training after to see if just data loading alone was causing it and that seems to be the case.
", "votes": 0, "replies": [] }, { "id": 1011413, "author_name": "Thomas Brandon", "author_url": "", "post_date": "2020-09-15T12:54:59.277000", "content": "\n\nmaybe it only starts accumulating after taking a certain number of steps? Is that what this is implying?
\n
PyTorch dataloader workers are created in the iterator. So restarting iteration will kill all the previous workers and release any leaked memory.
\nRestarting will also clear the LRU caches in the zarr dataset which otherwise seem liable to cause issues just through correct operation. With the default config of 16 workers each with a maximum cache of 1Gb that's 16Gb just for caching. Can't see any checks around available memory.
You can completely disable cache or reduce the memory footprint by tuning the parameters of the open() method in the ChunkedDataset
For me, a disabling caching solved the problem. Memory usage tends to fluctuate more now, but it doesn't accumulate and the process is no longer killed by the OS.
\nI didn't notice any measureable performance drop due to disabling caching.
I also didnt see any appreciable change in performance. I haven't inspected closely, but it seems like caching in this case isn't super useful. Maybe default should be set to False instead of True
", "votes": 0, "replies": [] }, { "id": 1020620, "author_name": "Luca Bergamini", "author_url": "", "post_date": "2020-09-21T10:00:12.197000", "content": "Yes, the issue is that zarr caches the raw chunk and not the decompressed one (no idea why that's not an option). That means it still needs to run blosc on the chunk even if it's cached
", "votes": 0, "replies": [] } ] }, { "id": 1008277, "author_name": "Khushal B", "author_url": "", "post_date": "2020-09-13T00:21:03.053000", "content": "I face the same issue when num_workers > 0 in the dataloader. I think it has to do with using dicts for loading the data. See here: https://github.com/pytorch/pytorch/issues/13246#issuecomment-445770039
I haven't tried it with the latest l5kit update yet though.
", "votes": 0, "replies": [] }, { "id": 1008039, "author_name": "ryches", "author_url": "", "post_date": "2020-09-12T18:00:24.023000", "content": "This is for the most part just code pulled from the example notebook from l5kit that was provided to us
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1008037": "https://www.kaggle.com/ryches/memory-leak\n\nI don't know if others have run into this but I have been fighting against a memory leak. I set up a minimal example to just iterate through the data and do nothing else. It isnt storing anything or accumulating gradients or anything, but memory usage seems to continue to drift overtime until eventually it kills the kernel. Have others run into this? Am I just doing something wrong?", "1056316": "Did anyone find the source of the memory leak? This keeps popping up from time to time!", "1008104": "The issue is not with your code, I had the same problem. After a few hours, my opened chrome tabs started to crash.. (I have 128Gb of RAM)\n\n\n\nI am using the latest code (l5kit repo's master branch) and it seems fixed now.", "1082898": "Are there any solutions for this? We are still experiencing this problem with the l5kit 1.1.0. It uses all my available physical memory (~11GB) in like 20 seconds and stay at 150-200MB free all the time.", "1011941": "I ran a data loader loop in a script (not within a notebook). It certainly eats up memory and CPU like crazy but seemed to stabilize. 4 workers stabilized at 6.4GB virtual, 3GB physical. It started at 4/2GB in first epoch and jumped to the stable value in epoch 2/3 and then stayed steady. \n\nBumping up the workers to 16, memory usage was 9GB virt, btw 3-4GB phys after epoch 3 with larger jumps per epoch. I killed it but suspect there is a steady state, just larger. There is likely a relationship between num workers and the steady state memory use. \n\nI feel the design as a while isn't optimal, each worker process zipping through the zarrs independently without co-ordination, but not sure how else you'd get something simple working without threading and tighter synchronization (needing C++).", "1008813": "I've been investigated this but It's quite difficult to pin this down to a specific component. It's not clear if this is:\n- a l5kit issue;\n- a torch issue;\n- a combination of both.\n\nWe use caching in the zarr, which may conflict with torch loaders (it's not clear whether the cache is entirely copied between workers).\n\nSomething you can try is disabling cache for the zarr dataset by using:\n\n`ChunkedDataset(\"path\").open(cached=False)`\n\nI'll take a deep look into this next week, as it's affecting multiple users", "1010197": "I may have found a work around. Change the max number of steps to 100 and correspondingly increase the number of epochs to make up for the same number of training examples. This dumps the memory after the first 100 iterations. This along with a batch size of 16 seems to work on the Kaggle kernel with GPU. However, the bottleneck is the rasterization, so it maybe better to just use CPU for this project. ", "1008277": "I face the same issue when `num_workers > 0` in the dataloader. I think it has to do with using dicts for loading the data. See here: [https://github.com/pytorch/pytorch/issues/13246#issuecomment-445770039](url)\n\nI haven't tried it with the latest l5kit update yet though.", "1008039": "This is for the most part just code pulled from the example notebook from l5kit that was provided to us" } }