`---------------------------------------------------------------------------
\nValueError Traceback (most recent call last)
\n in
\n 10
\n 11 from l5kit.data import LocalDataManager, ChunkedDataset
\n---> 12 from l5kit.dataset import AgentDataset, EgoDataset
\n 13 from l5kit.rasterization import build_rasterizer
~\\Anaconda3\\lib\\site-packages\\l5kit\\dataset__init__.py in
\n----> 1 from .agent import AgentDataset
\n 2 from .ego import EgoDataset
\n 3 from .select_agents import select_agents
\n 4
\n 5 all = [\"EgoDataset\", \"AgentDataset\", \"select_agents\"]
~\\Anaconda3\\lib\\site-packages\\l5kit\\dataset\\agent.py in
\n 10 from ..rasterization import Rasterizer
\n 11 from .ego import EgoDataset
\n---> 12 from .select_agents import TH_DISTANCE_AV, TH_EXTENT_RATIO, TH_YAW_DEGREE, select_agents
\n 13
\n 14 # WARNING: changing these values impact the number of instances selected for both train and inference!
~\\Anaconda3\\lib\\site-packages\\l5kit\\dataset\\select_agents.py in
\n 19 from l5kit.data.filter import _get_label_filter # TODO expose this without digging
\n 20
\n---> 21 multiprocessing.set_start_method(\"fork\", force=True) # this fix loop in python 3.8 on MacOS
\n 22 os.environ[\"BLOSC_NOLOCK\"] = \"1\" # this is required for multiprocessing
\n 23
~\\Anaconda3\\lib\\multiprocessing\\context.py in set_start_method(self, method, force)
\n 244 self._actual_context = None
\n 245 return
\n--> 246 self._actual_context = self.get_context(method)
\n 247
\n 248 def get_start_method(self, allow_none=False):
~\\Anaconda3\\lib\\multiprocessing\\context.py in get_context(self, method)
\n 236 return self._actual_context
\n 237 else:
\n--> 238 return super().get_context(method)
\n 239
\n 240 def set_start_method(self, method, force=False):
~\\Anaconda3\\lib\\multiprocessing\\context.py in get_context(self, method)
\n 190 ctx = _concrete_contexts[method]
\n 191 except KeyError:
\n--> 192 raise ValueError('cannot find context for %r' % method) from None
\n 193 ctx._check_available()
\n 194 return ctx
ValueError: cannot find context for 'fork'`
", "messages": [ { "id": 987083, "postDate": "2020-08-27T01:57:24.330Z", "content": "`---------------------------------------------------------------------------
\nValueError Traceback (most recent call last)
\n in
\n 10
\n 11 from l5kit.data import LocalDataManager, ChunkedDataset
\n---> 12 from l5kit.dataset import AgentDataset, EgoDataset
\n 13 from l5kit.rasterization import build_rasterizer
~\\Anaconda3\\lib\\site-packages\\l5kit\\dataset__init__.py in
\n----> 1 from .agent import AgentDataset
\n 2 from .ego import EgoDataset
\n 3 from .select_agents import select_agents
\n 4
\n 5 all = [\"EgoDataset\", \"AgentDataset\", \"select_agents\"]
~\\Anaconda3\\lib\\site-packages\\l5kit\\dataset\\agent.py in
\n 10 from ..rasterization import Rasterizer
\n 11 from .ego import EgoDataset
\n---> 12 from .select_agents import TH_DISTANCE_AV, TH_EXTENT_RATIO, TH_YAW_DEGREE, select_agents
\n 13
\n 14 # WARNING: changing these values impact the number of instances selected for both train and inference!
~\\Anaconda3\\lib\\site-packages\\l5kit\\dataset\\select_agents.py in
\n 19 from l5kit.data.filter import _get_label_filter # TODO expose this without digging
\n 20
\n---> 21 multiprocessing.set_start_method(\"fork\", force=True) # this fix loop in python 3.8 on MacOS
\n 22 os.environ[\"BLOSC_NOLOCK\"] = \"1\" # this is required for multiprocessing
\n 23
~\\Anaconda3\\lib\\multiprocessing\\context.py in set_start_method(self, method, force)
\n 244 self._actual_context = None
\n 245 return
\n--> 246 self._actual_context = self.get_context(method)
\n 247
\n 248 def get_start_method(self, allow_none=False):
~\\Anaconda3\\lib\\multiprocessing\\context.py in get_context(self, method)
\n 236 return self._actual_context
\n 237 else:
\n--> 238 return super().get_context(method)
\n 239
\n 240 def set_start_method(self, method, force=False):
~\\Anaconda3\\lib\\multiprocessing\\context.py in get_context(self, method)
\n 190 ctx = _concrete_contexts[method]
\n 191 except KeyError:
\n--> 192 raise ValueError('cannot find context for %r' % method) from None
\n 193 ctx._check_available()
\n 194 return ctx
ValueError: cannot find context for 'fork'`
", "rawMarkdown": "`---------------------------------------------------------------------------\nValueError Traceback (most recent call last)\nThe fork multiprocessing method is not supported in Windows (discussed a little here.
\nIt looks like the main use of this function is for creating a chopped validation dataset. So you may be able to just create the chopped dataset in linux but otherwise use windows. Though you may need to modify the sample code to directly use a pre-chopped validation dataset as I believe it will try to recreate it even if it exists (create_chopped_dataset will create a folder called sample_chopped_100 when you run it on the sample dataset, just use this as the key for dm.requires and load it as in the example, adding the mask and ground truth csv stuff).
You could use Kaggle to run create_chopped_dataset and then download that. Or you could set up WSL (Windows Subsystem for Linux) which lets you run a linux environment on windows easily. WSL2 has recently been released which has full linux support (some issues with WSL1 but may be fine for l5kit). This lets you run linux programs and input/output from your windows files. So would work well here. No GPU access but that is actually coming, available in latest windows insider builds so when released (hopefully next year) you'll be able to run the fuil Linux PyTorch/TF on Windows with CUDA. So worth learning as a programmer who uses windows.
Thank you so much for your detailed explanation.I am using WSL now,it worked well.đź‘Ť
", "rawMarkdown": "Thank you so much for your detailed explanation.I am using WSL now,it worked well.👍" }, { "id": 992120, "postDate": "2020-08-31T00:45:47.567Z", "content": "Can I ask how did you pip3 install l5kit for that WSL because the regular -r requirements.txt exits with code 1. Telling me that directory site-packages doesn't exist in home/usr/../python3.8/ and it doesn't have permissions. Which is true but most are getting installed into home//.local/…./python3.8/site-packages. So I tried changing --target=that directory. But then it exits with code 1, and says error: option --home not recognized. The entire error is a bit long.
", "rawMarkdown": "Can I ask how did you pip3 install l5kit for that WSL because the regular -r requirements.txt exits with code 1. Telling me that directory site-packages doesn't exist in home/usr/../python3.8/ and it doesn't have permissions. Which is true but most are getting installed into home/You use pip3 install --user l5kit to install to .local. Or you can use sudo pip3 install l5kit to install system-wide (sudo is super-user do which is basically run as administrator for linux). But installing things system-wide outside of the system package manager is not considered good practice.
Thanks --user worked perfectly.
", "rawMarkdown": "Thanks --user worked perfectly. " }, { "id": 993498, "postDate": "2020-09-01T01:24:25.540Z", "content": "@thomasbrandon are you sure it is possible to create chopped_data_100 from kaggle kernel? because 'chop_dataset function' is calling 'zarr_scenes_chop'. And It tries to write in input directory but input directory is read only. So we have to find work around in kaggle kernel right?
", "rawMarkdown": "@thomasbrandon are you sure it is possible to create chopped_data_100 from kaggle kernel? because 'chop_dataset function' is calling 'zarr_scenes_chop'. And It tries to write in input directory but input directory is read only. So we have to find work around in kaggle kernel right?" }, { "id": 993824, "postDate": "2020-09-01T06:59:38.067Z", "content": "Yeah, parent of location needs to be writeable as it doesn't let you set output location. I copied it into /tmp. I've only done that with the sample set running the Lyft example so may be some space issues with copying the full validate set. Should also be able to symlink it (something like !mkdir /tmp/lyft && ln -s {eval_data} /tmp/lyft/validate and then point create-chopped_dataset to /tmp/lyft/validate).
I don’t think this solution works. Or maybe I am just missing something. Thanks for the info though!
", "rawMarkdown": "I don’t think this solution works. Or maybe I am just missing something. Thanks for the info though! " }, { "id": 994522, "postDate": "2020-09-01T16:54:50.633Z", "content": "Both ways (copy and symlink) seem to work fine for me (creates and loads and I've evaluated with one, just on sample as noted). As shown here.
", "rawMarkdown": "Both ways (copy and symlink) seem to work fine for me (creates and loads and I've evaluated with one, just on sample as noted). As shown [here](https://www.kaggle.com/thomasbrandon/l5kit-chopped-dataset).", "votes": 1 }, { "id": 994527, "postDate": "2020-09-01T17:04:51.230Z", "content": "@thomasbrandon you are really kind for creating notebook. Thanks for that. I did manage to do this not so sophisticated like you showed. so I give you that. Question is can we do for validate.zarr? scene.zarr occupies more than 500 MB of HDD on kernel. The max HDD limit is 4 GB something. I tried it once and it said cant copy because limit exceeded. Am I still missing something? I will also post this comment on the notebook for others.
", "rawMarkdown": "@thomasbrandon you are really kind for creating notebook. Thanks for that. I did manage to do this not so sophisticated like you showed. so I give you that. Question is can we do for validate.zarr? scene.zarr occupies more than 500 MB of HDD on kernel. The max HDD limit is 4 GB something. I tried it once and it said cant copy because limit exceeded. Am I still missing something? I will also post this comment on the notebook for others. " }, { "id": 995414, "postDate": "2020-09-02T12:26:02.590Z", "content": "I think it should just fit. The chopped_100_validate dataset I have is 3.5Gb (the whole validate is 8.5G). So think that will fit within the limits for kernel outputs. You would need to symlink as won't fit both the full validate and the chopped (I gather your error was trying to copy the whole validate, the symlink method won't use any space for the original data).
", "rawMarkdown": "I think it should just fit. The chopped_100_validate dataset I have is 3.5Gb (the whole validate is 8.5G). So think that will fit within the limits for kernel outputs. You would need to symlink as won't fit both the full validate and the chopped (I gather your error was trying to copy the whole validate, the symlink method won't use any space for the original data).", "votes": 1 } ] }, { "id": 988355, "postDate": "2020-08-28T01:58:12.303Z", "content": "Here is the trick I used to fix it.
\nThe reason this does not work is windows have no multiprocessing option. ;-(
\nComment out these lines in the source code of the library
\nmultiprocessing.set_start_method(\"fork\", force=True) # this fix loop in python 3.8 on MacOSos.environ[\"BLOSC_NOLOCK\"] = \"1\" # this is required for multiprocessingUpvote if it helped, comment if not I will try my best to solve it.
", "rawMarkdown": "Here is the trick I used to fix it.\n\nThe reason this does not work is windows have no multiprocessing option. ;-(\n\nComment out these lines in the source code of the library\n\n21. `multiprocessing.set_start_method(\"fork\", force=True) # this fix loop in python 3.8 on MacOS`\n22. `os.environ[\"BLOSC_NOLOCK\"] = \"1\" # this is required for multiprocessing`\n\nUpvote if it helped, comment if not I will try my best to solve it.", "votes": 4 }, { "id": 988266, "postDate": "2020-08-27T23:48:04.833Z", "content": "\nnote that the package developer mentioned that issue here.
", "rawMarkdown": "@senkin13 ,\n\nnote that the package developer mentioned that issue [here](https://github.com/lyft/l5kit/issues/123#issuecomment-680825552).", "votes": 1 }, { "id": 988011, "postDate": "2020-08-27T17:38:14.087Z", "content": "It works if you set num_workers to zero in the DataLoader. It's not ideal, but it might be a way to save a chunked dataset without installing Linux.
", "rawMarkdown": "It works if you set num_workers to zero in the DataLoader. It's not ideal, but it might be a way to save a chunked dataset without installing Linux.", "votes": 2 }, { "id": 993012, "postDate": "2020-08-31T15:32:07.757Z", "content": "On windows I created an Anaconda environment and installed Pytorch 1.5.0 and Torchvison first. I then 'pip' installed l5kit and it seemed to work but was getting cannot find contact for 'fork'.
\nI then commented out only this line in the source code of the library
\nmultiprocessing.set_start_method(\"fork\", force=True) # this fix loop in python 3.8 on MacOS
This seems to be working fine without changing num_workers=0
May I ask you your Python version ?
\nI cannot work without num_workers=0. My version is Python3.8, torch 1.5.1+cu101 and torchvision 0.6.1+cu101.
To run l5kit on windows 10:
\nIf you get multiprocessing error while importing
\nComment out multiprocessing line
Set cfg['val_data_loader']['num_workers']=0
\nThat's it rest will work with cuda if you have pytorch cuda installed
\nFor conda users
\nIn a empty env…
\n1) conda install pytorch==1.5.0 torchvision==0.6.0 cudatoolkit=10.2 -c pytorch
\n2) pip install l5kit
@preethamrakshithp Fixed it
", "rawMarkdown": "@preethamrakshithp ~~I am running same version and have set num workers to 0. But I still have the error. The same described > https://github.com/lyft/l5kit/issues/130 here. Did you guys end up finding solution?~~ Fixed it", "votes": 1 } ] }, { "id": 987630, "postDate": "2020-08-27T12:12:26.547Z", "content": "same error
", "rawMarkdown": "same error", "replies": [ { "id": 987759, "postDate": "2020-08-27T13:49:30.743Z", "content": "I think it's not easy to use l5kit with windows,let's change our os to linux :)
", "rawMarkdown": "I think it's not easy to use l5kit with windows,let's change our os to linux :)", "votes": 2 }, { "id": 989510, "postDate": "2020-08-28T21:29:57.153Z", "content": "Errors even on Mac. Not same but kernel crashes.
", "rawMarkdown": "Errors even on Mac. Not same but kernel crashes." } ] }, { "id": 987088, "postDate": "2020-08-27T02:04:49.717Z", "content": "is it possible to save AgentDataset to other data format with kaggle notebook then download to local?
", "rawMarkdown": "is it possible to save AgentDataset to other data format with kaggle notebook then download to local?", "replies": [ { "id": 1001881, "postDate": "2020-09-07T16:37:34.423Z", "content": "you can check this out for that. https://www.kaggle.com/kneroma/zarr-files-and-l5kit-data-for-dummies
\nBTW how are you storing 22 GB data locally? I dont have that much space is there any other way?
The fork multiprocessing method is not supported in Windows (discussed a little here.
\nIt looks like the main use of this function is for creating a chopped validation dataset. So you may be able to just create the chopped dataset in linux but otherwise use windows. Though you may need to modify the sample code to directly use a pre-chopped validation dataset as I believe it will try to recreate it even if it exists (create_chopped_dataset will create a folder called sample_chopped_100 when you run it on the sample dataset, just use this as the key for dm.requires and load it as in the example, adding the mask and ground truth csv stuff).
You could use Kaggle to run create_chopped_dataset and then download that. Or you could set up WSL (Windows Subsystem for Linux) which lets you run a linux environment on windows easily. WSL2 has recently been released which has full linux support (some issues with WSL1 but may be fine for l5kit). This lets you run linux programs and input/output from your windows files. So would work well here. No GPU access but that is actually coming, available in latest windows insider builds so when released (hopefully next year) you'll be able to run the fuil Linux PyTorch/TF on Windows with CUDA. So worth learning as a programmer who uses windows.
Thank you so much for your detailed explanation.I am using WSL now,it worked well.đź‘Ť
", "votes": 0, "replies": [] }, { "id": 992120, "author_name": "Todd Chaney", "author_url": "", "post_date": "2020-08-31T00:45:47.567000", "content": "Can I ask how did you pip3 install l5kit for that WSL because the regular -r requirements.txt exits with code 1. Telling me that directory site-packages doesn't exist in home/usr/../python3.8/ and it doesn't have permissions. Which is true but most are getting installed into home//.local/…./python3.8/site-packages. So I tried changing --target=that directory. But then it exits with code 1, and says error: option --home not recognized. The entire error is a bit long.
", "votes": 0, "replies": [] }, { "id": 992683, "author_name": "Thomas Brandon", "author_url": "", "post_date": "2020-08-31T10:51:07.927000", "content": "You use pip3 install --user l5kit to install to .local. Or you can use sudo pip3 install l5kit to install system-wide (sudo is super-user do which is basically run as administrator for linux). But installing things system-wide outside of the system package manager is not considered good practice.
Thanks --user worked perfectly.
", "votes": 0, "replies": [] }, { "id": 993498, "author_name": "The Brown Iceman", "author_url": "", "post_date": "2020-09-01T01:24:25.540000", "content": "@thomasbrandon are you sure it is possible to create chopped_data_100 from kaggle kernel? because 'chop_dataset function' is calling 'zarr_scenes_chop'. And It tries to write in input directory but input directory is read only. So we have to find work around in kaggle kernel right?
", "votes": 0, "replies": [] }, { "id": 993824, "author_name": "Thomas Brandon", "author_url": "", "post_date": "2020-09-01T06:59:38.067000", "content": "Yeah, parent of location needs to be writeable as it doesn't let you set output location. I copied it into /tmp. I've only done that with the sample set running the Lyft example so may be some space issues with copying the full validate set. Should also be able to symlink it (something like !mkdir /tmp/lyft && ln -s {eval_data} /tmp/lyft/validate and then point create-chopped_dataset to /tmp/lyft/validate).
I don’t think this solution works. Or maybe I am just missing something. Thanks for the info though!
", "votes": 0, "replies": [] }, { "id": 994522, "author_name": "Thomas Brandon", "author_url": "", "post_date": "2020-09-01T16:54:50.633000", "content": "Both ways (copy and symlink) seem to work fine for me (creates and loads and I've evaluated with one, just on sample as noted). As shown here.
", "votes": 1, "replies": [] }, { "id": 994527, "author_name": "The Brown Iceman", "author_url": "", "post_date": "2020-09-01T17:04:51.230000", "content": "@thomasbrandon you are really kind for creating notebook. Thanks for that. I did manage to do this not so sophisticated like you showed. so I give you that. Question is can we do for validate.zarr? scene.zarr occupies more than 500 MB of HDD on kernel. The max HDD limit is 4 GB something. I tried it once and it said cant copy because limit exceeded. Am I still missing something? I will also post this comment on the notebook for others.
", "votes": 0, "replies": [] }, { "id": 995414, "author_name": "Thomas Brandon", "author_url": "", "post_date": "2020-09-02T12:26:02.590000", "content": "I think it should just fit. The chopped_100_validate dataset I have is 3.5Gb (the whole validate is 8.5G). So think that will fit within the limits for kernel outputs. You would need to symlink as won't fit both the full validate and the chopped (I gather your error was trying to copy the whole validate, the symlink method won't use any space for the original data).
", "votes": 1, "replies": [] } ] }, { "id": 988355, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2020-08-28T01:58:12.303000", "content": "Here is the trick I used to fix it.
\nThe reason this does not work is windows have no multiprocessing option. ;-(
\nComment out these lines in the source code of the library
\nmultiprocessing.set_start_method(\"fork\", force=True) # this fix loop in python 3.8 on MacOSos.environ[\"BLOSC_NOLOCK\"] = \"1\" # this is required for multiprocessingUpvote if it helped, comment if not I will try my best to solve it.
", "votes": 4, "replies": [] }, { "id": 988266, "author_name": "nosound", "author_url": "", "post_date": "2020-08-27T23:48:04.833000", "content": "\nnote that the package developer mentioned that issue here.
", "votes": 1, "replies": [] }, { "id": 988011, "author_name": "Brian Smith", "author_url": "", "post_date": "2020-08-27T17:38:14.087000", "content": "It works if you set num_workers to zero in the DataLoader. It's not ideal, but it might be a way to save a chunked dataset without installing Linux.
", "votes": 2, "replies": [] }, { "id": 993012, "author_name": "Srinivas Gopal Krishna", "author_url": "", "post_date": "2020-08-31T15:32:07.757000", "content": "On windows I created an Anaconda environment and installed Pytorch 1.5.0 and Torchvison first. I then 'pip' installed l5kit and it seemed to work but was getting cannot find contact for 'fork'.
\nI then commented out only this line in the source code of the library
\nmultiprocessing.set_start_method(\"fork\", force=True) # this fix loop in python 3.8 on MacOS
This seems to be working fine without changing num_workers=0
May I ask you your Python version ?
\nI cannot work without num_workers=0. My version is Python3.8, torch 1.5.1+cu101 and torchvision 0.6.1+cu101.
To run l5kit on windows 10:
\nIf you get multiprocessing error while importing
\nComment out multiprocessing line
Set cfg['val_data_loader']['num_workers']=0
\nThat's it rest will work with cuda if you have pytorch cuda installed
\nFor conda users
\nIn a empty env…
\n1) conda install pytorch==1.5.0 torchvision==0.6.0 cudatoolkit=10.2 -c pytorch
\n2) pip install l5kit
@preethamrakshithp Fixed it
", "votes": 1, "replies": [] } ] }, { "id": 987630, "author_name": "th-blitz", "author_url": "", "post_date": "2020-08-27T12:12:26.547000", "content": "same error
", "votes": 0, "replies": [ { "id": 987759, "author_name": "senkin13", "author_url": "", "post_date": "2020-08-27T13:49:30.743000", "content": "I think it's not easy to use l5kit with windows,let's change our os to linux :)
", "votes": 2, "replies": [] }, { "id": 989510, "author_name": "Srinivas Gopal Krishna", "author_url": "", "post_date": "2020-08-28T21:29:57.153000", "content": "Errors even on Mac. Not same but kernel crashes.
", "votes": 0, "replies": [] } ] }, { "id": 987088, "author_name": "senkin13", "author_url": "", "post_date": "2020-08-27T02:04:49.717000", "content": "is it possible to save AgentDataset to other data format with kaggle notebook then download to local?
", "votes": 0, "replies": [ { "id": 1001881, "author_name": "SujaydKhandekar", "author_url": "", "post_date": "2020-09-07T16:37:34.423000", "content": "you can check this out for that. https://www.kaggle.com/kneroma/zarr-files-and-l5kit-data-for-dummies
\nBTW how are you storing 22 GB data locally? I dont have that much space is there any other way?