I tried launching a kernel a few times in the last two hours - every time I end up with
\n\n\nTime\nLog Message \n2.71 [NbConvertApp] Converting notebook __notebook__.ipynb to notebook\n2.75 [NbConvertApp] Executing notebook with kernel: python3\n11.96 ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).\n
Pretty sure it's not a matter of my model size (re-running a previously successful kernel yields same error). Is anyone experiencing the same thing?
", "messages": [ { "id": "504624", "postDate": "03/31/2019 22:07:38", "content": "I tried launching a kernel a few times in the last two hours - every time I end up with
\n\n\nTime\nLog Message \n2.71 [NbConvertApp] Converting notebook __notebook__.ipynb to notebook\n2.75 [NbConvertApp] Executing notebook with kernel: python3\n11.96 ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).\n
Pretty sure it's not a matter of my model size (re-running a previously successful kernel yields same error). Is anyone experiencing the same thing?
", "rawMarkdown": "I tried launching a kernel a few times in the last two hours - every time I end up with \n\n```\nTime\nLog Message \n2.71 [NbConvertApp] Converting notebook __notebook__.ipynb to notebook\n2.75 [NbConvertApp] Executing notebook with kernel: python3\n11.96 ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).\n```\n\nPretty sure it's not a matter of my model size (re-running a previously successful kernel yields same error). Is anyone experiencing the same thing?", "votes": null }, { "id": "504795", "postDate": "04/01/2019 07:12:29", "content": "See https://www.kaggle.com/product-feedback/72606 (I also mentioned on this forum here https://www.kaggle.com/c/imet-2019-fgvc6/discussion/87074#504053) - hopefully this will be fixed.\nIn the meantime, I switched to a multi-threaded pytorch generator, works ok for images where libraries release the GIL anyway.
", "rawMarkdown": "See https://www.kaggle.com/product-feedback/72606 (I also mentioned on this forum here https://www.kaggle.com/c/imet-2019-fgvc6/discussion/87074#504053) - hopefully this will be fixed.\nIn the meantime, I switched to a multi-threaded pytorch generator, works ok for images where libraries release the GIL anyway.", "votes": null }, { "id": "504800", "postDate": "04/01/2019 07:24:27", "content": "Thanks!
", "rawMarkdown": "Thanks!", "votes": null }, { "id": "505528", "postDate": "04/02/2019 06:50:24", "content": "I stuck with the same problem using pytorch and have to set num_workers to 0.\nCan you share your code for image generator please?
", "rawMarkdown": "I stuck with the same problem using pytorch and have to set num_workers to 0.\nCan you share your code for image generator please?", "votes": null }, { "id": "505606", "postDate": "04/02/2019 09:18:51", "content": "Sure, here it is (python 3) - didn't test it much, does not support all features of default DataLoader, but seems to work for me:\n```\nfrom multiprocessing.pool import ThreadPool\nfrom torch.utils.data import DataLoader
\n\nclass ThreadingDataLoader(DataLoader):\n def iter(self):\n sample_iter = iter(self.batch_sampler)\n if self.num_workers == 0:\n for indices in sample_iter:\n yield self.collate_fn([self._get_item(i) for i in indices])\n else:\n prefetch = 1\n with ThreadPool(processes=self.num_workers) as pool:\n futures = []\n for indices in sample_iter:\n futures.append([pool.apply_async(self._get_item, args=(i,))\n for i in indices])\n if len(futures) > prefetch:\n yield self.collate_fn([f.get() for f in futures.pop(0)])\n for batch_futures in futures:\n yield self.collate_fn([f.get() for f in batch_futures])
\n\ndef _get_item(self, i):\n return self.dataset[i]\n```
", "rawMarkdown": "Sure, here it is (python 3) - didn't test it much, does not support all features of default DataLoader, but seems to work for me:\n```\nfrom multiprocessing.pool import ThreadPool\nfrom torch.utils.data import DataLoader\n\nclass ThreadingDataLoader(DataLoader):\n def __iter__(self):\n sample_iter = iter(self.batch_sampler)\n if self.num_workers == 0:\n for indices in sample_iter:\n yield self.collate_fn([self._get_item(i) for i in indices])\n else:\n prefetch = 1\n with ThreadPool(processes=self.num_workers) as pool:\n futures = []\n for indices in sample_iter:\n futures.append([pool.apply_async(self._get_item, args=(i,))\n for i in indices])\n if len(futures) > prefetch:\n yield self.collate_fn([f.get() for f in futures.pop(0)])\n for batch_futures in futures:\n yield self.collate_fn([f.get() for f in batch_futures])\n\n def _get_item(self, i):\n return self.dataset[i]\n```", "votes": null }, { "id": "505640", "postDate": "04/02/2019 10:35:24", "content": "Thanks!\nIts kinda sad when batch creation takes ~80% of training time.
", "rawMarkdown": "Thanks!\nIts kinda sad when batch creation takes ~80% of training time.", "votes": null }, { "id": "506269", "postDate": "04/03/2019 08:16:58", "content": "If you’re having kernel problems I feel bad for you son, I’ve got 99 problems but an unexpected bus error ain’t one.
", "rawMarkdown": "If you’re having kernel problems I feel bad for you son, I’ve got 99 problems but an unexpected bus error ain’t one.", "votes": null } ], "comments": [ { "id": 504795, "author_name": "lopuhin", "author_url": "", "post_date": "04/01/2019 07:12:29", "content": "See https://www.kaggle.com/product-feedback/72606 (I also mentioned on this forum here https://www.kaggle.com/c/imet-2019-fgvc6/discussion/87074#504053) - hopefully this will be fixed.\nIn the meantime, I switched to a multi-threaded pytorch generator, works ok for images where libraries release the GIL anyway.
", "votes": null, "replies": [ { "id": 504800, "author_name": "konradb", "author_url": "", "post_date": "04/01/2019 07:24:27", "content": "Thanks!
", "votes": null, "replies": [] }, { "id": 505528, "author_name": "vlad0922", "author_url": "", "post_date": "04/02/2019 06:50:24", "content": "I stuck with the same problem using pytorch and have to set num_workers to 0.\nCan you share your code for image generator please?
", "votes": null, "replies": [] }, { "id": 505606, "author_name": "lopuhin", "author_url": "", "post_date": "04/02/2019 09:18:51", "content": "Sure, here it is (python 3) - didn't test it much, does not support all features of default DataLoader, but seems to work for me:\n```\nfrom multiprocessing.pool import ThreadPool\nfrom torch.utils.data import DataLoader
\n\nclass ThreadingDataLoader(DataLoader):\n def iter(self):\n sample_iter = iter(self.batch_sampler)\n if self.num_workers == 0:\n for indices in sample_iter:\n yield self.collate_fn([self._get_item(i) for i in indices])\n else:\n prefetch = 1\n with ThreadPool(processes=self.num_workers) as pool:\n futures = []\n for indices in sample_iter:\n futures.append([pool.apply_async(self._get_item, args=(i,))\n for i in indices])\n if len(futures) > prefetch:\n yield self.collate_fn([f.get() for f in futures.pop(0)])\n for batch_futures in futures:\n yield self.collate_fn([f.get() for f in batch_futures])
\n\ndef _get_item(self, i):\n return self.dataset[i]\n```
", "votes": null, "replies": [] }, { "id": 505640, "author_name": "vlad0922", "author_url": "", "post_date": "04/02/2019 10:35:24", "content": "Thanks!\nIts kinda sad when batch creation takes ~80% of training time.
", "votes": null, "replies": [] } ] }, { "id": 506269, "author_name": "interneuron", "author_url": "", "post_date": "04/03/2019 08:16:58", "content": "If you’re having kernel problems I feel bad for you son, I’ve got 99 problems but an unexpected bus error ain’t one.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "504624": "I tried launching a kernel a few times in the last two hours - every time I end up with \n\n```\nTime\nLog Message \n2.71 [NbConvertApp] Converting notebook __notebook__.ipynb to notebook\n2.75 [NbConvertApp] Executing notebook with kernel: python3\n11.96 ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).\n```\n\nPretty sure it's not a matter of my model size (re-running a previously successful kernel yields same error). Is anyone experiencing the same thing?", "504795": "See https://www.kaggle.com/product-feedback/72606 (I also mentioned on this forum here https://www.kaggle.com/c/imet-2019-fgvc6/discussion/87074#504053) - hopefully this will be fixed.\nIn the meantime, I switched to a multi-threaded pytorch generator, works ok for images where libraries release the GIL anyway.", "504800": "Thanks!", "505528": "I stuck with the same problem using pytorch and have to set num_workers to 0.\nCan you share your code for image generator please?", "505606": "Sure, here it is (python 3) - didn't test it much, does not support all features of default DataLoader, but seems to work for me:\n```\nfrom multiprocessing.pool import ThreadPool\nfrom torch.utils.data import DataLoader\n\nclass ThreadingDataLoader(DataLoader):\n def __iter__(self):\n sample_iter = iter(self.batch_sampler)\n if self.num_workers == 0:\n for indices in sample_iter:\n yield self.collate_fn([self._get_item(i) for i in indices])\n else:\n prefetch = 1\n with ThreadPool(processes=self.num_workers) as pool:\n futures = []\n for indices in sample_iter:\n futures.append([pool.apply_async(self._get_item, args=(i,))\n for i in indices])\n if len(futures) > prefetch:\n yield self.collate_fn([f.get() for f in futures.pop(0)])\n for batch_futures in futures:\n yield self.collate_fn([f.get() for f in batch_futures])\n\n def _get_item(self, i):\n return self.dataset[i]\n```", "505640": "Thanks!\nIts kinda sad when batch creation takes ~80% of training time.", "506269": "If you’re having kernel problems I feel bad for you son, I’ve got 99 problems but an unexpected bus error ain’t one." }, "source": "meta" }