Hello!
\nMy notebook does some preprocessing tasks using joblib and saves results to disk.
\nOn my local machine, after tasks completion, preprocessed results take up about 35 Gb of disk space. And on session metrics of kaggle notebook they say that I can use up to 73.1 Gb.
\nWhen I ran my code on kaggle notebook a few days ago, everything was fine.
But today, when I was running the almost same code (on kaggle), first I got warning:
\n/opt/conda/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py:691: UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak.\n \"timeout or by a memory leak.\", UserWarning\nand then I got error:
\nException in thread Thread-5:\nTraceback (most recent call last):\n File \"/opt/conda/lib/python3.7/threading.py\", line 926, in _bootstrap_inner\n self.run()\n File \"/opt/conda/lib/python3.7/threading.py\", line 870, in run\n self._target(*self._args, **self._kwargs)\n File \"<ipython-input-7-ef91b9b8e8f9>\", line 306, in write_queue_messages\n output_file.write(f\"{exception_info[0]}: {exception_info[1]}\")\nOSError: [Errno 28] No space left on device\nI did factory reset and then set JOBLIB_TEMP_FOLDER=/tmp/<notebook_id>/joblib_tmp
\nand set save path for preprocessed data to /tmp/<notebook_id>/all_preprocessed
\nbut it didn't help.
Do you have any ideas why did it happen and how to circumvent this problem?
", "messages": [ { "id": "1325664", "postDate": "05/27/2021 23:11:34", "content": "Hello!
\nMy notebook does some preprocessing tasks using joblib and saves results to disk.
\nOn my local machine, after tasks completion, preprocessed results take up about 35 Gb of disk space. And on session metrics of kaggle notebook they say that I can use up to 73.1 Gb.
\nWhen I ran my code on kaggle notebook a few days ago, everything was fine.
But today, when I was running the almost same code (on kaggle), first I got warning:
\n/opt/conda/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py:691: UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak.\n \"timeout or by a memory leak.\", UserWarning\nand then I got error:
\nException in thread Thread-5:\nTraceback (most recent call last):\n File \"/opt/conda/lib/python3.7/threading.py\", line 926, in _bootstrap_inner\n self.run()\n File \"/opt/conda/lib/python3.7/threading.py\", line 870, in run\n self._target(*self._args, **self._kwargs)\n File \"<ipython-input-7-ef91b9b8e8f9>\", line 306, in write_queue_messages\n output_file.write(f\"{exception_info[0]}: {exception_info[1]}\")\nOSError: [Errno 28] No space left on device\nI did factory reset and then set JOBLIB_TEMP_FOLDER=/tmp/<notebook_id>/joblib_tmp
\nand set save path for preprocessed data to /tmp/<notebook_id>/all_preprocessed
\nbut it didn't help.
Do you have any ideas why did it happen and how to circumvent this problem?
", "rawMarkdown": "Hello!\n\nMy notebook does some preprocessing tasks using joblib and saves results to disk. \nOn my local machine, after tasks completion, preprocessed results take up about 35 Gb of disk space. And on session metrics of kaggle notebook they say that I can use up to 73.1 Gb. \nWhen I ran my code on kaggle notebook a few days ago, everything was fine. \n\nBut today, when I was running the almost same code (on kaggle), first I got warning:\n```\n/opt/conda/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py:691: UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak.\n \"timeout or by a memory leak.\", UserWarning\n```\n\nand then I got error:\n```\nException in thread Thread-5:\nTraceback (most recent call last):\n File \"/opt/conda/lib/python3.7/threading.py\", line 926, in _bootstrap_inner\n self.run()\n File \"/opt/conda/lib/python3.7/threading.py\", line 870, in run\n self._target(*self._args, **self._kwargs)\n File \"You are getting this error while running the kernel yourself right? If so, people are saying that there's some technical issue on kaggle today, it maybe related.
\nCheck it out
\nhttps://www.kaggle.com/c/birdclef-2021/discussion/242161
", "rawMarkdown": "You are getting this error while running the kernel yourself right? If so, people are saying that there's some technical issue on kaggle today, it maybe related.\n\nCheck it out\n\nhttps://www.kaggle.com/c/birdclef-2021/discussion/242161", "votes": null } ], "comments": [ { "id": 1325675, "author_name": "victorasso", "author_url": "", "post_date": "05/27/2021 23:49:18", "content": "You are getting this error while running the kernel yourself right? If so, people are saying that there's some technical issue on kaggle today, it maybe related.
\nCheck it out
\nhttps://www.kaggle.com/c/birdclef-2021/discussion/242161
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1325664": "Hello!\n\nMy notebook does some preprocessing tasks using joblib and saves results to disk. \nOn my local machine, after tasks completion, preprocessed results take up about 35 Gb of disk space. And on session metrics of kaggle notebook they say that I can use up to 73.1 Gb. \nWhen I ran my code on kaggle notebook a few days ago, everything was fine. \n\nBut today, when I was running the almost same code (on kaggle), first I got warning:\n```\n/opt/conda/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py:691: UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak.\n \"timeout or by a memory leak.\", UserWarning\n```\n\nand then I got error:\n```\nException in thread Thread-5:\nTraceback (most recent call last):\n File \"/opt/conda/lib/python3.7/threading.py\", line 926, in _bootstrap_inner\n self.run()\n File \"/opt/conda/lib/python3.7/threading.py\", line 870, in run\n self._target(*self._args, **self._kwargs)\n File \"