{
  "id": 230134,
  "title": "OOM error when using torch_xla",
  "url": "/competitions/bms-molecular-translation/discussion/230134",
  "author_name": "Prakash Pandey",
  "post_date": "2021-04-02T06:55:41.333000",
  "votes": 3,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Hello everyone !<br>\nI am trying to train my model on the TPU using torch_xla, but it shows \"out of memory\" error. I even tried with batch size = 1, but did not get through. Please help.</p>\n<p>The error message looks like the following :</p>\n<p>`ProcessExitedException                    Traceback (most recent call last)<br>\n in <br>\n      1 if <strong>name</strong> == '<strong>main</strong>':<br>\n      2     flags = {}<br>\n----&gt; 3     xmp.spawn(_mp_fn, args = (flags,), nprocs=8, start_method='fork')</p>\n<p>/opt/conda/lib/python3.7/site-packages/torch_xla/distributed/xla_multiprocessing.py in spawn(fn, args, nprocs, join, daemon, start_method)<br>\n    392         join=join,<br>\n    393         daemon=daemon,<br>\n--&gt; 394         start_method=start_method)<br>\n    395 <br>\n    396 </p>\n<p>/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py in start_processes(fn, args, nprocs, join, daemon, start_method)<br>\n    186 <br>\n    187     # Loop on join until it returns True or raises an exception.<br>\n--&gt; 188     while not context.join():<br>\n    189         pass<br>\n    190 </p>\n<p>/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py in join(self, timeout)<br>\n    134                     error_pid=failed_process.pid,<br>\n    135                     exit_code=exitcode,<br>\n--&gt; 136                     signal_name=name<br>\n    137                 )<br>\n    138             else:</p>\n<p>ProcessExitedException: process 4 terminated with signal SIGKILL`</p>",
  "messages": [
    {
      "id": 1260489,
      "postDate": "2021-04-02T06:55:41.333Z",
      "content": "<p>Hello everyone !<br>\nI am trying to train my model on the TPU using torch_xla, but it shows \"out of memory\" error. I even tried with batch size = 1, but did not get through. Please help.</p>\n<p>The error message looks like the following :</p>\n<p>`ProcessExitedException                    Traceback (most recent call last)<br>\n in <br>\n      1 if <strong>name</strong> == '<strong>main</strong>':<br>\n      2     flags = {}<br>\n----&gt; 3     xmp.spawn(_mp_fn, args = (flags,), nprocs=8, start_method='fork')</p>\n<p>/opt/conda/lib/python3.7/site-packages/torch_xla/distributed/xla_multiprocessing.py in spawn(fn, args, nprocs, join, daemon, start_method)<br>\n    392         join=join,<br>\n    393         daemon=daemon,<br>\n--&gt; 394         start_method=start_method)<br>\n    395 <br>\n    396 </p>\n<p>/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py in start_processes(fn, args, nprocs, join, daemon, start_method)<br>\n    186 <br>\n    187     # Loop on join until it returns True or raises an exception.<br>\n--&gt; 188     while not context.join():<br>\n    189         pass<br>\n    190 </p>\n<p>/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py in join(self, timeout)<br>\n    134                     error_pid=failed_process.pid,<br>\n    135                     exit_code=exitcode,<br>\n--&gt; 136                     signal_name=name<br>\n    137                 )<br>\n    138             else:</p>\n<p>ProcessExitedException: process 4 terminated with signal SIGKILL`</p>",
      "rawMarkdown": "Hello everyone !\nI am trying to train my model on the TPU using torch_xla, but it shows \"out of memory\" error. I even tried with batch size = 1, but did not get through. Please help.\n\nThe error message looks like the following :\n\n\n\n\n`ProcessExitedException                    Traceback (most recent call last)\n<ipython-input-35-f41d7fbf1c48> in <module>\n      1 if __name__ == '__main__':\n      2     flags = {}\n----> 3     xmp.spawn(_mp_fn, args = (flags,), nprocs=8, start_method='fork')\n\n/opt/conda/lib/python3.7/site-packages/torch_xla/distributed/xla_multiprocessing.py in spawn(fn, args, nprocs, join, daemon, start_method)\n    392         join=join,\n    393         daemon=daemon,\n--> 394         start_method=start_method)\n    395 \n    396 \n\n/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py in start_processes(fn, args, nprocs, join, daemon, start_method)\n    186 \n    187     # Loop on join until it returns True or raises an exception.\n--> 188     while not context.join():\n    189         pass\n    190 \n\n/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py in join(self, timeout)\n    134                     error_pid=failed_process.pid,\n    135                     exit_code=exitcode,\n--> 136                     signal_name=name\n    137                 )\n    138             else:\n\nProcessExitedException: process 4 terminated with signal SIGKILL`",
      "votes": 3
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1260489": "Hello everyone !\nI am trying to train my model on the TPU using torch_xla, but it shows \"out of memory\" error. I even tried with batch size = 1, but did not get through. Please help.\n\nThe error message looks like the following :\n\n\n\n\n`ProcessExitedException                    Traceback (most recent call last)\n<ipython-input-35-f41d7fbf1c48> in <module>\n      1 if __name__ == '__main__':\n      2     flags = {}\n----> 3     xmp.spawn(_mp_fn, args = (flags,), nprocs=8, start_method='fork')\n\n/opt/conda/lib/python3.7/site-packages/torch_xla/distributed/xla_multiprocessing.py in spawn(fn, args, nprocs, join, daemon, start_method)\n    392         join=join,\n    393         daemon=daemon,\n--> 394         start_method=start_method)\n    395 \n    396 \n\n/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py in start_processes(fn, args, nprocs, join, daemon, start_method)\n    186 \n    187     # Loop on join until it returns True or raises an exception.\n--> 188     while not context.join():\n    189         pass\n    190 \n\n/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py in join(self, timeout)\n    134                     error_pid=failed_process.pid,\n    135                     exit_code=exitcode,\n--> 136                     signal_name=name\n    137                 )\n    138             else:\n\nProcessExitedException: process 4 terminated with signal SIGKILL`"
  }
}