{
  "id": 163884,
  "title": "Using TPU's on Pytorch",
  "url": "/competitions/siim-isic-melanoma-classification/discussion/163884",
  "author_name": "",
  "post_date": "2020-07-03T19:41:25.001720200Z",
  "votes": 3,
  "comment_count": 1,
  "views": 0,
  "content": "<p>I've been trying to use the full capabilities that TPU's offer on Pytorch. However, I'm facing certain issues.</p>\n\n<p>```\n!curl <a href=\"https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py\">https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py</a> -o pytorch-xla-env-setup.py\n!python pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev\nimport torch_xla.core.xla_model as xm\nimport torch_xla.distributed.xla_multiprocessing as xmp</p>\n\n<p><code>\n</code>\ndef _mp_fn(rank, flags):\n    print(f\"TRAINING_FOLD_{i+1}.{rank+1}\")\n    xm.rendezvous('init')\n```</p>\n\n<p>```\nfor i in range(8):\n    print(f\"TRAINING_FOLD_{i+1}\")\n    xmp.spawn(_mp_fn, args=({},), nprocs=8, start_method='fork')\n    print(\"TRAINING_DONE\")</p>\n\n<p>```\nThe output that I get for the last cell is:</p>\n\n<p><code>\nTRAINING_FOLD_1\nTRAINING_FOLD_1.1\nTRAINING_FOLD_1.4\nTRAINING_FOLD_1.6\nTRAINING_FOLD_1.8\nTRAINING_FOLD_1.7\nTRAINING_FOLD_1.3\nTRAINING_FOLD_1.5\nTRAINING_FOLD_1.2\n</code></p>\n\n<p>The expected output is for the other seven training folds also to execute, but I get the following error:</p>\n\n<p><code>\nException in device=TPU:6: tensorflow/compiler/xla/xla_client/mesh_service.cc:299 : Failed to meet rendezvous 'init': Socket closed (14)\nTraceback (most recent call last):\n  File \"/opt/conda/lib/python3.7/site-packages/torch_xla/distributed/xla_multiprocessing.py\", line 235, in _mp_start_fn\n    _start_fn(index, pf_cfg, fn, args)\n  File \"/opt/conda/lib/python3.7/site-packages/torch_xla/distributed/xla_multiprocessing.py\", line 229, in _start_fn\n    fn(gindex, *args)\n  File \"&lt;ipython-input-4-8d7dca5e3f79&gt;\", line 3, in _mp_fn\n    xm.rendezvous('init')\n  File \"/opt/conda/lib/python3.7/site-packages/torch_xla/core/xla_model.py\", line 692, in rendezvous\n    return torch_xla._XLAC._xla_rendezvous(get_ordinal(), tag, payload, replicas)\nRuntimeError: tensorflow/compiler/xla/xla_client/mesh_service.cc:299 : Failed to meet rendezvous 'init': Socket closed (14)\n</code></p>\n\n<p>If I change <code>nprocs</code> to 1, then the code runs smoothly, however I'm not utilising the full capability of the GPU's.</p>\n\n<p>Any ideas?</p>",
  "messages": [
    {
      "id": "914362",
      "postDate": "07/03/2020 19:41:25",
      "content": "<p>I've been trying to use the full capabilities that TPU's offer on Pytorch. However, I'm facing certain issues.</p>\n\n<p>```\n!curl <a href=\"https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py\">https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py</a> -o pytorch-xla-env-setup.py\n!python pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev\nimport torch_xla.core.xla_model as xm\nimport torch_xla.distributed.xla_multiprocessing as xmp</p>\n\n<p><code>\n</code>\ndef _mp_fn(rank, flags):\n    print(f\"TRAINING_FOLD_{i+1}.{rank+1}\")\n    xm.rendezvous('init')\n```</p>\n\n<p>```\nfor i in range(8):\n    print(f\"TRAINING_FOLD_{i+1}\")\n    xmp.spawn(_mp_fn, args=({},), nprocs=8, start_method='fork')\n    print(\"TRAINING_DONE\")</p>\n\n<p>```\nThe output that I get for the last cell is:</p>\n\n<p><code>\nTRAINING_FOLD_1\nTRAINING_FOLD_1.1\nTRAINING_FOLD_1.4\nTRAINING_FOLD_1.6\nTRAINING_FOLD_1.8\nTRAINING_FOLD_1.7\nTRAINING_FOLD_1.3\nTRAINING_FOLD_1.5\nTRAINING_FOLD_1.2\n</code></p>\n\n<p>The expected output is for the other seven training folds also to execute, but I get the following error:</p>\n\n<p><code>\nException in device=TPU:6: tensorflow/compiler/xla/xla_client/mesh_service.cc:299 : Failed to meet rendezvous 'init': Socket closed (14)\nTraceback (most recent call last):\n  File \"/opt/conda/lib/python3.7/site-packages/torch_xla/distributed/xla_multiprocessing.py\", line 235, in _mp_start_fn\n    _start_fn(index, pf_cfg, fn, args)\n  File \"/opt/conda/lib/python3.7/site-packages/torch_xla/distributed/xla_multiprocessing.py\", line 229, in _start_fn\n    fn(gindex, *args)\n  File \"&lt;ipython-input-4-8d7dca5e3f79&gt;\", line 3, in _mp_fn\n    xm.rendezvous('init')\n  File \"/opt/conda/lib/python3.7/site-packages/torch_xla/core/xla_model.py\", line 692, in rendezvous\n    return torch_xla._XLAC._xla_rendezvous(get_ordinal(), tag, payload, replicas)\nRuntimeError: tensorflow/compiler/xla/xla_client/mesh_service.cc:299 : Failed to meet rendezvous 'init': Socket closed (14)\n</code></p>\n\n<p>If I change <code>nprocs</code> to 1, then the code runs smoothly, however I'm not utilising the full capability of the GPU's.</p>\n\n<p>Any ideas?</p>",
      "rawMarkdown": "I've been trying to use the full capabilities that TPU's offer on Pytorch. However, I'm facing certain issues.\n\n```\n!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py\n!python pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev\nimport torch_xla.core.xla_model as xm\nimport torch_xla.distributed.xla_multiprocessing as xmp\n\n```\n```\ndef _mp_fn(rank, flags):\n    print(f\"TRAINING_FOLD_{i+1}.{rank+1}\")\n    xm.rendezvous('init')\n```\n\n\n```\nfor i in range(8):\n    print(f\"TRAINING_FOLD_{i+1}\")\n    xmp.spawn(_mp_fn, args=({},), nprocs=8, start_method='fork')\n    print(\"TRAINING_DONE\")\n\n```\nThe output that I get for the last cell is:\n\n```\nTRAINING_FOLD_1\nTRAINING_FOLD_1.1\nTRAINING_FOLD_1.4\nTRAINING_FOLD_1.6\nTRAINING_FOLD_1.8\nTRAINING_FOLD_1.7\nTRAINING_FOLD_1.3\nTRAINING_FOLD_1.5\nTRAINING_FOLD_1.2\n```\n\nThe expected output is for the other seven training folds also to execute, but I get the following error:\n\n```\nException in device=TPU:6: tensorflow/compiler/xla/xla_client/mesh_service.cc:299 : Failed to meet rendezvous 'init': Socket closed (14)\nTraceback (most recent call last):\n  File \"/opt/conda/lib/python3.7/site-packages/torch_xla/distributed/xla_multiprocessing.py\", line 235, in _mp_start_fn\n    _start_fn(index, pf_cfg, fn, args)\n  File \"/opt/conda/lib/python3.7/site-packages/torch_xla/distributed/xla_multiprocessing.py\", line 229, in _start_fn\n    fn(gindex, *args)\n  File \"",
      "votes": null
    },
    {
      "id": "1030343",
      "postDate": "09/28/2020 15:27:25",
      "content": "<p>Hi, did you find a solution for this problem? I have the same issue, since 3 days.</p>",
      "rawMarkdown": "Hi, did you find a solution for this problem? I have the same issue, since 3 days.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1030343,
      "author_name": "rimzakhama",
      "author_url": "",
      "post_date": "09/28/2020 15:27:25",
      "content": "<p>Hi, did you find a solution for this problem? I have the same issue, since 3 days.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "914362": "I've been trying to use the full capabilities that TPU's offer on Pytorch. However, I'm facing certain issues.\n\n```\n!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py\n!python pytorch-xla-env-setup.py --version nightly --apt-packages libomp5 libopenblas-dev\nimport torch_xla.core.xla_model as xm\nimport torch_xla.distributed.xla_multiprocessing as xmp\n\n```\n```\ndef _mp_fn(rank, flags):\n    print(f\"TRAINING_FOLD_{i+1}.{rank+1}\")\n    xm.rendezvous('init')\n```\n\n\n```\nfor i in range(8):\n    print(f\"TRAINING_FOLD_{i+1}\")\n    xmp.spawn(_mp_fn, args=({},), nprocs=8, start_method='fork')\n    print(\"TRAINING_DONE\")\n\n```\nThe output that I get for the last cell is:\n\n```\nTRAINING_FOLD_1\nTRAINING_FOLD_1.1\nTRAINING_FOLD_1.4\nTRAINING_FOLD_1.6\nTRAINING_FOLD_1.8\nTRAINING_FOLD_1.7\nTRAINING_FOLD_1.3\nTRAINING_FOLD_1.5\nTRAINING_FOLD_1.2\n```\n\nThe expected output is for the other seven training folds also to execute, but I get the following error:\n\n```\nException in device=TPU:6: tensorflow/compiler/xla/xla_client/mesh_service.cc:299 : Failed to meet rendezvous 'init': Socket closed (14)\nTraceback (most recent call last):\n  File \"/opt/conda/lib/python3.7/site-packages/torch_xla/distributed/xla_multiprocessing.py\", line 235, in _mp_start_fn\n    _start_fn(index, pf_cfg, fn, args)\n  File \"/opt/conda/lib/python3.7/site-packages/torch_xla/distributed/xla_multiprocessing.py\", line 229, in _start_fn\n    fn(gindex, *args)\n  File \"",
    "1030343": "Hi, did you find a solution for this problem? I have the same issue, since 3 days."
  },
  "source": "meta"
}