{
  "id": 418651,
  "title": "NotFoundError: TPUs not found in the cluster.",
  "url": "/competitions/tpu-getting-started/discussion/418651",
  "author_name": "kalakagat",
  "post_date": "2023-06-21T21:14:54.053000",
  "votes": 2,
  "comment_count": 1,
  "views": null,
  "content": "<p><strong>It's strange but sometimes I can't launch TPU. What is the problem?</strong></p>\n<pre><code>:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver() \n    (, tpu.master())\n ValueError:\n    tpu = \n\n tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\n:\n    strategy = tf.distribute.get_strategy() \n\n(, strategy.num_replicas_in_sync)\n</code></pre>\n<p>Running on TPU  <br>\nINFO:tensorflow:Deallocate tpu buffers before initializing tpu system.<br>\nINFO:tensorflow:Initializing the TPU system: local</p>\n<p>InvalidArgumentError                      Traceback (most recent call last)<br>\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/tpu/tpu_strategy_util.py:126, in initialize_tpu_system(cluster_resolver)<br>\n    125 try:<br>\n--&gt; 126   with ops.device(tpu._tpu_system_device_name(job)):  # pylint: disable=protected-access<br>\n    127     output = _tpu_init_fn()</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:5526, in device(device_name_or_function)<br>\n   5523     raise RuntimeError(<br>\n   5524         \"tf.device does not support functions when eager execution \"<br>\n   5525         \"is enabled.\")<br>\n-&gt; 5526   return context.device(device_name_or_function)<br>\n   5527 elif executing_eagerly_outside_functions():</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:2348, in device(name)<br>\n   2329 \"\"\"Context-manager to force placement of operations and Tensors on a device.<br>\n   2330 <br>\n   2331 Example:<br>\n   (…)<br>\n   2346   Context manager for setting the device.<br>\n   2347 \"\"\"<br>\n-&gt; 2348 ensure_initialized()<br>\n   2349 return context().device(name)</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:2143, in ensure_initialized()<br>\n   2142 \"\"\"Initialize the context.\"\"\"<br>\n-&gt; 2143 context().ensure_initialized()</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:583, in Context.ensure_initialized(self)<br>\n    581   pywrap_tfe.TFE_ContextOptionsSetJitCompileRewrite(<br>\n    582       opts, self._jit_compile_rewrite)<br>\n--&gt; 583   context_handle = pywrap_tfe.TFE_NewContext(opts)<br>\n    584 finally:</p>\n<p>InvalidArgumentError: open(/dev/accel0): Invalid argument: Invalid argument; Couldn't open device: /dev/accel0; Unable to create Node RegisterInterface for node 0, config: device_path:      \"/dev/accel0\" mode: KERNEL debug_data_directory: \"\" dump_anomalies_only: true crash_in_debug_dump: false allow_core_dump: true; could not create driver instance</p>\n<p>During handling of the above exception, another exception occurred:</p>\n<p>NotFoundError                             Traceback (most recent call last)<br>\nCell In[4], line 10<br>\n      8 if tpu:<br>\n      9     tf.config.experimental_connect_to_cluster(tpu)<br>\n---&gt; 10     tf.tpu.experimental.initialize_tpu_system(tpu)<br>\n     11     strategy = tf.distribute.experimental.TPUStrategy(tpu)<br>\n     12 else:</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/tpu/tpu_strategy_util.py:130, in initialize_tpu_system(cluster_resolver)<br>\n    128   context.async_wait()<br>\n    129 except errors.InvalidArgumentError as e:<br>\n--&gt; 130   raise errors.NotFoundError(<br>\n    131       None, None,<br>\n    132       \"TPUs not found in the cluster. Failed in initialization: \"<br>\n    133       + str(e))<br>\n    134 finally:<br>\n    135   if run_eagerly is not None:</p>\n<p>NotFoundError: TPUs not found in the cluster. Failed in initialization: open(/dev/accel0): Invalid argument: Invalid argument; Couldn't open device: /dev/accel0; Unable to create Node RegisterInterface for node 0, config: device_path:      \"/dev/accel0\" mode: KERNEL debug_data_directory: \"\" dump_anomalies_only: true crash_in_debug_dump: false allow_core_dump: true; could not create driver instance</p>",
  "messages": [
    {
      "id": 2312378,
      "postDate": "2023-06-21T21:14:54.053Z",
      "content": "<p><strong>It's strange but sometimes I can't launch TPU. What is the problem?</strong></p>\n<pre><code>:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver() \n    (, tpu.master())\n ValueError:\n    tpu = \n\n tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\n:\n    strategy = tf.distribute.get_strategy() \n\n(, strategy.num_replicas_in_sync)\n</code></pre>\n<p>Running on TPU  <br>\nINFO:tensorflow:Deallocate tpu buffers before initializing tpu system.<br>\nINFO:tensorflow:Initializing the TPU system: local</p>\n<p>InvalidArgumentError                      Traceback (most recent call last)<br>\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/tpu/tpu_strategy_util.py:126, in initialize_tpu_system(cluster_resolver)<br>\n    125 try:<br>\n--&gt; 126   with ops.device(tpu._tpu_system_device_name(job)):  # pylint: disable=protected-access<br>\n    127     output = _tpu_init_fn()</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:5526, in device(device_name_or_function)<br>\n   5523     raise RuntimeError(<br>\n   5524         \"tf.device does not support functions when eager execution \"<br>\n   5525         \"is enabled.\")<br>\n-&gt; 5526   return context.device(device_name_or_function)<br>\n   5527 elif executing_eagerly_outside_functions():</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:2348, in device(name)<br>\n   2329 \"\"\"Context-manager to force placement of operations and Tensors on a device.<br>\n   2330 <br>\n   2331 Example:<br>\n   (…)<br>\n   2346   Context manager for setting the device.<br>\n   2347 \"\"\"<br>\n-&gt; 2348 ensure_initialized()<br>\n   2349 return context().device(name)</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:2143, in ensure_initialized()<br>\n   2142 \"\"\"Initialize the context.\"\"\"<br>\n-&gt; 2143 context().ensure_initialized()</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:583, in Context.ensure_initialized(self)<br>\n    581   pywrap_tfe.TFE_ContextOptionsSetJitCompileRewrite(<br>\n    582       opts, self._jit_compile_rewrite)<br>\n--&gt; 583   context_handle = pywrap_tfe.TFE_NewContext(opts)<br>\n    584 finally:</p>\n<p>InvalidArgumentError: open(/dev/accel0): Invalid argument: Invalid argument; Couldn't open device: /dev/accel0; Unable to create Node RegisterInterface for node 0, config: device_path:      \"/dev/accel0\" mode: KERNEL debug_data_directory: \"\" dump_anomalies_only: true crash_in_debug_dump: false allow_core_dump: true; could not create driver instance</p>\n<p>During handling of the above exception, another exception occurred:</p>\n<p>NotFoundError                             Traceback (most recent call last)<br>\nCell In[4], line 10<br>\n      8 if tpu:<br>\n      9     tf.config.experimental_connect_to_cluster(tpu)<br>\n---&gt; 10     tf.tpu.experimental.initialize_tpu_system(tpu)<br>\n     11     strategy = tf.distribute.experimental.TPUStrategy(tpu)<br>\n     12 else:</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/tpu/tpu_strategy_util.py:130, in initialize_tpu_system(cluster_resolver)<br>\n    128   context.async_wait()<br>\n    129 except errors.InvalidArgumentError as e:<br>\n--&gt; 130   raise errors.NotFoundError(<br>\n    131       None, None,<br>\n    132       \"TPUs not found in the cluster. Failed in initialization: \"<br>\n    133       + str(e))<br>\n    134 finally:<br>\n    135   if run_eagerly is not None:</p>\n<p>NotFoundError: TPUs not found in the cluster. Failed in initialization: open(/dev/accel0): Invalid argument: Invalid argument; Couldn't open device: /dev/accel0; Unable to create Node RegisterInterface for node 0, config: device_path:      \"/dev/accel0\" mode: KERNEL debug_data_directory: \"\" dump_anomalies_only: true crash_in_debug_dump: false allow_core_dump: true; could not create driver instance</p>",
      "rawMarkdown": "**It's strange but sometimes I can't launch TPU. What is the problem?**\n\n\n```python\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver() \n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() \n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)\n```\n\nRunning on TPU  \nINFO:tensorflow:Deallocate tpu buffers before initializing tpu system.\nINFO:tensorflow:Initializing the TPU system: local\n\nInvalidArgumentError                      Traceback (most recent call last)\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/tpu/tpu_strategy_util.py:126, in initialize_tpu_system(cluster_resolver)\n    125 try:\n--> 126   with ops.device(tpu._tpu_system_device_name(job)):  # pylint: disable=protected-access\n    127     output = _tpu_init_fn()\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:5526, in device(device_name_or_function)\n   5523     raise RuntimeError(\n   5524         \"tf.device does not support functions when eager execution \"\n   5525         \"is enabled.\")\n-> 5526   return context.device(device_name_or_function)\n   5527 elif executing_eagerly_outside_functions():\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:2348, in device(name)\n   2329 \"\"\"Context-manager to force placement of operations and Tensors on a device.\n   2330 \n   2331 Example:\n   (...)\n   2346   Context manager for setting the device.\n   2347 \"\"\"\n-> 2348 ensure_initialized()\n   2349 return context().device(name)\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:2143, in ensure_initialized()\n   2142 \"\"\"Initialize the context.\"\"\"\n-> 2143 context().ensure_initialized()\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:583, in Context.ensure_initialized(self)\n    581   pywrap_tfe.TFE_ContextOptionsSetJitCompileRewrite(\n    582       opts, self._jit_compile_rewrite)\n--> 583   context_handle = pywrap_tfe.TFE_NewContext(opts)\n    584 finally:\n\nInvalidArgumentError: open(/dev/accel0): Invalid argument: Invalid argument; Couldn't open device: /dev/accel0; Unable to create Node RegisterInterface for node 0, config: device_path: \t \"/dev/accel0\" mode: KERNEL debug_data_directory: \"\" dump_anomalies_only: true crash_in_debug_dump: false allow_core_dump: true; could not create driver instance\n\nDuring handling of the above exception, another exception occurred:\n\nNotFoundError                             Traceback (most recent call last)\nCell In[4], line 10\n      8 if tpu:\n      9     tf.config.experimental_connect_to_cluster(tpu)\n---> 10     tf.tpu.experimental.initialize_tpu_system(tpu)\n     11     strategy = tf.distribute.experimental.TPUStrategy(tpu)\n     12 else:\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/tpu/tpu_strategy_util.py:130, in initialize_tpu_system(cluster_resolver)\n    128   context.async_wait()\n    129 except errors.InvalidArgumentError as e:\n--> 130   raise errors.NotFoundError(\n    131       None, None,\n    132       \"TPUs not found in the cluster. Failed in initialization: \"\n    133       + str(e))\n    134 finally:\n    135   if run_eagerly is not None:\n\nNotFoundError: TPUs not found in the cluster. Failed in initialization: open(/dev/accel0): Invalid argument: Invalid argument; Couldn't open device: /dev/accel0; Unable to create Node RegisterInterface for node 0, config: device_path: \t \"/dev/accel0\" mode: KERNEL debug_data_directory: \"\" dump_anomalies_only: true crash_in_debug_dump: false allow_core_dump: true; could not create driver instance",
      "votes": 2
    },
    {
      "id": 3068897,
      "postDate": "2024-12-10T19:29:09.313Z",
      "content": "<p>Hello. I have the same problem. Have you solved this issue?</p>",
      "rawMarkdown": "Hello. I have the same problem. Have you solved this issue?\n"
    }
  ],
  "comments": [
    {
      "id": 3068897,
      "author_name": "Masha Kholodenko",
      "author_url": "",
      "post_date": "2024-12-10T19:29:09.313000",
      "content": "<p>Hello. I have the same problem. Have you solved this issue?</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2312378": "**It's strange but sometimes I can't launch TPU. What is the problem?**\n\n\n```python\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver() \n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy() \n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)\n```\n\nRunning on TPU  \nINFO:tensorflow:Deallocate tpu buffers before initializing tpu system.\nINFO:tensorflow:Initializing the TPU system: local\n\nInvalidArgumentError                      Traceback (most recent call last)\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/tpu/tpu_strategy_util.py:126, in initialize_tpu_system(cluster_resolver)\n    125 try:\n--> 126   with ops.device(tpu._tpu_system_device_name(job)):  # pylint: disable=protected-access\n    127     output = _tpu_init_fn()\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:5526, in device(device_name_or_function)\n   5523     raise RuntimeError(\n   5524         \"tf.device does not support functions when eager execution \"\n   5525         \"is enabled.\")\n-> 5526   return context.device(device_name_or_function)\n   5527 elif executing_eagerly_outside_functions():\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:2348, in device(name)\n   2329 \"\"\"Context-manager to force placement of operations and Tensors on a device.\n   2330 \n   2331 Example:\n   (...)\n   2346   Context manager for setting the device.\n   2347 \"\"\"\n-> 2348 ensure_initialized()\n   2349 return context().device(name)\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:2143, in ensure_initialized()\n   2142 \"\"\"Initialize the context.\"\"\"\n-> 2143 context().ensure_initialized()\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:583, in Context.ensure_initialized(self)\n    581   pywrap_tfe.TFE_ContextOptionsSetJitCompileRewrite(\n    582       opts, self._jit_compile_rewrite)\n--> 583   context_handle = pywrap_tfe.TFE_NewContext(opts)\n    584 finally:\n\nInvalidArgumentError: open(/dev/accel0): Invalid argument: Invalid argument; Couldn't open device: /dev/accel0; Unable to create Node RegisterInterface for node 0, config: device_path: \t \"/dev/accel0\" mode: KERNEL debug_data_directory: \"\" dump_anomalies_only: true crash_in_debug_dump: false allow_core_dump: true; could not create driver instance\n\nDuring handling of the above exception, another exception occurred:\n\nNotFoundError                             Traceback (most recent call last)\nCell In[4], line 10\n      8 if tpu:\n      9     tf.config.experimental_connect_to_cluster(tpu)\n---> 10     tf.tpu.experimental.initialize_tpu_system(tpu)\n     11     strategy = tf.distribute.experimental.TPUStrategy(tpu)\n     12 else:\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/tpu/tpu_strategy_util.py:130, in initialize_tpu_system(cluster_resolver)\n    128   context.async_wait()\n    129 except errors.InvalidArgumentError as e:\n--> 130   raise errors.NotFoundError(\n    131       None, None,\n    132       \"TPUs not found in the cluster. Failed in initialization: \"\n    133       + str(e))\n    134 finally:\n    135   if run_eagerly is not None:\n\nNotFoundError: TPUs not found in the cluster. Failed in initialization: open(/dev/accel0): Invalid argument: Invalid argument; Couldn't open device: /dev/accel0; Unable to create Node RegisterInterface for node 0, config: device_path: \t \"/dev/accel0\" mode: KERNEL debug_data_directory: \"\" dump_anomalies_only: true crash_in_debug_dump: false allow_core_dump: true; could not create driver instance",
    "3068897": "Hello. I have the same problem. Have you solved this issue?\n"
  }
}