{
  "id": 407819,
  "title": "Couldn't open device {TPU  Error} ",
  "url": "/competitions/birdclef-2023/discussion/407819",
  "author_name": "Balaji Selvaraj",
  "post_date": "2023-05-08T11:29:27.491000",
  "votes": 1,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Hi kagglers,</p>\n<p>I am using the public notebook with TPU</p>\n<pre><code>\nstrategy, CFG.device, tpu = get_device()\nCFG.replicas = strategy.num_replicas_in_sync\n</code></pre>\n<p>I am getting following error when I try to access the TPU's today.<br>\nThe same code didn't create any errors previously. <br>\nAny solution ?</p>\n<h2><strong>Error</strong></h2>\n<p>InvalidArgumentError                      Traceback (most recent call last)<br>\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/tpu/tpu_strategy_util.py:126, in initialize_tpu_system(cluster_resolver)<br>\n    125 try:<br>\n--&gt; 126   with ops.device(tpu._tpu_system_device_name(job)):  # pylint: disable=protected-access<br>\n    127     output = _tpu_init_fn()</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:5526, in device(device_name_or_function)<br>\n   5523     raise RuntimeError(<br>\n   5524         \"tf.device does not support functions when eager execution \"<br>\n   5525         \"is enabled.\")<br>\n-&gt; 5526   return context.device(device_name_or_function)<br>\n   5527 elif executing_eagerly_outside_functions():</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:2348, in device(name)<br>\n   2329 \"\"\"Context-manager to force placement of operations and Tensors on a device.<br>\n   2330 <br>\n   2331 Example:<br>\n   (…)<br>\n   2346   Context manager for setting the device.<br>\n   2347 \"\"\"<br>\n-&gt; 2348 ensure_initialized()<br>\n   2349 return context().device(name)</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:2143, in ensure_initialized()<br>\n   2142 \"\"\"Initialize the context.\"\"\"<br>\n-&gt; 2143 context().ensure_initialized()</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:583, in Context.ensure_initialized(self)<br>\n    581   pywrap_tfe.TFE_ContextOptionsSetJitCompileRewrite(<br>\n    582       opts, self._jit_compile_rewrite)<br>\n--&gt; 583   context_handle = pywrap_tfe.TFE_NewContext(opts)<br>\n    584 finally:</p>\n<p>InvalidArgumentError: open(/dev/accel0): Invalid argument: Invalid argument; Couldn't open device: /dev/accel0; Unable to create Node RegisterInterface for node 0, config: device_path:      \"/dev/accel0\" mode: KERNEL debug_data_directory: \"\" dump_anomalies_only: true crash_in_debug_dump: false allow_core_dump: true; could not create driver instance</p>\n<p>During handling of the above exception, another exception occurred:</p>\n<p>NotFoundError                             Traceback (most recent call last)<br>\nCell In[7], line 7, in get_device()<br>\n      5 try:<br>\n      6     # Connect to TPU<br>\n----&gt; 7     tpu = tf.distribute.cluster_resolver.TPUClusterResolver.connect(tpu=tpu) <br>\n      8     # Set TPU strategy</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/distribute/cluster_resolver/tpu/tpu_cluster_resolver.py:110, in TPUClusterResolver.connect(tpu, zone, project)<br>\n    109 from tensorflow.python.tpu import tpu_strategy_util  # pylint: disable=g-import-not-at-top<br>\n--&gt; 110 tpu_strategy_util.initialize_tpu_system(resolver)<br>\n    111 return resolver</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/tpu/tpu_strategy_util.py:130, in initialize_tpu_system(cluster_resolver)<br>\n    129 except errors.InvalidArgumentError as e:<br>\n--&gt; 130   raise errors.NotFoundError(<br>\n    131       None, None,<br>\n    132       \"TPUs not found in the cluster. Failed in initialization: \"<br>\n    133       + str(e))<br>\n    134 finally:</p>\n<p>NotFoundError: TPUs not found in the cluster. Failed in initialization: open(/dev/accel0): Invalid argument: Invalid argument; Couldn't open device: /dev/accel0; Unable to create Node RegisterInterface for node 0, config: device_path:      \"/dev/accel0\" mode: KERNEL debug_data_directory: \"\" dump_anomalies_only: true crash_in_debug_dump: false allow_core_dump: true; could not create driver instance</p>\n<p>During handling of the above exception, another exception occurred:</p>\n<p>InvalidArgumentError                      Traceback (most recent call last)<br>\nCell In[8], line 2<br>\n      1 # Initialize GPU/TPU/TPU-VM<br>\n----&gt; 2 strategy, CFG.device, tpu = get_device()<br>\n      3 CFG.replicas = strategy.num_replicas_in_sync</p>\n<p>Cell In[7], line 15, in get_device()<br>\n     12     device=CFG.device<br>\n     13 except:<br>\n     14     # If TPU is not available, detect GPUs<br>\n---&gt; 15     gpus = tf.config.list_logical_devices('GPU')<br>\n     16     ngpu = len(gpus)<br>\n     17      # Check number of GPUs</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/framework/config.py:478, in list_logical_devices(device_type)<br>\n    445 <a href=\"https://www.kaggle.com/tf\" target=\"_blank\">@tf</a>_export('config.list_logical_devices',<br>\n    446            'config.experimental.list_logical_devices')<br>\n    447 <a href=\"https://www.kaggle.com/deprecation.deprecated\" target=\"_blank\">@deprecation.deprecated</a>_endpoints('config.experimental.list_logical_devices')<br>\n    448 def list_logical_devices(device_type=None):<br>\n    449   \"\"\"Return a list of logical devices created by runtime.<br>\n    450 <br>\n    451   Logical devices may correspond to physical devices or remote devices in the<br>\n   (…)<br>\n    476     List of initialized <code>LogicalDevice</code>s<br>\n    477   \"\"\"<br>\n--&gt; 478   return context.context().list_logical_devices(device_type=device_type)</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:1582, in Context.list_logical_devices(self, device_type)<br>\n   1580 def list_logical_devices(self, device_type=None):<br>\n   1581   \"\"\"Return logical devices.\"\"\"<br>\n-&gt; 1582   self.ensure_initialized()<br>\n   1583   if device_type is None:<br>\n   1584     return list(self._logical_devices)</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:583, in Context.ensure_initialized(self)<br>\n    580   pywrap_tfe.TFE_ContextOptionsSetRunEagerOpAsFunction(opts, True)<br>\n    581   pywrap_tfe.TFE_ContextOptionsSetJitCompileRewrite(<br>\n    582       opts, self._jit_compile_rewrite)<br>\n--&gt; 583   context_handle = pywrap_tfe.TFE_NewContext(opts)<br>\n    584 finally:<br>\n    585   pywrap_tfe.TFE_DeleteContextOptions(opts)</p>\n<p>InvalidArgumentError: open(/dev/accel0): Invalid argument: Invalid argument; Couldn't open device: /dev/accel0; Unable to create Node RegisterInterface for node 0, config: device_path:      \"/dev/accel0\" mode: KERNEL debug_data_directory: \"\" dump_anomalies_only: true crash_in_debug_dump: false allow_core_dump: true; could not create driver instanc</p>",
  "messages": [
    {
      "id": 2250149,
      "postDate": "2023-05-08T11:29:27.493Z",
      "content": "<p>Hi kagglers,</p>\n<p>I am using the public notebook with TPU</p>\n<pre><code>\nstrategy, CFG.device, tpu = get_device()\nCFG.replicas = strategy.num_replicas_in_sync\n</code></pre>\n<p>I am getting following error when I try to access the TPU's today.<br>\nThe same code didn't create any errors previously. <br>\nAny solution ?</p>\n<h2><strong>Error</strong></h2>\n<p>InvalidArgumentError                      Traceback (most recent call last)<br>\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/tpu/tpu_strategy_util.py:126, in initialize_tpu_system(cluster_resolver)<br>\n    125 try:<br>\n--&gt; 126   with ops.device(tpu._tpu_system_device_name(job)):  # pylint: disable=protected-access<br>\n    127     output = _tpu_init_fn()</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:5526, in device(device_name_or_function)<br>\n   5523     raise RuntimeError(<br>\n   5524         \"tf.device does not support functions when eager execution \"<br>\n   5525         \"is enabled.\")<br>\n-&gt; 5526   return context.device(device_name_or_function)<br>\n   5527 elif executing_eagerly_outside_functions():</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:2348, in device(name)<br>\n   2329 \"\"\"Context-manager to force placement of operations and Tensors on a device.<br>\n   2330 <br>\n   2331 Example:<br>\n   (…)<br>\n   2346   Context manager for setting the device.<br>\n   2347 \"\"\"<br>\n-&gt; 2348 ensure_initialized()<br>\n   2349 return context().device(name)</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:2143, in ensure_initialized()<br>\n   2142 \"\"\"Initialize the context.\"\"\"<br>\n-&gt; 2143 context().ensure_initialized()</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:583, in Context.ensure_initialized(self)<br>\n    581   pywrap_tfe.TFE_ContextOptionsSetJitCompileRewrite(<br>\n    582       opts, self._jit_compile_rewrite)<br>\n--&gt; 583   context_handle = pywrap_tfe.TFE_NewContext(opts)<br>\n    584 finally:</p>\n<p>InvalidArgumentError: open(/dev/accel0): Invalid argument: Invalid argument; Couldn't open device: /dev/accel0; Unable to create Node RegisterInterface for node 0, config: device_path:      \"/dev/accel0\" mode: KERNEL debug_data_directory: \"\" dump_anomalies_only: true crash_in_debug_dump: false allow_core_dump: true; could not create driver instance</p>\n<p>During handling of the above exception, another exception occurred:</p>\n<p>NotFoundError                             Traceback (most recent call last)<br>\nCell In[7], line 7, in get_device()<br>\n      5 try:<br>\n      6     # Connect to TPU<br>\n----&gt; 7     tpu = tf.distribute.cluster_resolver.TPUClusterResolver.connect(tpu=tpu) <br>\n      8     # Set TPU strategy</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/distribute/cluster_resolver/tpu/tpu_cluster_resolver.py:110, in TPUClusterResolver.connect(tpu, zone, project)<br>\n    109 from tensorflow.python.tpu import tpu_strategy_util  # pylint: disable=g-import-not-at-top<br>\n--&gt; 110 tpu_strategy_util.initialize_tpu_system(resolver)<br>\n    111 return resolver</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/tpu/tpu_strategy_util.py:130, in initialize_tpu_system(cluster_resolver)<br>\n    129 except errors.InvalidArgumentError as e:<br>\n--&gt; 130   raise errors.NotFoundError(<br>\n    131       None, None,<br>\n    132       \"TPUs not found in the cluster. Failed in initialization: \"<br>\n    133       + str(e))<br>\n    134 finally:</p>\n<p>NotFoundError: TPUs not found in the cluster. Failed in initialization: open(/dev/accel0): Invalid argument: Invalid argument; Couldn't open device: /dev/accel0; Unable to create Node RegisterInterface for node 0, config: device_path:      \"/dev/accel0\" mode: KERNEL debug_data_directory: \"\" dump_anomalies_only: true crash_in_debug_dump: false allow_core_dump: true; could not create driver instance</p>\n<p>During handling of the above exception, another exception occurred:</p>\n<p>InvalidArgumentError                      Traceback (most recent call last)<br>\nCell In[8], line 2<br>\n      1 # Initialize GPU/TPU/TPU-VM<br>\n----&gt; 2 strategy, CFG.device, tpu = get_device()<br>\n      3 CFG.replicas = strategy.num_replicas_in_sync</p>\n<p>Cell In[7], line 15, in get_device()<br>\n     12     device=CFG.device<br>\n     13 except:<br>\n     14     # If TPU is not available, detect GPUs<br>\n---&gt; 15     gpus = tf.config.list_logical_devices('GPU')<br>\n     16     ngpu = len(gpus)<br>\n     17      # Check number of GPUs</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/framework/config.py:478, in list_logical_devices(device_type)<br>\n    445 <a href=\"https://www.kaggle.com/tf\" target=\"_blank\">@tf</a>_export('config.list_logical_devices',<br>\n    446            'config.experimental.list_logical_devices')<br>\n    447 <a href=\"https://www.kaggle.com/deprecation.deprecated\" target=\"_blank\">@deprecation.deprecated</a>_endpoints('config.experimental.list_logical_devices')<br>\n    448 def list_logical_devices(device_type=None):<br>\n    449   \"\"\"Return a list of logical devices created by runtime.<br>\n    450 <br>\n    451   Logical devices may correspond to physical devices or remote devices in the<br>\n   (…)<br>\n    476     List of initialized <code>LogicalDevice</code>s<br>\n    477   \"\"\"<br>\n--&gt; 478   return context.context().list_logical_devices(device_type=device_type)</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:1582, in Context.list_logical_devices(self, device_type)<br>\n   1580 def list_logical_devices(self, device_type=None):<br>\n   1581   \"\"\"Return logical devices.\"\"\"<br>\n-&gt; 1582   self.ensure_initialized()<br>\n   1583   if device_type is None:<br>\n   1584     return list(self._logical_devices)</p>\n<p>File /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:583, in Context.ensure_initialized(self)<br>\n    580   pywrap_tfe.TFE_ContextOptionsSetRunEagerOpAsFunction(opts, True)<br>\n    581   pywrap_tfe.TFE_ContextOptionsSetJitCompileRewrite(<br>\n    582       opts, self._jit_compile_rewrite)<br>\n--&gt; 583   context_handle = pywrap_tfe.TFE_NewContext(opts)<br>\n    584 finally:<br>\n    585   pywrap_tfe.TFE_DeleteContextOptions(opts)</p>\n<p>InvalidArgumentError: open(/dev/accel0): Invalid argument: Invalid argument; Couldn't open device: /dev/accel0; Unable to create Node RegisterInterface for node 0, config: device_path:      \"/dev/accel0\" mode: KERNEL debug_data_directory: \"\" dump_anomalies_only: true crash_in_debug_dump: false allow_core_dump: true; could not create driver instanc</p>",
      "rawMarkdown": "Hi kagglers,\n\nI am using the public notebook with TPU\n```python\n# Initialize GPU/TPU/TPU-VM\nstrategy, CFG.device, tpu = get_device()\nCFG.replicas = strategy.num_replicas_in_sync\n\n```\nI am getting following error when I try to access the TPU's today.\nThe same code didn't create any errors previously. \nAny solution ?\n\n\n**Error**\n---------------------------------------------------------------------------\nInvalidArgumentError                      Traceback (most recent call last)\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/tpu/tpu_strategy_util.py:126, in initialize_tpu_system(cluster_resolver)\n    125 try:\n--> 126   with ops.device(tpu._tpu_system_device_name(job)):  # pylint: disable=protected-access\n    127     output = _tpu_init_fn()\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:5526, in device(device_name_or_function)\n   5523     raise RuntimeError(\n   5524         \"tf.device does not support functions when eager execution \"\n   5525         \"is enabled.\")\n-> 5526   return context.device(device_name_or_function)\n   5527 elif executing_eagerly_outside_functions():\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:2348, in device(name)\n   2329 \"\"\"Context-manager to force placement of operations and Tensors on a device.\n   2330 \n   2331 Example:\n   (...)\n   2346   Context manager for setting the device.\n   2347 \"\"\"\n-> 2348 ensure_initialized()\n   2349 return context().device(name)\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:2143, in ensure_initialized()\n   2142 \"\"\"Initialize the context.\"\"\"\n-> 2143 context().ensure_initialized()\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:583, in Context.ensure_initialized(self)\n    581   pywrap_tfe.TFE_ContextOptionsSetJitCompileRewrite(\n    582       opts, self._jit_compile_rewrite)\n--> 583   context_handle = pywrap_tfe.TFE_NewContext(opts)\n    584 finally:\n\nInvalidArgumentError: open(/dev/accel0): Invalid argument: Invalid argument; Couldn't open device: /dev/accel0; Unable to create Node RegisterInterface for node 0, config: device_path: \t \"/dev/accel0\" mode: KERNEL debug_data_directory: \"\" dump_anomalies_only: true crash_in_debug_dump: false allow_core_dump: true; could not create driver instance\n\nDuring handling of the above exception, another exception occurred:\n\nNotFoundError                             Traceback (most recent call last)\nCell In[7], line 7, in get_device()\n      5 try:\n      6     # Connect to TPU\n----> 7     tpu = tf.distribute.cluster_resolver.TPUClusterResolver.connect(tpu=tpu) \n      8     # Set TPU strategy\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/distribute/cluster_resolver/tpu/tpu_cluster_resolver.py:110, in TPUClusterResolver.connect(tpu, zone, project)\n    109 from tensorflow.python.tpu import tpu_strategy_util  # pylint: disable=g-import-not-at-top\n--> 110 tpu_strategy_util.initialize_tpu_system(resolver)\n    111 return resolver\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/tpu/tpu_strategy_util.py:130, in initialize_tpu_system(cluster_resolver)\n    129 except errors.InvalidArgumentError as e:\n--> 130   raise errors.NotFoundError(\n    131       None, None,\n    132       \"TPUs not found in the cluster. Failed in initialization: \"\n    133       + str(e))\n    134 finally:\n\nNotFoundError: TPUs not found in the cluster. Failed in initialization: open(/dev/accel0): Invalid argument: Invalid argument; Couldn't open device: /dev/accel0; Unable to create Node RegisterInterface for node 0, config: device_path: \t \"/dev/accel0\" mode: KERNEL debug_data_directory: \"\" dump_anomalies_only: true crash_in_debug_dump: false allow_core_dump: true; could not create driver instance\n\nDuring handling of the above exception, another exception occurred:\n\nInvalidArgumentError                      Traceback (most recent call last)\nCell In[8], line 2\n      1 # Initialize GPU/TPU/TPU-VM\n----> 2 strategy, CFG.device, tpu = get_device()\n      3 CFG.replicas = strategy.num_replicas_in_sync\n\nCell In[7], line 15, in get_device()\n     12     device=CFG.device\n     13 except:\n     14     # If TPU is not available, detect GPUs\n---> 15     gpus = tf.config.list_logical_devices('GPU')\n     16     ngpu = len(gpus)\n     17      # Check number of GPUs\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/framework/config.py:478, in list_logical_devices(device_type)\n    445 @tf_export('config.list_logical_devices',\n    446            'config.experimental.list_logical_devices')\n    447 @deprecation.deprecated_endpoints('config.experimental.list_logical_devices')\n    448 def list_logical_devices(device_type=None):\n    449   \"\"\"Return a list of logical devices created by runtime.\n    450 \n    451   Logical devices may correspond to physical devices or remote devices in the\n   (...)\n    476     List of initialized `LogicalDevice`s\n    477   \"\"\"\n--> 478   return context.context().list_logical_devices(device_type=device_type)\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:1582, in Context.list_logical_devices(self, device_type)\n   1580 def list_logical_devices(self, device_type=None):\n   1581   \"\"\"Return logical devices.\"\"\"\n-> 1582   self.ensure_initialized()\n   1583   if device_type is None:\n   1584     return list(self._logical_devices)\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:583, in Context.ensure_initialized(self)\n    580   pywrap_tfe.TFE_ContextOptionsSetRunEagerOpAsFunction(opts, True)\n    581   pywrap_tfe.TFE_ContextOptionsSetJitCompileRewrite(\n    582       opts, self._jit_compile_rewrite)\n--> 583   context_handle = pywrap_tfe.TFE_NewContext(opts)\n    584 finally:\n    585   pywrap_tfe.TFE_DeleteContextOptions(opts)\n\nInvalidArgumentError: open(/dev/accel0): Invalid argument: Invalid argument; Couldn't open device: /dev/accel0; Unable to create Node RegisterInterface for node 0, config: device_path: \t \"/dev/accel0\" mode: KERNEL debug_data_directory: \"\" dump_anomalies_only: true crash_in_debug_dump: false allow_core_dump: true; could not create driver instanc",
      "votes": 1
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "2250149": "Hi kagglers,\n\nI am using the public notebook with TPU\n```python\n# Initialize GPU/TPU/TPU-VM\nstrategy, CFG.device, tpu = get_device()\nCFG.replicas = strategy.num_replicas_in_sync\n\n```\nI am getting following error when I try to access the TPU's today.\nThe same code didn't create any errors previously. \nAny solution ?\n\n\n**Error**\n---------------------------------------------------------------------------\nInvalidArgumentError                      Traceback (most recent call last)\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/tpu/tpu_strategy_util.py:126, in initialize_tpu_system(cluster_resolver)\n    125 try:\n--> 126   with ops.device(tpu._tpu_system_device_name(job)):  # pylint: disable=protected-access\n    127     output = _tpu_init_fn()\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:5526, in device(device_name_or_function)\n   5523     raise RuntimeError(\n   5524         \"tf.device does not support functions when eager execution \"\n   5525         \"is enabled.\")\n-> 5526   return context.device(device_name_or_function)\n   5527 elif executing_eagerly_outside_functions():\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:2348, in device(name)\n   2329 \"\"\"Context-manager to force placement of operations and Tensors on a device.\n   2330 \n   2331 Example:\n   (...)\n   2346   Context manager for setting the device.\n   2347 \"\"\"\n-> 2348 ensure_initialized()\n   2349 return context().device(name)\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:2143, in ensure_initialized()\n   2142 \"\"\"Initialize the context.\"\"\"\n-> 2143 context().ensure_initialized()\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:583, in Context.ensure_initialized(self)\n    581   pywrap_tfe.TFE_ContextOptionsSetJitCompileRewrite(\n    582       opts, self._jit_compile_rewrite)\n--> 583   context_handle = pywrap_tfe.TFE_NewContext(opts)\n    584 finally:\n\nInvalidArgumentError: open(/dev/accel0): Invalid argument: Invalid argument; Couldn't open device: /dev/accel0; Unable to create Node RegisterInterface for node 0, config: device_path: \t \"/dev/accel0\" mode: KERNEL debug_data_directory: \"\" dump_anomalies_only: true crash_in_debug_dump: false allow_core_dump: true; could not create driver instance\n\nDuring handling of the above exception, another exception occurred:\n\nNotFoundError                             Traceback (most recent call last)\nCell In[7], line 7, in get_device()\n      5 try:\n      6     # Connect to TPU\n----> 7     tpu = tf.distribute.cluster_resolver.TPUClusterResolver.connect(tpu=tpu) \n      8     # Set TPU strategy\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/distribute/cluster_resolver/tpu/tpu_cluster_resolver.py:110, in TPUClusterResolver.connect(tpu, zone, project)\n    109 from tensorflow.python.tpu import tpu_strategy_util  # pylint: disable=g-import-not-at-top\n--> 110 tpu_strategy_util.initialize_tpu_system(resolver)\n    111 return resolver\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/tpu/tpu_strategy_util.py:130, in initialize_tpu_system(cluster_resolver)\n    129 except errors.InvalidArgumentError as e:\n--> 130   raise errors.NotFoundError(\n    131       None, None,\n    132       \"TPUs not found in the cluster. Failed in initialization: \"\n    133       + str(e))\n    134 finally:\n\nNotFoundError: TPUs not found in the cluster. Failed in initialization: open(/dev/accel0): Invalid argument: Invalid argument; Couldn't open device: /dev/accel0; Unable to create Node RegisterInterface for node 0, config: device_path: \t \"/dev/accel0\" mode: KERNEL debug_data_directory: \"\" dump_anomalies_only: true crash_in_debug_dump: false allow_core_dump: true; could not create driver instance\n\nDuring handling of the above exception, another exception occurred:\n\nInvalidArgumentError                      Traceback (most recent call last)\nCell In[8], line 2\n      1 # Initialize GPU/TPU/TPU-VM\n----> 2 strategy, CFG.device, tpu = get_device()\n      3 CFG.replicas = strategy.num_replicas_in_sync\n\nCell In[7], line 15, in get_device()\n     12     device=CFG.device\n     13 except:\n     14     # If TPU is not available, detect GPUs\n---> 15     gpus = tf.config.list_logical_devices('GPU')\n     16     ngpu = len(gpus)\n     17      # Check number of GPUs\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/framework/config.py:478, in list_logical_devices(device_type)\n    445 @tf_export('config.list_logical_devices',\n    446            'config.experimental.list_logical_devices')\n    447 @deprecation.deprecated_endpoints('config.experimental.list_logical_devices')\n    448 def list_logical_devices(device_type=None):\n    449   \"\"\"Return a list of logical devices created by runtime.\n    450 \n    451   Logical devices may correspond to physical devices or remote devices in the\n   (...)\n    476     List of initialized `LogicalDevice`s\n    477   \"\"\"\n--> 478   return context.context().list_logical_devices(device_type=device_type)\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:1582, in Context.list_logical_devices(self, device_type)\n   1580 def list_logical_devices(self, device_type=None):\n   1581   \"\"\"Return logical devices.\"\"\"\n-> 1582   self.ensure_initialized()\n   1583   if device_type is None:\n   1584     return list(self._logical_devices)\n\nFile /usr/local/lib/python3.8/site-packages/tensorflow/python/eager/context.py:583, in Context.ensure_initialized(self)\n    580   pywrap_tfe.TFE_ContextOptionsSetRunEagerOpAsFunction(opts, True)\n    581   pywrap_tfe.TFE_ContextOptionsSetJitCompileRewrite(\n    582       opts, self._jit_compile_rewrite)\n--> 583   context_handle = pywrap_tfe.TFE_NewContext(opts)\n    584 finally:\n    585   pywrap_tfe.TFE_DeleteContextOptions(opts)\n\nInvalidArgumentError: open(/dev/accel0): Invalid argument: Invalid argument; Couldn't open device: /dev/accel0; Unable to create Node RegisterInterface for node 0, config: device_path: \t \"/dev/accel0\" mode: KERNEL debug_data_directory: \"\" dump_anomalies_only: true crash_in_debug_dump: false allow_core_dump: true; could not create driver instanc"
  }
}