I can't use TF with TPU, because today I suddenly got bunch of errors like
\n2021-09-11 09:51:16.151813: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib\n2021-09-11 09:51:16.151941: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n2021-09-11 09:51:22.020664: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set\n2021-09-11 09:51:22.023435: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib\n2021-09-11 09:51:22.023467: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)\n2021-09-11 09:51:22.023492: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (11dddf7bee45): /proc/driver/nvidia/version does not exist\n2021-09-11 09:51:22.025490: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA\nTo enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n2021-09-11 09:51:22.027151: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set\n2021-09-11 09:51:22.064686: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:301] Initialize GrpcChannelCache for job worker -> {0 -> 10.0.0.2:8470}\n2021-09-11 09:51:22.064754: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:301] Initialize GrpcChannelCache for job localhost -> {0 -> localhost:30043}\n2021-09-11 09:51:22.083529: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:301] Initialize GrpcChannelCache for job worker -> {0 -> 10.0.0.2:8470}\n2021-09-11 09:51:22.083583: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:301] Initialize GrpcChannelCache for job localhost -> {0 -> localhost:30043}\n2021-09-11 09:51:22.085070: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:411] Started server with target: grpc://localhost:30043\n2021-09-11 09:51:29.253986: I tensorflow/core/platform/cloud/google_auth_provider.cc:180] Attempting an empty bearer token since no token was retrieved from files, and GCE metadata check was skipped.\n2021-09-11 09:51:29.328709: I tensorflow/core/platform/cloud/google_auth_provider.cc:180] Attempting an empty bearer token since no token was retrieved from files, and GCE metadata check was skipped.\n2021-09-11 09:51:29.395006: I tensorflow/core/platform/cloud/google_auth_provider.cc:180] Attempting an empty bearer token since no token was retrieved from files, and GCE metadata check was skipped.\n2021-09-11 09:51:29.461165: I tensorflow/core/platform/cloud/google_auth_provider.cc:180] Attempting an empty bearer token since no token was retrieved from files, and GCE metadata check was skipped.\nAnd so on. Does anyone else observe the same thing?
", "messages": [ { "id": "1509443", "postDate": "09/11/2021 09:54:45", "content": "I can't use TF with TPU, because today I suddenly got bunch of errors like
\n2021-09-11 09:51:16.151813: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib\n2021-09-11 09:51:16.151941: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n2021-09-11 09:51:22.020664: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set\n2021-09-11 09:51:22.023435: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib\n2021-09-11 09:51:22.023467: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)\n2021-09-11 09:51:22.023492: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (11dddf7bee45): /proc/driver/nvidia/version does not exist\n2021-09-11 09:51:22.025490: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA\nTo enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n2021-09-11 09:51:22.027151: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set\n2021-09-11 09:51:22.064686: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:301] Initialize GrpcChannelCache for job worker -> {0 -> 10.0.0.2:8470}\n2021-09-11 09:51:22.064754: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:301] Initialize GrpcChannelCache for job localhost -> {0 -> localhost:30043}\n2021-09-11 09:51:22.083529: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:301] Initialize GrpcChannelCache for job worker -> {0 -> 10.0.0.2:8470}\n2021-09-11 09:51:22.083583: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:301] Initialize GrpcChannelCache for job localhost -> {0 -> localhost:30043}\n2021-09-11 09:51:22.085070: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:411] Started server with target: grpc://localhost:30043\n2021-09-11 09:51:29.253986: I tensorflow/core/platform/cloud/google_auth_provider.cc:180] Attempting an empty bearer token since no token was retrieved from files, and GCE metadata check was skipped.\n2021-09-11 09:51:29.328709: I tensorflow/core/platform/cloud/google_auth_provider.cc:180] Attempting an empty bearer token since no token was retrieved from files, and GCE metadata check was skipped.\n2021-09-11 09:51:29.395006: I tensorflow/core/platform/cloud/google_auth_provider.cc:180] Attempting an empty bearer token since no token was retrieved from files, and GCE metadata check was skipped.\n2021-09-11 09:51:29.461165: I tensorflow/core/platform/cloud/google_auth_provider.cc:180] Attempting an empty bearer token since no token was retrieved from files, and GCE metadata check was skipped.\nAnd so on. Does anyone else observe the same thing?
", "rawMarkdown": "I can't use TF with TPU, because today I suddenly got bunch of errors like\n```\n2021-09-11 09:51:16.151813: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib\n2021-09-11 09:51:16.151941: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n```\n\n```\n2021-09-11 09:51:22.020664: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set\n2021-09-11 09:51:22.023435: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib\n2021-09-11 09:51:22.023467: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)\n2021-09-11 09:51:22.023492: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (11dddf7bee45): /proc/driver/nvidia/version does not exist\n2021-09-11 09:51:22.025490: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA\nTo enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n2021-09-11 09:51:22.027151: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set\n2021-09-11 09:51:22.064686: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:301] Initialize GrpcChannelCache for job worker -> {0 -> 10.0.0.2:8470}\n2021-09-11 09:51:22.064754: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:301] Initialize GrpcChannelCache for job localhost -> {0 -> localhost:30043}\n2021-09-11 09:51:22.083529: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:301] Initialize GrpcChannelCache for job worker -> {0 -> 10.0.0.2:8470}\n2021-09-11 09:51:22.083583: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:301] Initialize GrpcChannelCache for job localhost -> {0 -> localhost:30043}\n2021-09-11 09:51:22.085070: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:411] Started server with target: grpc://localhost:30043\n```\n\n```\n2021-09-11 09:51:29.253986: I tensorflow/core/platform/cloud/google_auth_provider.cc:180] Attempting an empty bearer token since no token was retrieved from files, and GCE metadata check was skipped.\n2021-09-11 09:51:29.328709: I tensorflow/core/platform/cloud/google_auth_provider.cc:180] Attempting an empty bearer token since no token was retrieved from files, and GCE metadata check was skipped.\n2021-09-11 09:51:29.395006: I tensorflow/core/platform/cloud/google_auth_provider.cc:180] Attempting an empty bearer token since no token was retrieved from files, and GCE metadata check was skipped.\n2021-09-11 09:51:29.461165: I tensorflow/core/platform/cloud/google_auth_provider.cc:180] Attempting an empty bearer token since no token was retrieved from files, and GCE metadata check was skipped.\n```\n\nAnd so on. Does anyone else observe the same thing?", "votes": null }, { "id": "1509450", "postDate": "09/11/2021 10:07:37", "content": "I also have those errors, but TPU works normally. They seems to be related to lack of GPU in TPU enviroment.
", "rawMarkdown": "I also have those errors, but TPU works normally. They seems to be related to lack of GPU in TPU enviroment.", "votes": null }, { "id": "1509457", "postDate": "09/11/2021 10:22:19", "content": "Interesting. So you say that notebook will run correctly despite all these errors/warnings?
", "rawMarkdown": "Interesting. So you say that notebook will run correctly despite all these errors/warnings?", "votes": null }, { "id": "1509605", "postDate": "09/11/2021 13:06:01", "content": "I too faced these errors, but based on execution status of the kernel/CV/LB score of given kernel I can tell that everything works good in my case.
", "rawMarkdown": "I too faced these errors, but based on execution status of the kernel/CV/LB score of given kernel I can tell that everything works good in my case.", "votes": null }, { "id": "1509866", "postDate": "09/11/2021 18:29:48", "content": "It is weird to look for cuda when using TPU. This seems to be a little glitch on TensorFlow side. As long a syou don't have TPU related errors you should be fine.
", "rawMarkdown": "It is weird to look for cuda when using TPU. This seems to be a little glitch on TensorFlow side. As long a syou don't have TPU related errors you should be fine.", "votes": null }, { "id": "1511743", "postDate": "09/13/2021 16:42:37", "content": "These error messages shouldn't affect the execution of your notebook. It's just an artifact of Tensorflows build checking if a GPU is available, but it does not require it.
", "rawMarkdown": "These error messages shouldn't affect the execution of your notebook. It's just an artifact of Tensorflows build checking if a GPU is available, but it does not require it.", "votes": null }, { "id": "1559710", "postDate": "10/27/2021 07:08:31", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
I also have those errors, but TPU works normally. They seems to be related to lack of GPU in TPU enviroment.
", "votes": null, "replies": [ { "id": 1509457, "author_name": "atamazian", "author_url": "", "post_date": "09/11/2021 10:22:19", "content": "Interesting. So you say that notebook will run correctly despite all these errors/warnings?
", "votes": null, "replies": [] }, { "id": 1509605, "author_name": "martynoveduard", "author_url": "", "post_date": "09/11/2021 13:06:01", "content": "I too faced these errors, but based on execution status of the kernel/CV/LB score of given kernel I can tell that everything works good in my case.
", "votes": null, "replies": [] } ] }, { "id": 1509866, "author_name": "cpmpml", "author_url": "", "post_date": "09/11/2021 18:29:48", "content": "It is weird to look for cuda when using TPU. This seems to be a little glitch on TensorFlow side. As long a syou don't have TPU related errors you should be fine.
", "votes": null, "replies": [] }, { "id": 1511743, "author_name": "herbison", "author_url": "", "post_date": "09/13/2021 16:42:37", "content": "These error messages shouldn't affect the execution of your notebook. It's just an artifact of Tensorflows build checking if a GPU is available, but it does not require it.
", "votes": null, "replies": [] }, { "id": 1559710, "author_name": "zerafachris", "author_url": "", "post_date": "10/27/2021 07:08:31", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris