Organizers none of my notebooks with TPU work, I wrote about it before. In addition to me, several other people have faced this problem (Several discussions on this topic have appeared at the same time). Notebooks have worked well until recently. I do not see the reason for the error, Google Colab does NOT have this problem! This is not a problem for me because I use colab. But if only a part of the users encountered the problem, it is at least wrong from a sports point of view.
\nUnavailableError: 7 root error(s) found.
\n (0) Unavailable: {{function_node __inference_train_function_91692}} failed to connect to all addresses
\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
\n:{\"created\":\"@1612477044.916429830\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1612477044.916423940\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}
\n [[{{node StatefulPartitionedCall}}]]
\n [[MultiDeviceIteratorGetNextFromShard]]
\n [[RemoteCall]]
\n [[IteratorGetNext]]
\n [[Pad_12/_144]]
\n (1) Unavailable: {{function_node __inference_train_function_91692}} failed to connect to all addresses
\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
\n:{\"created\":\"@1612477044.916429830\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1612477044.916423940\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}
\n [[{{node StatefulPartitionedCall}}]]
\n [[MultiDeviceIteratorGetNextFromShard]]
\n [[RemoteCall]]
\n [[IteratorGetNext]]
\n [[IteratorGetNext_2/_12]]
\n (2) Unavailable: {{function_node __inference_train_function_91692}} failed to connect to all addresses
\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
\n:{\"created\":\"@1612477044.916429830\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1612477044.916423940\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}
\n [[{{node StatefulPartitionedCall}}]]
\n [[MultiDeviceIteratorGetNextFromShard]]
\n [[RemoteCall]]
\n [[IteratorGetNext]]
\n (3) Unavailable: {{function_node __inference_train_function_91692}} failed to connect to all addresses
\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
\n:{\"created\":\"@1612477044.916429830\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1612477044.916423940\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}
\n [[{{node StatefulPartitionedCall}}]]
\n [[MultiDeviceIteratorGetNextFromShard]]
\n [[RemoteCall]]
\n [[IteratorGetNext]]
\n [[tpu_compile_succeeded_assert/_4487815273615 … [truncated]
Maybe someone has already solved this problem?
", "messages": [ { "id": "1186564", "postDate": "02/04/2021 22:34:53", "content": "Organizers none of my notebooks with TPU work, I wrote about it before. In addition to me, several other people have faced this problem (Several discussions on this topic have appeared at the same time). Notebooks have worked well until recently. I do not see the reason for the error, Google Colab does NOT have this problem! This is not a problem for me because I use colab. But if only a part of the users encountered the problem, it is at least wrong from a sports point of view.
\nUnavailableError: 7 root error(s) found.
\n (0) Unavailable: {{function_node __inference_train_function_91692}} failed to connect to all addresses
\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
\n:{\"created\":\"@1612477044.916429830\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1612477044.916423940\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}
\n [[{{node StatefulPartitionedCall}}]]
\n [[MultiDeviceIteratorGetNextFromShard]]
\n [[RemoteCall]]
\n [[IteratorGetNext]]
\n [[Pad_12/_144]]
\n (1) Unavailable: {{function_node __inference_train_function_91692}} failed to connect to all addresses
\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
\n:{\"created\":\"@1612477044.916429830\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1612477044.916423940\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}
\n [[{{node StatefulPartitionedCall}}]]
\n [[MultiDeviceIteratorGetNextFromShard]]
\n [[RemoteCall]]
\n [[IteratorGetNext]]
\n [[IteratorGetNext_2/_12]]
\n (2) Unavailable: {{function_node __inference_train_function_91692}} failed to connect to all addresses
\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
\n:{\"created\":\"@1612477044.916429830\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1612477044.916423940\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}
\n [[{{node StatefulPartitionedCall}}]]
\n [[MultiDeviceIteratorGetNextFromShard]]
\n [[RemoteCall]]
\n [[IteratorGetNext]]
\n (3) Unavailable: {{function_node __inference_train_function_91692}} failed to connect to all addresses
\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
\n:{\"created\":\"@1612477044.916429830\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1612477044.916423940\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}
\n [[{{node StatefulPartitionedCall}}]]
\n [[MultiDeviceIteratorGetNextFromShard]]
\n [[RemoteCall]]
\n [[IteratorGetNext]]
\n [[tpu_compile_succeeded_assert/_4487815273615 … [truncated]
Maybe someone has already solved this problem?
", "rawMarkdown": "Organizers none of my notebooks with TPU work, I wrote about it before. In addition to me, several other people have faced this problem (Several discussions on this topic have appeared at the same time). Notebooks have worked well until recently. I do not see the reason for the error, Google Colab does NOT have this problem! This is not a problem for me because I use colab. But if only a part of the users encountered the problem, it is at least wrong from a sports point of view.\n\n\nUnavailableError: 7 root error(s) found.\n (0) Unavailable: {{function_node __inference_train_function_91692}} failed to connect to all addresses\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:\n:{\"created\":\"@1612477044.916429830\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1612477044.916423940\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}\n\t [[{{node StatefulPartitionedCall}}]]\n\t [[MultiDeviceIteratorGetNextFromShard]]\n\t [[RemoteCall]]\n\t [[IteratorGetNext]]\n\t [[Pad_12/_144]]\n (1) Unavailable: {{function_node __inference_train_function_91692}} failed to connect to all addresses\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:\n:{\"created\":\"@1612477044.916429830\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1612477044.916423940\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}\n\t [[{{node StatefulPartitionedCall}}]]\n\t [[MultiDeviceIteratorGetNextFromShard]]\n\t [[RemoteCall]]\n\t [[IteratorGetNext]]\n\t [[IteratorGetNext_2/_12]]\n (2) Unavailable: {{function_node __inference_train_function_91692}} failed to connect to all addresses\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:\n:{\"created\":\"@1612477044.916429830\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1612477044.916423940\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}\n\t [[{{node StatefulPartitionedCall}}]]\n\t [[MultiDeviceIteratorGetNextFromShard]]\n\t [[RemoteCall]]\n\t [[IteratorGetNext]]\n (3) Unavailable: {{function_node __inference_train_function_91692}} failed to connect to all addresses\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:\n:{\"created\":\"@1612477044.916429830\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1612477044.916423940\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}\n\t [[{{node StatefulPartitionedCall}}]]\n\t [[MultiDeviceIteratorGetNextFromShard]]\n\t [[RemoteCall]]\n\t [[IteratorGetNext]]\n\t [[tpu_compile_succeeded_assert/_4487815273615 ... [truncated]\n\n\nMaybe someone has already solved this problem?", "votes": null }, { "id": "1186568", "postDate": "02/04/2021 22:39:02", "content": "Another thought came to me: does anyone know the Kaggle team did not update the basic version of tensorflow? It is in the style of this library after the next update there are new incomprehensible errors that are unclear how to fix, it is easier to install the previous version :-).
", "rawMarkdown": "Another thought came to me: does anyone know the Kaggle team did not update the basic version of tensorflow? It is in the style of this library after the next update there are new incomprehensible errors that are unclear how to fix, it is easier to install the previous version :-).", "votes": null }, { "id": "1186707", "postDate": "02/05/2021 02:22:10", "content": "I faced the same problem. The docker file environment can't be changed with TPU. I don't know why. I guess it's because of the package upgrade.
", "rawMarkdown": "I faced the same problem. The docker file environment can't be changed with TPU. I don't know why. I guess it's because of the package upgrade.", "votes": null }, { "id": "1186850", "postDate": "02/05/2021 04:49:12", "content": "There is a workaround available now. See comments by Kaggle support staff in :
\nhttps://www.kaggle.com/c/rfcx-species-audio-detection/discussion/216408
Another thought came to me: does anyone know the Kaggle team did not update the basic version of tensorflow? It is in the style of this library after the next update there are new incomprehensible errors that are unclear how to fix, it is easier to install the previous version :-).
", "votes": null, "replies": [] }, { "id": 1186707, "author_name": "gengzll", "author_url": "", "post_date": "02/05/2021 02:22:10", "content": "I faced the same problem. The docker file environment can't be changed with TPU. I don't know why. I guess it's because of the package upgrade.
", "votes": null, "replies": [] }, { "id": 1186850, "author_name": "allohvk", "author_url": "", "post_date": "02/05/2021 04:49:12", "content": "There is a workaround available now. See comments by Kaggle support staff in :
\nhttps://www.kaggle.com/c/rfcx-species-audio-detection/discussion/216408