Hi to all. Today I get an error on different notebooks:
\nUnavailableError: 4 root error(s) found.
\n (0) Unavailable: {{function_node __inference_train_function_102325}} failed to connect to all addresses
\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
\n:{\"created\":\"@1612285052.764414505\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1612285052.764412286\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}
\n [[{{node StatefulPartitionedCall}}]]
\n [[MultiDeviceIteratorGetNextFromShard]]
\n [[RemoteCall]]
\n [[IteratorGetNext]]
\n [[Pad_14/_162]]
\n (1) Unavailable: {{function_node __inference_train_function_102325}} failed to connect to all addresses
\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
\n:{\"created\":\"@1612285052.764414505\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1612285052.764412286\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}
\n [[{{node StatefulPartitionedCall}}]]
\n [[MultiDeviceIteratorGetNextFromShard]]
\n [[RemoteCall]]
\n [[IteratorGetNext]]
\n [[cluster_train_function/_execute_4_0/_195]]
\n (2) Unavailable: {{function_node __inference_train_function_102325}} failed to connect to all addresses
\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
\n:{\"created\":\"@1612285052.764414505\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1612285052.764412286\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}
\n [[{{node StatefulPartitionedCall}}]]
\n [[MultiDeviceIteratorGetNextFromShard]]
\n [[RemoteCall]]
\n [[IteratorGetNext]]
\n [[Pad_12/_144]]
\n (3) Unavailable: {{function_node __inference_train_function_102325}} failed to connect to all addresses
\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
\n:{\"created\":\"@1612285052.764414505\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1612285052.764412286\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}
\n [[{{node StatefulPartitionedCall}}]]
\n [[MultiDeviceIteratorGetNextFromShard]]
\n [[RemoteCall]]
\n [[IteratorGetNext]]
\n0 succ … [truncated]
Іs it just that I have such problems, or is it related to general problems in Kaggle …
", "messages": [ { "id": "1182976", "postDate": "02/02/2021 17:05:44", "content": "Hi to all. Today I get an error on different notebooks:
\nUnavailableError: 4 root error(s) found.
\n (0) Unavailable: {{function_node __inference_train_function_102325}} failed to connect to all addresses
\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
\n:{\"created\":\"@1612285052.764414505\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1612285052.764412286\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}
\n [[{{node StatefulPartitionedCall}}]]
\n [[MultiDeviceIteratorGetNextFromShard]]
\n [[RemoteCall]]
\n [[IteratorGetNext]]
\n [[Pad_14/_162]]
\n (1) Unavailable: {{function_node __inference_train_function_102325}} failed to connect to all addresses
\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
\n:{\"created\":\"@1612285052.764414505\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1612285052.764412286\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}
\n [[{{node StatefulPartitionedCall}}]]
\n [[MultiDeviceIteratorGetNextFromShard]]
\n [[RemoteCall]]
\n [[IteratorGetNext]]
\n [[cluster_train_function/_execute_4_0/_195]]
\n (2) Unavailable: {{function_node __inference_train_function_102325}} failed to connect to all addresses
\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
\n:{\"created\":\"@1612285052.764414505\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1612285052.764412286\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}
\n [[{{node StatefulPartitionedCall}}]]
\n [[MultiDeviceIteratorGetNextFromShard]]
\n [[RemoteCall]]
\n [[IteratorGetNext]]
\n [[Pad_12/_144]]
\n (3) Unavailable: {{function_node __inference_train_function_102325}} failed to connect to all addresses
\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
\n:{\"created\":\"@1612285052.764414505\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1612285052.764412286\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}
\n [[{{node StatefulPartitionedCall}}]]
\n [[MultiDeviceIteratorGetNextFromShard]]
\n [[RemoteCall]]
\n [[IteratorGetNext]]
\n0 succ … [truncated]
Іs it just that I have such problems, or is it related to general problems in Kaggle …
", "rawMarkdown": "Hi to all. Today I get an error on different notebooks:\n\nUnavailableError: 4 root error(s) found.\n (0) Unavailable: {{function_node __inference_train_function_102325}} failed to connect to all addresses\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:\n:{\"created\":\"@1612285052.764414505\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1612285052.764412286\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}\n\t [[{{node StatefulPartitionedCall}}]]\n\t [[MultiDeviceIteratorGetNextFromShard]]\n\t [[RemoteCall]]\n\t [[IteratorGetNext]]\n\t [[Pad_14/_162]]\n (1) Unavailable: {{function_node __inference_train_function_102325}} failed to connect to all addresses\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:\n:{\"created\":\"@1612285052.764414505\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1612285052.764412286\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}\n\t [[{{node StatefulPartitionedCall}}]]\n\t [[MultiDeviceIteratorGetNextFromShard]]\n\t [[RemoteCall]]\n\t [[IteratorGetNext]]\n\t [[cluster_train_function/_execute_4_0/_195]]\n (2) Unavailable: {{function_node __inference_train_function_102325}} failed to connect to all addresses\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:\n:{\"created\":\"@1612285052.764414505\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1612285052.764412286\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}\n\t [[{{node StatefulPartitionedCall}}]]\n\t [[MultiDeviceIteratorGetNextFromShard]]\n\t [[RemoteCall]]\n\t [[IteratorGetNext]]\n\t [[Pad_12/_144]]\n (3) Unavailable: {{function_node __inference_train_function_102325}} failed to connect to all addresses\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:\n:{\"created\":\"@1612285052.764414505\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1612285052.764412286\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}\n\t [[{{node StatefulPartitionedCall}}]]\n\t [[MultiDeviceIteratorGetNextFromShard]]\n\t [[RemoteCall]]\n\t [[IteratorGetNext]]\n0 succ ... [truncated]\n\n\nІs it just that I have such problems, or is it related to general problems in Kaggle ...", "votes": null }, { "id": "1183575", "postDate": "02/03/2021 05:20:20", "content": "Same happend to me. May be due to kaggle have updated their environment.
", "rawMarkdown": "Same happend to me. May be due to kaggle have updated their environment.", "votes": null }, { "id": "1183819", "postDate": "02/03/2021 08:34:42", "content": "See some analysis & possible workaround here: https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/216408
", "rawMarkdown": "See some analysis & possible workaround here: https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/216408", "votes": null }, { "id": "1184346", "postDate": "02/03/2021 13:53:26", "content": "i just realized i posted a similar thread. Yeah all my kernels are down at this point as well…
\nedit: it seems like there are some updates to the docker images for the kernel. and this seems to affect the TPU kernel only
\nedit2: just disable tpu and run with gpu + older docker env. seems to work now
", "rawMarkdown": "i just realized i posted a similar thread. Yeah all my kernels are down at this point as well...\n\nedit: it seems like there are some updates to the docker images for the kernel. and this seems to affect the TPU kernel only\n\nedit2: just disable tpu and run with gpu + older docker env. seems to work now", "votes": null }, { "id": "1185360", "postDate": "02/04/2021 06:06:06", "content": "Both the old and new docker images dont work with TPU. The status 14 GPRC unavailable error is typically linked to firewall/proxy settings. gRPC is google's Remote procedure call platform which tf uses to connect the master with the slaves. Seems like some pip downloads could mess it up (for e.g. grpcio - https://stackoverflow.com/questions/57397723/grpc-client-failing-to-connect-to-server-with-tls-certificates - though I noticed it isnt used in our case..but probably some other library could have changed which does not support some service used by gRPC). This isnt the first time this is happening on Kaggle. ..see for e.g. https://www.kaggle.com/c/alaska2-image-steganalysis/discussion/168316. Of course, Torch users are not impacted…Most of the winning solutions (in past competitions) were Torch based..I am guessing same applies here too which explains why not too many people seem bothered about this issue
", "rawMarkdown": "Both the old and new docker images dont work with TPU. The status 14 GPRC unavailable error is typically linked to firewall/proxy settings. gRPC is google's Remote procedure call platform which tf uses to connect the master with the slaves. Seems like some pip downloads could mess it up (for e.g. grpcio - https://stackoverflow.com/questions/57397723/grpc-client-failing-to-connect-to-server-with-tls-certificates - though I noticed it isnt used in our case..but probably some other library could have changed which does not support some service used by gRPC). This isnt the first time this is happening on Kaggle. ..see for e.g. https://www.kaggle.com/c/alaska2-image-steganalysis/discussion/168316. Of course, Torch users are not impacted...Most of the winning solutions (in past competitions) were Torch based..I am guessing same applies here too which explains why not too many people seem bothered about this issue", "votes": null }, { "id": "1186851", "postDate": "02/05/2021 04:50:15", "content": "There is a workaround available now. See comments by Kaggle support staff in :
\nhttps://www.kaggle.com/c/rfcx-species-audio-detection/discussion/216408
Same happend to me. May be due to kaggle have updated their environment.
", "votes": null, "replies": [] }, { "id": 1183819, "author_name": "allohvk", "author_url": "", "post_date": "02/03/2021 08:34:42", "content": "See some analysis & possible workaround here: https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/216408
", "votes": null, "replies": [] }, { "id": 1184346, "author_name": "germmie", "author_url": "", "post_date": "02/03/2021 13:53:26", "content": "i just realized i posted a similar thread. Yeah all my kernels are down at this point as well…
\nedit: it seems like there are some updates to the docker images for the kernel. and this seems to affect the TPU kernel only
\nedit2: just disable tpu and run with gpu + older docker env. seems to work now
", "votes": null, "replies": [ { "id": 1185360, "author_name": "allohvk", "author_url": "", "post_date": "02/04/2021 06:06:06", "content": "Both the old and new docker images dont work with TPU. The status 14 GPRC unavailable error is typically linked to firewall/proxy settings. gRPC is google's Remote procedure call platform which tf uses to connect the master with the slaves. Seems like some pip downloads could mess it up (for e.g. grpcio - https://stackoverflow.com/questions/57397723/grpc-client-failing-to-connect-to-server-with-tls-certificates - though I noticed it isnt used in our case..but probably some other library could have changed which does not support some service used by gRPC). This isnt the first time this is happening on Kaggle. ..see for e.g. https://www.kaggle.com/c/alaska2-image-steganalysis/discussion/168316. Of course, Torch users are not impacted…Most of the winning solutions (in past competitions) were Torch based..I am guessing same applies here too which explains why not too many people seem bothered about this issue
", "votes": null, "replies": [] } ] }, { "id": 1186851, "author_name": "allohvk", "author_url": "", "post_date": "02/05/2021 04:50:15", "content": "There is a workaround available now. See comments by Kaggle support staff in :
\nhttps://www.kaggle.com/c/rfcx-species-audio-detection/discussion/216408