{
  "id": 226703,
  "title": "Facing \"UnavailableError\" while trying to fit model with Keras ImageDataGenerator",
  "url": "/competitions/plant-pathology-2021-fgvc8/discussion/226703",
  "author_name": "",
  "post_date": "2021-03-17T11:23:36.630185300Z",
  "votes": 1,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Hi,</p>\n<p>I am getting this error when I try to train the model with ImageDataGenerator. But how ever if I use GPU I am not facing any such issues. Does anyone has any workaround for this?</p>\n<pre><code>def get_generators():\n    datagen=ImageDataGenerator(rescale = 1./255,\n                                      shear_range = 0.2,\n                                      zoom_range = 0.2,\n                                      horizontal_flip = True)\n\n    test_datagen=ImageDataGenerator(rescale = 1./255,\n                                      shear_range = 0.2,\n                                      zoom_range = 0.2,\n                                      horizontal_flip = True)\n\n    #Train\n    train_generator=datagen.flow_from_dataframe(\n    dataframe=train_data[:15000],\n    directory=train_dir,\n    x_col=\"image\",\n    y_col=list(labels.unique()),\n    batch_size=32,\n    seed=42,\n    shuffle=True,\n    class_mode=\"raw\",\n    target_size=(IMG_SIZE,IMG_SIZE))\n\n    #Valid\n    valid_generator=datagen.flow_from_dataframe(\n    dataframe=train_data[15000:],\n    directory=train_dir,\n    x_col=\"image\",\n    y_col=list(labels.unique()),\n    batch_size=32,\n    seed=42,\n    shuffle=True,\n    class_mode=\"raw\",\n    target_size=(IMG_SIZE,IMG_SIZE))\n    return train_generator, valid_generator\n\n\n\n# Enable tpu if avialable\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)\n\ntrain_generator, val_generator = get_generators()\nwith strategy.scope():\n    model = build_model()\nhistory=model.fit_generator(train_generator,steps_per_epoch=len(train_generator),epochs=10,validation_data=val_generator,validation_steps=len(val_generator), verbose=1)\n</code></pre>\n<p>Error::</p>\n<pre><code>UnavailableError: 6 root error(s) found.\n  (0) Unavailable: {{function_node __inference_train_function_217367}} failed to connect to all addresses\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:\n:{\"created\":\"@1615979512.481645132\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1615979512.481643365\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}\n     [[{{node MultiDeviceIteratorGetNextFromShard}}]]\n     [[RemoteCall]]\n     [[IteratorGetNextAsOptional]]\n     [[cluster_train_function/_execute_7_0/_447]]\n  (1) Unavailable: {{function_node __inference_train_function_217367}} failed to connect to all addresses\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:\n:{\"created\":\"@1615979512.481645132\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1615979512.481643365\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}\n     [[{{node MultiDeviceIteratorGetNextFromShard}}]]\n     [[RemoteCall]]\n     [[IteratorGetNextAsOptional]]\n     [[ConstantFoldingCtrl/cond_4/switch_pred/_38_1/_99]]\n  (2) Unavailable: {{function_node __inference_train_function_217367}} failed to connect to all addresses\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:\n:{\"created\":\"@1615979512.481645132\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1615979512.481643365\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}\n     [[{{node MultiDeviceIteratorGetNextFromShard}}]]\n     [[RemoteCall]]\n     [[IteratorGetNextAsOptional]]\n     [[Pad_4/paddings/_228]]\n  (3) Unavailable: {{function_node __inference_train_function_217367}} failed to connect to all addresses\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:\n:{\"created\":\"@1615979512.481645132\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1615979512.481643365\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}\n     [[{{node MultiDeviceIteratorGetNextFromShard}}]]\n     [[RemoteCall]]\n     [[IteratorGetNextAsOptional]]\n     [[Pad_12/paddings/_236]]\n  (4) Unavaila ... [truncated]\n</code></pre>",
  "messages": [
    {
      "id": "1242059",
      "postDate": "03/17/2021 11:23:36",
      "content": "<p>Hi,</p>\n<p>I am getting this error when I try to train the model with ImageDataGenerator. But how ever if I use GPU I am not facing any such issues. Does anyone has any workaround for this?</p>\n<pre><code>def get_generators():\n    datagen=ImageDataGenerator(rescale = 1./255,\n                                      shear_range = 0.2,\n                                      zoom_range = 0.2,\n                                      horizontal_flip = True)\n\n    test_datagen=ImageDataGenerator(rescale = 1./255,\n                                      shear_range = 0.2,\n                                      zoom_range = 0.2,\n                                      horizontal_flip = True)\n\n    #Train\n    train_generator=datagen.flow_from_dataframe(\n    dataframe=train_data[:15000],\n    directory=train_dir,\n    x_col=\"image\",\n    y_col=list(labels.unique()),\n    batch_size=32,\n    seed=42,\n    shuffle=True,\n    class_mode=\"raw\",\n    target_size=(IMG_SIZE,IMG_SIZE))\n\n    #Valid\n    valid_generator=datagen.flow_from_dataframe(\n    dataframe=train_data[15000:],\n    directory=train_dir,\n    x_col=\"image\",\n    y_col=list(labels.unique()),\n    batch_size=32,\n    seed=42,\n    shuffle=True,\n    class_mode=\"raw\",\n    target_size=(IMG_SIZE,IMG_SIZE))\n    return train_generator, valid_generator\n\n\n\n# Enable tpu if avialable\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)\n\ntrain_generator, val_generator = get_generators()\nwith strategy.scope():\n    model = build_model()\nhistory=model.fit_generator(train_generator,steps_per_epoch=len(train_generator),epochs=10,validation_data=val_generator,validation_steps=len(val_generator), verbose=1)\n</code></pre>\n<p>Error::</p>\n<pre><code>UnavailableError: 6 root error(s) found.\n  (0) Unavailable: {{function_node __inference_train_function_217367}} failed to connect to all addresses\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:\n:{\"created\":\"@1615979512.481645132\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1615979512.481643365\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}\n     [[{{node MultiDeviceIteratorGetNextFromShard}}]]\n     [[RemoteCall]]\n     [[IteratorGetNextAsOptional]]\n     [[cluster_train_function/_execute_7_0/_447]]\n  (1) Unavailable: {{function_node __inference_train_function_217367}} failed to connect to all addresses\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:\n:{\"created\":\"@1615979512.481645132\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1615979512.481643365\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}\n     [[{{node MultiDeviceIteratorGetNextFromShard}}]]\n     [[RemoteCall]]\n     [[IteratorGetNextAsOptional]]\n     [[ConstantFoldingCtrl/cond_4/switch_pred/_38_1/_99]]\n  (2) Unavailable: {{function_node __inference_train_function_217367}} failed to connect to all addresses\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:\n:{\"created\":\"@1615979512.481645132\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1615979512.481643365\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}\n     [[{{node MultiDeviceIteratorGetNextFromShard}}]]\n     [[RemoteCall]]\n     [[IteratorGetNextAsOptional]]\n     [[Pad_4/paddings/_228]]\n  (3) Unavailable: {{function_node __inference_train_function_217367}} failed to connect to all addresses\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:\n:{\"created\":\"@1615979512.481645132\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1615979512.481643365\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}\n     [[{{node MultiDeviceIteratorGetNextFromShard}}]]\n     [[RemoteCall]]\n     [[IteratorGetNextAsOptional]]\n     [[Pad_12/paddings/_236]]\n  (4) Unavaila ... [truncated]\n</code></pre>",
      "rawMarkdown": "Hi,\n\nI am getting this error when I try to train the model with ImageDataGenerator. But how ever if I use GPU I am not facing any such issues. Does anyone has any workaround for this?\n\n```\ndef get_generators():\n    datagen=ImageDataGenerator(rescale = 1./255,\n                                      shear_range = 0.2,\n                                      zoom_range = 0.2,\n                                      horizontal_flip = True)\n\n    test_datagen=ImageDataGenerator(rescale = 1./255,\n                                      shear_range = 0.2,\n                                      zoom_range = 0.2,\n                                      horizontal_flip = True)\n\n    #Train\n    train_generator=datagen.flow_from_dataframe(\n    dataframe=train_data[:15000],\n    directory=train_dir,\n    x_col=\"image\",\n    y_col=list(labels.unique()),\n    batch_size=32,\n    seed=42,\n    shuffle=True,\n    class_mode=\"raw\",\n    target_size=(IMG_SIZE,IMG_SIZE))\n\n    #Valid\n    valid_generator=datagen.flow_from_dataframe(\n    dataframe=train_data[15000:],\n    directory=train_dir,\n    x_col=\"image\",\n    y_col=list(labels.unique()),\n    batch_size=32,\n    seed=42,\n    shuffle=True,\n    class_mode=\"raw\",\n    target_size=(IMG_SIZE,IMG_SIZE))\n    return train_generator, valid_generator\n\n\n\n# Enable tpu if avialable\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)\n\ntrain_generator, val_generator = get_generators()\nwith strategy.scope():\n    model = build_model()\nhistory=model.fit_generator(train_generator,steps_per_epoch=len(train_generator),epochs=10,validation_data=val_generator,validation_steps=len(val_generator), verbose=1)\n```\n\n\nError::\n\n```\nUnavailableError: 6 root error(s) found.\n  (0) Unavailable: {{function_node __inference_train_function_217367}} failed to connect to all addresses\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:\n:{\"created\":\"@1615979512.481645132\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1615979512.481643365\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}\n\t [[{{node MultiDeviceIteratorGetNextFromShard}}]]\n\t [[RemoteCall]]\n\t [[IteratorGetNextAsOptional]]\n\t [[cluster_train_function/_execute_7_0/_447]]\n  (1) Unavailable: {{function_node __inference_train_function_217367}} failed to connect to all addresses\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:\n:{\"created\":\"@1615979512.481645132\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1615979512.481643365\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}\n\t [[{{node MultiDeviceIteratorGetNextFromShard}}]]\n\t [[RemoteCall]]\n\t [[IteratorGetNextAsOptional]]\n\t [[ConstantFoldingCtrl/cond_4/switch_pred/_38_1/_99]]\n  (2) Unavailable: {{function_node __inference_train_function_217367}} failed to connect to all addresses\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:\n:{\"created\":\"@1615979512.481645132\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1615979512.481643365\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}\n\t [[{{node MultiDeviceIteratorGetNextFromShard}}]]\n\t [[RemoteCall]]\n\t [[IteratorGetNextAsOptional]]\n\t [[Pad_4/paddings/_228]]\n  (3) Unavailable: {{function_node __inference_train_function_217367}} failed to connect to all addresses\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:\n:{\"created\":\"@1615979512.481645132\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1615979512.481643365\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}\n\t [[{{node MultiDeviceIteratorGetNextFromShard}}]]\n\t [[RemoteCall]]\n\t [[IteratorGetNextAsOptional]]\n\t [[Pad_12/paddings/_236]]\n  (4) Unavaila ... [truncated]\n```",
      "votes": null
    },
    {
      "id": "1249075",
      "postDate": "03/23/2021 04:47:24",
      "content": "<p>For training with TPU you can refer my <a href=\"https://www.kaggle.com/shanmukh05/plant-pathology-2k21-baseline-tpu-training\" target=\"_blank\">notebook</a> : I think we can't use <code>ImageDatagenerator</code> and <code>flow_from_dataframe</code> with TPU.</p>",
      "rawMarkdown": "For training with TPU you can refer my [notebook](https://www.kaggle.com/shanmukh05/plant-pathology-2k21-baseline-tpu-training) : I think we can't use `ImageDatagenerator` and `flow_from_dataframe` with TPU.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1249075,
      "author_name": "shanmukh05",
      "author_url": "",
      "post_date": "03/23/2021 04:47:24",
      "content": "<p>For training with TPU you can refer my <a href=\"https://www.kaggle.com/shanmukh05/plant-pathology-2k21-baseline-tpu-training\" target=\"_blank\">notebook</a> : I think we can't use <code>ImageDatagenerator</code> and <code>flow_from_dataframe</code> with TPU.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1242059": "Hi,\n\nI am getting this error when I try to train the model with ImageDataGenerator. But how ever if I use GPU I am not facing any such issues. Does anyone has any workaround for this?\n\n```\ndef get_generators():\n    datagen=ImageDataGenerator(rescale = 1./255,\n                                      shear_range = 0.2,\n                                      zoom_range = 0.2,\n                                      horizontal_flip = True)\n\n    test_datagen=ImageDataGenerator(rescale = 1./255,\n                                      shear_range = 0.2,\n                                      zoom_range = 0.2,\n                                      horizontal_flip = True)\n\n    #Train\n    train_generator=datagen.flow_from_dataframe(\n    dataframe=train_data[:15000],\n    directory=train_dir,\n    x_col=\"image\",\n    y_col=list(labels.unique()),\n    batch_size=32,\n    seed=42,\n    shuffle=True,\n    class_mode=\"raw\",\n    target_size=(IMG_SIZE,IMG_SIZE))\n\n    #Valid\n    valid_generator=datagen.flow_from_dataframe(\n    dataframe=train_data[15000:],\n    directory=train_dir,\n    x_col=\"image\",\n    y_col=list(labels.unique()),\n    batch_size=32,\n    seed=42,\n    shuffle=True,\n    class_mode=\"raw\",\n    target_size=(IMG_SIZE,IMG_SIZE))\n    return train_generator, valid_generator\n\n\n\n# Enable tpu if avialable\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)\n\ntrain_generator, val_generator = get_generators()\nwith strategy.scope():\n    model = build_model()\nhistory=model.fit_generator(train_generator,steps_per_epoch=len(train_generator),epochs=10,validation_data=val_generator,validation_steps=len(val_generator), verbose=1)\n```\n\n\nError::\n\n```\nUnavailableError: 6 root error(s) found.\n  (0) Unavailable: {{function_node __inference_train_function_217367}} failed to connect to all addresses\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:\n:{\"created\":\"@1615979512.481645132\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1615979512.481643365\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}\n\t [[{{node MultiDeviceIteratorGetNextFromShard}}]]\n\t [[RemoteCall]]\n\t [[IteratorGetNextAsOptional]]\n\t [[cluster_train_function/_execute_7_0/_447]]\n  (1) Unavailable: {{function_node __inference_train_function_217367}} failed to connect to all addresses\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:\n:{\"created\":\"@1615979512.481645132\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1615979512.481643365\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}\n\t [[{{node MultiDeviceIteratorGetNextFromShard}}]]\n\t [[RemoteCall]]\n\t [[IteratorGetNextAsOptional]]\n\t [[ConstantFoldingCtrl/cond_4/switch_pred/_38_1/_99]]\n  (2) Unavailable: {{function_node __inference_train_function_217367}} failed to connect to all addresses\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:\n:{\"created\":\"@1615979512.481645132\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1615979512.481643365\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}\n\t [[{{node MultiDeviceIteratorGetNextFromShard}}]]\n\t [[RemoteCall]]\n\t [[IteratorGetNextAsOptional]]\n\t [[Pad_4/paddings/_228]]\n  (3) Unavailable: {{function_node __inference_train_function_217367}} failed to connect to all addresses\nAdditional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:\n:{\"created\":\"@1615979512.481645132\",\"description\":\"Failed to pick subchannel\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":4143,\"referenced_errors\":[{\"created\":\"@1615979512.481643365\",\"description\":\"failed to connect to all addresses\",\"file\":\"third_party/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc\",\"file_line\":398,\"grpc_status\":14}]}\n\t [[{{node MultiDeviceIteratorGetNextFromShard}}]]\n\t [[RemoteCall]]\n\t [[IteratorGetNextAsOptional]]\n\t [[Pad_12/paddings/_236]]\n  (4) Unavaila ... [truncated]\n```",
    "1249075": "For training with TPU you can refer my [notebook](https://www.kaggle.com/shanmukh05/plant-pathology-2k21-baseline-tpu-training) : I think we can't use `ImageDatagenerator` and `flow_from_dataframe` with TPU."
  },
  "source": "meta"
}