{
  "id": 154879,
  "title": "Error when using tensorboard with TPU",
  "url": "/competitions/jigsaw-multilingual-toxic-comment-classification/discussion/154879",
  "author_name": "",
  "post_date": "2020-05-30T07:24:48.439732800Z",
  "votes": null,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hi kagglers, </p>\n\n<p>I encountered the following error when trying to use tf.keras.callbacks.TensorBoard when using TPU:</p>\n\n<p>*NotFoundError: Container worker does not exist. (Could not find resource: worker/_AnonymousVar1656)\n    Encountered when executing an operation using EagerExecutor. This error cancels all future operations and poisons their output tensors.*</p>\n\n<p>The same code seems working when I test using CPU.</p>\n\n<p>Has anyone set up tensorboard suceesfully set up tensorboard with TPU?</p>\n\n<p>Thank you!</p>",
  "messages": [
    {
      "id": "867346",
      "postDate": "05/30/2020 07:24:48",
      "content": "<p>Hi kagglers, </p>\n\n<p>I encountered the following error when trying to use tf.keras.callbacks.TensorBoard when using TPU:</p>\n\n<p>*NotFoundError: Container worker does not exist. (Could not find resource: worker/_AnonymousVar1656)\n    Encountered when executing an operation using EagerExecutor. This error cancels all future operations and poisons their output tensors.*</p>\n\n<p>The same code seems working when I test using CPU.</p>\n\n<p>Has anyone set up tensorboard suceesfully set up tensorboard with TPU?</p>\n\n<p>Thank you!</p>",
      "rawMarkdown": "Hi kagglers, \n\nI encountered the following error when trying to use tf.keras.callbacks.TensorBoard when using TPU:\n\n*NotFoundError: Container worker does not exist. (Could not find resource: worker/_AnonymousVar1656)\n\tEncountered when executing an operation using EagerExecutor. This error cancels all future operations and poisons their output tensors.*\n\nThe same code seems working when I test using CPU.\n \nHas anyone set up tensorboard suceesfully set up tensorboard with TPU?\n\nThank you!",
      "votes": null
    },
    {
      "id": "872200",
      "postDate": "06/03/2020 01:50:34",
      "content": "<p>Just out of curiosity, how do you usually use Tensorboard with Kaggle notebooks ?</p>",
      "rawMarkdown": "Just out of curiosity, how do you usually use Tensorboard with Kaggle notebooks ?",
      "votes": null
    },
    {
      "id": "872886",
      "postDate": "06/03/2020 15:25:42",
      "content": "<p>I haven't tried using Tensorboard on Kaggle notebooks before, so just followed notebook: <a href=\"https://www.kaggle.com/aagundez/using-tensorboard-in-kaggle-kernels\">https://www.kaggle.com/aagundez/using-tensorboard-in-kaggle-kernels</a>\nI used:\n%load_ext tensorboard.notebook\nthen use:\ntf.keras.callbacks.TensorBoard during training to write files TensorBoard needs\nThis seems to work when TPU is turned off, but gave me the error above when using TPU.</p>",
      "rawMarkdown": "I haven't tried using Tensorboard on Kaggle notebooks before, so just followed notebook: https://www.kaggle.com/aagundez/using-tensorboard-in-kaggle-kernels\nI used:\n%load_ext tensorboard.notebook\nthen use:\ntf.keras.callbacks.TensorBoard during training to write files TensorBoard needs\nThis seems to work when TPU is turned off, but gave me the error above when using TPU.",
      "votes": null
    },
    {
      "id": "879996",
      "postDate": "06/09/2020 22:33:54",
      "content": "<p>OK, I understand now. Since TPU cannot write their Tensorboard logs back to the VM, this is not expected to work.</p>",
      "rawMarkdown": "OK, I understand now. Since TPU cannot write their Tensorboard logs back to the VM, this is not expected to work.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 872200,
      "author_name": "mgorner",
      "author_url": "",
      "post_date": "06/03/2020 01:50:34",
      "content": "<p>Just out of curiosity, how do you usually use Tensorboard with Kaggle notebooks ?</p>",
      "votes": null,
      "replies": [
        {
          "id": 872886,
          "author_name": "benhzn07",
          "author_url": "",
          "post_date": "06/03/2020 15:25:42",
          "content": "<p>I haven't tried using Tensorboard on Kaggle notebooks before, so just followed notebook: <a href=\"https://www.kaggle.com/aagundez/using-tensorboard-in-kaggle-kernels\">https://www.kaggle.com/aagundez/using-tensorboard-in-kaggle-kernels</a>\nI used:\n%load_ext tensorboard.notebook\nthen use:\ntf.keras.callbacks.TensorBoard during training to write files TensorBoard needs\nThis seems to work when TPU is turned off, but gave me the error above when using TPU.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 879996,
          "author_name": "mgorner",
          "author_url": "",
          "post_date": "06/09/2020 22:33:54",
          "content": "<p>OK, I understand now. Since TPU cannot write their Tensorboard logs back to the VM, this is not expected to work.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "867346": "Hi kagglers, \n\nI encountered the following error when trying to use tf.keras.callbacks.TensorBoard when using TPU:\n\n*NotFoundError: Container worker does not exist. (Could not find resource: worker/_AnonymousVar1656)\n\tEncountered when executing an operation using EagerExecutor. This error cancels all future operations and poisons their output tensors.*\n\nThe same code seems working when I test using CPU.\n \nHas anyone set up tensorboard suceesfully set up tensorboard with TPU?\n\nThank you!",
    "872200": "Just out of curiosity, how do you usually use Tensorboard with Kaggle notebooks ?",
    "872886": "I haven't tried using Tensorboard on Kaggle notebooks before, so just followed notebook: https://www.kaggle.com/aagundez/using-tensorboard-in-kaggle-kernels\nI used:\n%load_ext tensorboard.notebook\nthen use:\ntf.keras.callbacks.TensorBoard during training to write files TensorBoard needs\nThis seems to work when TPU is turned off, but gave me the error above when using TPU.",
    "879996": "OK, I understand now. Since TPU cannot write their Tensorboard logs back to the VM, this is not expected to work."
  },
  "source": "meta"
}