{
  "id": 162341,
  "title": "Colab TPU Error: UnavailableError: Socket closed",
  "url": "/competitions/alaska2-image-steganalysis/discussion/162341",
  "author_name": "",
  "post_date": "2020-06-28T14:55:13.776682800Z",
  "votes": 5,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I got this error while training with TPU:</p>\n\n<p><code>\nUnavailableError: Socket closed\nAdditional GRPC error information:\n{\"created\":\"@1593347904.565869993\",\"description\":\"Error received from peer ipv4:10.26.3.74:8470\",\"file\":\"external/com_github_grpc_grpc/src/core/lib/surface/call.cc\",\"file_line\":1056,\"grpc_message\":\"Socket closed\",\"grpc_status\":14}\n</code></p>\n\n<p>Since the error occurs after about 40 minutes of training, I suspect the problem stems from using .cache().</p>\n\n<p>How did you guys solve the problem? Did removing .cache() help? Will that make the training much slower?</p>\n\n<p>Thanks!</p>",
  "messages": [
    {
      "id": "905443",
      "postDate": "06/28/2020 14:55:13",
      "content": "<p>I got this error while training with TPU:</p>\n\n<p><code>\nUnavailableError: Socket closed\nAdditional GRPC error information:\n{\"created\":\"@1593347904.565869993\",\"description\":\"Error received from peer ipv4:10.26.3.74:8470\",\"file\":\"external/com_github_grpc_grpc/src/core/lib/surface/call.cc\",\"file_line\":1056,\"grpc_message\":\"Socket closed\",\"grpc_status\":14}\n</code></p>\n\n<p>Since the error occurs after about 40 minutes of training, I suspect the problem stems from using .cache().</p>\n\n<p>How did you guys solve the problem? Did removing .cache() help? Will that make the training much slower?</p>\n\n<p>Thanks!</p>",
      "rawMarkdown": "I got this error while training with TPU:\n\n```\nUnavailableError: Socket closed\nAdditional GRPC error information:\n{\"created\":\"@1593347904.565869993\",\"description\":\"Error received from peer ipv4:10.26.3.74:8470\",\"file\":\"external/com_github_grpc_grpc/src/core/lib/surface/call.cc\",\"file_line\":1056,\"grpc_message\":\"Socket closed\",\"grpc_status\":14}\n```\n\nSince the error occurs after about 40 minutes of training, I suspect the problem stems from using .cache().\n\nHow did you guys solve the problem? Did removing .cache() help? Will that make the training much slower?\n\nThanks!",
      "votes": null
    },
    {
      "id": "906023",
      "postDate": "06/29/2020 03:19:00",
      "content": "<p>Removing .cache() indeed solved the problem and slowed the training down radically. If you are using a TPU, how did you solve this problem? Did you just use incomplete dataset?</p>",
      "rawMarkdown": "Removing .cache() indeed solved the problem and slowed the training down radically. If you are using a TPU, how did you solve this problem? Did you just use incomplete dataset?",
      "votes": null
    },
    {
      "id": "913888",
      "postDate": "07/03/2020 13:57:16",
      "content": "<p>Same Problem..\nIs this problem solved??\nThank you</p>",
      "rawMarkdown": "Same Problem..\nIs this problem solved??\nThank you",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 906023,
      "author_name": "tonychenxyz",
      "author_url": "",
      "post_date": "06/29/2020 03:19:00",
      "content": "<p>Removing .cache() indeed solved the problem and slowed the training down radically. If you are using a TPU, how did you solve this problem? Did you just use incomplete dataset?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 913888,
      "author_name": "gyanendradas",
      "author_url": "",
      "post_date": "07/03/2020 13:57:16",
      "content": "<p>Same Problem..\nIs this problem solved??\nThank you</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "905443": "I got this error while training with TPU:\n\n```\nUnavailableError: Socket closed\nAdditional GRPC error information:\n{\"created\":\"@1593347904.565869993\",\"description\":\"Error received from peer ipv4:10.26.3.74:8470\",\"file\":\"external/com_github_grpc_grpc/src/core/lib/surface/call.cc\",\"file_line\":1056,\"grpc_message\":\"Socket closed\",\"grpc_status\":14}\n```\n\nSince the error occurs after about 40 minutes of training, I suspect the problem stems from using .cache().\n\nHow did you guys solve the problem? Did removing .cache() help? Will that make the training much slower?\n\nThanks!",
    "906023": "Removing .cache() indeed solved the problem and slowed the training down radically. If you are using a TPU, how did you solve this problem? Did you just use incomplete dataset?",
    "913888": "Same Problem..\nIs this problem solved??\nThank you"
  },
  "source": "meta"
}