{
  "id": 145948,
  "title": "\"UnavailableError: Socket closed\"",
  "url": "/competitions/jigsaw-multilingual-toxic-comment-classification/discussion/145948",
  "author_name": "",
  "post_date": "2020-04-25T07:34:24.793905800Z",
  "votes": 2,
  "comment_count": 9,
  "views": 0,
  "content": "<blockquote>\n  <p>/opt/conda/lib/python3.6/site-packages/six.py in raise_from(value, from_value)\n  UnavailableError: Socket closed</p>\n</blockquote>\n\n<p>I get this error when i do forward pass. This issue just came yesterday, everything was working great 2 days ago. \n<strong>Details</strong>: I'm using tensorflow 2.1, i have ~12 minutes data preprocessing before i do anything with the model (in case that's a timeout issue), my model has ~550m. parameters, it gets raised on both commit and edit modes. \nIs anyone else having the same problem? What do you think could cause it? </p>",
  "messages": [
    {
      "id": "820163",
      "postDate": "04/25/2020 07:34:24",
      "content": "<blockquote>\n  <p>/opt/conda/lib/python3.6/site-packages/six.py in raise_from(value, from_value)\n  UnavailableError: Socket closed</p>\n</blockquote>\n\n<p>I get this error when i do forward pass. This issue just came yesterday, everything was working great 2 days ago. \n<strong>Details</strong>: I'm using tensorflow 2.1, i have ~12 minutes data preprocessing before i do anything with the model (in case that's a timeout issue), my model has ~550m. parameters, it gets raised on both commit and edit modes. \nIs anyone else having the same problem? What do you think could cause it? </p>",
      "rawMarkdown": "&gt; /opt/conda/lib/python3.6/site-packages/six.py in raise_from(value, from_value)\nUnavailableError: Socket closed\n\nI get this error when i do forward pass. This issue just came yesterday, everything was working great 2 days ago. \n**Details**: I'm using tensorflow 2.1, i have ~12 minutes data preprocessing before i do anything with the model (in case that's a timeout issue), my model has ~550m. parameters, it gets raised on both commit and edit modes. \nIs anyone else having the same problem? What do you think could cause it?",
      "votes": null
    },
    {
      "id": "820224",
      "postDate": "04/25/2020 08:50:32",
      "content": "<p>I have the same problem. It has started ~2 days ago too. I don't have any informations to fix it currently</p>",
      "rawMarkdown": "I have the same problem. It has started ~2 days ago too. I don't have any informations to fix it currently",
      "votes": null
    },
    {
      "id": "820331",
      "postDate": "04/25/2020 10:49:32",
      "content": "<p>Check this thread's comments.\n<a href=\"https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/139586\">https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/139586</a></p>",
      "rawMarkdown": "Check this thread's comments.\nhttps://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/139586",
      "votes": null
    },
    {
      "id": "820419",
      "postDate": "04/25/2020 12:32:29",
      "content": "<p>Hi, thanks a lot, this thread really helped me. The issue for me was that i used float labels with binary cross entropy loss, adding <code>labels = labels.round().astype(int)</code> solved the issue. Apparently it seems like calculating negative log likelihood loss on TPU with float targets isn't supported on tensorflow 2.1. I tried using the nightly version of tensorflow, but it produced even more weirder errors. I guess i'll treat that as a constraint. :)</p>",
      "rawMarkdown": "Hi, thanks a lot, this thread really helped me. The issue for me was that i used float labels with binary cross entropy loss, adding `labels = labels.round().astype(int)` solved the issue. Apparently it seems like calculating negative log likelihood loss on TPU with float targets isn't supported on tensorflow 2.1. I tried using the nightly version of tensorflow, but it produced even more weirder errors. I guess i'll treat that as a constraint. :)",
      "votes": null
    },
    {
      "id": "829725",
      "postDate": "05/02/2020 03:48:00",
      "content": "<p>can you try explicitly use float32 ? perhaps the problem is on float64 only</p>",
      "rawMarkdown": "can you try explicitly use float32 ? perhaps the problem is on float64 only",
      "votes": null
    },
    {
      "id": "834820",
      "postDate": "05/05/2020 19:47:00",
      "content": "<p>That might be it. TPU support only 3 number formats: float32, int64, bfloat16</p>",
      "rawMarkdown": "That might be it. TPU support only 3 number formats: float32, int64, bfloat16",
      "votes": null
    },
    {
      "id": "890943",
      "postDate": "06/17/2020 19:28:30",
      "content": "<p>I have this problem as well. I have <code>x_train</code>, <code>y_train</code>, <code>x_val</code>and <code>y_val</code>set to <code>np.int64</code>, but it doesn't help. Preprocessing in my case takes ~5 min. What can be done to fix that?</p>",
      "rawMarkdown": "I have this problem as well. I have `x_train`, `y_train`, `x_val `and `y_val `set to `np.int64`, but it doesn't help. Preprocessing in my case takes ~5 min. What can be done to fix that?",
      "votes": null
    },
    {
      "id": "890948",
      "postDate": "06/17/2020 19:31:18",
      "content": "<p>My error message is <code>Socket closed\nAdditional GRPC error information:\n{\"created\":\"@1592422022.678307948\",\"description\":\"Error received from peer ipv4:10.0.0.2:8470\",\"file\":\"external/com_github_grpc_grpc/src/core/lib/surface/call.cc\",\"file_line\":1056,\"grpc_message\":\"Socket closed\",\"grpc_status\":14}</code></p>",
      "rawMarkdown": "My error message is `Socket closed\nAdditional GRPC error information:\n{\"created\":\"@1592422022.678307948\",\"description\":\"Error received from peer ipv4:10.0.0.2:8470\",\"file\":\"external/com_github_grpc_grpc/src/core/lib/surface/call.cc\",\"file_line\":1056,\"grpc_message\":\"Socket closed\",\"grpc_status\":14}`",
      "votes": null
    },
    {
      "id": "891546",
      "postDate": "06/18/2020 08:47:40",
      "content": "<p>Tried today, and got a little different error message:\n<code>/opt/conda/lib/python3.7/site-packages/six.py in raise_from(value, from_value)</code>\n<code>UnavailableError: Socket closed</code></p>",
      "rawMarkdown": "Tried today, and got a little different error message:\n`/opt/conda/lib/python3.7/site-packages/six.py in raise_from(value, from_value)`\n`UnavailableError: Socket closed`",
      "votes": null
    },
    {
      "id": "891585",
      "postDate": "06/18/2020 09:24:44",
      "content": "<p>I found a reason of this error. It's usage of a custom <code>Attention</code> tensorflow layer from <a href=\"https://www.kaggle.com/rftexas/gru-lstm-rnn-101\">this</a> notebook.</p>",
      "rawMarkdown": "I found a reason of this error. It's usage of a custom `Attention` tensorflow layer from [this](https://www.kaggle.com/rftexas/gru-lstm-rnn-101) notebook.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 820224,
      "author_name": "moutroll",
      "author_url": "",
      "post_date": "04/25/2020 08:50:32",
      "content": "<p>I have the same problem. It has started ~2 days ago too. I don't have any informations to fix it currently</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 820331,
      "author_name": "ipythonx",
      "author_url": "",
      "post_date": "04/25/2020 10:49:32",
      "content": "<p>Check this thread's comments.\n<a href=\"https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/139586\">https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/139586</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 820419,
          "author_name": "theonekeyg",
          "author_url": "",
          "post_date": "04/25/2020 12:32:29",
          "content": "<p>Hi, thanks a lot, this thread really helped me. The issue for me was that i used float labels with binary cross entropy loss, adding <code>labels = labels.round().astype(int)</code> solved the issue. Apparently it seems like calculating negative log likelihood loss on TPU with float targets isn't supported on tensorflow 2.1. I tried using the nightly version of tensorflow, but it produced even more weirder errors. I guess i'll treat that as a constraint. :)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 829725,
          "author_name": "ratthachat",
          "author_url": "",
          "post_date": "05/02/2020 03:48:00",
          "content": "<p>can you try explicitly use float32 ? perhaps the problem is on float64 only</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 834820,
          "author_name": "mgorner",
          "author_url": "",
          "post_date": "05/05/2020 19:47:00",
          "content": "<p>That might be it. TPU support only 3 number formats: float32, int64, bfloat16</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 890943,
      "author_name": "atamazian",
      "author_url": "",
      "post_date": "06/17/2020 19:28:30",
      "content": "<p>I have this problem as well. I have <code>x_train</code>, <code>y_train</code>, <code>x_val</code>and <code>y_val</code>set to <code>np.int64</code>, but it doesn't help. Preprocessing in my case takes ~5 min. What can be done to fix that?</p>",
      "votes": null,
      "replies": [
        {
          "id": 890948,
          "author_name": "atamazian",
          "author_url": "",
          "post_date": "06/17/2020 19:31:18",
          "content": "<p>My error message is <code>Socket closed\nAdditional GRPC error information:\n{\"created\":\"@1592422022.678307948\",\"description\":\"Error received from peer ipv4:10.0.0.2:8470\",\"file\":\"external/com_github_grpc_grpc/src/core/lib/surface/call.cc\",\"file_line\":1056,\"grpc_message\":\"Socket closed\",\"grpc_status\":14}</code></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 891546,
          "author_name": "atamazian",
          "author_url": "",
          "post_date": "06/18/2020 08:47:40",
          "content": "<p>Tried today, and got a little different error message:\n<code>/opt/conda/lib/python3.7/site-packages/six.py in raise_from(value, from_value)</code>\n<code>UnavailableError: Socket closed</code></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 891585,
          "author_name": "atamazian",
          "author_url": "",
          "post_date": "06/18/2020 09:24:44",
          "content": "<p>I found a reason of this error. It's usage of a custom <code>Attention</code> tensorflow layer from <a href=\"https://www.kaggle.com/rftexas/gru-lstm-rnn-101\">this</a> notebook.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "820163": "&gt; /opt/conda/lib/python3.6/site-packages/six.py in raise_from(value, from_value)\nUnavailableError: Socket closed\n\nI get this error when i do forward pass. This issue just came yesterday, everything was working great 2 days ago. \n**Details**: I'm using tensorflow 2.1, i have ~12 minutes data preprocessing before i do anything with the model (in case that's a timeout issue), my model has ~550m. parameters, it gets raised on both commit and edit modes. \nIs anyone else having the same problem? What do you think could cause it?",
    "820224": "I have the same problem. It has started ~2 days ago too. I don't have any informations to fix it currently",
    "820331": "Check this thread's comments.\nhttps://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/139586",
    "820419": "Hi, thanks a lot, this thread really helped me. The issue for me was that i used float labels with binary cross entropy loss, adding `labels = labels.round().astype(int)` solved the issue. Apparently it seems like calculating negative log likelihood loss on TPU with float targets isn't supported on tensorflow 2.1. I tried using the nightly version of tensorflow, but it produced even more weirder errors. I guess i'll treat that as a constraint. :)",
    "829725": "can you try explicitly use float32 ? perhaps the problem is on float64 only",
    "834820": "That might be it. TPU support only 3 number formats: float32, int64, bfloat16",
    "890943": "I have this problem as well. I have `x_train`, `y_train`, `x_val `and `y_val `set to `np.int64`, but it doesn't help. Preprocessing in my case takes ~5 min. What can be done to fix that?",
    "890948": "My error message is `Socket closed\nAdditional GRPC error information:\n{\"created\":\"@1592422022.678307948\",\"description\":\"Error received from peer ipv4:10.0.0.2:8470\",\"file\":\"external/com_github_grpc_grpc/src/core/lib/surface/call.cc\",\"file_line\":1056,\"grpc_message\":\"Socket closed\",\"grpc_status\":14}`",
    "891546": "Tried today, and got a little different error message:\n`/opt/conda/lib/python3.7/site-packages/six.py in raise_from(value, from_value)`\n`UnavailableError: Socket closed`",
    "891585": "I found a reason of this error. It's usage of a custom `Attention` tensorflow layer from [this](https://www.kaggle.com/rftexas/gru-lstm-rnn-101) notebook."
  },
  "source": "meta"
}