While calling model.fit(tf_dataset,...) I'm getting the following error:
UnavailableError: 2 root error(s) found.\n (0) Unavailable: Socket closed\n (1) Unavailable: Unable to find a context_id matching the specified one (6747970432017632323). Perhaps the worker was restarted, or the context was GC'd?\n0 successful operations.\n0 derived errors ignored.\nEach TfRecord is only 4 MB size and represents the tf.float32 matrix of [500,2000] size. The case is very hard to debug and playing with batch size, parallel reads, and data type (float64, int32, etc) doesn't help.
\nDecreasing the matrix size to [1,2000] didn't help either.
Your hints are appreciated.
", "messages": [ { "id": "1472611", "postDate": "08/15/2021 02:19:22", "content": "While calling model.fit(tf_dataset,...) I'm getting the following error:
UnavailableError: 2 root error(s) found.\n (0) Unavailable: Socket closed\n (1) Unavailable: Unable to find a context_id matching the specified one (6747970432017632323). Perhaps the worker was restarted, or the context was GC'd?\n0 successful operations.\n0 derived errors ignored.\nEach TfRecord is only 4 MB size and represents the tf.float32 matrix of [500,2000] size. The case is very hard to debug and playing with batch size, parallel reads, and data type (float64, int32, etc) doesn't help.
\nDecreasing the matrix size to [1,2000] didn't help either.
Your hints are appreciated.
", "rawMarkdown": "While calling `model.fit(tf_dataset,...)` I'm getting the following error:\n\n```\nUnavailableError: 2 root error(s) found.\n (0) Unavailable: Socket closed\n (1) Unavailable: Unable to find a context_id matching the specified one (6747970432017632323). Perhaps the worker was restarted, or the context was GC'd?\n0 successful operations.\n0 derived errors ignored.\n```\n\nEach TfRecord is only 4 MB size and represents the **tf.float32** matrix of **[500,2000]** size. The case is very hard to debug and playing with batch size, parallel reads, and data type (float64, int32, etc) doesn't help.\nDecreasing the matrix size to [1,2000] didn't help either.\n\nYour hints are appreciated.", "votes": null }, { "id": "1472660", "postDate": "08/15/2021 03:44:51", "content": "Curious, what happens when you run it with GPU instead of TPU as your notebook backend?
\nhttps://github.com/tensorflow/tensorflow/issues/36136#issuecomment-590090660
\nIn this Github thread people are saying this is most likely something to do with your host call method. Try testing each of your functions separately to see if there are any issues.
\nIf possible, you could provide some more specific code so we can help debug your issue?
", "rawMarkdown": "Curious, what happens when you run it with GPU instead of TPU as your notebook backend?\n\nhttps://github.com/tensorflow/tensorflow/issues/36136#issuecomment-590090660\n\nIn this Github thread people are saying this is most likely something to do with your host call method. Try testing each of your functions separately to see if there are any issues.\n\nIf possible, you could provide some more specific code so we can help debug your issue?", "votes": null }, { "id": "1473233", "postDate": "08/15/2021 12:54:19", "content": "@d223chen with GPU it worked. If you want, I can share the code with you.
", "rawMarkdown": "d223chen with GPU it worked. If you want, I can share the code with you.", "votes": null }, { "id": "1473241", "postDate": "08/15/2021 12:58:04", "content": "Sure, maybe if you can add to the post
", "rawMarkdown": "Sure, maybe if you can add to the post", "votes": null } ], "comments": [ { "id": 1472660, "author_name": "d223chen", "author_url": "", "post_date": "08/15/2021 03:44:51", "content": "Curious, what happens when you run it with GPU instead of TPU as your notebook backend?
\nhttps://github.com/tensorflow/tensorflow/issues/36136#issuecomment-590090660
\nIn this Github thread people are saying this is most likely something to do with your host call method. Try testing each of your functions separately to see if there are any issues.
\nIf possible, you could provide some more specific code so we can help debug your issue?
", "votes": null, "replies": [] }, { "id": 1473233, "author_name": "jhasanov", "author_url": "", "post_date": "08/15/2021 12:54:19", "content": "@d223chen with GPU it worked. If you want, I can share the code with you.
", "votes": null, "replies": [ { "id": 1473241, "author_name": "d223chen", "author_url": "", "post_date": "08/15/2021 12:58:04", "content": "Sure, maybe if you can add to the post
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1472611": "While calling `model.fit(tf_dataset,...)` I'm getting the following error:\n\n```\nUnavailableError: 2 root error(s) found.\n (0) Unavailable: Socket closed\n (1) Unavailable: Unable to find a context_id matching the specified one (6747970432017632323). Perhaps the worker was restarted, or the context was GC'd?\n0 successful operations.\n0 derived errors ignored.\n```\n\nEach TfRecord is only 4 MB size and represents the **tf.float32** matrix of **[500,2000]** size. The case is very hard to debug and playing with batch size, parallel reads, and data type (float64, int32, etc) doesn't help.\nDecreasing the matrix size to [1,2000] didn't help either.\n\nYour hints are appreciated.", "1472660": "Curious, what happens when you run it with GPU instead of TPU as your notebook backend?\n\nhttps://github.com/tensorflow/tensorflow/issues/36136#issuecomment-590090660\n\nIn this Github thread people are saying this is most likely something to do with your host call method. Try testing each of your functions separately to see if there are any issues.\n\nIf possible, you could provide some more specific code so we can help debug your issue?", "1473233": "d223chen with GPU it worked. If you want, I can share the code with you.", "1473241": "Sure, maybe if you can add to the post" }, "source": "meta" }