anyone getting this error with tensorflow tpu? :
\n\nAbortedError: RecvTensor expects a different device incarnation: 6679287771369086306 vs. 15380175649607391670. Your worker job (\"/job:tpu_worker/replica:0/task:0\") was probably restarted. Check your worker job for the reason why it was restarted.\nAdditional GRPC error information:\n{\"created\":\"@1588269568.641962411\",\"description\":\"Error received from peer\",\"file\":\"external/grpc/src/core/lib/surface/call.cc\",\"file_line\":1039,\"grpc_message\":\"RecvTensor expects a different device incarnation: 6679287771369086306 vs. 15380175649607391670. Your worker job (\"/job:tpu_worker/replica:0/task:0\") was probably restarted. Check your worker job for the reason why it was restarted.\",\"grpc_status\":10}
\n\ni only get this error with a subset of the data that i created for model training\ni checked carefully,the dataframe has 2 columns comment_text and toxic and not a single row is empty\nall rows contains comment and corresponding label and i don't know why i am getting that error, even i get that error with xhlulu's kernel,i just change his train set with my train set, anyone knows how to solve this problem?\ni love the beautiful <3 tensorflow community <3 \ni asked several questions in that community about problems like this and never got help that's why seeking someone here who can help me with this,thank you
", "messages": [ { "id": 828051, "postDate": "2020-04-30T18:19:22.060Z", "content": "anyone getting this error with tensorflow tpu? :
\n\nAbortedError: RecvTensor expects a different device incarnation: 6679287771369086306 vs. 15380175649607391670. Your worker job (\"/job:tpu_worker/replica:0/task:0\") was probably restarted. Check your worker job for the reason why it was restarted.\nAdditional GRPC error information:\n{\"created\":\"@1588269568.641962411\",\"description\":\"Error received from peer\",\"file\":\"external/grpc/src/core/lib/surface/call.cc\",\"file_line\":1039,\"grpc_message\":\"RecvTensor expects a different device incarnation: 6679287771369086306 vs. 15380175649607391670. Your worker job (\"/job:tpu_worker/replica:0/task:0\") was probably restarted. Check your worker job for the reason why it was restarted.\",\"grpc_status\":10}
\n\ni only get this error with a subset of the data that i created for model training\ni checked carefully,the dataframe has 2 columns comment_text and toxic and not a single row is empty\nall rows contains comment and corresponding label and i don't know why i am getting that error, even i get that error with xhlulu's kernel,i just change his train set with my train set, anyone knows how to solve this problem?\ni love the beautiful <3 tensorflow community <3 \ni asked several questions in that community about problems like this and never got help that's why seeking someone here who can help me with this,thank you
", "rawMarkdown": "anyone getting this error with tensorflow tpu? : \n\n**AbortedError: RecvTensor expects a different device incarnation: 6679287771369086306 vs. 15380175649607391670. Your worker job (\"/job:tpu_worker/replica:0/task:0\") was probably restarted. Check your worker job for the reason why it was restarted.\nAdditional GRPC error information:\n{\"created\":\"@1588269568.641962411\",\"description\":\"Error received from peer\",\"file\":\"external/grpc/src/core/lib/surface/call.cc\",\"file_line\":1039,\"grpc_message\":\"RecvTensor expects a different device incarnation: 6679287771369086306 vs. 15380175649607391670. Your worker job (\"/job:tpu_worker/replica:0/task:0\") was probably restarted. Check your worker job for the reason why it was restarted.\",\"grpc_status\":10}**\n\ni only get this error with a subset of the data that i created for model training\ni checked carefully,the dataframe has 2 columns comment_text and toxic and not a single row is empty\nall rows contains comment and corresponding label and i don't know why i am getting that error, even i get that error with xhlulu's kernel,i just change his train set with my train set, anyone knows how to solve this problem?\ni love the beautiful <3 tensorflow community <3 \ni asked several questions in that community about problems like this and never got help that's why seeking someone here who can help me with this,thank you", "votes": 5 }, { "id": 830064, "postDate": "2020-05-02T09:37:38.100Z", "rawMarkdown": "", "votes": -5, "isDeleted": true } ], "comments": [ { "id": 830064, "author_name": "", "author_url": "", "post_date": "2020-05-02T09:37:38.100000", "content": "", "votes": -5, "replies": [] } ], "raw_markdown_by_id": { "828051": "anyone getting this error with tensorflow tpu? : \n\n**AbortedError: RecvTensor expects a different device incarnation: 6679287771369086306 vs. 15380175649607391670. Your worker job (\"/job:tpu_worker/replica:0/task:0\") was probably restarted. Check your worker job for the reason why it was restarted.\nAdditional GRPC error information:\n{\"created\":\"@1588269568.641962411\",\"description\":\"Error received from peer\",\"file\":\"external/grpc/src/core/lib/surface/call.cc\",\"file_line\":1039,\"grpc_message\":\"RecvTensor expects a different device incarnation: 6679287771369086306 vs. 15380175649607391670. Your worker job (\"/job:tpu_worker/replica:0/task:0\") was probably restarted. Check your worker job for the reason why it was restarted.\",\"grpc_status\":10}**\n\ni only get this error with a subset of the data that i created for model training\ni checked carefully,the dataframe has 2 columns comment_text and toxic and not a single row is empty\nall rows contains comment and corresponding label and i don't know why i am getting that error, even i get that error with xhlulu's kernel,i just change his train set with my train set, anyone knows how to solve this problem?\ni love the beautiful <3 tensorflow community <3 \ni asked several questions in that community about problems like this and never got help that's why seeking someone here who can help me with this,thank you", "830064": "" } }