{
  "id": 144836,
  "title": "Committing Notebook taking more than 1 hour",
  "url": "/competitions/flower-classification-with-tpus/discussion/144836",
  "author_name": "",
  "post_date": "2020-04-20T18:56:08.846228400Z",
  "votes": 2,
  "comment_count": 3,
  "views": 0,
  "content": "<p>@martingorner \nHi, i am trying to commit my notebook, but since last one hour same message showing GCE check failed is being displayed. I left my notebook as it is. Next day i saw commit operation was failed.\nPlease help me as i am unable to commit my version. </p>",
  "messages": [
    {
      "id": "814554",
      "postDate": "04/20/2020 18:56:08",
      "content": "<p>@martingorner \nHi, i am trying to commit my notebook, but since last one hour same message showing GCE check failed is being displayed. I left my notebook as it is. Next day i saw commit operation was failed.\nPlease help me as i am unable to commit my version. </p>",
      "rawMarkdown": "martingorner \nHi, i am trying to commit my notebook, but since last one hour same message showing GCE check failed is being displayed. I left my notebook as it is. Next day i saw commit operation was failed.\nPlease help me as i am unable to commit my version.",
      "votes": null
    },
    {
      "id": "815365",
      "postDate": "04/21/2020 13:37:55",
      "content": "<p>It failed with following log messages. At time 10166 error is coming but i have no idea about it.  Please guide me <a href=\"/martingorner\">@martingorner</a> </p>\n\n<p>Time\nLine #\nLog Message\n3.0s\n1\n[NbConvertApp] Converting notebook <strong>notebook</strong>.ipynb to notebook\n6.1s\n2\n[NbConvertApp] Executing notebook with kernel: python3\n14.3s\n3\n2020-04-20 17:18:06.289469: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX512F\n14.3s\n4\n2020-04-20 17:18:06.314414: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2000179999 Hz\n14.3s\n5\n2020-04-20 17:18:06.315135: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x558c1cfbef00 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n14.3s\n6\n2020-04-20 17:18:06.315181: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version\n14.3s\n7\n2020-04-20 17:18:06.344279: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:300] Initialize GrpcChannelCache for job worker -&gt; {0 -&gt; 10.0.0.2:8470}\n14.3s\n8\n2020-04-20 17:18:06.344339: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:300] Initialize GrpcChannelCache for job localhost -&gt; {0 -&gt; localhost:30013}\n14.3s\n9\n2020-04-20 17:18:06.359935: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:300] Initialize GrpcChannelCache for job worker -&gt; {0 -&gt; 10.0.0.2:8470}\n14.3s\n10\n2020-04-20 17:18:06.359995: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:300] Initialize GrpcChannelCache for job localhost -&gt; {0 -&gt; localhost:30013}\n14.3s\n11\n2020-04-20 17:18:06.366237: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:390] Started server with target: grpc://localhost:30013\n21.6s\n12\n2020-04-20 17:18:13.653909: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n21.7s\n13\n2020-04-20 17:18:13.704236: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n21.7s\n14\n2020-04-20 17:18:13.741587: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n10166.2s\n15\n<strong>*</strong>*2020-04-20 20:07:18.218870: W tensorflow/core/distributed_runtime/eager/remote_tensor_handle_data.cc:75] Unable to destroy remote tensor handles. If you are running a tf.function, it usually indicates some op in the graph gets an error: RecvTensor expects a different device incarnation: 14797989987728605988 vs. 1799738036951955635. Your worker job (\"/job:tpu_worker/replica:0/task:0\") was probably restarted. Check your worker job for the reason why it was restarted.\n10166.2s\n16\nAdditional GRPC error information:\n10166.2s\n17\n{\"created\":\"@1587413238.218478572\",\"description\":\"Error received from peer\",\"file\":\"external/grpc/src/core/lib/surface/call.cc\",\"file_line\":1039,\"grpc_message\":\"RecvTensor expects a different device incarnation: 14797989987728605988 vs. 1799738036951955635. Your worker job (\"/job:tpu_worker/replica:0/task:0\") was probably restarted. Check your worker job for the reason why it was restarted.\",\"grpc_status\":10}\n10169.6s*<strong>*</strong>\n18\n2020-04-20 20:07:21.664715: W ./tensorflow/core/distributed_runtime/eager/destroy_tensor_handle_node.h:79] Ignoring an error encountered when deleting remote tensors handles: Invalid argument: Unable to find a context_id matching the specified one (12193600234898793608). Perhaps the worker was restarted, or the context was GC'd?\n10169.6s\n19\nAdditional GRPC error information:\n10169.6s\n20\n{\"created\":\"@1587413241.664555067\",\"description\":\"Error received from peer\",\"file\":\"external/grpc/src/core/lib/surface/call.cc\",\"file_line\":1039,\"grpc_message\":\"Unable to find a context_id matching the specified one (12193600234898793608). Perhaps the worker was restarted, or the context was GC'd?\",\"grpc_status\":3}\n10180.0s\n21\n[NbConvertApp] Writing 158697 bytes to <strong>notebook</strong>.ipynb\n10180.9s\n22\n[NbConvertApp] Converting notebook <strong>notebook</strong>.ipynb to html\n10182.4s\n23\n[NbConvertApp] Writing 508223 bytes to <strong>results</strong>.html\n10182.4s\n24\n10182.4s\n26\nComplete. Exited with code 0.</p>",
      "rawMarkdown": "It failed with following log messages. At time 10166 error is coming but i have no idea about it.  Please guide me @martingorner \n\nTime\nLine #\nLog Message\n3.0s\n1\n[NbConvertApp] Converting notebook __notebook__.ipynb to notebook\n6.1s\n2\n[NbConvertApp] Executing notebook with kernel: python3\n14.3s\n3\n2020-04-20 17:18:06.289469: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX512F\n14.3s\n4\n2020-04-20 17:18:06.314414: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2000179999 Hz\n14.3s\n5\n2020-04-20 17:18:06.315135: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x558c1cfbef00 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n14.3s\n6\n2020-04-20 17:18:06.315181: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version\n14.3s\n7\n2020-04-20 17:18:06.344279: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:300] Initialize GrpcChannelCache for job worker -&gt; {0 -&gt; 10.0.0.2:8470}\n14.3s\n8\n2020-04-20 17:18:06.344339: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:300] Initialize GrpcChannelCache for job localhost -&gt; {0 -&gt; localhost:30013}\n14.3s\n9\n2020-04-20 17:18:06.359935: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:300] Initialize GrpcChannelCache for job worker -&gt; {0 -&gt; 10.0.0.2:8470}\n14.3s\n10\n2020-04-20 17:18:06.359995: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:300] Initialize GrpcChannelCache for job localhost -&gt; {0 -&gt; localhost:30013}\n14.3s\n11\n2020-04-20 17:18:06.366237: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:390] Started server with target: grpc://localhost:30013\n21.6s\n12\n2020-04-20 17:18:13.653909: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n21.7s\n13\n2020-04-20 17:18:13.704236: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n21.7s\n14\n2020-04-20 17:18:13.741587: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n10166.2s\n15\n******2020-04-20 20:07:18.218870: W tensorflow/core/distributed_runtime/eager/remote_tensor_handle_data.cc:75] Unable to destroy remote tensor handles. If you are running a tf.function, it usually indicates some op in the graph gets an error: RecvTensor expects a different device incarnation: 14797989987728605988 vs. 1799738036951955635. Your worker job (\"/job:tpu_worker/replica:0/task:0\") was probably restarted. Check your worker job for the reason why it was restarted.\n10166.2s\n16\nAdditional GRPC error information:\n10166.2s\n17\n{\"created\":\"@1587413238.218478572\",\"description\":\"Error received from peer\",\"file\":\"external/grpc/src/core/lib/surface/call.cc\",\"file_line\":1039,\"grpc_message\":\"RecvTensor expects a different device incarnation: 14797989987728605988 vs. 1799738036951955635. Your worker job (\"/job:tpu_worker/replica:0/task:0\") was probably restarted. Check your worker job for the reason why it was restarted.\",\"grpc_status\":10}\n10169.6s******\n18\n2020-04-20 20:07:21.664715: W ./tensorflow/core/distributed_runtime/eager/destroy_tensor_handle_node.h:79] Ignoring an error encountered when deleting remote tensors handles: Invalid argument: Unable to find a context_id matching the specified one (12193600234898793608). Perhaps the worker was restarted, or the context was GC'd?\n10169.6s\n19\nAdditional GRPC error information:\n10169.6s\n20\n{\"created\":\"@1587413241.664555067\",\"description\":\"Error received from peer\",\"file\":\"external/grpc/src/core/lib/surface/call.cc\",\"file_line\":1039,\"grpc_message\":\"Unable to find a context_id matching the specified one (12193600234898793608). Perhaps the worker was restarted, or the context was GC'd?\",\"grpc_status\":3}\n10180.0s\n21\n[NbConvertApp] Writing 158697 bytes to __notebook__.ipynb\n10180.9s\n22\n[NbConvertApp] Converting notebook __notebook__.ipynb to html\n10182.4s\n23\n[NbConvertApp] Writing 508223 bytes to __results__.html\n10182.4s\n24\n10182.4s\n26\nComplete. Exited with code 0.",
      "votes": null
    },
    {
      "id": "867308",
      "postDate": "05/30/2020 06:48:31",
      "content": "<p>Hi! <a href=\"/pawanverma\">@pawanverma</a>  I'm having the same problem. Did you find any solution?\nIn my case, the bucket can be accessed publicly without authentication. I don't understand why TensorFlow is expecting an authentication token when I commit the notebook.</p>\n\n<p><code>\n2020-05-30 03:56:35.631435: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n56.9s\n</code></p>",
      "rawMarkdown": "Hi! @pawanverma  I'm having the same problem. Did you find any solution?\nIn my case, the bucket can be accessed publicly without authentication. I don't understand why TensorFlow is expecting an authentication token when I commit the notebook.\n\n\n```\n2020-05-30 03:56:35.631435: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n56.9s\n```",
      "votes": null
    },
    {
      "id": "867344",
      "postDate": "05/30/2020 07:22:24",
      "content": "<p>Hi, as per Kaggle team this is a harmless message and will b removed in future versions.  However,  the time taken to commit the notebook depends on the model training as well and for this notebook with two models ensemble it took around 2 hrs to commit. </p>",
      "rawMarkdown": "Hi, as per Kaggle team this is a harmless message and will b removed in future versions.  However,  the time taken to commit the notebook depends on the model training as well and for this notebook with two models ensemble it took around 2 hrs to commit.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 815365,
      "author_name": "pawanverma",
      "author_url": "",
      "post_date": "04/21/2020 13:37:55",
      "content": "<p>It failed with following log messages. At time 10166 error is coming but i have no idea about it.  Please guide me <a href=\"/martingorner\">@martingorner</a> </p>\n\n<p>Time\nLine #\nLog Message\n3.0s\n1\n[NbConvertApp] Converting notebook <strong>notebook</strong>.ipynb to notebook\n6.1s\n2\n[NbConvertApp] Executing notebook with kernel: python3\n14.3s\n3\n2020-04-20 17:18:06.289469: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX512F\n14.3s\n4\n2020-04-20 17:18:06.314414: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2000179999 Hz\n14.3s\n5\n2020-04-20 17:18:06.315135: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x558c1cfbef00 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n14.3s\n6\n2020-04-20 17:18:06.315181: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version\n14.3s\n7\n2020-04-20 17:18:06.344279: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:300] Initialize GrpcChannelCache for job worker -&gt; {0 -&gt; 10.0.0.2:8470}\n14.3s\n8\n2020-04-20 17:18:06.344339: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:300] Initialize GrpcChannelCache for job localhost -&gt; {0 -&gt; localhost:30013}\n14.3s\n9\n2020-04-20 17:18:06.359935: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:300] Initialize GrpcChannelCache for job worker -&gt; {0 -&gt; 10.0.0.2:8470}\n14.3s\n10\n2020-04-20 17:18:06.359995: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:300] Initialize GrpcChannelCache for job localhost -&gt; {0 -&gt; localhost:30013}\n14.3s\n11\n2020-04-20 17:18:06.366237: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:390] Started server with target: grpc://localhost:30013\n21.6s\n12\n2020-04-20 17:18:13.653909: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n21.7s\n13\n2020-04-20 17:18:13.704236: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n21.7s\n14\n2020-04-20 17:18:13.741587: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n10166.2s\n15\n<strong>*</strong>*2020-04-20 20:07:18.218870: W tensorflow/core/distributed_runtime/eager/remote_tensor_handle_data.cc:75] Unable to destroy remote tensor handles. If you are running a tf.function, it usually indicates some op in the graph gets an error: RecvTensor expects a different device incarnation: 14797989987728605988 vs. 1799738036951955635. Your worker job (\"/job:tpu_worker/replica:0/task:0\") was probably restarted. Check your worker job for the reason why it was restarted.\n10166.2s\n16\nAdditional GRPC error information:\n10166.2s\n17\n{\"created\":\"@1587413238.218478572\",\"description\":\"Error received from peer\",\"file\":\"external/grpc/src/core/lib/surface/call.cc\",\"file_line\":1039,\"grpc_message\":\"RecvTensor expects a different device incarnation: 14797989987728605988 vs. 1799738036951955635. Your worker job (\"/job:tpu_worker/replica:0/task:0\") was probably restarted. Check your worker job for the reason why it was restarted.\",\"grpc_status\":10}\n10169.6s*<strong>*</strong>\n18\n2020-04-20 20:07:21.664715: W ./tensorflow/core/distributed_runtime/eager/destroy_tensor_handle_node.h:79] Ignoring an error encountered when deleting remote tensors handles: Invalid argument: Unable to find a context_id matching the specified one (12193600234898793608). Perhaps the worker was restarted, or the context was GC'd?\n10169.6s\n19\nAdditional GRPC error information:\n10169.6s\n20\n{\"created\":\"@1587413241.664555067\",\"description\":\"Error received from peer\",\"file\":\"external/grpc/src/core/lib/surface/call.cc\",\"file_line\":1039,\"grpc_message\":\"Unable to find a context_id matching the specified one (12193600234898793608). Perhaps the worker was restarted, or the context was GC'd?\",\"grpc_status\":3}\n10180.0s\n21\n[NbConvertApp] Writing 158697 bytes to <strong>notebook</strong>.ipynb\n10180.9s\n22\n[NbConvertApp] Converting notebook <strong>notebook</strong>.ipynb to html\n10182.4s\n23\n[NbConvertApp] Writing 508223 bytes to <strong>results</strong>.html\n10182.4s\n24\n10182.4s\n26\nComplete. Exited with code 0.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 867308,
      "author_name": "tchaye59",
      "author_url": "",
      "post_date": "05/30/2020 06:48:31",
      "content": "<p>Hi! <a href=\"/pawanverma\">@pawanverma</a>  I'm having the same problem. Did you find any solution?\nIn my case, the bucket can be accessed publicly without authentication. I don't understand why TensorFlow is expecting an authentication token when I commit the notebook.</p>\n\n<p><code>\n2020-05-30 03:56:35.631435: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n56.9s\n</code></p>",
      "votes": null,
      "replies": [
        {
          "id": 867344,
          "author_name": "pawanverma",
          "author_url": "",
          "post_date": "05/30/2020 07:22:24",
          "content": "<p>Hi, as per Kaggle team this is a harmless message and will b removed in future versions.  However,  the time taken to commit the notebook depends on the model training as well and for this notebook with two models ensemble it took around 2 hrs to commit. </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "814554": "martingorner \nHi, i am trying to commit my notebook, but since last one hour same message showing GCE check failed is being displayed. I left my notebook as it is. Next day i saw commit operation was failed.\nPlease help me as i am unable to commit my version.",
    "815365": "It failed with following log messages. At time 10166 error is coming but i have no idea about it.  Please guide me @martingorner \n\nTime\nLine #\nLog Message\n3.0s\n1\n[NbConvertApp] Converting notebook __notebook__.ipynb to notebook\n6.1s\n2\n[NbConvertApp] Executing notebook with kernel: python3\n14.3s\n3\n2020-04-20 17:18:06.289469: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX512F\n14.3s\n4\n2020-04-20 17:18:06.314414: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2000179999 Hz\n14.3s\n5\n2020-04-20 17:18:06.315135: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x558c1cfbef00 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n14.3s\n6\n2020-04-20 17:18:06.315181: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version\n14.3s\n7\n2020-04-20 17:18:06.344279: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:300] Initialize GrpcChannelCache for job worker -&gt; {0 -&gt; 10.0.0.2:8470}\n14.3s\n8\n2020-04-20 17:18:06.344339: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:300] Initialize GrpcChannelCache for job localhost -&gt; {0 -&gt; localhost:30013}\n14.3s\n9\n2020-04-20 17:18:06.359935: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:300] Initialize GrpcChannelCache for job worker -&gt; {0 -&gt; 10.0.0.2:8470}\n14.3s\n10\n2020-04-20 17:18:06.359995: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:300] Initialize GrpcChannelCache for job localhost -&gt; {0 -&gt; localhost:30013}\n14.3s\n11\n2020-04-20 17:18:06.366237: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:390] Started server with target: grpc://localhost:30013\n21.6s\n12\n2020-04-20 17:18:13.653909: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n21.7s\n13\n2020-04-20 17:18:13.704236: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n21.7s\n14\n2020-04-20 17:18:13.741587: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n10166.2s\n15\n******2020-04-20 20:07:18.218870: W tensorflow/core/distributed_runtime/eager/remote_tensor_handle_data.cc:75] Unable to destroy remote tensor handles. If you are running a tf.function, it usually indicates some op in the graph gets an error: RecvTensor expects a different device incarnation: 14797989987728605988 vs. 1799738036951955635. Your worker job (\"/job:tpu_worker/replica:0/task:0\") was probably restarted. Check your worker job for the reason why it was restarted.\n10166.2s\n16\nAdditional GRPC error information:\n10166.2s\n17\n{\"created\":\"@1587413238.218478572\",\"description\":\"Error received from peer\",\"file\":\"external/grpc/src/core/lib/surface/call.cc\",\"file_line\":1039,\"grpc_message\":\"RecvTensor expects a different device incarnation: 14797989987728605988 vs. 1799738036951955635. Your worker job (\"/job:tpu_worker/replica:0/task:0\") was probably restarted. Check your worker job for the reason why it was restarted.\",\"grpc_status\":10}\n10169.6s******\n18\n2020-04-20 20:07:21.664715: W ./tensorflow/core/distributed_runtime/eager/destroy_tensor_handle_node.h:79] Ignoring an error encountered when deleting remote tensors handles: Invalid argument: Unable to find a context_id matching the specified one (12193600234898793608). Perhaps the worker was restarted, or the context was GC'd?\n10169.6s\n19\nAdditional GRPC error information:\n10169.6s\n20\n{\"created\":\"@1587413241.664555067\",\"description\":\"Error received from peer\",\"file\":\"external/grpc/src/core/lib/surface/call.cc\",\"file_line\":1039,\"grpc_message\":\"Unable to find a context_id matching the specified one (12193600234898793608). Perhaps the worker was restarted, or the context was GC'd?\",\"grpc_status\":3}\n10180.0s\n21\n[NbConvertApp] Writing 158697 bytes to __notebook__.ipynb\n10180.9s\n22\n[NbConvertApp] Converting notebook __notebook__.ipynb to html\n10182.4s\n23\n[NbConvertApp] Writing 508223 bytes to __results__.html\n10182.4s\n24\n10182.4s\n26\nComplete. Exited with code 0.",
    "867308": "Hi! @pawanverma  I'm having the same problem. Did you find any solution?\nIn my case, the bucket can be accessed publicly without authentication. I don't understand why TensorFlow is expecting an authentication token when I commit the notebook.\n\n\n```\n2020-05-30 03:56:35.631435: W tensorflow/core/platform/cloud/google_auth_provider.cc:178] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with \"Not found: Could not locate the credentials file.\". Retrieving token from GCE failed with \"Cancelled: GCE check skipped due to presence of $NO_GCE_CHECK environment variable.\".\n56.9s\n```",
    "867344": "Hi, as per Kaggle team this is a harmless message and will b removed in future versions.  However,  the time taken to commit the notebook depends on the model training as well and for this notebook with two models ensemble it took around 2 hrs to commit."
  },
  "source": "meta"
}