{
  "id": 126113,
  "title": "Upgrading from Colab TPU to paid TPU",
  "url": "/competitions/tensorflow2-question-answering/discussion/126113",
  "author_name": "",
  "post_date": "2020-01-15T19:02:10.983548800Z",
  "votes": null,
  "comment_count": 4,
  "views": 0,
  "content": "<p>I've been using Colab with a TPU over the past few days and it's been great. Now, I'd like to upgrade to a paid TPU on GCE. I've been following the quickstart here: <a href=\"https://cloud.google.com/tpu/docs/quickstart\">https://cloud.google.com/tpu/docs/quickstart</a></p>\n\n<p>I've created a storage bucket and a TPU using tpu up. Now would like to connect to the TPU using my notebook, hosted with GCP AI Notebooks (<a href=\"https://console.cloud.google.com/ai-platform/notebooks\">https://console.cloud.google.com/ai-platform/notebooks</a>). When I was on Colab, I could access it using:</p>\n\n<p><code>\ncluster_resolver = tf.distribute.cluster_resolver.TPUClusterResolver()\ntf.config.experimental_connect_to_cluster(cluster_resolver)\n</code></p>\n\n<p>When I run the same line in a GCP AI Notebook...</p>\n\n<p><code>\ncluster_resolver = tf.distribute.cluster_resolver.TPUClusterResolver('MyTPUName')\ntf.config.experimental_connect_to_cluster(cluster_resolver)\n</code></p>\n\n<p>...it throws this error</p>\n\n<p>HttpError: HttpError 403 when requesting https://tpu.googleapis.com/v1/projects/local-dialect-[MyProjectID]/locations/us-east1-c/nodes/[MyTPUName]?alt=json returned \"Location us-east1-c is not found or access is unauthorized.\"</p>",
  "messages": [
    {
      "id": "719714",
      "postDate": "01/15/2020 19:02:10",
      "content": "<p>I've been using Colab with a TPU over the past few days and it's been great. Now, I'd like to upgrade to a paid TPU on GCE. I've been following the quickstart here: <a href=\"https://cloud.google.com/tpu/docs/quickstart\">https://cloud.google.com/tpu/docs/quickstart</a></p>\n\n<p>I've created a storage bucket and a TPU using tpu up. Now would like to connect to the TPU using my notebook, hosted with GCP AI Notebooks (<a href=\"https://console.cloud.google.com/ai-platform/notebooks\">https://console.cloud.google.com/ai-platform/notebooks</a>). When I was on Colab, I could access it using:</p>\n\n<p><code>\ncluster_resolver = tf.distribute.cluster_resolver.TPUClusterResolver()\ntf.config.experimental_connect_to_cluster(cluster_resolver)\n</code></p>\n\n<p>When I run the same line in a GCP AI Notebook...</p>\n\n<p><code>\ncluster_resolver = tf.distribute.cluster_resolver.TPUClusterResolver('MyTPUName')\ntf.config.experimental_connect_to_cluster(cluster_resolver)\n</code></p>\n\n<p>...it throws this error</p>\n\n<p>HttpError: HttpError 403 when requesting https://tpu.googleapis.com/v1/projects/local-dialect-[MyProjectID]/locations/us-east1-c/nodes/[MyTPUName]?alt=json returned \"Location us-east1-c is not found or access is unauthorized.\"</p>",
      "rawMarkdown": "I've been using Colab with a TPU over the past few days and it's been great. Now, I'd like to upgrade to a paid TPU on GCE. I've been following the quickstart here: https://cloud.google.com/tpu/docs/quickstart\n\nI've created a storage bucket and a TPU using tpu up. Now would like to connect to the TPU using my notebook, hosted with GCP AI Notebooks (https://console.cloud.google.com/ai-platform/notebooks). When I was on Colab, I could access it using:\n\n```\ncluster_resolver = tf.distribute.cluster_resolver.TPUClusterResolver()\ntf.config.experimental_connect_to_cluster(cluster_resolver)\n```\n\nWhen I run the same line in a GCP AI Notebook...\n\n```\ncluster_resolver = tf.distribute.cluster_resolver.TPUClusterResolver('MyTPUName')\ntf.config.experimental_connect_to_cluster(cluster_resolver)\n```\n\n...it throws this error\n\nHttpError: HttpError 403 when requesting https://tpu.googleapis.com/v1/projects/local-dialect-[MyProjectID]/locations/us-east1-c/nodes/[MyTPUName]?alt=json returned \"Location us-east1-c is not found or access is unauthorized.\"",
      "votes": null
    },
    {
      "id": "719822",
      "postDate": "01/15/2020 22:11:42",
      "content": "<p>Hello <a href=\"/alonbochman\">@alonbochman</a>, I believe that this is due to insufficient authentication scopes or in other words, you didn't provide enough or the right arguments to access that TPU.</p>\n\n<p>Please refer to <a href=\"https://www.tensorflow.org/api_docs/python/tf/distribute/cluster_resolver/TPUClusterResolver?version=stable\">tf.distribute.cluster_resolver.TPUClusterResolver</a> and try to provide the TPUClusterResolver with all the arguments required to connect.</p>\n\n<p>Hope this helps.</p>",
      "rawMarkdown": "Hello @alonbochman, I believe that this is due to insufficient authentication scopes or in other words, you didn't provide enough or the right arguments to access that TPU.\n\nPlease refer to [tf.distribute.cluster_resolver.TPUClusterResolver](https://www.tensorflow.org/api_docs/python/tf/distribute/cluster_resolver/TPUClusterResolver?version=stable) and try to provide the TPUClusterResolver with all the arguments required to connect.\n\nHope this helps.",
      "votes": null
    },
    {
      "id": "719842",
      "postDate": "01/15/2020 23:06:03",
      "content": "<p>Thanks <a href=\"/msheriey\">@msheriey</a>. I just realized I didn't post the full HTTP error - now edited. Does the TPU have to be in the same zone as the AI notebook?</p>",
      "rawMarkdown": "Thanks @msheriey. I just realized I didn't post the full HTTP error - now edited. Does the TPU have to be in the same zone as the AI notebook?",
      "votes": null
    },
    {
      "id": "719846",
      "postDate": "01/15/2020 23:16:54",
      "content": "<p>Yes, if not then you've to explicitly provide the zone argument.</p>\n\n<p>Quote from <a href=\"https://www.tensorflow.org/api_docs/python/tf/distribute/cluster_resolver/TPUClusterResolver\">tf.distribute.cluster_resolver.TPUClusterResolver</a>\n&gt; zone: Zone where the TPUs are located. If omitted or empty, we will assume that the zone of the TPU is the same as the zone of the GCE VM, which we will try to discover from the GCE metadata service.</p>\n\n<p>Here is a snippet from my code:\n<code>\ntpu_cluster_resolver = tf.distribute.cluster_resolver.TPUClusterResolver(\n        FLAGS.tpu_name, \n        zone=FLAGS.tpu_zone, \n        project=FLAGS.gcp_project)\n</code></p>",
      "rawMarkdown": "Yes, if not then you've to explicitly provide the zone argument.\n\nQuote from [tf.distribute.cluster_resolver.TPUClusterResolver](https://www.tensorflow.org/api_docs/python/tf/distribute/cluster_resolver/TPUClusterResolver)\n&gt; zone: Zone where the TPUs are located. If omitted or empty, we will assume that the zone of the TPU is the same as the zone of the GCE VM, which we will try to discover from the GCE metadata service.\n\nHere is a snippet from my code:\n```\ntpu_cluster_resolver = tf.distribute.cluster_resolver.TPUClusterResolver(\n        FLAGS.tpu_name, \n        zone=FLAGS.tpu_zone, \n        project=FLAGS.gcp_project)\n```",
      "votes": null
    },
    {
      "id": "719908",
      "postDate": "01/16/2020 01:11:34",
      "content": "<p>That worked. Thank you sir! I've been struggling with that one.</p>",
      "rawMarkdown": "That worked. Thank you sir! I've been struggling with that one.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 719822,
      "author_name": "msheriey",
      "author_url": "",
      "post_date": "01/15/2020 22:11:42",
      "content": "<p>Hello <a href=\"/alonbochman\">@alonbochman</a>, I believe that this is due to insufficient authentication scopes or in other words, you didn't provide enough or the right arguments to access that TPU.</p>\n\n<p>Please refer to <a href=\"https://www.tensorflow.org/api_docs/python/tf/distribute/cluster_resolver/TPUClusterResolver?version=stable\">tf.distribute.cluster_resolver.TPUClusterResolver</a> and try to provide the TPUClusterResolver with all the arguments required to connect.</p>\n\n<p>Hope this helps.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 719842,
      "author_name": "alonbochman",
      "author_url": "",
      "post_date": "01/15/2020 23:06:03",
      "content": "<p>Thanks <a href=\"/msheriey\">@msheriey</a>. I just realized I didn't post the full HTTP error - now edited. Does the TPU have to be in the same zone as the AI notebook?</p>",
      "votes": null,
      "replies": [
        {
          "id": 719846,
          "author_name": "msheriey",
          "author_url": "",
          "post_date": "01/15/2020 23:16:54",
          "content": "<p>Yes, if not then you've to explicitly provide the zone argument.</p>\n\n<p>Quote from <a href=\"https://www.tensorflow.org/api_docs/python/tf/distribute/cluster_resolver/TPUClusterResolver\">tf.distribute.cluster_resolver.TPUClusterResolver</a>\n&gt; zone: Zone where the TPUs are located. If omitted or empty, we will assume that the zone of the TPU is the same as the zone of the GCE VM, which we will try to discover from the GCE metadata service.</p>\n\n<p>Here is a snippet from my code:\n<code>\ntpu_cluster_resolver = tf.distribute.cluster_resolver.TPUClusterResolver(\n        FLAGS.tpu_name, \n        zone=FLAGS.tpu_zone, \n        project=FLAGS.gcp_project)\n</code></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 719908,
          "author_name": "alonbochman",
          "author_url": "",
          "post_date": "01/16/2020 01:11:34",
          "content": "<p>That worked. Thank you sir! I've been struggling with that one.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "719714": "I've been using Colab with a TPU over the past few days and it's been great. Now, I'd like to upgrade to a paid TPU on GCE. I've been following the quickstart here: https://cloud.google.com/tpu/docs/quickstart\n\nI've created a storage bucket and a TPU using tpu up. Now would like to connect to the TPU using my notebook, hosted with GCP AI Notebooks (https://console.cloud.google.com/ai-platform/notebooks). When I was on Colab, I could access it using:\n\n```\ncluster_resolver = tf.distribute.cluster_resolver.TPUClusterResolver()\ntf.config.experimental_connect_to_cluster(cluster_resolver)\n```\n\nWhen I run the same line in a GCP AI Notebook...\n\n```\ncluster_resolver = tf.distribute.cluster_resolver.TPUClusterResolver('MyTPUName')\ntf.config.experimental_connect_to_cluster(cluster_resolver)\n```\n\n...it throws this error\n\nHttpError: HttpError 403 when requesting https://tpu.googleapis.com/v1/projects/local-dialect-[MyProjectID]/locations/us-east1-c/nodes/[MyTPUName]?alt=json returned \"Location us-east1-c is not found or access is unauthorized.\"",
    "719822": "Hello @alonbochman, I believe that this is due to insufficient authentication scopes or in other words, you didn't provide enough or the right arguments to access that TPU.\n\nPlease refer to [tf.distribute.cluster_resolver.TPUClusterResolver](https://www.tensorflow.org/api_docs/python/tf/distribute/cluster_resolver/TPUClusterResolver?version=stable) and try to provide the TPUClusterResolver with all the arguments required to connect.\n\nHope this helps.",
    "719842": "Thanks @msheriey. I just realized I didn't post the full HTTP error - now edited. Does the TPU have to be in the same zone as the AI notebook?",
    "719846": "Yes, if not then you've to explicitly provide the zone argument.\n\nQuote from [tf.distribute.cluster_resolver.TPUClusterResolver](https://www.tensorflow.org/api_docs/python/tf/distribute/cluster_resolver/TPUClusterResolver)\n&gt; zone: Zone where the TPUs are located. If omitted or empty, we will assume that the zone of the TPU is the same as the zone of the GCE VM, which we will try to discover from the GCE metadata service.\n\nHere is a snippet from my code:\n```\ntpu_cluster_resolver = tf.distribute.cluster_resolver.TPUClusterResolver(\n        FLAGS.tpu_name, \n        zone=FLAGS.tpu_zone, \n        project=FLAGS.gcp_project)\n```",
    "719908": "That worked. Thank you sir! I've been struggling with that one."
  },
  "source": "meta"
}