\n
![\"screen\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F505153%2F52cbf1d04c5005b24066083505a7d3a0%2FTPU%20Kaggle.png?generation=1610542331030073&alt=media\")
Hello everybody. I tried to use my own dataset with TPU, but I got an error:
\n\n\nInvalidArgumentError: Unsuccessful TensorSliceReader constructor: Failed to get matching files on gs://kds-c807dd8fdf3be5c96c8d3def09e6d089b02512e1a5782e252179db6c/nets/cropnet_feature_vector_cassava_disease_V1_1/variables/variables: Permission denied: Error executing an HTTP request: HTTP response code 403 with body '{
\n
\n \"error\": {
\n \"code\": 403,
\n \"message\": \"service-467472385656@cloud-tpu.iam.gserviceaccount.com does not have storage.objects.list access to the Google Cloud Storage bucket.\",
\n \"errors\": [
\n {
\n \"message\": \"service-467472385656@cloud-tpu.iam.gserviceaccount.com does not have storage.objects.list access to the Google Cloud Storage bucket.\",
\n \"domain\": \"global\",
\n \"reason\": \"forbidden\"
\n }
\n ]
\n }
\n }
\n '
\n when reading gs://kds-c807dd8fdf3be5c96c8d3def09e6d089b02512e1a5782e252179db6c/nets/cropnet_feature_vector_cassava_disease_V1_1/variables>
Here is the dataset
\nAnd simple code for test:
import tensorflow as tf\nfrom kaggle_datasets import KaggleDatasets\nimport tensorflow_hub as hub\nfrom kaggle_secrets import UserSecretsClient\nuser_secrets = UserSecretsClient()\nuser_credential = user_secrets.get_gcloud_credential()\nprint(user_credential)\n\"\"\"{\"client_id\":\"32555940559.apps.googleusercontent.com\",\"client_secret\":\"Zm****2\",\"refresh_token\":\"1//05q*******SLU\",\"type\":\"authorized_user\"}\"\"\"\nuser_secrets.set_tensorflow_credential(user_credential)\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('cassava-leaf-disease-classification')\nGCS_PATH_TO_SAVEDMODEL = KaggleDatasets().get_gcs_path('tf-hub')\n\ntry:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver() \n print('Running on TPU ', tpu.master())\nexcept ValueError:\n tpu = None\n\nif tpu:\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n strategy = tf.distribute.get_strategy() \nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)\n\ntf.keras.backend.clear_session()\nmodels = [GCS_PATH_TO_SAVEDMODEL + '/nets/' + x for x in tf.io.gfile.listdir(GCS_PATH_TO_SAVEDMODEL + '/nets/')]\n\nwith strategy.scope():\n for model in models:\n if model.startswith('efficientnet'):\n continue\n print(f'Load model {model}')\n loaded_model = tf.saved_model.load(model)\n base_model = hub.KerasLayer(loaded_model, trainable=True)\nI have a Google cloud account and I've already logged in:
\n\n\nubscriber66@cloudshell:~$ gcloud auth list
\n
\n Credentialed Accounts
\n ACTIVE ACCOUNT\n
\n- subscriber66@gmail.com
\n
\n To set the active account, run:
\n $ gcloud config set accountACCOUNT>
And I linked Google account to kaggle
\n
So, what I'm doing wrong and what do I need to do to make it all work?
", "messages": [ { "id": "1151612", "postDate": "01/13/2021 13:01:03", "content": "Hello everybody. I tried to use my own dataset with TPU, but I got an error:
\n\n\nInvalidArgumentError: Unsuccessful TensorSliceReader constructor: Failed to get matching files on gs://kds-c807dd8fdf3be5c96c8d3def09e6d089b02512e1a5782e252179db6c/nets/cropnet_feature_vector_cassava_disease_V1_1/variables/variables: Permission denied: Error executing an HTTP request: HTTP response code 403 with body '{
\n
\n \"error\": {
\n \"code\": 403,
\n \"message\": \"service-467472385656@cloud-tpu.iam.gserviceaccount.com does not have storage.objects.list access to the Google Cloud Storage bucket.\",
\n \"errors\": [
\n {
\n \"message\": \"service-467472385656@cloud-tpu.iam.gserviceaccount.com does not have storage.objects.list access to the Google Cloud Storage bucket.\",
\n \"domain\": \"global\",
\n \"reason\": \"forbidden\"
\n }
\n ]
\n }
\n }
\n '
\n when reading gs://kds-c807dd8fdf3be5c96c8d3def09e6d089b02512e1a5782e252179db6c/nets/cropnet_feature_vector_cassava_disease_V1_1/variables>
Here is the dataset
\nAnd simple code for test:
import tensorflow as tf\nfrom kaggle_datasets import KaggleDatasets\nimport tensorflow_hub as hub\nfrom kaggle_secrets import UserSecretsClient\nuser_secrets = UserSecretsClient()\nuser_credential = user_secrets.get_gcloud_credential()\nprint(user_credential)\n\"\"\"{\"client_id\":\"32555940559.apps.googleusercontent.com\",\"client_secret\":\"Zm****2\",\"refresh_token\":\"1//05q*******SLU\",\"type\":\"authorized_user\"}\"\"\"\nuser_secrets.set_tensorflow_credential(user_credential)\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('cassava-leaf-disease-classification')\nGCS_PATH_TO_SAVEDMODEL = KaggleDatasets().get_gcs_path('tf-hub')\n\ntry:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver() \n print('Running on TPU ', tpu.master())\nexcept ValueError:\n tpu = None\n\nif tpu:\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n strategy = tf.distribute.get_strategy() \nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)\n\ntf.keras.backend.clear_session()\nmodels = [GCS_PATH_TO_SAVEDMODEL + '/nets/' + x for x in tf.io.gfile.listdir(GCS_PATH_TO_SAVEDMODEL + '/nets/')]\n\nwith strategy.scope():\n for model in models:\n if model.startswith('efficientnet'):\n continue\n print(f'Load model {model}')\n loaded_model = tf.saved_model.load(model)\n base_model = hub.KerasLayer(loaded_model, trainable=True)\nI have a Google cloud account and I've already logged in:
\n\n\nubscriber66@cloudshell:~$ gcloud auth list
\n
\n Credentialed Accounts
\n ACTIVE ACCOUNT\n
\n- subscriber66@gmail.com
\n
\n To set the active account, run:
\n $ gcloud config set accountACCOUNT>
And I linked Google account to kaggle
\n
So, what I'm doing wrong and what do I need to do to make it all work?
", "rawMarkdown": "Hello everybody. I tried to use my own dataset with TPU, but I got an error:\n> InvalidArgumentError: Unsuccessful TensorSliceReader constructor: Failed to get matching files on gs://kds-c807dd8fdf3be5c96c8d3def09e6d089b02512e1a5782e252179db6c/nets/cropnet_feature_vector_cassava_disease_V1_1/variables/variables: Permission denied: Error executing an HTTP request: HTTP response code 403 with body '{\n \"error\": {\n \"code\": 403,\n \"message\": \"service-467472385656@cloud-tpu.iam.gserviceaccount.com does not have storage.objects.list access to the Google Cloud Storage bucket.\",\n \"errors\": [\n {\n \"message\": \"service-467472385656@cloud-tpu.iam.gserviceaccount.com does not have storage.objects.list access to the Google Cloud Storage bucket.\",\n \"domain\": \"global\",\n \"reason\": \"forbidden\"\n }\n ]\n }\n}\n'\n\t when reading gs://kds-c807dd8fdf3be5c96c8d3def09e6d089b02512e1a5782e252179db6c/nets/cropnet_feature_vector_cassava_disease_V1_1/variables> \n\nHere is the [dataset](https://www.kaggle.com/stahh007/tf-hub)\nAnd simple code for test:\n```\nimport tensorflow as tf\nfrom kaggle_datasets import KaggleDatasets\nimport tensorflow_hub as hub\nfrom kaggle_secrets import UserSecretsClient\nuser_secrets = UserSecretsClient()\nuser_credential = user_secrets.get_gcloud_credential()\nprint(user_credential)\n\"\"\"{\"client_id\":\"32555940559.apps.googleusercontent.com\",\"client_secret\":\"Zm****2\",\"refresh_token\":\"1//05q*******SLU\",\"type\":\"authorized_user\"}\"\"\"\nuser_secrets.set_tensorflow_credential(user_credential)\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('cassava-leaf-disease-classification')\nGCS_PATH_TO_SAVEDMODEL = KaggleDatasets().get_gcs_path('tf-hub')\n\ntry:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver() \n print('Running on TPU ', tpu.master())\nexcept ValueError:\n tpu = None\n\nif tpu:\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n strategy = tf.distribute.get_strategy() \nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)\n\ntf.keras.backend.clear_session()\nmodels = [GCS_PATH_TO_SAVEDMODEL + '/nets/' + x for x in tf.io.gfile.listdir(GCS_PATH_TO_SAVEDMODEL + '/nets/')]\n\nwith strategy.scope():\n for model in models:\n if model.startswith('efficientnet'):\n continue\n print(f'Load model {model}')\n loaded_model = tf.saved_model.load(model)\n base_model = hub.KerasLayer(loaded_model, trainable=True)\n```\n\nI have a Google cloud account and I've already logged in:\n> ubscriber66@cloudshell:~$ gcloud auth list\n Credentialed Accounts\nACTIVE ACCOUNT\n* subscriber66@gmail.com\nTo set the active account, run:\n $ gcloud config set account `ACCOUNT`> \n\nAnd I linked Google account to kaggle\n\n\nSo, what I'm doing wrong and what do I need to do to make it all work?", "votes": null }, { "id": "1153493", "postDate": "01/14/2021 23:58:11", "content": "Try to initialize the TPU before the user secrets. Just had a similar issue working with a private dataset and this worked for me.
", "rawMarkdown": "Try to initialize the TPU before the user secrets. Just had a similar issue working with a private dataset and this worked for me.", "votes": null }, { "id": "1154065", "postDate": "01/15/2021 11:30:28", "content": "@morodertobias many thanks! It works. Later I will share a notebook with full code.
", "rawMarkdown": "morodertobias many thanks! It works. Later I will share a notebook with full code.", "votes": null } ], "comments": [ { "id": 1153493, "author_name": "morodertobias", "author_url": "", "post_date": "01/14/2021 23:58:11", "content": "Try to initialize the TPU before the user secrets. Just had a similar issue working with a private dataset and this worked for me.
", "votes": null, "replies": [ { "id": 1154065, "author_name": "stahh007", "author_url": "", "post_date": "01/15/2021 11:30:28", "content": "@morodertobias many thanks! It works. Later I will share a notebook with full code.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1151612": "Hello everybody. I tried to use my own dataset with TPU, but I got an error:\n> InvalidArgumentError: Unsuccessful TensorSliceReader constructor: Failed to get matching files on gs://kds-c807dd8fdf3be5c96c8d3def09e6d089b02512e1a5782e252179db6c/nets/cropnet_feature_vector_cassava_disease_V1_1/variables/variables: Permission denied: Error executing an HTTP request: HTTP response code 403 with body '{\n \"error\": {\n \"code\": 403,\n \"message\": \"service-467472385656@cloud-tpu.iam.gserviceaccount.com does not have storage.objects.list access to the Google Cloud Storage bucket.\",\n \"errors\": [\n {\n \"message\": \"service-467472385656@cloud-tpu.iam.gserviceaccount.com does not have storage.objects.list access to the Google Cloud Storage bucket.\",\n \"domain\": \"global\",\n \"reason\": \"forbidden\"\n }\n ]\n }\n}\n'\n\t when reading gs://kds-c807dd8fdf3be5c96c8d3def09e6d089b02512e1a5782e252179db6c/nets/cropnet_feature_vector_cassava_disease_V1_1/variables> \n\nHere is the [dataset](https://www.kaggle.com/stahh007/tf-hub)\nAnd simple code for test:\n```\nimport tensorflow as tf\nfrom kaggle_datasets import KaggleDatasets\nimport tensorflow_hub as hub\nfrom kaggle_secrets import UserSecretsClient\nuser_secrets = UserSecretsClient()\nuser_credential = user_secrets.get_gcloud_credential()\nprint(user_credential)\n\"\"\"{\"client_id\":\"32555940559.apps.googleusercontent.com\",\"client_secret\":\"Zm****2\",\"refresh_token\":\"1//05q*******SLU\",\"type\":\"authorized_user\"}\"\"\"\nuser_secrets.set_tensorflow_credential(user_credential)\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('cassava-leaf-disease-classification')\nGCS_PATH_TO_SAVEDMODEL = KaggleDatasets().get_gcs_path('tf-hub')\n\ntry:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver() \n print('Running on TPU ', tpu.master())\nexcept ValueError:\n tpu = None\n\nif tpu:\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n strategy = tf.distribute.get_strategy() \nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)\n\ntf.keras.backend.clear_session()\nmodels = [GCS_PATH_TO_SAVEDMODEL + '/nets/' + x for x in tf.io.gfile.listdir(GCS_PATH_TO_SAVEDMODEL + '/nets/')]\n\nwith strategy.scope():\n for model in models:\n if model.startswith('efficientnet'):\n continue\n print(f'Load model {model}')\n loaded_model = tf.saved_model.load(model)\n base_model = hub.KerasLayer(loaded_model, trainable=True)\n```\n\nI have a Google cloud account and I've already logged in:\n> ubscriber66@cloudshell:~$ gcloud auth list\n Credentialed Accounts\nACTIVE ACCOUNT\n* subscriber66@gmail.com\nTo set the active account, run:\n $ gcloud config set account `ACCOUNT`> \n\nAnd I linked Google account to kaggle\n\n\nSo, what I'm doing wrong and what do I need to do to make it all work?", "1153493": "Try to initialize the TPU before the user secrets. Just had a similar issue working with a private dataset and this worked for me.", "1154065": "morodertobias many thanks! It works. Later I will share a notebook with full code." }, "source": "meta" }