{
  "id": 312630,
  "title": "Create Dataset for using TPU",
  "url": "/competitions/happy-whale-and-dolphin/discussion/312630",
  "author_name": "",
  "post_date": "2022-03-13T06:45:43.105756800Z",
  "votes": 4,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hello everyone, i'm just created my private dataset for training my model on TPU, I don't know what wrong with my dataset or my configuration.<br>\nI know that TPU need dataset have to be on Google cloud. </p>\n<ul>\n<li>1. i ran this to get my GCS_PATH</li>\n</ul>\n<pre><code>from kaggle_datasets import KaggleDatasets \nGCS_PATH = KaggleDatasets().get_gcs_path() \nVALID_FILENAMES = tf.io.gfile.glob(GCS_PATH+'/**') \nVALID_FILENAMES\n</code></pre>\n<p>it returns list of my tfrec along with  <code>GCS_PATH</code><br>\nlike this </p>\n<pre><code>['gs://kds-216eb3347dfe9f2aa540176dead2dd2db6de473a95ae18bd93490c2d/happywhale-2022-test-0-2796.tfrec',\n 'gs://kds-216eb3347dfe9f2aa540176dead2dd2db6de473a95ae18bd93490c2d/happywhale-2022-test-1-2796.tfrec',\n 'gs://kds-216eb3347dfe9f2aa540176dead2dd2db6de473a95ae18bd93490c2d/happywhale-2022-test-2-2796.tfrec',\n 'gs://kds-216eb3347dfe9f2aa540176dead2dd2db6de473a95ae18bd93490c2d/happywhale-2022-test-3-2796.tfrec',...]\n</code></pre>\n<ul>\n<li>2. Next, i ran this </li>\n</ul>\n<pre><code>!gsutil ls $GCS_PATH\n</code></pre>\n<p>instead of getting the same result as <code>VALID_FILENAMES</code> , i got <br>\n<code>ServiceException: 401 Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket.</code></p>\n<ul>\n<li>3. What's i have tried:<br>\n<strong>i attach my Google Cloud SDK on my notebook.</strong><br>\n<strong>i tried publish the dataset.</strong><br>\n<strong>i tried to allocate TPU in the notebook.</strong></li>\n</ul>\n<p>Idk what's wrong with my dataset. 😓It seem like there are people struggled like me. I'm so confused when read all of the discussion and API documentations, but nothing works at all. Can anyone help me and others with this problem. Thank you very much for helping me out.</p>",
  "messages": [
    {
      "id": "1720806",
      "postDate": "03/13/2022 06:45:43",
      "content": "<p>Hello everyone, i'm just created my private dataset for training my model on TPU, I don't know what wrong with my dataset or my configuration.<br>\nI know that TPU need dataset have to be on Google cloud. </p>\n<ul>\n<li>1. i ran this to get my GCS_PATH</li>\n</ul>\n<pre><code>from kaggle_datasets import KaggleDatasets \nGCS_PATH = KaggleDatasets().get_gcs_path() \nVALID_FILENAMES = tf.io.gfile.glob(GCS_PATH+'/**') \nVALID_FILENAMES\n</code></pre>\n<p>it returns list of my tfrec along with  <code>GCS_PATH</code><br>\nlike this </p>\n<pre><code>['gs://kds-216eb3347dfe9f2aa540176dead2dd2db6de473a95ae18bd93490c2d/happywhale-2022-test-0-2796.tfrec',\n 'gs://kds-216eb3347dfe9f2aa540176dead2dd2db6de473a95ae18bd93490c2d/happywhale-2022-test-1-2796.tfrec',\n 'gs://kds-216eb3347dfe9f2aa540176dead2dd2db6de473a95ae18bd93490c2d/happywhale-2022-test-2-2796.tfrec',\n 'gs://kds-216eb3347dfe9f2aa540176dead2dd2db6de473a95ae18bd93490c2d/happywhale-2022-test-3-2796.tfrec',...]\n</code></pre>\n<ul>\n<li>2. Next, i ran this </li>\n</ul>\n<pre><code>!gsutil ls $GCS_PATH\n</code></pre>\n<p>instead of getting the same result as <code>VALID_FILENAMES</code> , i got <br>\n<code>ServiceException: 401 Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket.</code></p>\n<ul>\n<li>3. What's i have tried:<br>\n<strong>i attach my Google Cloud SDK on my notebook.</strong><br>\n<strong>i tried publish the dataset.</strong><br>\n<strong>i tried to allocate TPU in the notebook.</strong></li>\n</ul>\n<p>Idk what's wrong with my dataset. 😓It seem like there are people struggled like me. I'm so confused when read all of the discussion and API documentations, but nothing works at all. Can anyone help me and others with this problem. Thank you very much for helping me out.</p>",
      "rawMarkdown": "Hello everyone, i'm just created my private dataset for training my model on TPU, I don't know what wrong with my dataset or my configuration.\nI know that TPU need dataset have to be on Google cloud. \n\n- 1. i ran this to get my GCS_PATH\n```\nfrom kaggle_datasets import KaggleDatasets \nGCS_PATH = KaggleDatasets().get_gcs_path() \nVALID_FILENAMES = tf.io.gfile.glob(GCS_PATH+'/**') \nVALID_FILENAMES\n```\nit returns list of my tfrec along with  `GCS_PATH`\nlike this \n```\n['gs://kds-216eb3347dfe9f2aa540176dead2dd2db6de473a95ae18bd93490c2d/happywhale-2022-test-0-2796.tfrec',\n 'gs://kds-216eb3347dfe9f2aa540176dead2dd2db6de473a95ae18bd93490c2d/happywhale-2022-test-1-2796.tfrec',\n 'gs://kds-216eb3347dfe9f2aa540176dead2dd2db6de473a95ae18bd93490c2d/happywhale-2022-test-2-2796.tfrec',\n 'gs://kds-216eb3347dfe9f2aa540176dead2dd2db6de473a95ae18bd93490c2d/happywhale-2022-test-3-2796.tfrec',...]\n```\n\n- 2. Next, i ran this \n```\n!gsutil ls $GCS_PATH\n```\ninstead of getting the same result as `VALID_FILENAMES` , i got \n`ServiceException: 401 Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket.`\n\n- 3. What's i have tried:\n**i attach my Google Cloud SDK on my notebook.**\n**i tried publish the dataset.**\n**i tried to allocate TPU in the notebook.**\n\nIdk what's wrong with my dataset. 😓It seem like there are people struggled like me. I'm so confused when read all of the discussion and API documentations, but nothing works at all. Can anyone help me and others with this problem. Thank you very much for helping me out.",
      "votes": null
    },
    {
      "id": "1720808",
      "postDate": "03/13/2022 06:51:30",
      "content": "<p>Hello! </p>\n<p>Looking at the error, it seems you haven't connected your GCP account? I found this <a href=\"https://towardsdatascience.com/importing-kaggle-training-data-into-google-cloud-storage-70830a439048\" target=\"_blank\">blog</a> with detailed steps that might help!</p>",
      "rawMarkdown": "Hello! \n\nLooking at the error, it seems you haven't connected your GCP account? I found this [blog](https://towardsdatascience.com/importing-kaggle-training-data-into-google-cloud-storage-70830a439048) with detailed steps that might help!",
      "votes": null
    },
    {
      "id": "1720810",
      "postDate": "03/13/2022 06:54:01",
      "content": "<p>Thank you, i'll try it out.</p>",
      "rawMarkdown": "Thank you, i'll try it out.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1720808,
      "author_name": "init27",
      "author_url": "",
      "post_date": "03/13/2022 06:51:30",
      "content": "<p>Hello! </p>\n<p>Looking at the error, it seems you haven't connected your GCP account? I found this <a href=\"https://towardsdatascience.com/importing-kaggle-training-data-into-google-cloud-storage-70830a439048\" target=\"_blank\">blog</a> with detailed steps that might help!</p>",
      "votes": null,
      "replies": [
        {
          "id": 1720810,
          "author_name": "locbaop",
          "author_url": "",
          "post_date": "03/13/2022 06:54:01",
          "content": "<p>Thank you, i'll try it out.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1720806": "Hello everyone, i'm just created my private dataset for training my model on TPU, I don't know what wrong with my dataset or my configuration.\nI know that TPU need dataset have to be on Google cloud. \n\n- 1. i ran this to get my GCS_PATH\n```\nfrom kaggle_datasets import KaggleDatasets \nGCS_PATH = KaggleDatasets().get_gcs_path() \nVALID_FILENAMES = tf.io.gfile.glob(GCS_PATH+'/**') \nVALID_FILENAMES\n```\nit returns list of my tfrec along with  `GCS_PATH`\nlike this \n```\n['gs://kds-216eb3347dfe9f2aa540176dead2dd2db6de473a95ae18bd93490c2d/happywhale-2022-test-0-2796.tfrec',\n 'gs://kds-216eb3347dfe9f2aa540176dead2dd2db6de473a95ae18bd93490c2d/happywhale-2022-test-1-2796.tfrec',\n 'gs://kds-216eb3347dfe9f2aa540176dead2dd2db6de473a95ae18bd93490c2d/happywhale-2022-test-2-2796.tfrec',\n 'gs://kds-216eb3347dfe9f2aa540176dead2dd2db6de473a95ae18bd93490c2d/happywhale-2022-test-3-2796.tfrec',...]\n```\n\n- 2. Next, i ran this \n```\n!gsutil ls $GCS_PATH\n```\ninstead of getting the same result as `VALID_FILENAMES` , i got \n`ServiceException: 401 Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket.`\n\n- 3. What's i have tried:\n**i attach my Google Cloud SDK on my notebook.**\n**i tried publish the dataset.**\n**i tried to allocate TPU in the notebook.**\n\nIdk what's wrong with my dataset. 😓It seem like there are people struggled like me. I'm so confused when read all of the discussion and API documentations, but nothing works at all. Can anyone help me and others with this problem. Thank you very much for helping me out.",
    "1720808": "Hello! \n\nLooking at the error, it seems you haven't connected your GCP account? I found this [blog](https://towardsdatascience.com/importing-kaggle-training-data-into-google-cloud-storage-70830a439048) with detailed steps that might help!",
    "1720810": "Thank you, i'll try it out."
  },
  "source": "meta"
}