{
  "id": 135217,
  "title": "tf.data.Dataset.list_files() not working on TPU",
  "url": "/competitions/flower-classification-with-tpus/discussion/135217",
  "author_name": "Shashank Shekhar",
  "post_date": "2020-03-12T16:45:52.854000",
  "votes": 1,
  "comment_count": 7,
  "views": 0,
  "content": "<p>while loading data through <code>tf.data.Dataset.list_files()</code>, I am getting an UnimplementedError like this.\n<code>UnimplementedError: File system scheme '[local]' not implemented (file: '/kaggle/input/flowers-104/images/*/*.jpg') [Op:Greater] name: match_not_empty</code></p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1793891%2F8be9e5f3568045e28b11d4374c3487cb%2Ferror.png?generation=1584031548968972&amp;alt=media\" alt=\"\"></p>",
  "messages": [
    {
      "id": 770185,
      "postDate": "2020-03-12T16:45:52.853Z",
      "content": "<p>while loading data through <code>tf.data.Dataset.list_files()</code>, I am getting an UnimplementedError like this.\n<code>UnimplementedError: File system scheme '[local]' not implemented (file: '/kaggle/input/flowers-104/images/*/*.jpg') [Op:Greater] name: match_not_empty</code></p>\n\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1793891%2F8be9e5f3568045e28b11d4374c3487cb%2Ferror.png?generation=1584031548968972&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "while loading data through `tf.data.Dataset.list_files()`, I am getting an UnimplementedError like this.\n`UnimplementedError: File system scheme '[local]' not implemented (file: '/kaggle/input/flowers-104/images/*/*.jpg') [Op:Greater] name: match_not_empty`\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1793891%2F8be9e5f3568045e28b11d4374c3487cb%2Ferror.png?generation=1584031548968972&amp;alt=media)\n",
      "votes": 1
    },
    {
      "id": 835973,
      "postDate": "2020-05-06T16:24:19.500Z",
      "content": "<p>Martin,\nI am using GCS_PATH. I am able to list the files into variable 'filenames', but even then I got same error.\nI tried to run map function on TFRecord dataset. But it  is throwing the error.\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1887153%2F1ee4b4fdd6d6d822d6350688683f294e%2F000003.jpg?generation=1588782245673664&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "Martin,\nI am using GCS_PATH. I am able to list the files into variable 'filenames', but even then I got same error.\nI tried to run map function on TFRecord dataset. But it  is throwing the error.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1887153%2F1ee4b4fdd6d6d822d6350688683f294e%2F000003.jpg?generation=1588782245673664&amp;alt=media)\n",
      "replies": [
        {
          "id": 836055,
          "postDate": "2020-05-06T17:27:14.113Z",
          "content": "<p>Your GCS path should look like gs://...</p>",
          "rawMarkdown": "Your GCS path should look like gs://...",
          "votes": 1
        },
        {
          "id": 836482,
          "postDate": "2020-05-07T02:52:21.750Z",
          "content": "<p>Martin, thanks for the pointer.\nActually the command given in TPU help page, is not resolving the absolute path.\ninitially I used \nfilenames = tf.io.gfile.glob('GCS_PATH/*')\nall files listed, but GCS_PATH not resolved.</p>\n\n<p>After your reply I used explicitely gcs path string, then it is working, and the error resolved.\nfilenames = tf.io.gfile.glob('gs://kds-55d....../*')</p>",
          "rawMarkdown": "Martin, thanks for the pointer.\nActually the command given in TPU help page, is not resolving the absolute path.\ninitially I used \nfilenames = tf.io.gfile.glob('GCS_PATH/*')\nall files listed, but GCS_PATH not resolved.\n\nAfter your reply I used explicitely gcs path string, then it is working, and the error resolved.\nfilenames = tf.io.gfile.glob('gs://kds-55d....../*')"
        },
        {
          "id": 837677,
          "postDate": "2020-05-08T00:31:15.477Z",
          "content": "<p>In the <a href=\"https://www.kaggle.com/docs/tpu\">TPU doc</a>, the code is:\n<code>\nGCS_PATH = KaggleDatasets().get_gcs_path()\ngcs_pattern = GCS_PATH + '/tfrecords-jpeg-331x331/*.tfrec'\nfilename_list =  tf.io.gfile.glob(gcs_pattern)\ndataset = tf.data.TFRecordDataset(filenames)\n...\n</code></p>",
          "rawMarkdown": "In the [TPU doc](https://www.kaggle.com/docs/tpu), the code is:\n```\nGCS_PATH = KaggleDatasets().get_gcs_path()\ngcs_pattern = GCS_PATH + '/tfrecords-jpeg-331x331/*.tfrec'\nfilename_list =  tf.io.gfile.glob(gcs_pattern)\ndataset = tf.data.TFRecordDataset(filenames)\n...\n```"
        }
      ]
    },
    {
      "id": 770356,
      "postDate": "2020-03-12T20:59:53.480Z",
      "content": "<p>TPUs can only load data from GCS (Google Cloud Storage).\nYou can get the GCS path of your Kaggle dataset with <code>KaggleDatasets().get_gcs_path()</code></p>",
      "rawMarkdown": "TPUs can only load data from GCS (Google Cloud Storage).\nYou can get the GCS path of your Kaggle dataset with `KaggleDatasets().get_gcs_path()`",
      "replies": [
        {
          "id": 770553,
          "postDate": "2020-03-13T04:55:22.573Z",
          "content": "<p>Ok thanks a lot.</p>",
          "rawMarkdown": "Ok thanks a lot."
        }
      ]
    },
    {
      "id": 836478,
      "postDate": "2020-05-07T02:49:50.080Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 835973,
      "author_name": "Prakash",
      "author_url": "",
      "post_date": "2020-05-06T16:24:19.500000",
      "content": "<p>Martin,\nI am using GCS_PATH. I am able to list the files into variable 'filenames', but even then I got same error.\nI tried to run map function on TFRecord dataset. But it  is throwing the error.\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1887153%2F1ee4b4fdd6d6d822d6350688683f294e%2F000003.jpg?generation=1588782245673664&amp;alt=media\" alt=\"\"></p>",
      "votes": 0,
      "replies": [
        {
          "id": 836055,
          "author_name": "Martin Görner",
          "author_url": "",
          "post_date": "2020-05-06T17:27:14.113000",
          "content": "<p>Your GCS path should look like gs://...</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 836482,
          "author_name": "Prakash",
          "author_url": "",
          "post_date": "2020-05-07T02:52:21.750000",
          "content": "<p>Martin, thanks for the pointer.\nActually the command given in TPU help page, is not resolving the absolute path.\ninitially I used \nfilenames = tf.io.gfile.glob('GCS_PATH/*')\nall files listed, but GCS_PATH not resolved.</p>\n\n<p>After your reply I used explicitely gcs path string, then it is working, and the error resolved.\nfilenames = tf.io.gfile.glob('gs://kds-55d....../*')</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 837677,
          "author_name": "Martin Görner",
          "author_url": "",
          "post_date": "2020-05-08T00:31:15.477000",
          "content": "<p>In the <a href=\"https://www.kaggle.com/docs/tpu\">TPU doc</a>, the code is:\n<code>\nGCS_PATH = KaggleDatasets().get_gcs_path()\ngcs_pattern = GCS_PATH + '/tfrecords-jpeg-331x331/*.tfrec'\nfilename_list =  tf.io.gfile.glob(gcs_pattern)\ndataset = tf.data.TFRecordDataset(filenames)\n...\n</code></p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 770356,
      "author_name": "Martin Görner",
      "author_url": "",
      "post_date": "2020-03-12T20:59:53.480000",
      "content": "<p>TPUs can only load data from GCS (Google Cloud Storage).\nYou can get the GCS path of your Kaggle dataset with <code>KaggleDatasets().get_gcs_path()</code></p>",
      "votes": 0,
      "replies": [
        {
          "id": 770553,
          "author_name": "Shashank Shekhar",
          "author_url": "",
          "post_date": "2020-03-13T04:55:22.573000",
          "content": "<p>Ok thanks a lot.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 836478,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-05-07T02:49:50.080000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "770185": "while loading data through `tf.data.Dataset.list_files()`, I am getting an UnimplementedError like this.\n`UnimplementedError: File system scheme '[local]' not implemented (file: '/kaggle/input/flowers-104/images/*/*.jpg') [Op:Greater] name: match_not_empty`\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1793891%2F8be9e5f3568045e28b11d4374c3487cb%2Ferror.png?generation=1584031548968972&amp;alt=media)\n",
    "835973": "Martin,\nI am using GCS_PATH. I am able to list the files into variable 'filenames', but even then I got same error.\nI tried to run map function on TFRecord dataset. But it  is throwing the error.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1887153%2F1ee4b4fdd6d6d822d6350688683f294e%2F000003.jpg?generation=1588782245673664&amp;alt=media)\n",
    "770356": "TPUs can only load data from GCS (Google Cloud Storage).\nYou can get the GCS path of your Kaggle dataset with `KaggleDatasets().get_gcs_path()`",
    "836478": ""
  }
}