{
  "id": 309016,
  "title": "How to load dataset in colab by GCS? ",
  "url": "/competitions/happy-whale-and-dolphin/discussion/309016",
  "author_name": "",
  "post_date": "2022-02-21T12:58:38.973378Z",
  "votes": 5,
  "comment_count": 12,
  "views": 0,
  "content": "<p>Hi, I want to use kaggle dataset in colab, but due to disk limitation, I have to load data by GCS：</p>\n<pre><code>from kaggle_datasets import KaggleDatasets\nGCS_PATH = KaggleDatasets().get_gcs_path()\n</code></pre>\n<p>when I ran:</p>\n<pre><code>pd.read_csv(GCS_PATH+'/sample_submission.csv')\n</code></pre>\n<p>everything is ok</p>\n<p>but when I ran:</p>\n<pre><code>from PIL import Image\nImage.open(GCS_PATH+'/test_images/000110707af0ba.jpg')\n</code></pre>\n<p>error occurs:</p>\n<pre><code>FileNotFoundError: [Errno 2] No such file or directory: 'gs://kds-ec2a2e61dd5345d807d47adc26eb93dcc160dd3b4da4084f45a859f0/test_images/000110707af0ba.jpg'\n</code></pre>\n<p>How to fix this? Thank you !</p>",
  "messages": [
    {
      "id": "1699797",
      "postDate": "02/21/2022 12:58:38",
      "content": "<p>Hi, I want to use kaggle dataset in colab, but due to disk limitation, I have to load data by GCS：</p>\n<pre><code>from kaggle_datasets import KaggleDatasets\nGCS_PATH = KaggleDatasets().get_gcs_path()\n</code></pre>\n<p>when I ran:</p>\n<pre><code>pd.read_csv(GCS_PATH+'/sample_submission.csv')\n</code></pre>\n<p>everything is ok</p>\n<p>but when I ran:</p>\n<pre><code>from PIL import Image\nImage.open(GCS_PATH+'/test_images/000110707af0ba.jpg')\n</code></pre>\n<p>error occurs:</p>\n<pre><code>FileNotFoundError: [Errno 2] No such file or directory: 'gs://kds-ec2a2e61dd5345d807d47adc26eb93dcc160dd3b4da4084f45a859f0/test_images/000110707af0ba.jpg'\n</code></pre>\n<p>How to fix this? Thank you !</p>",
      "rawMarkdown": "Hi, I want to use kaggle dataset in colab, but due to disk limitation, I have to load data by GCS：\n```\nfrom kaggle_datasets import KaggleDatasets\nGCS_PATH = KaggleDatasets().get_gcs_path()\n```\n\nwhen I ran:\n```\npd.read_csv(GCS_PATH+'/sample_submission.csv')\n```\neverything is ok\n\nbut when I ran:\n```\nfrom PIL import Image\nImage.open(GCS_PATH+'/test_images/000110707af0ba.jpg')\n```\nerror occurs:\n```\nFileNotFoundError: [Errno 2] No such file or directory: 'gs://kds-ec2a2e61dd5345d807d47adc26eb93dcc160dd3b4da4084f45a859f0/test_images/000110707af0ba.jpg'\n```\nHow to fix this? Thank you !",
      "votes": null
    },
    {
      "id": "1699814",
      "postDate": "02/21/2022 13:12:12",
      "content": "<p>there are many shared notebooks. search for it.</p>\n<p>you get gcs_path,  then  paste and use it in colab to access the dataset.</p>",
      "rawMarkdown": "there are many shared notebooks. search for it.\n\nyou get gcs_path,  then  paste and use it in colab to access the dataset.",
      "votes": null
    },
    {
      "id": "1699841",
      "postDate": "02/21/2022 13:33:27",
      "content": "<p>Yes, I found some notebooks, but still got this, I'll keep looking , Thakns! </p>",
      "rawMarkdown": "Yes, I found some notebooks, but still got this, I'll keep looking , Thakns!",
      "votes": null
    },
    {
      "id": "1699851",
      "postDate": "02/21/2022 13:38:46",
      "content": "<p>gcs path only valid for some time, if It doesn't work, you should get the new gcs_path from kaggle platform and then update it on colab.</p>",
      "rawMarkdown": "gcs path only valid for some time, if It doesn't work, you should get the new gcs_path from kaggle platform and then update it on colab.",
      "votes": null
    },
    {
      "id": "1699857",
      "postDate": "02/21/2022 13:43:21",
      "content": "<p>I use this two line:</p>\n<pre><code>from kaggle_datasets import KaggleDatasets\nGCS_PATH = KaggleDatasets().get_gcs_path()\n</code></pre>\n<p>to get the latest gcs_path.</p>\n<p>but still  <strong>FileNotFound</strong>😭</p>",
      "rawMarkdown": "I use this two line:\n```\nfrom kaggle_datasets import KaggleDatasets\nGCS_PATH = KaggleDatasets().get_gcs_path()\n```\nto get the latest gcs_path.\n\nbut still  **FileNotFound**😭",
      "votes": null
    },
    {
      "id": "1699870",
      "postDate": "02/21/2022 13:48:38",
      "content": "<p>KaggleDatasets().get_gcs_path('tensorflow-great-barrier-reef')</p>\n<p>you should have add dataset in your kaggle notebook first, and then put the dataset name as parameter!</p>\n<p>then copy and paste the gcs from kaggle to colab.</p>",
      "rawMarkdown": "KaggleDatasets().get_gcs_path('tensorflow-great-barrier-reef')\n\nyou should have add dataset in your kaggle notebook first, and then put the dataset name as parameter!\n\nthen copy and paste the gcs from kaggle to colab.",
      "votes": null
    },
    {
      "id": "1699875",
      "postDate": "02/21/2022 13:54:01",
      "content": "<p>Okay, I wiil try that, thanks!</p>",
      "rawMarkdown": "Okay, I wiil try that, thanks!",
      "votes": null
    },
    {
      "id": "1699888",
      "postDate": "02/21/2022 14:02:59",
      "content": "<p>Hi, this is my screenshot, it still don't work:<br>\n<img src=\"https://www.cdnjson.com/images/2022/02/21/-2022-02-21-215959.png\" alt=\"-2022-02-21-215959.png\"></p>",
      "rawMarkdown": "Hi, this is my screenshot, it still don't work:\n![-2022-02-21-215959.png](https://www.cdnjson.com/images/2022/02/21/-2022-02-21-215959.png)",
      "votes": null
    },
    {
      "id": "1701238",
      "postDate": "02/22/2022 15:50:39",
      "content": "<p>To use jpeg files rather than existing TFRecords, have a look at this tutorial provided by Kaggle for the Flower identification competition using a TPU.</p>\n<p><a href=\"https://www.kaggle.com/ryanholbrook/create-your-first-submission\" target=\"_blank\">https://www.kaggle.com/ryanholbrook/create-your-first-submission</a></p>",
      "rawMarkdown": "To use jpeg files rather than existing TFRecords, have a look at this tutorial provided by Kaggle for the Flower identification competition using a TPU.\n\nhttps://www.kaggle.com/ryanholbrook/create-your-first-submission",
      "votes": null
    },
    {
      "id": "1701848",
      "postDate": "02/23/2022 05:31:42",
      "content": "<p>can't view the screenshot.  but I guess that It is the Image from PIL not working. <br>\nyou probably need tf.io to do the job.</p>",
      "rawMarkdown": "can't view the screenshot.  but I guess that It is the Image from PIL not working. \nyou probably need tf.io to do the job.",
      "votes": null
    },
    {
      "id": "1701938",
      "postDate": "02/23/2022 07:17:36",
      "content": "<p>But I want to use PyTorch.😂</p>",
      "rawMarkdown": "But I want to use PyTorch.😂",
      "votes": null
    },
    {
      "id": "1704162",
      "postDate": "02/25/2022 08:46:34",
      "content": "<p>then probably need to download or copy to gdrive.</p>",
      "rawMarkdown": "then probably need to download or copy to gdrive.",
      "votes": null
    },
    {
      "id": "1730414",
      "postDate": "03/21/2022 08:18:51",
      "content": "<p>how can i get gcs_path</p>",
      "rawMarkdown": "how can i get gcs_path",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1699814,
      "author_name": "dragonzhang",
      "author_url": "",
      "post_date": "02/21/2022 13:12:12",
      "content": "<p>there are many shared notebooks. search for it.</p>\n<p>you get gcs_path,  then  paste and use it in colab to access the dataset.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1699841,
          "author_name": "rainfalllove",
          "author_url": "",
          "post_date": "02/21/2022 13:33:27",
          "content": "<p>Yes, I found some notebooks, but still got this, I'll keep looking , Thakns! </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1699851,
          "author_name": "dragonzhang",
          "author_url": "",
          "post_date": "02/21/2022 13:38:46",
          "content": "<p>gcs path only valid for some time, if It doesn't work, you should get the new gcs_path from kaggle platform and then update it on colab.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1699857,
          "author_name": "rainfalllove",
          "author_url": "",
          "post_date": "02/21/2022 13:43:21",
          "content": "<p>I use this two line:</p>\n<pre><code>from kaggle_datasets import KaggleDatasets\nGCS_PATH = KaggleDatasets().get_gcs_path()\n</code></pre>\n<p>to get the latest gcs_path.</p>\n<p>but still  <strong>FileNotFound</strong>😭</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1699870,
          "author_name": "dragonzhang",
          "author_url": "",
          "post_date": "02/21/2022 13:48:38",
          "content": "<p>KaggleDatasets().get_gcs_path('tensorflow-great-barrier-reef')</p>\n<p>you should have add dataset in your kaggle notebook first, and then put the dataset name as parameter!</p>\n<p>then copy and paste the gcs from kaggle to colab.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1699875,
          "author_name": "rainfalllove",
          "author_url": "",
          "post_date": "02/21/2022 13:54:01",
          "content": "<p>Okay, I wiil try that, thanks!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1699888,
          "author_name": "rainfalllove",
          "author_url": "",
          "post_date": "02/21/2022 14:02:59",
          "content": "<p>Hi, this is my screenshot, it still don't work:<br>\n<img src=\"https://www.cdnjson.com/images/2022/02/21/-2022-02-21-215959.png\" alt=\"-2022-02-21-215959.png\"></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1701848,
          "author_name": "dragonzhang",
          "author_url": "",
          "post_date": "02/23/2022 05:31:42",
          "content": "<p>can't view the screenshot.  but I guess that It is the Image from PIL not working. <br>\nyou probably need tf.io to do the job.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1701938,
          "author_name": "rainfalllove",
          "author_url": "",
          "post_date": "02/23/2022 07:17:36",
          "content": "<p>But I want to use PyTorch.😂</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1704162,
          "author_name": "dragonzhang",
          "author_url": "",
          "post_date": "02/25/2022 08:46:34",
          "content": "<p>then probably need to download or copy to gdrive.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1730414,
          "author_name": "",
          "author_url": "",
          "post_date": "03/21/2022 08:18:51",
          "content": "<p>how can i get gcs_path</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1701238,
      "author_name": "mutantspore",
      "author_url": "",
      "post_date": "02/22/2022 15:50:39",
      "content": "<p>To use jpeg files rather than existing TFRecords, have a look at this tutorial provided by Kaggle for the Flower identification competition using a TPU.</p>\n<p><a href=\"https://www.kaggle.com/ryanholbrook/create-your-first-submission\" target=\"_blank\">https://www.kaggle.com/ryanholbrook/create-your-first-submission</a></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1699797": "Hi, I want to use kaggle dataset in colab, but due to disk limitation, I have to load data by GCS：\n```\nfrom kaggle_datasets import KaggleDatasets\nGCS_PATH = KaggleDatasets().get_gcs_path()\n```\n\nwhen I ran:\n```\npd.read_csv(GCS_PATH+'/sample_submission.csv')\n```\neverything is ok\n\nbut when I ran:\n```\nfrom PIL import Image\nImage.open(GCS_PATH+'/test_images/000110707af0ba.jpg')\n```\nerror occurs:\n```\nFileNotFoundError: [Errno 2] No such file or directory: 'gs://kds-ec2a2e61dd5345d807d47adc26eb93dcc160dd3b4da4084f45a859f0/test_images/000110707af0ba.jpg'\n```\nHow to fix this? Thank you !",
    "1699814": "there are many shared notebooks. search for it.\n\nyou get gcs_path,  then  paste and use it in colab to access the dataset.",
    "1699841": "Yes, I found some notebooks, but still got this, I'll keep looking , Thakns!",
    "1699851": "gcs path only valid for some time, if It doesn't work, you should get the new gcs_path from kaggle platform and then update it on colab.",
    "1699857": "I use this two line:\n```\nfrom kaggle_datasets import KaggleDatasets\nGCS_PATH = KaggleDatasets().get_gcs_path()\n```\nto get the latest gcs_path.\n\nbut still  **FileNotFound**😭",
    "1699870": "KaggleDatasets().get_gcs_path('tensorflow-great-barrier-reef')\n\nyou should have add dataset in your kaggle notebook first, and then put the dataset name as parameter!\n\nthen copy and paste the gcs from kaggle to colab.",
    "1699875": "Okay, I wiil try that, thanks!",
    "1699888": "Hi, this is my screenshot, it still don't work:\n![-2022-02-21-215959.png](https://www.cdnjson.com/images/2022/02/21/-2022-02-21-215959.png)",
    "1701238": "To use jpeg files rather than existing TFRecords, have a look at this tutorial provided by Kaggle for the Flower identification competition using a TPU.\n\nhttps://www.kaggle.com/ryanholbrook/create-your-first-submission",
    "1701848": "can't view the screenshot.  but I guess that It is the Image from PIL not working. \nyou probably need tf.io to do the job.",
    "1701938": "But I want to use PyTorch.😂",
    "1704162": "then probably need to download or copy to gdrive.",
    "1730414": "how can i get gcs_path"
  },
  "source": "meta"
}