I want to use this Kaggle competition big dataset in my google collab notebook, how can I do that without downloading it to my collab notebook. I know about this one(https://stackoverflow.com/questions/49310470/using-kaggle-datasets-in-google-colab), but the problem is it\"ll download the data(correct me if I'm wrong) to my collab notebook and I don't have enough internet data to spend on that, is there any way to fetch the data without downloading, maybe something like a URL or an API just to read without downloading.
", "messages": [ { "id": "1073201", "postDate": "11/09/2020 09:01:08", "content": "I want to use this Kaggle competition big dataset in my google collab notebook, how can I do that without downloading it to my collab notebook. I know about this one(https://stackoverflow.com/questions/49310470/using-kaggle-datasets-in-google-colab), but the problem is it\"ll download the data(correct me if I'm wrong) to my collab notebook and I don't have enough internet data to spend on that, is there any way to fetch the data without downloading, maybe something like a URL or an API just to read without downloading.
", "rawMarkdown": "I want to use this Kaggle competition big dataset in my google collab notebook, how can I do that without downloading it to my collab notebook. I know about this one(https://stackoverflow.com/questions/49310470/using-kaggle-datasets-in-google-colab), but the problem is it\"ll download the data(correct me if I'm wrong) to my collab notebook and I don't have enough internet data to spend on that, is there any way to fetch the data without downloading, maybe something like a URL or an API just to read without downloading.", "votes": null }, { "id": "1073366", "postDate": "11/09/2020 13:12:38", "content": "Bro whether you use the api, url or download the data, it will have to travel from the source to your notebook thereby consuming your internet. Sorry but I cant think of a solution. Kaggle gpu quota over?
", "rawMarkdown": "Bro whether you use the api, url or download the data, it will have to travel from the source to your notebook thereby consuming your internet. Sorry but I cant think of a solution. Kaggle gpu quota over?", "votes": null }, { "id": "1073443", "postDate": "11/09/2020 14:33:46", "content": "Not necessarily true, right? TF's can read data from buckets when we use TPU's for e.g.
", "rawMarkdown": "Not necessarily true, right? TF's can read data from buckets when we use TPU's for e.g.", "votes": null }, { "id": "1073464", "postDate": "11/09/2020 14:44:59", "content": "Is \"I don't have enough internet data\" a problem? Because the Kaggle data is hosted on Google Cloud Storage, and when you download to your notebook it is still hosted on GCS. Unless you download it to your machine then you are using your internet.
", "rawMarkdown": "Is \"I don't have enough internet data\" a problem? Because the Kaggle data is hosted on Google Cloud Storage, and when you download to your notebook it is still hosted on GCS. Unless you download it to your machine then you are using your internet.", "votes": null }, { "id": "1073499", "postDate": "11/09/2020 15:19:10", "content": "That doesn't transport data from GCS to TPU memory?
", "rawMarkdown": "That doesn't transport data from GCS to TPU memory?", "votes": null }, { "id": "1073681", "postDate": "11/09/2020 19:44:47", "content": "But if you are using Colab TPU, the data won't flow through your local PC.
", "rawMarkdown": "But if you are using Colab TPU, the data won't flow through your local PC.", "votes": null }, { "id": "1073685", "postDate": "11/09/2020 19:55:27", "content": "Just download your kaggle API json file named \"kaggle.json\" by following the process in this link and place it in your \"Colab Notebooks\" folder in \"Google Drive\".
\nMount your Google drive in Colab and run the following snippet.
\n!pip install -q kaggle\n\n%cd /content/drive/My\\ Drive/Colab\\ Notebooks/\n%cp -av kaggle.json /content/\n%cd /content/\n\n!mkdir ~/.kaggle\n!cp kaggle.json ~/.kaggle/\n!chmod 600 ~/.kaggle/kaggle.json \n!kaggle competitions download -c riiid-test-answer-prediction\n\n!mkdir train\n!unzip train.csv.zip -d /content/\nIt'll download directly to the Colab VM (Hence no load on your internet connection). Takes about a min or so.
", "rawMarkdown": "Just download your kaggle API json file named \"kaggle.json\" by following the process in this [link](https://github.com/Kaggle/kaggle-api#:~:text=To%20use%20the%20Kaggle%20API,file%20containing%20your%20API%20credentials.) and place it in your \"Colab Notebooks\" folder in \"Google Drive\".\n\nMount your Google drive in Colab and run the following snippet.\n\n```\n!pip install -q kaggle\n\n%cd /content/drive/My\\ Drive/Colab\\ Notebooks/\n%cp -av kaggle.json /content/\n%cd /content/\n\n!mkdir ~/.kaggle\n!cp kaggle.json ~/.kaggle/\n!chmod 600 ~/.kaggle/kaggle.json \n!kaggle competitions download -c riiid-test-answer-prediction\n\n!mkdir train\n!unzip train.csv.zip -d /content/\n```\n\n\nIt'll download directly to the Colab VM (Hence no load on your internet connection). Takes about a min or so.", "votes": null } ], "comments": [ { "id": 1073366, "author_name": "abhimanyud", "author_url": "", "post_date": "11/09/2020 13:12:38", "content": "Bro whether you use the api, url or download the data, it will have to travel from the source to your notebook thereby consuming your internet. Sorry but I cant think of a solution. Kaggle gpu quota over?
", "votes": null, "replies": [ { "id": 1073443, "author_name": "adityaecdrid", "author_url": "", "post_date": "11/09/2020 14:33:46", "content": "Not necessarily true, right? TF's can read data from buckets when we use TPU's for e.g.
", "votes": null, "replies": [] }, { "id": 1073499, "author_name": "abhimanyud", "author_url": "", "post_date": "11/09/2020 15:19:10", "content": "That doesn't transport data from GCS to TPU memory?
", "votes": null, "replies": [] }, { "id": 1073681, "author_name": "yihdarshieh", "author_url": "", "post_date": "11/09/2020 19:44:47", "content": "But if you are using Colab TPU, the data won't flow through your local PC.
", "votes": null, "replies": [] } ] }, { "id": 1073464, "author_name": "aeryss", "author_url": "", "post_date": "11/09/2020 14:44:59", "content": "Is \"I don't have enough internet data\" a problem? Because the Kaggle data is hosted on Google Cloud Storage, and when you download to your notebook it is still hosted on GCS. Unless you download it to your machine then you are using your internet.
", "votes": null, "replies": [] }, { "id": 1073685, "author_name": "abdurrafae", "author_url": "", "post_date": "11/09/2020 19:55:27", "content": "Just download your kaggle API json file named \"kaggle.json\" by following the process in this link and place it in your \"Colab Notebooks\" folder in \"Google Drive\".
\nMount your Google drive in Colab and run the following snippet.
\n!pip install -q kaggle\n\n%cd /content/drive/My\\ Drive/Colab\\ Notebooks/\n%cp -av kaggle.json /content/\n%cd /content/\n\n!mkdir ~/.kaggle\n!cp kaggle.json ~/.kaggle/\n!chmod 600 ~/.kaggle/kaggle.json \n!kaggle competitions download -c riiid-test-answer-prediction\n\n!mkdir train\n!unzip train.csv.zip -d /content/\nIt'll download directly to the Colab VM (Hence no load on your internet connection). Takes about a min or so.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1073201": "I want to use this Kaggle competition big dataset in my google collab notebook, how can I do that without downloading it to my collab notebook. I know about this one(https://stackoverflow.com/questions/49310470/using-kaggle-datasets-in-google-colab), but the problem is it\"ll download the data(correct me if I'm wrong) to my collab notebook and I don't have enough internet data to spend on that, is there any way to fetch the data without downloading, maybe something like a URL or an API just to read without downloading.", "1073366": "Bro whether you use the api, url or download the data, it will have to travel from the source to your notebook thereby consuming your internet. Sorry but I cant think of a solution. Kaggle gpu quota over?", "1073443": "Not necessarily true, right? TF's can read data from buckets when we use TPU's for e.g.", "1073464": "Is \"I don't have enough internet data\" a problem? Because the Kaggle data is hosted on Google Cloud Storage, and when you download to your notebook it is still hosted on GCS. Unless you download it to your machine then you are using your internet.", "1073499": "That doesn't transport data from GCS to TPU memory?", "1073681": "But if you are using Colab TPU, the data won't flow through your local PC.", "1073685": "Just download your kaggle API json file named \"kaggle.json\" by following the process in this [link](https://github.com/Kaggle/kaggle-api#:~:text=To%20use%20the%20Kaggle%20API,file%20containing%20your%20API%20credentials.) and place it in your \"Colab Notebooks\" folder in \"Google Drive\".\n\nMount your Google drive in Colab and run the following snippet.\n\n```\n!pip install -q kaggle\n\n%cd /content/drive/My\\ Drive/Colab\\ Notebooks/\n%cp -av kaggle.json /content/\n%cd /content/\n\n!mkdir ~/.kaggle\n!cp kaggle.json ~/.kaggle/\n!chmod 600 ~/.kaggle/kaggle.json \n!kaggle competitions download -c riiid-test-answer-prediction\n\n!mkdir train\n!unzip train.csv.zip -d /content/\n```\n\n\nIt'll download directly to the Colab VM (Hence no load on your internet connection). Takes about a min or so." }, "source": "meta" }