Hey guys, I'm a new Kaggler and I use Google Colab to import datasets to my Google Drive, however Google Colab only allows a total of 107 GB of data to be downloaded and this dataset is about 150 GB. I also lack local disk space so downloading the data set locally is not an option. How should I go about downloading datasets?
", "messages": [ { "id": 1244426, "postDate": "2021-03-19T01:32:44.837Z", "content": "Well the dataset is about 150 GB but it contains both TF records and png files. You probably don't need both of them so the solution would be to download only files you need (either TF records or png files depending on your pipeline).
\nI couldn't find a way how to download only a specific folder with Kaggle API or multiple files using wildcard but if you are downloading TF records you can probably just download the files one by one (see the Kaggle API github)
\nOr you can connect from your colab notebook to the competition Google Cloud Storage directly and download anything you need. To do so you have to first run this in the competition notebook here on kaggle:
\nfrom kaggle_datasets import KaggleDatasets\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('hpa-single-cell-image-classification')\nprint (GCS_DS_PATH)\nwhich will print out something like this
\ngs://kds-a7dd8b09d52950714bee1818d843af117accc125d4d289d9afcebfab
\n(the path does change from time to time so if it stops working in your colab just rerun the code in your kaggle notebook)
and then in your colab you can connect to the GCS and download what you want
\n!mkdir /home/data/train\n!mkdir /home/data/test\n\n!pip install gcsfs \nimport gcsfs\n\nfs = gcsfs.GCSFileSystem(project='my-project')\ndata_gcs = 'kds-a7dd8b09d52950714bee1818d843af117accc125d4d289d9afcebfab'\nfs.get(f\"{data_gcs}/train/*.png\", '/home/data/train/')\nfs.get(f\"{data_gcs}/test/*.png\", '/home/data/test/')\nI am not in this competition but i successfully used this in a different one so it should work just fine.
\nHope it helps. Good luck
Hi @michaln and @mgurevich,
\nThank you for sharing your steps
\nI am just trying to understand what is the reason for downloading the dataset in Google drive.
\nYou can train the model from Google Cloud Storage directly.
\nYou just need to change the path of the images.
\nHere @tt0721 and me trying to find the best way to do it.
\nhttps://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/229962
All the best
\nHappy Learning 😀
Hi @faisalalsrheed,
\nI am no expert in TF/PyTorch, GCP/GCS so you can correct me if i am wrong :-)
\ni definitely agree that if you are using Tensorflow you don't need to download the dataset at all and you can just access the data from GCS directly. I am not sure thought that the same works for PyTorch (i am not aware that there is a direct way and quick google search didn't give me a solution :-D ).
\nAnother reason that comes to my mind is that you might want to preprocess the data in some way and loading the files directly from GCS might not be supported by the library you want to use. Typically you would have to download the file from GCS anyway or read it as a bytes and then give it to the library.
\nI believe recently i saw a topic here on Kaggle that there are some fees in case you are accessing GCS that is in different region then your server on GCP. If you are using colab then this does not concern you but if you use GCP then you should keep that in mind. (no idea what the fees are, when they are charged and if they would be charged in this case, i might be totally wrong again here).
\nI would say in general the Tensorflow makes it nice and simple so if you want to use the original data and your pipeline is in TF then there might be no reason to download the dataset at all. If you are not using TF though then downloading it locally might be your best choice. Especially if you would read the data repeatedly as accessing the data from GCS might be much slower (as far as i know TF prefetches the data in parallel to avoid possible speed issues).
\nIn my case when using PyTorch i open images using cv2 or PIL and neither of them can read data from GCS directly. I would have to connect to the bucket, read the file as bytes and pass the bytes to the library. I am too lazy for that and i don't want to deal with GCS at all so i rather download the data. If there is a better and more efficient way I would love to know :-D
", "rawMarkdown": "Hi @faisalalsrheed,\n\nI am no expert in TF/PyTorch, GCP/GCS so you can correct me if i am wrong :-)\n\ni definitely agree that if you are using Tensorflow you don't need to download the dataset at all and you can just access the data from GCS directly. I am not sure thought that the same works for PyTorch (i am not aware that there is a direct way and quick google search didn't give me a solution :-D ).\n\nAnother reason that comes to my mind is that you might want to preprocess the data in some way and loading the files directly from GCS might not be supported by the library you want to use. Typically you would have to download the file from GCS anyway or read it as a bytes and then give it to the library. \n\nI believe recently i saw a topic here on Kaggle that there are some fees in case you are accessing GCS that is in different region then your server on GCP. If you are using colab then this does not concern you but if you use GCP then you should keep that in mind. (no idea what the fees are, when they are charged and if they would be charged in this case, i might be totally wrong again here).\n\nI would say in general the Tensorflow makes it nice and simple so if you want to use the original data and your pipeline is in TF then there might be no reason to download the dataset at all. If you are not using TF though then downloading it locally might be your best choice. Especially if you would read the data repeatedly as accessing the data from GCS might be much slower (as far as i know TF prefetches the data in parallel to avoid possible speed issues).\n\nIn my case when using PyTorch i open images using cv2 or PIL and neither of them can read data from GCS directly. I would have to connect to the bucket, read the file as bytes and pass the bytes to the library. I am too lazy for that and i don't want to deal with GCS at all so i rather download the data. If there is a better and more efficient way I would love to know :-D", "votes": 1 }, { "id": 1262682, "postDate": "2021-04-04T15:39:18.763Z", "content": "Thank you @michaln,
\nI see the point now.
\nYou raised some great questions. I will keep looking for the answers.
\nall the best :)
Hello! I ended up with following method:
\n1) Download dataset to a local machine
\n2) Remove all unnecessary data (in my case it was tfrecords)
\n3) Resize images to a size suitable for your needs (2048x2048 or 1024x1024 or even less)
\n4) Make a zip archive with all these data
\n5) Buy some space on google drive. Also buy colab pro (after that you will have 150Gb on a colab machine)
\n6) mount gdrive to your colab machine and unzip the archive:
from google.colab import drive\ndrive.mount('/content/drive')\n!unzip -q /content/drive/MyDrive/kaggle/train_1024_1024.zip -d data\nin my case unzip took about 20 minutes. After that you can train your models on colab
", "rawMarkdown": "Hello! I ended up with following method:\n1) Download dataset to a local machine\n2) Remove all unnecessary data (in my case it was tfrecords)\n3) Resize images to a size suitable for your needs (2048x2048 or 1024x1024 or even less)\n4) Make a zip archive with all these data\n5) Buy some space on google drive. Also buy colab pro (after that you will have 150Gb on a colab machine)\n6) mount gdrive to your colab machine and unzip the archive:\n```\nfrom google.colab import drive\ndrive.mount('/content/drive')\n!unzip -q /content/drive/MyDrive/kaggle/train_1024_1024.zip -d data\n```\nin my case unzip took about 20 minutes. After that you can train your models on colab", "votes": 3, "replies": [ { "id": 1261100, "postDate": "2021-04-02T17:34:42.690Z", "content": "Hello! Are you training model without problem on google drive and colab?
\nWhen I try to train model using cell-wise images, I get a drive error when loading the image.(I believe this is caused by having too many images.)
\nI would like to know how to solve it if you know. (Or you don’t use cell-wise images?)
Thanks!
", "rawMarkdown": "Hello! Are you training model without problem on google drive and colab?\n\nWhen I try to train model using cell-wise images, I get a drive error when loading the image.(I believe this is caused by having too many images.)\nI would like to know how to solve it if you know. (Or you don’t use cell-wise images?)\n\nThanks!" }, { "id": 1263157, "postDate": "2021-04-05T06:14:14.760Z", "content": "Hello, I had a problem with google drive when I tried to download each image separately - I think in that way you exceed qouta pretty fast.
\nTry following approach:
\n1) zip all you images
\n2) put archive on gdrive
\n3) download archive to your colab instance
\n4) extract images to the folder at colab instance (not gdrive)
Thank you so much!
", "rawMarkdown": "Thank you so much!" }, { "id": 1263341, "postDate": "2021-04-05T10:29:48.023Z", "content": "Hi @mgurevich
\nThnak you for sharing.
\nYou can access files in Drive in a number of ways, including:\nMounting your Google Drive in the runtime's virtual machine\nUsing a wrapper around the API such as PyDrive\nUsing the native REST API\nhttps://colab.research.google.com/notebooks/io.ipynb
\nWhich one did you use?
", "rawMarkdown": "Hi @mgurevich \n\nThnak you for sharing.\n\n```\nYou can access files in Drive in a number of ways, including:\nMounting your Google Drive in the runtime's virtual machine\nUsing a wrapper around the API such as PyDrive\nUsing the native REST API\n```\nhttps://colab.research.google.com/notebooks/io.ipynb\n\nWhich one did you use?" }, { "id": 1263433, "postDate": "2021-04-05T12:13:27.680Z", "content": "Hello, I mounted Google Drive using following snippet:
\nfrom google.colab import drive\ndrive.mount('/content/drive', force_remount=True)\nHey guys, I'm a new Kaggler and I use Google Colab to import datasets to my Google Drive, however Google Colab only allows a total of 107 GB of data to be downloaded and this dataset is about 150 GB. I also lack local disk space so downloading the data set locally is not an option. How should I go about downloading datasets?
", "rawMarkdown": "Hey guys, I'm a new Kaggler and I use Google Colab to import datasets to my Google Drive, however Google Colab only allows a total of 107 GB of data to be downloaded and this dataset is about 150 GB. I also lack local disk space so downloading the data set locally is not an option. How should I go about downloading datasets?", "votes": 2 } ], "comments": [ { "id": 1244426, "author_name": "Michal", "author_url": "", "post_date": "2021-03-19T01:32:44.837000", "content": "Well the dataset is about 150 GB but it contains both TF records and png files. You probably don't need both of them so the solution would be to download only files you need (either TF records or png files depending on your pipeline).
\nI couldn't find a way how to download only a specific folder with Kaggle API or multiple files using wildcard but if you are downloading TF records you can probably just download the files one by one (see the Kaggle API github)
\nOr you can connect from your colab notebook to the competition Google Cloud Storage directly and download anything you need. To do so you have to first run this in the competition notebook here on kaggle:
\nfrom kaggle_datasets import KaggleDatasets\nGCS_DS_PATH = KaggleDatasets().get_gcs_path('hpa-single-cell-image-classification')\nprint (GCS_DS_PATH)\nwhich will print out something like this
\ngs://kds-a7dd8b09d52950714bee1818d843af117accc125d4d289d9afcebfab
\n(the path does change from time to time so if it stops working in your colab just rerun the code in your kaggle notebook)
and then in your colab you can connect to the GCS and download what you want
\n!mkdir /home/data/train\n!mkdir /home/data/test\n\n!pip install gcsfs \nimport gcsfs\n\nfs = gcsfs.GCSFileSystem(project='my-project')\ndata_gcs = 'kds-a7dd8b09d52950714bee1818d843af117accc125d4d289d9afcebfab'\nfs.get(f\"{data_gcs}/train/*.png\", '/home/data/train/')\nfs.get(f\"{data_gcs}/test/*.png\", '/home/data/test/')\nI am not in this competition but i successfully used this in a different one so it should work just fine.
\nHope it helps. Good luck
Hi @michaln and @mgurevich,
\nThank you for sharing your steps
\nI am just trying to understand what is the reason for downloading the dataset in Google drive.
\nYou can train the model from Google Cloud Storage directly.
\nYou just need to change the path of the images.
\nHere @tt0721 and me trying to find the best way to do it.
\nhttps://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/229962
All the best
\nHappy Learning 😀
Hi @faisalalsrheed,
\nI am no expert in TF/PyTorch, GCP/GCS so you can correct me if i am wrong :-)
\ni definitely agree that if you are using Tensorflow you don't need to download the dataset at all and you can just access the data from GCS directly. I am not sure thought that the same works for PyTorch (i am not aware that there is a direct way and quick google search didn't give me a solution :-D ).
\nAnother reason that comes to my mind is that you might want to preprocess the data in some way and loading the files directly from GCS might not be supported by the library you want to use. Typically you would have to download the file from GCS anyway or read it as a bytes and then give it to the library.
\nI believe recently i saw a topic here on Kaggle that there are some fees in case you are accessing GCS that is in different region then your server on GCP. If you are using colab then this does not concern you but if you use GCP then you should keep that in mind. (no idea what the fees are, when they are charged and if they would be charged in this case, i might be totally wrong again here).
\nI would say in general the Tensorflow makes it nice and simple so if you want to use the original data and your pipeline is in TF then there might be no reason to download the dataset at all. If you are not using TF though then downloading it locally might be your best choice. Especially if you would read the data repeatedly as accessing the data from GCS might be much slower (as far as i know TF prefetches the data in parallel to avoid possible speed issues).
\nIn my case when using PyTorch i open images using cv2 or PIL and neither of them can read data from GCS directly. I would have to connect to the bucket, read the file as bytes and pass the bytes to the library. I am too lazy for that and i don't want to deal with GCS at all so i rather download the data. If there is a better and more efficient way I would love to know :-D
", "votes": 1, "replies": [] }, { "id": 1262682, "author_name": "Faisal Alsrheed", "author_url": "", "post_date": "2021-04-04T15:39:18.763000", "content": "Thank you @michaln,
\nI see the point now.
\nYou raised some great questions. I will keep looking for the answers.
\nall the best :)
Hello! I ended up with following method:
\n1) Download dataset to a local machine
\n2) Remove all unnecessary data (in my case it was tfrecords)
\n3) Resize images to a size suitable for your needs (2048x2048 or 1024x1024 or even less)
\n4) Make a zip archive with all these data
\n5) Buy some space on google drive. Also buy colab pro (after that you will have 150Gb on a colab machine)
\n6) mount gdrive to your colab machine and unzip the archive:
from google.colab import drive\ndrive.mount('/content/drive')\n!unzip -q /content/drive/MyDrive/kaggle/train_1024_1024.zip -d data\nin my case unzip took about 20 minutes. After that you can train your models on colab
", "votes": 3, "replies": [ { "id": 1261100, "author_name": "tsujino", "author_url": "", "post_date": "2021-04-02T17:34:42.690000", "content": "Hello! Are you training model without problem on google drive and colab?
\nWhen I try to train model using cell-wise images, I get a drive error when loading the image.(I believe this is caused by having too many images.)
\nI would like to know how to solve it if you know. (Or you don’t use cell-wise images?)
Thanks!
", "votes": 0, "replies": [] }, { "id": 1263157, "author_name": "Mikhail Gurevich", "author_url": "", "post_date": "2021-04-05T06:14:14.760000", "content": "Hello, I had a problem with google drive when I tried to download each image separately - I think in that way you exceed qouta pretty fast.
\nTry following approach:
\n1) zip all you images
\n2) put archive on gdrive
\n3) download archive to your colab instance
\n4) extract images to the folder at colab instance (not gdrive)
Thank you so much!
", "votes": 0, "replies": [] }, { "id": 1263341, "author_name": "Faisal Alsrheed", "author_url": "", "post_date": "2021-04-05T10:29:48.023000", "content": "Hi @mgurevich
\nThnak you for sharing.
\nYou can access files in Drive in a number of ways, including:\nMounting your Google Drive in the runtime's virtual machine\nUsing a wrapper around the API such as PyDrive\nUsing the native REST API\nhttps://colab.research.google.com/notebooks/io.ipynb
\nWhich one did you use?
", "votes": 0, "replies": [] }, { "id": 1263433, "author_name": "Mikhail Gurevich", "author_url": "", "post_date": "2021-04-05T12:13:27.680000", "content": "Hello, I mounted Google Drive using following snippet:
\nfrom google.colab import drive\ndrive.mount('/content/drive', force_remount=True)\n