When I download dataset to google drive by API command and try to unzip those zip files, some of those data will be missing.
\nI'm using unzip command, on colab notebook.
\nI can confirm that unzipping is working on colab, but I can't see some of the data on drive.
\n(I'm colab pro user and subscribe 200GB strage.)
Does anyone know how to solve it?
\nIs anyone else using colab without problems?
P.S.
\nIt may just be a slow reflection on google drive.
\nI checked after a while and the data showed up fine.
Great job bronze :))))
\nYou must be very good in google colab now :)
", "rawMarkdown": "@tt0721 \n\nGreat job bronze :))))\n\nYou must be very good in google colab now :)", "votes": 1, "replies": [ { "id": 1304953, "postDate": "2021-05-13T03:00:46.270Z", "content": "Thanks! it's thanks to your advice:)
\nGreat job bronze, you too!
Hi @tt0721
\nYou don't need to download the dataset to your google drive.
\nThe dataset already in Google Cloud Storage, I used it for GPU and of course TPU
\nsee this comment from @mgornergoogle
\n\"To access the data, run KaggleDatasets().get_gcs_path() on Kaggle once. It will return a GCS bucket name like 'gs://xyz12234567890'. That's where Kaggle stores the attached dataset.\"
\nhttps://www.kaggle.com/c/flower-classification-with-tpus/discussion/130194
\nSo You can find the path using this code:
\npath_gcs = KaggleDatasets().get_gcs_path('hpa-single-cell-image-classification')
\nprint(path_gcs)
See here :
\nhttps://www.kaggle.com/drcapa/human-protein-atlas-tpu-tutorial
All the best
", "rawMarkdown": "Hi @tt0721 \n\nYou don't need to download the dataset to your google drive.\n\nThe dataset already in Google Cloud Storage, I used it for GPU and of course TPU \n\nsee this comment from @mgornergoogle\n\n\"To access the data, run KaggleDatasets().get_gcs_path() on Kaggle once. It will return a GCS bucket name like 'gs://xyz12234567890'. That's where Kaggle stores the attached dataset.\"\n\nhttps://www.kaggle.com/c/flower-classification-with-tpus/discussion/130194\n\n\nSo You can find the path using this code:\n\npath_gcs = KaggleDatasets().get_gcs_path('hpa-single-cell-image-classification')\nprint(path_gcs)\n\nSee here :\nhttps://www.kaggle.com/drcapa/human-protein-atlas-tpu-tutorial\n\nAll the best ", "votes": 1, "replies": [ { "id": 1260342, "postDate": "2021-04-02T03:23:25.643Z", "content": "Very helpful! I'll try it!
\nThank you very much!
Hi, @faisalalsrheed
\nSorry for the newbie question, but please let me know if you can.
When using GPU, is it possible to load data from GCS directly onto colab?
\nOr do I need to copy it to Drive once?
I was able to get the bucket name, but I don't know what to do from there.
\nI've never used TPU before, so the explanation in the link you provided is not very clear for me:(
\nThanks!
Hi @tt0721,
\nIt is my pleasure 🙂
\nI am learning too about using Google Colab for Kaggle.
\nIt is an important skill to have when you do many competitions at the same time.
You can see here how I used it for past competitions ( it work with GPU and TPU)
\nhttps://colab.research.google.com/drive/1tb73j5SB9oryTZsPGvH2I7rhPrvPUJcV?usp=sharing
Here you can find another way to do it:
\nhttps://www.kaggle.com/amyjang/tensorflow-pneumonia-classification-on-x-rays
see this part of the code :
\nfilenames = tf.io.gfile.glob(str(GCS_PATH + '/chest_xray/train/*/*'))\nfilenames.extend(tf.io.gfile.glob(str(GCS_PATH + '/chest_xray/val/*/*')))\nsee this link to understand the reason why \"TPUs read training data exclusively from GCS (Google Cloud Storage).\"
\nhttps://www.kaggle.com/docs/tpu#tpu3
Please keep sharing with me your questions and Tips &Tricks you find.
\nso in the end, we can share our Tips &Tricks in public Kaggle notebook
like this one
\nhttps://www.kaggle.com/faisalalsrheed/kaggle-to-colab-tips-tricks
Happy Learning 😀
", "rawMarkdown": "Hi @tt0721, \n\nIt is my pleasure 🙂\n\nI am learning too about using Google Colab for Kaggle. \nIt is an important skill to have when you do many competitions at the same time.\n\nYou can see here how I used it for past competitions ( it work with GPU and TPU)\nhttps://colab.research.google.com/drive/1tb73j5SB9oryTZsPGvH2I7rhPrvPUJcV?usp=sharing\n\nHere you can find another way to do it:\nhttps://www.kaggle.com/amyjang/tensorflow-pneumonia-classification-on-x-rays\n\nsee this part of the code :\n\n```\nfilenames = tf.io.gfile.glob(str(GCS_PATH + '/chest_xray/train/*/*'))\nfilenames.extend(tf.io.gfile.glob(str(GCS_PATH + '/chest_xray/val/*/*')))\n```\n\n\nsee this link to understand the reason why \"TPUs read training data exclusively from GCS (Google Cloud Storage).\"\nhttps://www.kaggle.com/docs/tpu#tpu3\n\nPlease keep sharing with me your questions and Tips &Tricks you find.\nso in the end, we can share our Tips &Tricks in public Kaggle notebook \n\nlike this one \nhttps://www.kaggle.com/faisalalsrheed/kaggle-to-colab-tips-tricks\n\nHappy Learning 😀" }, { "id": 1263334, "postDate": "2021-04-05T10:26:54.503Z", "content": "Hi, @faisalalsrheed
\nSorry for taking me so long to get back to you.
\nThank you for teaching me so kindly.
\nI'm learning a lot!
\nI'll check out the link you posted.
Maybe you run into exceeding your storage quota, find the explanation here: https://research.google.com/colaboratory/faq.html
", "rawMarkdown": "Maybe you run into exceeding your storage quota, find the explanation here: https://research.google.com/colaboratory/faq.html", "votes": 1, "replies": [ { "id": 1260341, "postDate": "2021-04-02T03:22:26.487Z", "content": "It's seems that it's not fault in strage quota.
\nIt sometimes happens samely when dataset is small:(
\nThank you for your answer!
When I download dataset to google drive by API command and try to unzip those zip files, some of those data will be missing.
\nI'm using unzip command, on colab notebook.
\nI can confirm that unzipping is working on colab, but I can't see some of the data on drive.
\n(I'm colab pro user and subscribe 200GB strage.)
Does anyone know how to solve it?
\nIs anyone else using colab without problems?
P.S.
\nIt may just be a slow reflection on google drive.
\nI checked after a while and the data showed up fine.
unzip command, on colab notebook.\nI can confirm that unzipping is working on colab, but I can't see some of the data on drive.\n(I'm colab pro user and subscribe 200GB strage.)\n\nDoes anyone know how to solve it?\nIs anyone else using colab without problems?\n\n---\nP.S.\nIt may just be a slow reflection on google drive.\nI checked after a while and the data showed up fine.",
"votes": 2
}
],
"comments": [
{
"id": 1303220,
"author_name": "Faisal Alsrheed",
"author_url": "",
"post_date": "2021-05-12T01:12:14.213000",
"content": "\nGreat job bronze :))))
\nYou must be very good in google colab now :)
", "votes": 1, "replies": [ { "id": 1304953, "author_name": "tsujino", "author_url": "", "post_date": "2021-05-13T03:00:46.270000", "content": "Thanks! it's thanks to your advice:)
\nGreat job bronze, you too!
Hi @tt0721
\nYou don't need to download the dataset to your google drive.
\nThe dataset already in Google Cloud Storage, I used it for GPU and of course TPU
\nsee this comment from @mgornergoogle
\n\"To access the data, run KaggleDatasets().get_gcs_path() on Kaggle once. It will return a GCS bucket name like 'gs://xyz12234567890'. That's where Kaggle stores the attached dataset.\"
\nhttps://www.kaggle.com/c/flower-classification-with-tpus/discussion/130194
\nSo You can find the path using this code:
\npath_gcs = KaggleDatasets().get_gcs_path('hpa-single-cell-image-classification')
\nprint(path_gcs)
See here :
\nhttps://www.kaggle.com/drcapa/human-protein-atlas-tpu-tutorial
All the best
", "votes": 1, "replies": [ { "id": 1260342, "author_name": "tsujino", "author_url": "", "post_date": "2021-04-02T03:23:25.643000", "content": "Very helpful! I'll try it!
\nThank you very much!
Hi, @faisalalsrheed
\nSorry for the newbie question, but please let me know if you can.
When using GPU, is it possible to load data from GCS directly onto colab?
\nOr do I need to copy it to Drive once?
I was able to get the bucket name, but I don't know what to do from there.
\nI've never used TPU before, so the explanation in the link you provided is not very clear for me:(
\nThanks!
Hi @tt0721,
\nIt is my pleasure 🙂
\nI am learning too about using Google Colab for Kaggle.
\nIt is an important skill to have when you do many competitions at the same time.
You can see here how I used it for past competitions ( it work with GPU and TPU)
\nhttps://colab.research.google.com/drive/1tb73j5SB9oryTZsPGvH2I7rhPrvPUJcV?usp=sharing
Here you can find another way to do it:
\nhttps://www.kaggle.com/amyjang/tensorflow-pneumonia-classification-on-x-rays
see this part of the code :
\nfilenames = tf.io.gfile.glob(str(GCS_PATH + '/chest_xray/train/*/*'))\nfilenames.extend(tf.io.gfile.glob(str(GCS_PATH + '/chest_xray/val/*/*')))\nsee this link to understand the reason why \"TPUs read training data exclusively from GCS (Google Cloud Storage).\"
\nhttps://www.kaggle.com/docs/tpu#tpu3
Please keep sharing with me your questions and Tips &Tricks you find.
\nso in the end, we can share our Tips &Tricks in public Kaggle notebook
like this one
\nhttps://www.kaggle.com/faisalalsrheed/kaggle-to-colab-tips-tricks
Happy Learning 😀
", "votes": 0, "replies": [] }, { "id": 1263334, "author_name": "tsujino", "author_url": "", "post_date": "2021-04-05T10:26:54.503000", "content": "Hi, @faisalalsrheed
\nSorry for taking me so long to get back to you.
\nThank you for teaching me so kindly.
\nI'm learning a lot!
\nI'll check out the link you posted.
Maybe you run into exceeding your storage quota, find the explanation here: https://research.google.com/colaboratory/faq.html
", "votes": 1, "replies": [ { "id": 1260341, "author_name": "tsujino", "author_url": "", "post_date": "2021-04-02T03:22:26.487000", "content": "It's seems that it's not fault in strage quota.
\nIt sometimes happens samely when dataset is small:(
\nThank you for your answer!
unzip command, on colab notebook.\nI can confirm that unzipping is working on colab, but I can't see some of the data on drive.\n(I'm colab pro user and subscribe 200GB strage.)\n\nDoes anyone know how to solve it?\nIs anyone else using colab without problems?\n\n---\nP.S.\nIt may just be a slow reflection on google drive.\nI checked after a while and the data showed up fine."
}
}