![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F5211312%2Fddf78a64b050835e4d05a5b2d52d22e2%2FAdnotacja%202020-06-17%20145601.png?generation=1592398583806100&alt=media\")
Hello!
\n\nI'm pretty new to deep learning and I am taking part in this competition to learn how to make my own object detection model. I just wonder how to handle so large images dataset. I am using Google Colab with mounted Google Drive. Is there any possibility to download Open Images Dataset directly to Google Drive instead of downloading it to my local directory and then uploading it to Google Drive?
\n\nIf you have any better ideas to use this dataset with Google Colab feel free to share your thoughts ;)
", "messages": [ { "id": 871922, "postDate": "2020-06-02T18:18:30.820Z", "content": "Hello!
\n\nI'm pretty new to deep learning and I am taking part in this competition to learn how to make my own object detection model. I just wonder how to handle so large images dataset. I am using Google Colab with mounted Google Drive. Is there any possibility to download Open Images Dataset directly to Google Drive instead of downloading it to my local directory and then uploading it to Google Drive?
\n\nIf you have any better ideas to use this dataset with Google Colab feel free to share your thoughts ;)
", "rawMarkdown": "Hello!\n\nI'm pretty new to deep learning and I am taking part in this competition to learn how to make my own object detection model. I just wonder how to handle so large images dataset. I am using Google Colab with mounted Google Drive. Is there any possibility to download Open Images Dataset directly to Google Drive instead of downloading it to my local directory and then uploading it to Google Drive? \n\nIf you have any better ideas to use this dataset with Google Colab feel free to share your thoughts ;)", "votes": 4 }, { "id": 1230431, "postDate": "2021-03-08T06:03:57.913Z", "content": "Im always facing problem when i unzip my file on colab. it always show error
\nNo such file or directory: '/content/drive/My_drive/Kaggle/Object Detection/object.zip'
\nbut zip file always in folder and my path name is always right
You have a space in \"Object Detection\" you have to handle it. On the other hand I think you might have \"MyDrive\" and not \"My_drive\", you should check that. And synchronize with drive as @krzysztofsadowski mentioned.
", "rawMarkdown": "You have a space in \"Object Detection\" you have to handle it. On the other hand I think you might have \"MyDrive\" and not \"My_drive\", you should check that. And synchronize with drive as @krzysztofsadowski mentioned.", "votes": 1 }, { "id": 1230630, "postDate": "2021-03-08T09:43:46.057Z", "content": "But the same code run automatically when i run again and again. i think there is a problem in colab
", "rawMarkdown": "But the same code run automatically when i run again and again. i think there is a problem in colab" }, { "id": 1230857, "postDate": "2021-03-08T14:13:16.337Z", "content": "I updated my answer. What do you mean \"run automatically\" ? Just stop your session and restart it and run again.
", "rawMarkdown": "I updated my answer. What do you mean \"run automatically\" ? Just stop your session and restart it and run again.", "votes": -1 } ] }, { "id": 945176, "postDate": "2020-07-25T16:11:21.837Z", "content": "Check out this article to download OpenImages dataset in google drive using Colab.\nDownloading OpenImages Dataset in Google Drive using Colab
", "rawMarkdown": "Check out this article to download OpenImages dataset in google drive using Colab.\n[Downloading OpenImages Dataset in Google Drive using Colab](https://medium.com/@mringupt/downloading-openimages-dataset-in-google-drive-using-colab-132502f0dbde)", "replies": [ { "id": 951757, "postDate": "2020-07-30T11:45:54.863Z", "content": "That is a good documentation. Just to note that for training+validation we need all the classes. Otherwise, for test we can download one or two only.
", "rawMarkdown": "That is a good documentation. Just to note that for training+validation we need all the classes. Otherwise, for test we can download one or two only." } ] }, { "id": 872120, "postDate": "2020-06-02T23:44:04.247Z", "content": "Yes, you can download in Google Colab and then directly move the dataset to Google Drive. There's no need to download in your local directory.
", "rawMarkdown": "Yes, you can download in Google Colab and then directly move the dataset to Google Drive. There's no need to download in your local directory.", "replies": [ { "id": 872376, "postDate": "2020-06-03T06:17:07.890Z", "content": "I found a way to download files to Google Colab by\n!wget link_to_file\nand then moving it to Drive. Thanks!
can you give code snippet for this
", "rawMarkdown": "can you give code snippet for this\n" }, { "id": 890330, "postDate": "2020-06-17T13:00:32.620Z", "content": "You need firstly mount your Drive https://towardsdatascience.com/downloading-datasets-into-google-drive-via-google-colab-bcb1b30b0166 or more simply just by clicking in menu \n
Then in cell run:\nimport os\nos.chdir('/content/drive/My Drive/YOUR_PATH/
and you download files by running in cell:\n!wget https://datasets.appen.com/appen_datasets/open-images/zip_files_copy/train_00.zip
\n\n!wget https://datasets.appen.com/appen_datasets/open-images/zip_files_copy/train_01.zip
\n\n!wget https://datasets.appen.com/appen_datasets/open-images/zip_files_copy/train_02.zip
\n\n!wget https://datasets.appen.com/appen_datasets/open-images/zip_files_copy/train_03.zip
\n\n!wget https://datasets.appen.com/appen_datasets/open-images/zip_files_copy/train_04.zip
\n\n!wget https://datasets.appen.com/appen_datasets/open-images/zip_files_copy/train_05.zip
\n\n!wget https://datasets.appen.com/appen_datasets/open-images/zip_files_copy/train_06.zip
\n\n!wget https://datasets.appen.com/appen_datasets/open-images/zip_files_copy/train_07.zip
\n\n!wget https://datasets.appen.com/appen_datasets/open-images/zip_files_copy/train_08.zip
\n\n!wget https://datasets.appen.com/appen_datasets/open-images/zip_files_copy/validation.zip
\n\n!wget https://storage.googleapis.com/openimages/v5/test-annotations-bbox.csv
\n\n!wget https://datasets.appen.com/appen_datasets/open-images/test-images.csv
\n\n!wget https://storage.googleapis.com/openimages/v6/oidv6-train-annotations-bbox.csv
\n\n!wget https://datasets.figure-eight.com/figure_eight_datasets/open-images/train-images-boxable.csv
\n\n!wget https://storage.googleapis.com/openimages/v5/validation-annotations-bbox.csv
\n\n!wget https://datasets.figure-eight.com/figure_eight_datasets/open-images/validation-images.csv`
\n\nUnzipping:\n!unzip file_name.zip -d \"/content/drive/My Drive/YOUR_PATH/\"
How much storage will this require?
", "rawMarkdown": "How much storage will this require?" }, { "id": 903463, "postDate": "2020-06-26T20:59:50.163Z", "content": "Whole dataset is about 500GBs. After gaining more experience I don't recommend using Google Drive to store this dataset - it's too slow and too unstable.
", "rawMarkdown": "Whole dataset is about 500GBs. After gaining more experience I don't recommend using Google Drive to store this dataset - it's too slow and too unstable." }, { "id": 904362, "postDate": "2020-06-27T15:10:02.463Z", "content": "@krzysztofsadowski Have you found a better alternative?
", "rawMarkdown": "@krzysztofsadowski Have you found a better alternative? " }, { "id": 904391, "postDate": "2020-06-27T15:32:29.720Z", "content": "Google Cloud has 300$ to spend as free tier if you are new user and has really nice Deep Learning VMs. You couldn't spend this cash for GPU, but you could spend this at processors, ram, disk space etc.
", "rawMarkdown": "Google Cloud has 300$ to spend as free tier if you are new user and has really nice Deep Learning VMs. You couldn't spend this cash for GPU, but you could spend this at processors, ram, disk space etc." }, { "id": 960577, "postDate": "2020-08-06T14:12:26.437Z", "content": "Did it work? Because for me it did not, I tried in Kaggle Notebook using GPU accelerator and could not download a single batch of the training dataset. It seems it is too big (about 58Gb each). Are we to use TPU and end the images directly from the source ?
", "rawMarkdown": "Did it work? Because for me it did not, I tried in Kaggle Notebook using GPU accelerator and could not download a single batch of the training dataset. It seems it is too big (about 58Gb each). Are we to use TPU and end the images directly from the source ? " } ] }, { "id": 872375, "postDate": "2020-06-03T06:16:35.720Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1230431, "author_name": "Shivam Kr. Singh", "author_url": "", "post_date": "2021-03-08T06:03:57.913000", "content": "Im always facing problem when i unzip my file on colab. it always show error
\nNo such file or directory: '/content/drive/My_drive/Kaggle/Object Detection/object.zip'
\nbut zip file always in folder and my path name is always right
You have a space in \"Object Detection\" you have to handle it. On the other hand I think you might have \"MyDrive\" and not \"My_drive\", you should check that. And synchronize with drive as @krzysztofsadowski mentioned.
", "votes": 1, "replies": [] }, { "id": 1230630, "author_name": "Shivam Kr. Singh", "author_url": "", "post_date": "2021-03-08T09:43:46.057000", "content": "But the same code run automatically when i run again and again. i think there is a problem in colab
", "votes": 0, "replies": [] }, { "id": 1230857, "author_name": "Catadanna", "author_url": "", "post_date": "2021-03-08T14:13:16.337000", "content": "I updated my answer. What do you mean \"run automatically\" ? Just stop your session and restart it and run again.
", "votes": -1, "replies": [] } ] }, { "id": 945176, "author_name": "Mrinali Gupta", "author_url": "", "post_date": "2020-07-25T16:11:21.837000", "content": "Check out this article to download OpenImages dataset in google drive using Colab.\nDownloading OpenImages Dataset in Google Drive using Colab
", "votes": 0, "replies": [ { "id": 951757, "author_name": "Catadanna", "author_url": "", "post_date": "2020-07-30T11:45:54.863000", "content": "That is a good documentation. Just to note that for training+validation we need all the classes. Otherwise, for test we can download one or two only.
", "votes": 0, "replies": [] } ] }, { "id": 872120, "author_name": "Himanshu Agarwal", "author_url": "", "post_date": "2020-06-02T23:44:04.247000", "content": "Yes, you can download in Google Colab and then directly move the dataset to Google Drive. There's no need to download in your local directory.
", "votes": 0, "replies": [ { "id": 872376, "author_name": "Krzysztof Sadowski", "author_url": "", "post_date": "2020-06-03T06:17:07.890000", "content": "I found a way to download files to Google Colab by\n!wget link_to_file\nand then moving it to Drive. Thanks!
can you give code snippet for this
", "votes": 0, "replies": [] }, { "id": 890330, "author_name": "Krzysztof Sadowski", "author_url": "", "post_date": "2020-06-17T13:00:32.620000", "content": "You need firstly mount your Drive https://towardsdatascience.com/downloading-datasets-into-google-drive-via-google-colab-bcb1b30b0166 or more simply just by clicking in menu \n
Then in cell run:\nimport os\nos.chdir('/content/drive/My Drive/YOUR_PATH/
and you download files by running in cell:\n!wget https://datasets.appen.com/appen_datasets/open-images/zip_files_copy/train_00.zip
\n\n!wget https://datasets.appen.com/appen_datasets/open-images/zip_files_copy/train_01.zip
\n\n!wget https://datasets.appen.com/appen_datasets/open-images/zip_files_copy/train_02.zip
\n\n!wget https://datasets.appen.com/appen_datasets/open-images/zip_files_copy/train_03.zip
\n\n!wget https://datasets.appen.com/appen_datasets/open-images/zip_files_copy/train_04.zip
\n\n!wget https://datasets.appen.com/appen_datasets/open-images/zip_files_copy/train_05.zip
\n\n!wget https://datasets.appen.com/appen_datasets/open-images/zip_files_copy/train_06.zip
\n\n!wget https://datasets.appen.com/appen_datasets/open-images/zip_files_copy/train_07.zip
\n\n!wget https://datasets.appen.com/appen_datasets/open-images/zip_files_copy/train_08.zip
\n\n!wget https://datasets.appen.com/appen_datasets/open-images/zip_files_copy/validation.zip
\n\n!wget https://storage.googleapis.com/openimages/v5/test-annotations-bbox.csv
\n\n!wget https://datasets.appen.com/appen_datasets/open-images/test-images.csv
\n\n!wget https://storage.googleapis.com/openimages/v6/oidv6-train-annotations-bbox.csv
\n\n!wget https://datasets.figure-eight.com/figure_eight_datasets/open-images/train-images-boxable.csv
\n\n!wget https://storage.googleapis.com/openimages/v5/validation-annotations-bbox.csv
\n\n!wget https://datasets.figure-eight.com/figure_eight_datasets/open-images/validation-images.csv`
\n\nUnzipping:\n!unzip file_name.zip -d \"/content/drive/My Drive/YOUR_PATH/\"
How much storage will this require?
", "votes": 0, "replies": [] }, { "id": 903463, "author_name": "Krzysztof Sadowski", "author_url": "", "post_date": "2020-06-26T20:59:50.163000", "content": "Whole dataset is about 500GBs. After gaining more experience I don't recommend using Google Drive to store this dataset - it's too slow and too unstable.
", "votes": 0, "replies": [] }, { "id": 904362, "author_name": "Chris Murphy", "author_url": "", "post_date": "2020-06-27T15:10:02.463000", "content": "@krzysztofsadowski Have you found a better alternative?
", "votes": 0, "replies": [] }, { "id": 904391, "author_name": "Krzysztof Sadowski", "author_url": "", "post_date": "2020-06-27T15:32:29.720000", "content": "Google Cloud has 300$ to spend as free tier if you are new user and has really nice Deep Learning VMs. You couldn't spend this cash for GPU, but you could spend this at processors, ram, disk space etc.
", "votes": 0, "replies": [] }, { "id": 960577, "author_name": "Catadanna", "author_url": "", "post_date": "2020-08-06T14:12:26.437000", "content": "Did it work? Because for me it did not, I tried in Kaggle Notebook using GPU accelerator and could not download a single batch of the training dataset. It seems it is too big (about 58Gb each). Are we to use TPU and end the images directly from the source ?
", "votes": 0, "replies": [] } ] }, { "id": 872375, "author_name": "", "author_url": "", "post_date": "2020-06-03T06:16:35.720000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "871922": "Hello!\n\nI'm pretty new to deep learning and I am taking part in this competition to learn how to make my own object detection model. I just wonder how to handle so large images dataset. I am using Google Colab with mounted Google Drive. Is there any possibility to download Open Images Dataset directly to Google Drive instead of downloading it to my local directory and then uploading it to Google Drive? \n\nIf you have any better ideas to use this dataset with Google Colab feel free to share your thoughts ;)", "1230431": "Im always facing problem when i unzip my file on colab. it always show error \nNo such file or directory: '/content/drive/My_drive/Kaggle/Object Detection/object.zip'\nbut zip file always in folder and my path name is always right", "945176": "Check out this article to download OpenImages dataset in google drive using Colab.\n[Downloading OpenImages Dataset in Google Drive using Colab](https://medium.com/@mringupt/downloading-openimages-dataset-in-google-drive-using-colab-132502f0dbde)", "872120": "Yes, you can download in Google Colab and then directly move the dataset to Google Drive. There's no need to download in your local directory.", "872375": "" } }