If anyone is having issues with getting training Images(512gb) to your Google Cloud platform Bucket. Follow these directions(I am using Linux):
\n\nIf anyone is having issues with getting training Images(512gb) to your Google Cloud platform Bucket. Follow these directions(I am using Linux):
\n\nGreat help. Btw, how many hours did you take to download entire Images(512gb) ? For me, download speed is about 250KiB/s .... it took an hour to download just 1GB.
", "rawMarkdown": "Great help. Btw, how many hours did you take to download entire Images(512gb) ? For me, download speed is about 250KiB/s .... it took an hour to download just 1GB.", "votes": null }, { "id": "390676", "postDate": "09/20/2018 16:14:36", "content": "Just to be clear, you have to download the dataset to your local hard drive anyway, right? Or is it possible to download straight to the bucket?
", "rawMarkdown": "Just to be clear, you have to download the dataset to your local hard drive anyway, right? Or is it possible to download straight to the bucket?", "votes": null }, { "id": "391446", "postDate": "09/21/2018 20:51:22", "content": "it took me almost two-three nights.. :)
", "rawMarkdown": "it took me almost two-three nights.. :)", "votes": null }, { "id": "391448", "postDate": "09/21/2018 20:53:01", "content": "There is and I've tried to transfer data from AWS to Google Bucket (there are instructions on Data page) but it did not work well. It kept getting error and transfer would get canceled. Downloading straight from AWS to local hard drive seems to be the only method that works reliably from my experience.
", "rawMarkdown": "There is and I've tried to transfer data from AWS to Google Bucket (there are instructions on Data page) but it did not work well. It kept getting error and transfer would get canceled. Downloading straight from AWS to local hard drive seems to be the only method that works reliably from my experience.", "votes": null }, { "id": "392139", "postDate": "09/23/2018 05:12:40", "content": "You can directly transfer data from AWS to Google Bucket, following above instruction. I tried that way. However, I was not able to adjust upload threshold and download speed was too slow. So I downloaded the dataset to local hard drive.
", "rawMarkdown": "You can directly transfer data from AWS to Google Bucket, following above instruction. I tried that way. However, I was not able to adjust upload threshold and download speed was too slow. So I downloaded the dataset to local hard drive.", "votes": null }, { "id": "392140", "postDate": "09/23/2018 05:13:54", "content": "Thanks. I tried to download dataset directly to my local drive and it works.
", "rawMarkdown": "Thanks. I tried to download dataset directly to my local drive and it works.", "votes": null }, { "id": "393907", "postDate": "09/26/2018 03:35:48", "content": "Is it better not to bother with google buckets at all, and just add a persistent disk to your google cloud VM? Would that be better for performance?
", "rawMarkdown": "Is it better not to bother with google buckets at all, and just add a persistent disk to your google cloud VM? Would that be better for performance?", "votes": null }, { "id": "394518", "postDate": "09/27/2018 02:28:21", "content": "Thank you for the short instructions, but I'm getting some slow transfer speeds (<200KiB/s) when uploading the data to my GCP Bucket. It's not feasible for me to download to my local machine then upload to my bucket like some have suggested; upload speed is still too slow and not enough extra storage space.
\n\nIs there a way to directly upload to my GCP bucket from AWS with faster speeds? I've been going through a lot of the documentation and I haven't come across anything that worked for me (I'm pretty new to datasets above 10GB). I've tried Google's transfer tool but I get an Invalid access key. Make sure the access key for your S3 bucket is correct. error even after creating an IAM for AWS (AmazonS3FullAccess permission).
Any help would of course be greatly appreciated!
", "rawMarkdown": "Thank you for the short instructions, but I'm getting some slow transfer speeds (<200KiB/s) when uploading the data to my GCP Bucket. It's not feasible for me to download to my local machine then upload to my bucket like some have suggested; upload speed is still too slow and not enough extra storage space.\n\nIs there a way to directly upload to my GCP bucket from AWS with faster speeds? I've been going through a lot of the documentation and I haven't come across anything that worked for me (I'm pretty new to datasets above 10GB). I've tried Google's [transfer tool][1] but I get an `Invalid access key. Make sure the access key for your S3 bucket is correct.` error even after creating an IAM for AWS (AmazonS3FullAccess permission).\n\nAny help would of course be greatly appreciated!\n\n [1]: https://console.cloud.google.com/storage/transfer/", "votes": null }, { "id": "403547", "postDate": "10/13/2018 21:52:10", "content": "there is also an uploaded ans shared to public gs resource: gs://inclusive-images_challenge\nhttps://console.cloud.google.com/storage/browser/inclusive-images_challenge
", "rawMarkdown": "there is also an uploaded ans shared to public gs resource: gs://inclusive-images_challenge\nhttps://console.cloud.google.com/storage/browser/inclusive-images_challenge", "votes": null }, { "id": "408885", "postDate": "10/23/2018 15:18:54", "content": "Thank you for posting. I didn't know ..
", "rawMarkdown": "Thank you for posting. I didn't know ..", "votes": null }, { "id": "409320", "postDate": "10/24/2018 05:40:34", "content": "Hi Andrei, you mean train dataset are publicly available on this site: public gs resource: gs://inclusive-images_challenge? If so, could you please walkthrough how to access this site? Thanks!
", "rawMarkdown": "Hi Andrei, you mean train dataset are publicly available on this site: **public gs resource: gs://inclusive-images_challenge**? If so, could you please walkthrough how to access this site? Thanks!", "votes": null }, { "id": "410991", "postDate": "10/27/2018 05:13:18", "content": "@ancestral_recall: Exactly, the shared resource is funded by Kaggle. And will be online and available until promo credits expire or end what happens sooner. You may either use url paths https://console.cloud.google.com/storage/browser/inclusive-images_challenge or gsutil tool to access the dataset. Moreover, feel free to find more details at the discussion thread: https://www.kaggle.com/c/inclusive-images-challenge/discussion/69192 . There you will find a basic walkthrough on how to start with nvidia digits and with labeling images. The thread reflects the progress in training and classification on the current date.
", "rawMarkdown": "ancestral_recall: Exactly, the shared resource is funded by Kaggle. And will be online and available until promo credits expire or end what happens sooner. You may either use url paths https://console.cloud.google.com/storage/browser/inclusive-images_challenge or gsutil tool to access the dataset. Moreover, feel free to find more details at the discussion thread: https://www.kaggle.com/c/inclusive-images-challenge/discussion/69192 . There you will find a basic walkthrough on how to start with nvidia digits and with labeling images. The thread reflects the progress in training and classification on the current date.", "votes": null } ], "comments": [ { "id": 389980, "author_name": "seungwon", "author_url": "", "post_date": "09/19/2018 14:49:05", "content": "Great help. Btw, how many hours did you take to download entire Images(512gb) ? For me, download speed is about 250KiB/s .... it took an hour to download just 1GB.
", "votes": null, "replies": [ { "id": 391446, "author_name": "lemonista", "author_url": "", "post_date": "09/21/2018 20:51:22", "content": "it took me almost two-three nights.. :)
", "votes": null, "replies": [] }, { "id": 392140, "author_name": "seungwon", "author_url": "", "post_date": "09/23/2018 05:13:54", "content": "Thanks. I tried to download dataset directly to my local drive and it works.
", "votes": null, "replies": [] } ] }, { "id": 390676, "author_name": "lroszkowiak", "author_url": "", "post_date": "09/20/2018 16:14:36", "content": "Just to be clear, you have to download the dataset to your local hard drive anyway, right? Or is it possible to download straight to the bucket?
", "votes": null, "replies": [ { "id": 391448, "author_name": "lemonista", "author_url": "", "post_date": "09/21/2018 20:53:01", "content": "There is and I've tried to transfer data from AWS to Google Bucket (there are instructions on Data page) but it did not work well. It kept getting error and transfer would get canceled. Downloading straight from AWS to local hard drive seems to be the only method that works reliably from my experience.
", "votes": null, "replies": [] }, { "id": 392139, "author_name": "seungwon", "author_url": "", "post_date": "09/23/2018 05:12:40", "content": "You can directly transfer data from AWS to Google Bucket, following above instruction. I tried that way. However, I was not able to adjust upload threshold and download speed was too slow. So I downloaded the dataset to local hard drive.
", "votes": null, "replies": [] } ] }, { "id": 393907, "author_name": "kk1694", "author_url": "", "post_date": "09/26/2018 03:35:48", "content": "Is it better not to bother with google buckets at all, and just add a persistent disk to your google cloud VM? Would that be better for performance?
", "votes": null, "replies": [] }, { "id": 394518, "author_name": "mrgeislinger", "author_url": "", "post_date": "09/27/2018 02:28:21", "content": "Thank you for the short instructions, but I'm getting some slow transfer speeds (<200KiB/s) when uploading the data to my GCP Bucket. It's not feasible for me to download to my local machine then upload to my bucket like some have suggested; upload speed is still too slow and not enough extra storage space.
\n\nIs there a way to directly upload to my GCP bucket from AWS with faster speeds? I've been going through a lot of the documentation and I haven't come across anything that worked for me (I'm pretty new to datasets above 10GB). I've tried Google's transfer tool but I get an Invalid access key. Make sure the access key for your S3 bucket is correct. error even after creating an IAM for AWS (AmazonS3FullAccess permission).
Any help would of course be greatly appreciated!
", "votes": null, "replies": [ { "id": 403547, "author_name": "avolod", "author_url": "", "post_date": "10/13/2018 21:52:10", "content": "there is also an uploaded ans shared to public gs resource: gs://inclusive-images_challenge\nhttps://console.cloud.google.com/storage/browser/inclusive-images_challenge
", "votes": null, "replies": [] }, { "id": 408885, "author_name": "reachkishore", "author_url": "", "post_date": "10/23/2018 15:18:54", "content": "Thank you for posting. I didn't know ..
", "votes": null, "replies": [] }, { "id": 409320, "author_name": "", "author_url": "", "post_date": "10/24/2018 05:40:34", "content": "Hi Andrei, you mean train dataset are publicly available on this site: public gs resource: gs://inclusive-images_challenge? If so, could you please walkthrough how to access this site? Thanks!
", "votes": null, "replies": [] }, { "id": 410991, "author_name": "avolod", "author_url": "", "post_date": "10/27/2018 05:13:18", "content": "@ancestral_recall: Exactly, the shared resource is funded by Kaggle. And will be online and available until promo credits expire or end what happens sooner. You may either use url paths https://console.cloud.google.com/storage/browser/inclusive-images_challenge or gsutil tool to access the dataset. Moreover, feel free to find more details at the discussion thread: https://www.kaggle.com/c/inclusive-images-challenge/discussion/69192 . There you will find a basic walkthrough on how to start with nvidia digits and with labeling images. The thread reflects the progress in training and classification on the current date.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "389838": "If anyone is having issues with getting training Images(512gb) to your Google Cloud platform Bucket. Follow these directions(I am using Linux):\n\n 1. Mount your Google Cloud Platform Bucket to your file System.\n 2. Install gcsfuse, follow these directions: https://github.com/GoogleCloudPlatform/gcsfuse/blob/master/docs/installing.md\n 3. When you try to run gcsfuse(**gcsfuse my-bucket /path/to/mount**) you might get this error: \n“could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.”\n 4. run this command to obtain user access credentials **gcloud auth application-default login** and click the link to login.\n 5. run **gcsfuse my-bucket /path/to/mount** ,now it should successfully mount.\n 6. Now that your bucket is mounted you can run:\n**aws s3 --no-sign-request sync s3://open-images-dataset/train [target_dir/train]**\nyou should replace **[target_dir/train]** with your **/path/to/mount**. The images will then be uploaded to your Google Cloud platform bucket!", "389980": "Great help. Btw, how many hours did you take to download entire Images(512gb) ? For me, download speed is about 250KiB/s .... it took an hour to download just 1GB.", "390676": "Just to be clear, you have to download the dataset to your local hard drive anyway, right? Or is it possible to download straight to the bucket?", "391446": "it took me almost two-three nights.. :)", "391448": "There is and I've tried to transfer data from AWS to Google Bucket (there are instructions on Data page) but it did not work well. It kept getting error and transfer would get canceled. Downloading straight from AWS to local hard drive seems to be the only method that works reliably from my experience.", "392139": "You can directly transfer data from AWS to Google Bucket, following above instruction. I tried that way. However, I was not able to adjust upload threshold and download speed was too slow. So I downloaded the dataset to local hard drive.", "392140": "Thanks. I tried to download dataset directly to my local drive and it works.", "393907": "Is it better not to bother with google buckets at all, and just add a persistent disk to your google cloud VM? Would that be better for performance?", "394518": "Thank you for the short instructions, but I'm getting some slow transfer speeds (<200KiB/s) when uploading the data to my GCP Bucket. It's not feasible for me to download to my local machine then upload to my bucket like some have suggested; upload speed is still too slow and not enough extra storage space.\n\nIs there a way to directly upload to my GCP bucket from AWS with faster speeds? I've been going through a lot of the documentation and I haven't come across anything that worked for me (I'm pretty new to datasets above 10GB). I've tried Google's [transfer tool][1] but I get an `Invalid access key. Make sure the access key for your S3 bucket is correct.` error even after creating an IAM for AWS (AmazonS3FullAccess permission).\n\nAny help would of course be greatly appreciated!\n\n [1]: https://console.cloud.google.com/storage/transfer/", "403547": "there is also an uploaded ans shared to public gs resource: gs://inclusive-images_challenge\nhttps://console.cloud.google.com/storage/browser/inclusive-images_challenge", "408885": "Thank you for posting. I didn't know ..", "409320": "Hi Andrei, you mean train dataset are publicly available on this site: **public gs resource: gs://inclusive-images_challenge**? If so, could you please walkthrough how to access this site? Thanks!", "410991": "ancestral_recall: Exactly, the shared resource is funded by Kaggle. And will be online and available until promo credits expire or end what happens sooner. You may either use url paths https://console.cloud.google.com/storage/browser/inclusive-images_challenge or gsutil tool to access the dataset. Moreover, feel free to find more details at the discussion thread: https://www.kaggle.com/c/inclusive-images-challenge/discussion/69192 . There you will find a basic walkthrough on how to start with nvidia digits and with labeling images. The thread reflects the progress in training and classification on the current date." }, "source": "meta" }