Can anyone let me know the access key to transfer the dataset from Amazon s3 bucket to google cloud bucket!Thanks in advance.
", "messages": [ { "id": 389925, "postDate": "2018-09-19T13:19:36.653Z", "content": "Can anyone let me know the access key to transfer the dataset from Amazon s3 bucket to google cloud bucket!Thanks in advance.
", "rawMarkdown": "Can anyone let me know the access key to transfer the dataset from Amazon s3 bucket to google cloud bucket!Thanks in advance.", "votes": 1 }, { "id": 390706, "postDate": "2018-09-20T17:17:04.550Z", "content": "Is there any way I can transfer the dataset into my google cloud bucket instead of downloading 513 gb onto my local system??\nMoverover download seems to fail a lot of times! Please help!
", "rawMarkdown": "Is there any way I can transfer the dataset into my google cloud bucket instead of downloading 513 gb onto my local system??\nMoverover download seems to fail a lot of times! Please help!", "replies": [ { "id": 391452, "postDate": "2018-09-21T20:56:49.367Z", "content": "https://github.com/cvdfoundation/open-images-dataset#download-images-with-bounding-boxes-annotations
\n\nCheck out \"Download Full Dataset With Google Storage Transfer\". It did not work for me tho
", "rawMarkdown": "https://github.com/cvdfoundation/open-images-dataset#download-images-with-bounding-boxes-annotations\n\nCheck out \"Download Full Dataset With Google Storage Transfer\". It did not work for me tho", "votes": 2 } ] }, { "id": 389953, "postDate": "2018-09-19T14:03:09.817Z", "content": "You can get access key and secret key by creating AWS account and go to AWS service called IAM(https://aws.amazon.com/iam/?nc1=h_ls) Then create new user and assign access to be AmazonCloudDirectoryFullAccess. But careful, Transfer only work for gsutil.
", "rawMarkdown": "You can get access key and secret key by creating AWS account and go to AWS service called IAM(https://aws.amazon.com/iam/?nc1=h_ls) Then create new user and assign access to be AmazonCloudDirectoryFullAccess. But careful, Transfer only work for gsutil.", "replies": [ { "id": 394533, "postDate": "2018-09-27T03:06:35.327Z", "content": "Do you know if we can use Google's transfer tool/page with this permission to get the AWS S3 bucket into Google Cloud bucket? I only get the error: Invalid access key. Make sure the access key for your S3 bucket is correct.
Otherwise if gsutil is the way to go, do you have any documentation/instructions to point to on how to do this for an AWS S3 bucket to a Google Cloud bucket?
", "rawMarkdown": "Do you know if we can use [Google's transfer tool/page][1] with this permission to get the AWS S3 bucket into Google Cloud bucket? I only get the error: `Invalid access key. Make sure the access key for your S3 bucket is correct.`\n\nOtherwise if **gsutil** is the way to go, do you have any documentation/instructions to point to on how to do this for an AWS S3 bucket to a Google Cloud bucket?\n\n [1]: https://console.cloud.google.com/storage/transfer/" }, { "id": 394567, "postDate": "2018-09-27T04:57:25.353Z", "content": "I got the same problem when I used Google's transfer tool but work for gsutil.
\n\nFor my opinion ,There is 2 ways.
\n\n2.Use awscli install to your system. This way you can directly transfer to your bucket via mounted folder and it is too slow. So my solution I create new instance with 1000gb disk size and then transfer it to bucket. Hope it helps:)
", "rawMarkdown": "I got the same problem when I used Google's transfer tool but work for gsutil.\n\nFor my opinion ,There is 2 ways.\n\n 1. Use gsutil tool to transfer directly from s3 to gs using this command. \n\" gsutil -m -o GSUtil:parallel_composite_upload_threshold=150M cp -r s3://open-images-dataset/train gs://your_bucket_name \"\nbut note that by using gsutil you need to config your access key and secret key to make transfer work and AWS will charge you after you exceed transfer quota.\n\n2.Use awscli install to your system. This way you can directly transfer to your bucket via mounted folder and it is too slow. So my solution I create new instance with 1000gb disk size and then transfer it to bucket. Hope it helps:)", "votes": 1 }, { "id": 394811, "postDate": "2018-09-27T13:51:31.143Z", "content": "Reading the documentation, I didn't realize that you could use gsutil in this to transfer from an AWS S3! I'll have to check this out after work. You mentioned that AWS has a transfer quota. Do you know where that's stated? I don't want to expect one price and get another.
\n\nSomeone else also suggested creating an instance then transferring the data to a bucket; would this mean you'd have to do the cost of the storage on the VM instance and from the bucket?
\n\nThanks for the help!
", "rawMarkdown": "Reading the documentation, I didn't realize that you could use **gsutil** in this to transfer from an AWS S3! I'll have to check this out after work. You mentioned that AWS has a transfer quota. Do you know where that's stated? I don't want to expect one price and get another.\n\nSomeone else also suggested [creating an instance then transferring the data to a bucket][1]; would this mean you'd have to do the cost of the storage on the VM instance *and* from the bucket?\n\nThanks for the help!\n\n [1]: https://www.kaggle.com/c/inclusive-images-challenge/discussion/66955#394753" }, { "id": 395168, "postDate": "2018-09-28T04:35:23.873Z", "content": "Just wanted to do a quick update on this thread that the command from Vadepals above is working for me! It's still not as fast as I'd want (~5 days to transfer +500GB) but maybe that is due to my instance an other factors. I'll have to play with it / do some more research.
\n\nOne part I'm sure about is the pricing for transferring data from an AWS S3 bucket. I think this is the reference I'm looking for but I'm still unsure: https://aws.amazon.com/s3/pricing/#Data_Transfer_Pricing. Depending on the region, it looks like I'm looking at $11-$80 to transfer the full dataset needed for the competition.
\n\nI had to put in my access key & secret key using this guide: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html. I assume that this will end up charging me. I don't know if there is a free method using gsutil since I would get an error without my AWS access key & secret key (IAM).
", "rawMarkdown": "Just wanted to do a quick update on this thread that the command from [Vadepals][1] above is working for me! It's still not as fast as I'd want (~5 days to transfer +500GB) but maybe that is due to my instance an other factors. I'll have to play with it / do some more research.\n\nOne part I'm sure about is the pricing for transferring data from an AWS S3 bucket. I think this is the reference I'm looking for but I'm still unsure: https://aws.amazon.com/s3/pricing/#Data_Transfer_Pricing. Depending on the region, it looks like I'm looking at $11-$80 to transfer the full dataset needed for the competition. \n\nI had to put in my access key & secret key using this guide: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html. I assume that this will end up charging me. I don't know if there is a free method using **gsutil** since I would get an error without my AWS access key & secret key (IAM).\n\n\n [1]: https://www.kaggle.com/haha2536" } ] }, { "id": 389928, "postDate": "2018-09-19T13:33:19.507Z", "content": "Hi Rajath,
\n\nYou will have to first download the data from Amazon S3 bucket and then transfer it to a Google Cloud bucket.\nSee instructions here:\nhttps://www.kaggle.com/c/inclusive-images-challenge#Data-Download-&-Getting-Started
", "rawMarkdown": "Hi Rajath,\n\nYou will have to first download the data from Amazon S3 bucket and then transfer it to a Google Cloud bucket.\nSee instructions here:\nhttps://www.kaggle.com/c/inclusive-images-challenge#Data-Download-&-Getting-Started" }, { "id": 399098, "postDate": "2018-10-05T07:14:28.847Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 390706, "author_name": "Rajath S", "author_url": "", "post_date": "2018-09-20T17:17:04.550000", "content": "Is there any way I can transfer the dataset into my google cloud bucket instead of downloading 513 gb onto my local system??\nMoverover download seems to fail a lot of times! Please help!
", "votes": 0, "replies": [ { "id": 391452, "author_name": "Yejin Kim", "author_url": "", "post_date": "2018-09-21T20:56:49.367000", "content": "https://github.com/cvdfoundation/open-images-dataset#download-images-with-bounding-boxes-annotations
\n\nCheck out \"Download Full Dataset With Google Storage Transfer\". It did not work for me tho
", "votes": 2, "replies": [] } ] }, { "id": 389953, "author_name": "Vadepals", "author_url": "", "post_date": "2018-09-19T14:03:09.817000", "content": "You can get access key and secret key by creating AWS account and go to AWS service called IAM(https://aws.amazon.com/iam/?nc1=h_ls) Then create new user and assign access to be AmazonCloudDirectoryFullAccess. But careful, Transfer only work for gsutil.
", "votes": 0, "replies": [ { "id": 394533, "author_name": "Victor Geislinger", "author_url": "", "post_date": "2018-09-27T03:06:35.327000", "content": "Do you know if we can use Google's transfer tool/page with this permission to get the AWS S3 bucket into Google Cloud bucket? I only get the error: Invalid access key. Make sure the access key for your S3 bucket is correct.
Otherwise if gsutil is the way to go, do you have any documentation/instructions to point to on how to do this for an AWS S3 bucket to a Google Cloud bucket?
", "votes": 0, "replies": [] }, { "id": 394567, "author_name": "Vadepals", "author_url": "", "post_date": "2018-09-27T04:57:25.353000", "content": "I got the same problem when I used Google's transfer tool but work for gsutil.
\n\nFor my opinion ,There is 2 ways.
\n\n2.Use awscli install to your system. This way you can directly transfer to your bucket via mounted folder and it is too slow. So my solution I create new instance with 1000gb disk size and then transfer it to bucket. Hope it helps:)
", "votes": 1, "replies": [] }, { "id": 394811, "author_name": "Victor Geislinger", "author_url": "", "post_date": "2018-09-27T13:51:31.143000", "content": "Reading the documentation, I didn't realize that you could use gsutil in this to transfer from an AWS S3! I'll have to check this out after work. You mentioned that AWS has a transfer quota. Do you know where that's stated? I don't want to expect one price and get another.
\n\nSomeone else also suggested creating an instance then transferring the data to a bucket; would this mean you'd have to do the cost of the storage on the VM instance and from the bucket?
\n\nThanks for the help!
", "votes": 0, "replies": [] }, { "id": 395168, "author_name": "Victor Geislinger", "author_url": "", "post_date": "2018-09-28T04:35:23.873000", "content": "Just wanted to do a quick update on this thread that the command from Vadepals above is working for me! It's still not as fast as I'd want (~5 days to transfer +500GB) but maybe that is due to my instance an other factors. I'll have to play with it / do some more research.
\n\nOne part I'm sure about is the pricing for transferring data from an AWS S3 bucket. I think this is the reference I'm looking for but I'm still unsure: https://aws.amazon.com/s3/pricing/#Data_Transfer_Pricing. Depending on the region, it looks like I'm looking at $11-$80 to transfer the full dataset needed for the competition.
\n\nI had to put in my access key & secret key using this guide: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html. I assume that this will end up charging me. I don't know if there is a free method using gsutil since I would get an error without my AWS access key & secret key (IAM).
", "votes": 0, "replies": [] } ] }, { "id": 389928, "author_name": "Pallavi", "author_url": "", "post_date": "2018-09-19T13:33:19.507000", "content": "Hi Rajath,
\n\nYou will have to first download the data from Amazon S3 bucket and then transfer it to a Google Cloud bucket.\nSee instructions here:\nhttps://www.kaggle.com/c/inclusive-images-challenge#Data-Download-&-Getting-Started
", "votes": 0, "replies": [] }, { "id": 399098, "author_name": "", "author_url": "", "post_date": "2018-10-05T07:14:28.847000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "389925": "Can anyone let me know the access key to transfer the dataset from Amazon s3 bucket to google cloud bucket!Thanks in advance.", "390706": "Is there any way I can transfer the dataset into my google cloud bucket instead of downloading 513 gb onto my local system??\nMoverover download seems to fail a lot of times! Please help!", "389953": "You can get access key and secret key by creating AWS account and go to AWS service called IAM(https://aws.amazon.com/iam/?nc1=h_ls) Then create new user and assign access to be AmazonCloudDirectoryFullAccess. But careful, Transfer only work for gsutil.", "389928": "Hi Rajath,\n\nYou will have to first download the data from Amazon S3 bucket and then transfer it to a Google Cloud bucket.\nSee instructions here:\nhttps://www.kaggle.com/c/inclusive-images-challenge#Data-Download-&-Getting-Started", "399098": "" } }