{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# This is a simple script to move Kaggle files to Amazon Web Services (AWS) S3 bucket\n\n1. If you look at the first code cell you will see that you will need the following data: \"aws_access_key_id\", \"aws_secret_access_key\", \"aws_region\" -- below will show you how to obtain these pieces of data. \n\n2. **If you do not have an AWS account go here and sign up:**\nhttps://aws.amazon.com/ Then create an AWS S3 account and bucket. \nAWS S3 storage service: https://aws.amazon.com/s3/\n\n3. If (or once) you have an AWS account, [Follow these instructions in order](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) to obtain your access key and secret access key. Save those as a CSV so you can fill in the Secrets info.\n\n4. Then you need to select a region. Select the one closest to your location:\nhttps://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-available-regions The closest region to me is: us-east-2. Feel free to use. \n\n5. You will need to go to the menu above and click Add-ons>Secrets - to add all the information you gathered above. \n\nSee example - label is each of these: \"aws_access_key_id\", \"aws_secret_access_key\", \"aws_region\" and the value obtained above.\n[![Screen-Shot-2022-02-19-at-1-19-18-PM.png](https://i.postimg.cc/Z5BWsY88/Screen-Shot-2022-02-19-at-1-19-18-PM.png)](https://postimg.cc/68K6yX1q)\n\n**If you are a startup you can get free credits here:**\nhttps://aws.amazon.com/startups/\nhttps://aws.amazon.com/activate/\n\n**Depending on the size of your files it can take an hour or two. Enjoy!**","metadata":{}},{"cell_type":"code","source":"from kaggle_secrets import UserSecretsClient\nuser_secrets = UserSecretsClient()\naws_id = user_secrets.get_secret(\"aws_access_key_id\")\naws_key = user_secrets.get_secret(\"aws_secret_access_key\")\naws_region = user_secrets.get_secret(\"aws_region\")","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-02-19T16:17:01.296063Z","iopub.execute_input":"2022-02-19T16:17:01.296996Z","iopub.status.idle":"2022-02-19T16:17:02.301044Z","shell.execute_reply.started":"2022-02-19T16:17:01.296944Z","shell.execute_reply":"2022-02-19T16:17:02.300332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#If your Secret worked properly you should see a print\n# of your current region below\n\naws_region","metadata":{"execution":{"iopub.status.busy":"2022-02-19T16:17:38.90187Z","iopub.execute_input":"2022-02-19T16:17:38.902827Z","iopub.status.idle":"2022-02-19T16:17:38.912386Z","shell.execute_reply.started":"2022-02-19T16:17:38.902763Z","shell.execute_reply":"2022-02-19T16:17:38.911224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Below you will see:\n\nbucket_name = ''.join(['happywhales', str(uuid.uuid4())])\n\nPut in whatever you want as a name instead of happywhales. Don't add / or - as this can break the API. ","metadata":{}},{"cell_type":"code","source":"import boto3\nimport uuid\nimport os\n\ns3 = boto3.resource(\n    's3',\n    aws_access_key_id=aws_id,\n    aws_secret_access_key=aws_key,\n)\n\ns3_client = boto3.client(\n    's3',\n    aws_access_key_id=aws_id,\n    aws_secret_access_key=aws_key,\n)\nbucket_name = ''.join(['happywhales', str(uuid.uuid4())])\nbucket_response = s3_client.create_bucket(Bucket=bucket_name)\n\nresponse = s3_client.list_buckets()\n\n# Output the bucket names\n#print('Existing buckets:')\n#for bucket in response['Buckets']:\n#    print(f'  {bucket[\"Name\"]}')\n\n\nfor dirname, _, filenames in os.walk('/kaggle/input/'):\n    for filename in filenames:\n#        print(os.path.join(dirname, filename))\n        file_name = os.path.join(dirname, filename)\n        response = s3_client.upload_file(file_name, bucket_name, filename)\n\n# Let us get some feedback: the list of the objects of our Bucket is very verbose and we can\n# check that everything is OK\n\ns3_client.list_objects_v2(Bucket=bucket_name)","metadata":{"execution":{"iopub.status.busy":"2022-02-19T16:19:02.93911Z","iopub.execute_input":"2022-02-19T16:19:02.939454Z","iopub.status.idle":"2022-02-19T16:19:02.99349Z","shell.execute_reply.started":"2022-02-19T16:19:02.939417Z","shell.execute_reply":"2022-02-19T16:19:02.992065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And that is it! If you have questions ask in comments below and feel free to upvote if you liked or found this useful!","metadata":{}}]}