![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2279276%2F453237172d08efbe8ef215e370a4b916%2Faa.png?generation=1581179644119628&alt=media\")
Hello!
\n\nThis cheat-sheet is to use Kaggle Datasets (thanks to the Kagglers who have spent their time and energy to create those) that usually range <20GB with AWS SageMaker. Note that this discussion totally avoids S3 bucket, rather, uses local SageMaker SSD for storage, which is persistent until you delete the notebook instance. You can also download the complete Deep Fake dataset to SSD (will require to extend the SSD size to ~1TB for download and extraction), but I would suggest using Bucket in that case. Also, note that SSD's charge $0.14/GB/month, so be mindful about it.
\n\n
ml.p2.xlarge for now.![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2279276%2F2e2eb61f02c2dbb425dc451af5e55c89%2Fcc.png?generation=1581179930547863&alt=media\")
unzip command.![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2279276%2F104c0b641c5284a00422c3f5c80dcfb2%2Fbb.png?generation=1581179664415396&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2279276%2F934536506022c740a8d0a451781eca75%2FAnnotation%202020-02-08%20111803.png?generation=1581178866181343&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2279276%2Fec2106c54599c06f3438f1434487b4b8%2FAnnotation%202020-02-08%20112113.png?generation=1581178997560153&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2279276%2F59c86badca9974c331f8a1938995dfc6%2FAnnotation%202020-02-08%20112113.png?generation=1581179062731461&alt=media\")
Now you can use it as a Linux VM.
\n\nJupyter Lab don't forget to cd to SageMaker folder\n\ncd SageMaker/\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2279276%2F99e63c72aab7cdb3d0ee4401cdb6534b%2Fdd.png?generation=1581180022918076&alt=media\")
The links below are generated using CurlWGet (https://chrome.google.com/webstore/detail/curlwget/jmocjfidanebdlinpbcdkcmgdifblncg?hl=en). \n- Install this Chrom extension in your local machine.\n- Start downloading a database to your local machine.\n- Click on the extension logo in the top right of your browser, where you can copy the download link.\n- After that, you can stop the downloading process in the local machine.\n- Please note that download the dataset after you cd SageMaker
Original link for more information: https://www.kaggle.com/dagnelies/deepfake-faces/version/1
\n\n\nwget --header=\"Host: storage.googleapis.com\" --header=\"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36\" --header=\"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\" --header=\"Accept-Language: en-US,en;q=0.9,th;q=0.8,zh-CN;q=0.7,zh;q=0.6\" --header=\"Referer: https://www.kaggle.com/\" --header=\"Cookie: cuntwars_user_id=jilW0PIXf0; _ga=GA1.3.2052114344.1570426301\" --header=\"Connection: keep-alive\" \"https://storage.googleapis.com/kaggle-data-sets/464091/872229/upload/faces_155.zip?GoogleAccessId=web-data@kaggle-161607.iam.gserviceaccount.com&Expires=1581436324&Signature=b88jcLo6oIvIhVVxXM6A9sllTz%2BKF8D3wkE055dk0ltO%2FSnIRkMTlrGvUuODFJY3V4OSo0NBVOVpeKLcAaueOzkRiFAZkLvay6CwqlZa0D%2FhpA65XBje08rsAQDPWxWPQf62ze8zVhHg8q2i7KiUX59QNMmhRws6YsWe6XyRREzQS0CZ3Lum0%2F3bnCgYGws1wm98B9vZQvu5GTUelZpw1ycZFkTXSL1tAEQBPQWJKxAK55VNT7zqpnOkeeWsVeRjckBlMbgE%2BY8dFyjMsf1s5Z8lKjeXkBb3RVxomS6FvtaiIrcksl2RGmZ76dWAJ6aoNJa11E311R1H1D67eNIv%2BQ%3D%3D&response-content-disposition=attachment%3B+filename%3Dfaces_155.zip\" -O \"faces_155.zip\" -c\n
\nwget --header=\"Host: storage.googleapis.com\" --header=\"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36\" --header=\"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\" --header=\"Accept-Language: en-US,en;q=0.9,th;q=0.8,zh-CN;q=0.7,zh;q=0.6\" --header=\"Referer: https://www.kaggle.com/\" --header=\"Cookie: cuntwars_user_id=jilW0PIXf0; _ga=GA1.3.2052114344.1570426301\" --header=\"Connection: keep-alive\" \"https://storage.googleapis.com/kaggle-data-sets/464091/872229/compressed/metadata.csv.zip?GoogleAccessId=web-data@kaggle-161607.iam.gserviceaccount.com&Expires=1581436497&Signature=a3FaRIXdUJDf5T%2Ba1hLucidVob0w%2BNiSKyDip1JPZZ3HkSYw%2B7VwW50vypeiPP0aDVzALQQmMwOD5JQxhFmT0Mllqd44x7S4Fkzn3JhRhiP8jkJUb7Zx8fXEU9wzNGns5GY%2BhndqL%2BdEPD3O3rI42ImkUhpb%2FGrjBifdnGNkqH1jbCP5YUpacO0tRA17HAQ0n0GRJ%2BzMjiI8n%2FxArl6fGsJe8yotmB0yU%2F44Su0OH5NIqMRd5r%2F%2FgghzP6oaTsLsLKK0qhUrh4GkfPxbW3%2BdT0S71tiZzWl4qCx3ZZA4JfBL%2BAc2DzoT36d2vl0%2BNIdBiEDgmO1dAXHu1HXNN8PWDQ%3D%3D&response-content-disposition=attachment%3B+filename%3Dmetadata.csv.zip\" -O \"metadata.csv.zip\" -c\n
Original link for more information: https://www.kaggle.com/unkownhihi/deepfake
\n\n\nwget --header=\"Host: storage.googleapis.com\" --header=\"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36\" --header=\"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\" --header=\"Accept-Language: en-US,en;q=0.9,th;q=0.8,zh-CN;q=0.7,zh;q=0.6\" --header=\"Referer: https://www.kaggle.com/\" --header=\"Cookie: cuntwars_user_id=jilW0PIXf0; _ga=GA1.3.2052114344.1570426301\" --header=\"Connection: keep-alive\" \"https://storage.googleapis.com/kaggle-data-sets/451078/893807/bundle/archive.zip?GoogleAccessId=web-data@kaggle-161607.iam.gserviceaccount.com&Expires=1581436617&Signature=pn%2BfeEb6m5dYz76c4sSMF%2FE2uEY%2FFAzNBfVn3hwckOIc6hLXqWvwpHzS%2FWaCdNFBsqjxIAoPY5vXmmgoEjNB7dTJzQZ5thipQ3P%2Fhjs9RtPU8dCNXWrzMM8iNku%2BNmvdKoHcAWS3DE3LVaO93Z77TfwucSMohOl4AS7Ps%2Bu1AX5SU%2BUQSJ8gKPFsQ2PET11oIyVpYDHffnCakrh3HSUzVfKT6h4DrcP8VXxE57agT5fwdf1aa8kgwao4AfLY8BtL8ZryZf5YM6z8JL%2B7mOD6d8XPZpyBvfny4uOPlphvK6TvWkMjlJRcf8dSZWvKnLOH1B3NGuG12iyYGBY4XZfBqQ%3D%3D&response-content-disposition=attachment%3B+filename%3Ddeepfake.zip\" -O \"deepfake.zip\" -c\n
Hello!
\n\nThis cheat-sheet is to use Kaggle Datasets (thanks to the Kagglers who have spent their time and energy to create those) that usually range <20GB with AWS SageMaker. Note that this discussion totally avoids S3 bucket, rather, uses local SageMaker SSD for storage, which is persistent until you delete the notebook instance. You can also download the complete Deep Fake dataset to SSD (will require to extend the SSD size to ~1TB for download and extraction), but I would suggest using Bucket in that case. Also, note that SSD's charge $0.14/GB/month, so be mindful about it.
\n\n
ml.p2.xlarge for now.
unzip command.
Now you can use it as a Linux VM.
\n\nJupyter Lab don't forget to cd to SageMaker folder\n\ncd SageMaker/\n\nThe links below are generated using CurlWGet (https://chrome.google.com/webstore/detail/curlwget/jmocjfidanebdlinpbcdkcmgdifblncg?hl=en). \n- Install this Chrom extension in your local machine.\n- Start downloading a database to your local machine.\n- Click on the extension logo in the top right of your browser, where you can copy the download link.\n- After that, you can stop the downloading process in the local machine.\n- Please note that download the dataset after you cd SageMaker
Original link for more information: https://www.kaggle.com/dagnelies/deepfake-faces/version/1
\n\n\nwget --header=\"Host: storage.googleapis.com\" --header=\"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36\" --header=\"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\" --header=\"Accept-Language: en-US,en;q=0.9,th;q=0.8,zh-CN;q=0.7,zh;q=0.6\" --header=\"Referer: https://www.kaggle.com/\" --header=\"Cookie: cuntwars_user_id=jilW0PIXf0; _ga=GA1.3.2052114344.1570426301\" --header=\"Connection: keep-alive\" \"https://storage.googleapis.com/kaggle-data-sets/464091/872229/upload/faces_155.zip?GoogleAccessId=web-data@kaggle-161607.iam.gserviceaccount.com&Expires=1581436324&Signature=b88jcLo6oIvIhVVxXM6A9sllTz%2BKF8D3wkE055dk0ltO%2FSnIRkMTlrGvUuODFJY3V4OSo0NBVOVpeKLcAaueOzkRiFAZkLvay6CwqlZa0D%2FhpA65XBje08rsAQDPWxWPQf62ze8zVhHg8q2i7KiUX59QNMmhRws6YsWe6XyRREzQS0CZ3Lum0%2F3bnCgYGws1wm98B9vZQvu5GTUelZpw1ycZFkTXSL1tAEQBPQWJKxAK55VNT7zqpnOkeeWsVeRjckBlMbgE%2BY8dFyjMsf1s5Z8lKjeXkBb3RVxomS6FvtaiIrcksl2RGmZ76dWAJ6aoNJa11E311R1H1D67eNIv%2BQ%3D%3D&response-content-disposition=attachment%3B+filename%3Dfaces_155.zip\" -O \"faces_155.zip\" -c\n
\nwget --header=\"Host: storage.googleapis.com\" --header=\"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36\" --header=\"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\" --header=\"Accept-Language: en-US,en;q=0.9,th;q=0.8,zh-CN;q=0.7,zh;q=0.6\" --header=\"Referer: https://www.kaggle.com/\" --header=\"Cookie: cuntwars_user_id=jilW0PIXf0; _ga=GA1.3.2052114344.1570426301\" --header=\"Connection: keep-alive\" \"https://storage.googleapis.com/kaggle-data-sets/464091/872229/compressed/metadata.csv.zip?GoogleAccessId=web-data@kaggle-161607.iam.gserviceaccount.com&Expires=1581436497&Signature=a3FaRIXdUJDf5T%2Ba1hLucidVob0w%2BNiSKyDip1JPZZ3HkSYw%2B7VwW50vypeiPP0aDVzALQQmMwOD5JQxhFmT0Mllqd44x7S4Fkzn3JhRhiP8jkJUb7Zx8fXEU9wzNGns5GY%2BhndqL%2BdEPD3O3rI42ImkUhpb%2FGrjBifdnGNkqH1jbCP5YUpacO0tRA17HAQ0n0GRJ%2BzMjiI8n%2FxArl6fGsJe8yotmB0yU%2F44Su0OH5NIqMRd5r%2F%2FgghzP6oaTsLsLKK0qhUrh4GkfPxbW3%2BdT0S71tiZzWl4qCx3ZZA4JfBL%2BAc2DzoT36d2vl0%2BNIdBiEDgmO1dAXHu1HXNN8PWDQ%3D%3D&response-content-disposition=attachment%3B+filename%3Dmetadata.csv.zip\" -O \"metadata.csv.zip\" -c\n
Original link for more information: https://www.kaggle.com/unkownhihi/deepfake
\n\n\nwget --header=\"Host: storage.googleapis.com\" --header=\"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36\" --header=\"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\" --header=\"Accept-Language: en-US,en;q=0.9,th;q=0.8,zh-CN;q=0.7,zh;q=0.6\" --header=\"Referer: https://www.kaggle.com/\" --header=\"Cookie: cuntwars_user_id=jilW0PIXf0; _ga=GA1.3.2052114344.1570426301\" --header=\"Connection: keep-alive\" \"https://storage.googleapis.com/kaggle-data-sets/451078/893807/bundle/archive.zip?GoogleAccessId=web-data@kaggle-161607.iam.gserviceaccount.com&Expires=1581436617&Signature=pn%2BfeEb6m5dYz76c4sSMF%2FE2uEY%2FFAzNBfVn3hwckOIc6hLXqWvwpHzS%2FWaCdNFBsqjxIAoPY5vXmmgoEjNB7dTJzQZ5thipQ3P%2Fhjs9RtPU8dCNXWrzMM8iNku%2BNmvdKoHcAWS3DE3LVaO93Z77TfwucSMohOl4AS7Ps%2Bu1AX5SU%2BUQSJ8gKPFsQ2PET11oIyVpYDHffnCakrh3HSUzVfKT6h4DrcP8VXxE57agT5fwdf1aa8kgwao4AfLY8BtL8ZryZf5YM6z8JL%2B7mOD6d8XPZpyBvfny4uOPlphvK6TvWkMjlJRcf8dSZWvKnLOH1B3NGuG12iyYGBY4XZfBqQ%3D%3D&response-content-disposition=attachment%3B+filename%3Ddeepfake.zip\" -O \"deepfake.zip\" -c\n
Thanks for your cheat-sheet! Very clear ๐
", "rawMarkdown": "Thanks for your cheat-sheet! Very clear ๐", "votes": null }, { "id": "739978", "postDate": "02/08/2020 16:57:54", "content": "Thank you @phunghieu ! I am also planning to include your data, but may not be suitable for local SSD due to size. Hopefully if I make a discussion topic on the S3 bucket, I can include that!
", "rawMarkdown": "Thank you @phunghieu ! I am also planning to include your data, but may not be suitable for local SSD due to size. Hopefully if I make a discussion topic on the S3 bucket, I can include that!", "votes": null }, { "id": "739997", "postDate": "02/08/2020 17:23:12", "content": "This sounds great! I definitely need a way to move my datasets to S3 bucket whenever I receive credits from AWS (in case my request is accepted). BTW, thank you for spending time and effort to write this tutorial, it very helpful to me who is unfamiliar with AWS.
", "rawMarkdown": "This sounds great! I definitely need a way to move my datasets to S3 bucket whenever I receive credits from AWS (in case my request is accepted). BTW, thank you for spending time and effort to write this tutorial, it very helpful to me who is unfamiliar with AWS.", "votes": null }, { "id": "739999", "postDate": "02/08/2020 17:25:53", "content": "You are welcome :))
", "rawMarkdown": "You are welcome :))", "votes": null }, { "id": "740599", "postDate": "02/09/2020 15:16:10", "content": "@debanga Thanks for your good cheatsheet on AWS SageMaker which is really helpful for the beginners to the AWS cloud like me .
Can I transfer the data(472GB) toS3 bucket directly without saving to locally.
Hi, @saisrinivasreddy Thanks... I havenโt done that, and still looking for if someone can help :) But if I figure out will post. But, as I mentioned you can always save to the SageMaker SSD and transfer, OR you can create a VM in AWS download the data and transfer.
", "rawMarkdown": "Hi, @saisrinivasreddy Thanks... I havenโt done that, and still looking for if someone can help :) But if I figure out will post. But, as I mentioned you can always save to the SageMaker SSD and transfer, OR you can create a VM in AWS download the data and transfer.", "votes": null }, { "id": "740659", "postDate": "02/09/2020 16:13:45", "content": "As of now i am transfering data from sagemaker to S3 which is a time taking process(wget url --> AWS SageMaker --> S3). But direct process (wget url --> S3) saves time.
\n\nThanks @debanga
", "rawMarkdown": "As of now i am transfering data from sagemaker to S3 which is a time taking process(wget url --> AWS SageMaker --> S3). But direct process (wget url --> S3) saves time.\n\nThanks @debanga", "votes": null }, { "id": "741039", "postDate": "02/10/2020 05:24:46", "content": "Thanks for Sharing
", "rawMarkdown": "Thanks for Sharing", "votes": null }, { "id": "933683", "postDate": "07/17/2020 22:41:42", "content": "Thanks so much. Used it for another dataset download. It worked perfectly.
", "rawMarkdown": "Thanks so much. Used it for another dataset download. It worked perfectly.", "votes": null }, { "id": "1145265", "postDate": "01/09/2021 03:07:17", "content": "Thank You. It was very helpful.๐
", "rawMarkdown": "Thank You. It was very helpful.๐", "votes": null } ], "comments": [ { "id": 933683, "author_name": "meenaliyer", "author_url": "", "post_date": "07/17/2020 22:41:42", "content": "Thanks so much. Used it for another dataset download. It worked perfectly.
", "votes": null, "replies": [] }, { "id": 1145265, "author_name": "mohithgowdahr", "author_url": "", "post_date": "01/09/2021 03:07:17", "content": "Thank You. It was very helpful.๐
", "votes": null, "replies": [] }, { "id": 739977, "author_name": "phunghieu", "author_url": "", "post_date": "02/08/2020 16:55:36", "content": "Thanks for your cheat-sheet! Very clear ๐
", "votes": null, "replies": [ { "id": 739978, "author_name": "debanga", "author_url": "", "post_date": "02/08/2020 16:57:54", "content": "Thank you @phunghieu ! I am also planning to include your data, but may not be suitable for local SSD due to size. Hopefully if I make a discussion topic on the S3 bucket, I can include that!
", "votes": null, "replies": [] }, { "id": 739997, "author_name": "phunghieu", "author_url": "", "post_date": "02/08/2020 17:23:12", "content": "This sounds great! I definitely need a way to move my datasets to S3 bucket whenever I receive credits from AWS (in case my request is accepted). BTW, thank you for spending time and effort to write this tutorial, it very helpful to me who is unfamiliar with AWS.
", "votes": null, "replies": [] }, { "id": 739999, "author_name": "debanga", "author_url": "", "post_date": "02/08/2020 17:25:53", "content": "You are welcome :))
", "votes": null, "replies": [] } ] }, { "id": 740599, "author_name": "saisrinivasreddy", "author_url": "", "post_date": "02/09/2020 15:16:10", "content": "@debanga Thanks for your good cheatsheet on AWS SageMaker which is really helpful for the beginners to the AWS cloud like me .
Can I transfer the data(472GB) toS3 bucket directly without saving to locally.
Hi, @saisrinivasreddy Thanks... I havenโt done that, and still looking for if someone can help :) But if I figure out will post. But, as I mentioned you can always save to the SageMaker SSD and transfer, OR you can create a VM in AWS download the data and transfer.
", "votes": null, "replies": [] }, { "id": 740659, "author_name": "saisrinivasreddy", "author_url": "", "post_date": "02/09/2020 16:13:45", "content": "As of now i am transfering data from sagemaker to S3 which is a time taking process(wget url --> AWS SageMaker --> S3). But direct process (wget url --> S3) saves time.
\n\nThanks @debanga
", "votes": null, "replies": [] } ] }, { "id": 741039, "author_name": "aravindhkv123", "author_url": "", "post_date": "02/10/2020 05:24:46", "content": "Thanks for Sharing
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "739975": "# Using AWS Sagemaker with Existing Kaggle Datasets\nHello!\n\nThis cheat-sheet is to use Kaggle Datasets (thanks to the Kagglers who have spent their time and energy to create those) that usually range <20GB with AWS SageMaker. Note that this discussion totally avoids S3 bucket, rather, uses local SageMaker SSD for storage, which is persistent until you delete the notebook instance. You can also download the complete Deep Fake dataset to SSD (will require to extend the SSD size to ~1TB for download and extraction), but I would suggest using Bucket in that case. Also, note that SSD's charge $0.14/GB/month, so be mindful about it.\n\n## Create a SageMaker instance\n\n\n\n### Notes:\n- The GPU instances are not available by default. Please contact AWS support center. I requested for ```ml.p2.xlarge``` for now.\n\n\n\n- Don't forget to increase the size of the SageMaker SSD, depending on the size of the dataset you are downloading x2 times, as you will also need to extract it! For extracting use ```unzip``` command.\n\n\n\n\n\n## Open Jupyter Lab\n\n\n\n\n## Open terminal from Launcher\n\n\n\n\n\n\nNow you can use it as a Linux VM.\n\n### Note:\n- For your data to be visible in the folder browser in ```Jupyter Lab``` don't forget to ```cd``` to SageMaker folder\n```\ncd SageMaker/\n```\n\n\n\n## Download a Processed Dataset (Kaggle or own) to SageMaker VM\n\nThe links below are generated using ``` CurlWGet``` (https://chrome.google.com/webstore/detail/curlwget/jmocjfidanebdlinpbcdkcmgdifblncg?hl=en). \n- Install this Chrom extension in your local machine.\n- Start downloading a database to your local machine.\n- Click on the extension logo in the top right of your browser, where you can copy the download link.\n- After that, you can stop the downloading process in the local machine.\n- Please note that download the dataset after you ```cd SageMaker```\n\n## Kaggle Dataset 1\n\nOriginal link for more information: https://www.kaggle.com/dagnelies/deepfake-faces/version/1\n\n### Use following command in VM to extract data to the current folder\n\n```\nwget --header=\"Host: storage.googleapis.com\" --header=\"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36\" --header=\"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\" --header=\"Accept-Language: en-US,en;q=0.9,th;q=0.8,zh-CN;q=0.7,zh;q=0.6\" --header=\"Referer: https://www.kaggle.com/\" --header=\"Cookie: cuntwars_user_id=jilW0PIXf0; _ga=GA1.3.2052114344.1570426301\" --header=\"Connection: keep-alive\" \"https://storage.googleapis.com/kaggle-data-sets/464091/872229/upload/faces_155.zip?GoogleAccessId=web-data@kaggle-161607.iam.gserviceaccount.com&Expires=1581436324&Signature=b88jcLo6oIvIhVVxXM6A9sllTz%2BKF8D3wkE055dk0ltO%2FSnIRkMTlrGvUuODFJY3V4OSo0NBVOVpeKLcAaueOzkRiFAZkLvay6CwqlZa0D%2FhpA65XBje08rsAQDPWxWPQf62ze8zVhHg8q2i7KiUX59QNMmhRws6YsWe6XyRREzQS0CZ3Lum0%2F3bnCgYGws1wm98B9vZQvu5GTUelZpw1ycZFkTXSL1tAEQBPQWJKxAK55VNT7zqpnOkeeWsVeRjckBlMbgE%2BY8dFyjMsf1s5Z8lKjeXkBb3RVxomS6FvtaiIrcksl2RGmZ76dWAJ6aoNJa11E311R1H1D67eNIv%2BQ%3D%3D&response-content-disposition=attachment%3B+filename%3Dfaces_155.zip\" -O \"faces_155.zip\" -c\n```\n### Use following command in VM to extract metadata to the current folder\n\n```\nwget --header=\"Host: storage.googleapis.com\" --header=\"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36\" --header=\"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\" --header=\"Accept-Language: en-US,en;q=0.9,th;q=0.8,zh-CN;q=0.7,zh;q=0.6\" --header=\"Referer: https://www.kaggle.com/\" --header=\"Cookie: cuntwars_user_id=jilW0PIXf0; _ga=GA1.3.2052114344.1570426301\" --header=\"Connection: keep-alive\" \"https://storage.googleapis.com/kaggle-data-sets/464091/872229/compressed/metadata.csv.zip?GoogleAccessId=web-data@kaggle-161607.iam.gserviceaccount.com&Expires=1581436497&Signature=a3FaRIXdUJDf5T%2Ba1hLucidVob0w%2BNiSKyDip1JPZZ3HkSYw%2B7VwW50vypeiPP0aDVzALQQmMwOD5JQxhFmT0Mllqd44x7S4Fkzn3JhRhiP8jkJUb7Zx8fXEU9wzNGns5GY%2BhndqL%2BdEPD3O3rI42ImkUhpb%2FGrjBifdnGNkqH1jbCP5YUpacO0tRA17HAQ0n0GRJ%2BzMjiI8n%2FxArl6fGsJe8yotmB0yU%2F44Su0OH5NIqMRd5r%2F%2FgghzP6oaTsLsLKK0qhUrh4GkfPxbW3%2BdT0S71tiZzWl4qCx3ZZA4JfBL%2BAc2DzoT36d2vl0%2BNIdBiEDgmO1dAXHu1HXNN8PWDQ%3D%3D&response-content-disposition=attachment%3B+filename%3Dmetadata.csv.zip\" -O \"metadata.csv.zip\" -c\n```\n\n## Kaggle Dataset 2\n\nOriginal link for more information: https://www.kaggle.com/unkownhihi/deepfake\n\n### Use following command in VM to extract data + metadata to the current folder \n\n```\nwget --header=\"Host: storage.googleapis.com\" --header=\"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36\" --header=\"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\" --header=\"Accept-Language: en-US,en;q=0.9,th;q=0.8,zh-CN;q=0.7,zh;q=0.6\" --header=\"Referer: https://www.kaggle.com/\" --header=\"Cookie: cuntwars_user_id=jilW0PIXf0; _ga=GA1.3.2052114344.1570426301\" --header=\"Connection: keep-alive\" \"https://storage.googleapis.com/kaggle-data-sets/451078/893807/bundle/archive.zip?GoogleAccessId=web-data@kaggle-161607.iam.gserviceaccount.com&Expires=1581436617&Signature=pn%2BfeEb6m5dYz76c4sSMF%2FE2uEY%2FFAzNBfVn3hwckOIc6hLXqWvwpHzS%2FWaCdNFBsqjxIAoPY5vXmmgoEjNB7dTJzQZ5thipQ3P%2Fhjs9RtPU8dCNXWrzMM8iNku%2BNmvdKoHcAWS3DE3LVaO93Z77TfwucSMohOl4AS7Ps%2Bu1AX5SU%2BUQSJ8gKPFsQ2PET11oIyVpYDHffnCakrh3HSUzVfKT6h4DrcP8VXxE57agT5fwdf1aa8kgwao4AfLY8BtL8ZryZf5YM6z8JL%2B7mOD6d8XPZpyBvfny4uOPlphvK6TvWkMjlJRcf8dSZWvKnLOH1B3NGuG12iyYGBY4XZfBqQ%3D%3D&response-content-disposition=attachment%3B+filename%3Ddeepfake.zip\" -O \"deepfake.zip\" -c\n```\n\n# References:\n- https://www.kaggle.com/c/deepfake-detection-challenge/discussion/126156", "739977": "Thanks for your cheat-sheet! Very clear ๐", "739978": "Thank you @phunghieu ! I am also planning to include your data, but may not be suitable for local SSD due to size. Hopefully if I make a discussion topic on the S3 bucket, I can include that!", "739997": "This sounds great! I definitely need a way to move my datasets to S3 bucket whenever I receive credits from AWS (in case my request is accepted). BTW, thank you for spending time and effort to write this tutorial, it very helpful to me who is unfamiliar with AWS.", "739999": "You are welcome :))", "740599": "debanga Thanks for your good cheatsheet on ``AWS SageMaker`` which is really helpful for the beginners to the AWS cloud like me .\n\nCan I transfer the ``data(472GB)`` to`` S3`` bucket directly without saving to locally.", "740613": "Hi, @saisrinivasreddy Thanks... I havenโt done that, and still looking for if someone can help :) But if I figure out will post. But, as I mentioned you can always save to the SageMaker SSD and transfer, OR you can create a VM in AWS download the data and transfer.", "740659": "As of now i am transfering data from sagemaker to S3 which is a time taking process(wget url --> AWS SageMaker --> S3). But direct process (wget url --> S3) saves time.\n\nThanks @debanga", "741039": "Thanks for Sharing", "933683": "Thanks so much. Used it for another dataset download. It worked perfectly.", "1145265": "Thank You. It was very helpful.๐" }, "source": "meta" }