{
  "id": 124373,
  "title": "Beginner question: Upload data on AWS",
  "url": "/competitions/deepfake-detection-challenge/discussion/124373",
  "author_name": "",
  "post_date": "2020-01-03T18:52:38.503848300Z",
  "votes": 6,
  "comment_count": 8,
  "views": 0,
  "content": "<p>Hi kagglers,</p>\n\n<p>For those who work on AWS (as I intend to do, if I get the credits) did you simply uploaded the entire data on AWS ? won't it take a huge time du to internet bandwith? is there a simpler/faster way to access data from AWS?</p>\n\n<p>What is your advice to get the AWS credits since we first have to be on the leaderboard, how to train the model without good hardware? </p>\n\n<p>Thanks</p>",
  "messages": [
    {
      "id": "709636",
      "postDate": "01/03/2020 18:52:38",
      "content": "<p>Hi kagglers,</p>\n\n<p>For those who work on AWS (as I intend to do, if I get the credits) did you simply uploaded the entire data on AWS ? won't it take a huge time du to internet bandwith? is there a simpler/faster way to access data from AWS?</p>\n\n<p>What is your advice to get the AWS credits since we first have to be on the leaderboard, how to train the model without good hardware? </p>\n\n<p>Thanks</p>",
      "rawMarkdown": "Hi kagglers,\n\nFor those who work on AWS (as I intend to do, if I get the credits) did you simply uploaded the entire data on AWS ? won't it take a huge time du to internet bandwith? is there a simpler/faster way to access data from AWS?\n\nWhat is your advice to get the AWS credits since we first have to be on the leaderboard, how to train the model without good hardware? \n\nThanks",
      "votes": null
    },
    {
      "id": "709812",
      "postDate": "01/04/2020 00:33:53",
      "content": "<p>To be on the leaderboard, submit a submission that labeled all the videos 0.5. The benefits from this way is you either do not need hardware, and when you have to get a higher score second time in order to get another pool of credit, it will be easier.\nThis is my first time using AWS too. I did it this way:\nI started a whatever ec2 instance(and mount an EBS volume) and download it on the instance. It took about a hour to download the entire dataset. That is the quickest way I could think of. The EBS volume will stay forever(if you don't delete it).\nHope this helps!</p>",
      "rawMarkdown": "To be on the leaderboard, submit a submission that labeled all the videos 0.5. The benefits from this way is you either do not need hardware, and when you have to get a higher score second time in order to get another pool of credit, it will be easier.\nThis is my first time using AWS too. I did it this way:\nI started a whatever ec2 instance(and mount an EBS volume) and download it on the instance. It took about a hour to download the entire dataset. That is the quickest way I could think of. The EBS volume will stay forever(if you don't delete it).\nHope this helps!",
      "votes": null
    },
    {
      "id": "709813",
      "postDate": "01/04/2020 00:36:45",
      "content": "<p>I have received my team initial credits already.   If you submitted form before deadline and had a submission you should see $500 in your aws account.</p>\n\n<p>If you had no submission - you would not have received the credits.</p>\n\n<p>Follow the directions for a credit request by 1/23 for the next batch.</p>\n\n<p>But I second your question - what is the fastest way to get the full data set into an aws bucket on our account?</p>",
      "rawMarkdown": "I have received my team initial credits already.   If you submitted form before deadline and had a submission you should see $500 in your aws account.\n\nIf you had no submission - you would not have received the credits.\n\nFollow the directions for a credit request by 1/23 for the next batch.\n\nBut I second your question - what is the fastest way to get the full data set into an aws bucket on our account?",
      "votes": null
    },
    {
      "id": "710238",
      "postDate": "01/04/2020 13:49:29",
      "content": "<p>I created ec2 instance, and downloaded parts (not the full thing) then upload to s3 bucket .. still in the process</p>",
      "rawMarkdown": "I created ec2 instance, and downloaded parts (not the full thing) then upload to s3 bucket .. still in the process",
      "votes": null
    },
    {
      "id": "710371",
      "postDate": "01/04/2020 16:48:56",
      "content": "<p>Thanks for you answers. in the Getting started, they suggest to access the dataset on a GCS Bucket, Does it mean we have to upload it there,or is there any already feeded GCS Bucket?</p>",
      "rawMarkdown": "Thanks for you answers. in the Getting started, they suggest to access the dataset on a GCS Bucket, Does it mean we have to upload it there,or is there any already feeded GCS Bucket?",
      "votes": null
    },
    {
      "id": "710373",
      "postDate": "01/04/2020 16:49:50",
      "content": "<p>Thank you, I will test this strategy.</p>",
      "rawMarkdown": "Thank you, I will test this strategy.",
      "votes": null
    },
    {
      "id": "710541",
      "postDate": "01/04/2020 21:44:44",
      "content": "<p>hey ..  how did you download the data to the ec2 instance ? using wget ? </p>",
      "rawMarkdown": "hey ..  how did you download the data to the ec2 instance ? using wget ?",
      "votes": null
    },
    {
      "id": "710545",
      "postDate": "01/04/2020 21:50:54",
      "content": "<p>Yep! wget using CurlWget plugin in google chrome.</p>",
      "rawMarkdown": "Yep! wget using CurlWget plugin in google chrome.",
      "votes": null
    },
    {
      "id": "710589",
      "postDate": "01/04/2020 23:46:38",
      "content": "<p>Thanks</p>",
      "rawMarkdown": "Thanks",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 709812,
      "author_name": "unkownhihi",
      "author_url": "",
      "post_date": "01/04/2020 00:33:53",
      "content": "<p>To be on the leaderboard, submit a submission that labeled all the videos 0.5. The benefits from this way is you either do not need hardware, and when you have to get a higher score second time in order to get another pool of credit, it will be easier.\nThis is my first time using AWS too. I did it this way:\nI started a whatever ec2 instance(and mount an EBS volume) and download it on the instance. It took about a hour to download the entire dataset. That is the quickest way I could think of. The EBS volume will stay forever(if you don't delete it).\nHope this helps!</p>",
      "votes": null,
      "replies": [
        {
          "id": 710373,
          "author_name": "kamstats",
          "author_url": "",
          "post_date": "01/04/2020 16:49:50",
          "content": "<p>Thank you, I will test this strategy.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 710541,
          "author_name": "ankitmanocha",
          "author_url": "",
          "post_date": "01/04/2020 21:44:44",
          "content": "<p>hey ..  how did you download the data to the ec2 instance ? using wget ? </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 710545,
          "author_name": "unkownhihi",
          "author_url": "",
          "post_date": "01/04/2020 21:50:54",
          "content": "<p>Yep! wget using CurlWget plugin in google chrome.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 710589,
          "author_name": "kamstats",
          "author_url": "",
          "post_date": "01/04/2020 23:46:38",
          "content": "<p>Thanks</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 709813,
      "author_name": "pcjimmmy",
      "author_url": "",
      "post_date": "01/04/2020 00:36:45",
      "content": "<p>I have received my team initial credits already.   If you submitted form before deadline and had a submission you should see $500 in your aws account.</p>\n\n<p>If you had no submission - you would not have received the credits.</p>\n\n<p>Follow the directions for a credit request by 1/23 for the next batch.</p>\n\n<p>But I second your question - what is the fastest way to get the full data set into an aws bucket on our account?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 710238,
      "author_name": "basharallabadi",
      "author_url": "",
      "post_date": "01/04/2020 13:49:29",
      "content": "<p>I created ec2 instance, and downloaded parts (not the full thing) then upload to s3 bucket .. still in the process</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 710371,
      "author_name": "kamstats",
      "author_url": "",
      "post_date": "01/04/2020 16:48:56",
      "content": "<p>Thanks for you answers. in the Getting started, they suggest to access the dataset on a GCS Bucket, Does it mean we have to upload it there,or is there any already feeded GCS Bucket?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "709636": "Hi kagglers,\n\nFor those who work on AWS (as I intend to do, if I get the credits) did you simply uploaded the entire data on AWS ? won't it take a huge time du to internet bandwith? is there a simpler/faster way to access data from AWS?\n\nWhat is your advice to get the AWS credits since we first have to be on the leaderboard, how to train the model without good hardware? \n\nThanks",
    "709812": "To be on the leaderboard, submit a submission that labeled all the videos 0.5. The benefits from this way is you either do not need hardware, and when you have to get a higher score second time in order to get another pool of credit, it will be easier.\nThis is my first time using AWS too. I did it this way:\nI started a whatever ec2 instance(and mount an EBS volume) and download it on the instance. It took about a hour to download the entire dataset. That is the quickest way I could think of. The EBS volume will stay forever(if you don't delete it).\nHope this helps!",
    "709813": "I have received my team initial credits already.   If you submitted form before deadline and had a submission you should see $500 in your aws account.\n\nIf you had no submission - you would not have received the credits.\n\nFollow the directions for a credit request by 1/23 for the next batch.\n\nBut I second your question - what is the fastest way to get the full data set into an aws bucket on our account?",
    "710238": "I created ec2 instance, and downloaded parts (not the full thing) then upload to s3 bucket .. still in the process",
    "710371": "Thanks for you answers. in the Getting started, they suggest to access the dataset on a GCS Bucket, Does it mean we have to upload it there,or is there any already feeded GCS Bucket?",
    "710373": "Thank you, I will test this strategy.",
    "710541": "hey ..  how did you download the data to the ec2 instance ? using wget ?",
    "710545": "Yep! wget using CurlWget plugin in google chrome.",
    "710589": "Thanks"
  },
  "source": "meta"
}