{
  "id": 126156,
  "title": "Using AWS for DFDC",
  "url": "/competitions/deepfake-detection-challenge/discussion/126156",
  "author_name": "AWS MLRA",
  "post_date": "2020-01-16T03:05:48.436000",
  "votes": 23,
  "comment_count": 15,
  "views": 0,
  "content": "<p>Dear DFDC participants, </p>\n\n<p>We have put together a tutorial on using AWS for your participation in DFDC:\n<a href=\"https://github.com/data-science-on-aws/deep-fake-detection-challenge\">https://github.com/data-science-on-aws/deep-fake-detection-challenge</a></p>\n\n<p>Hope this information helps!</p>",
  "messages": [
    {
      "id": 719966,
      "postDate": "2020-01-16T03:05:48.437Z",
      "content": "<p>Dear DFDC participants, </p>\n\n<p>We have put together a tutorial on using AWS for your participation in DFDC:\n<a href=\"https://github.com/data-science-on-aws/deep-fake-detection-challenge\">https://github.com/data-science-on-aws/deep-fake-detection-challenge</a></p>\n\n<p>Hope this information helps!</p>",
      "rawMarkdown": "Dear DFDC participants, \n\nWe have put together a tutorial on using AWS for your participation in DFDC:\nhttps://github.com/data-science-on-aws/deep-fake-detection-challenge\n\nHope this information helps!",
      "votes": 22
    },
    {
      "id": 769155,
      "postDate": "2020-03-11T15:35:44.390Z",
      "content": "<p>Is there a way to get the training video data to AWS directly?  Without having to download to my local computer, then re-upload to AWS?</p>",
      "rawMarkdown": "Is there a way to get the training video data to AWS directly?  Without having to download to my local computer, then re-upload to AWS?",
      "replies": [
        {
          "id": 775232,
          "postDate": "2020-03-16T12:42:29.643Z",
          "content": "<p><a href=\"/awsmlra\">@awsmlra</a> would it be a possibility that I create an ami &amp; make it public with the extracted videos to share? As far as I know public images should be free of charge for me, right? I know S3 would be the preferred method to store this huge amount of data, but it might be good for newcomers to aws... </p>",
          "rawMarkdown": "@awsmlra would it be a possibility that I create an ami &amp; make it public with the extracted videos to share? As far as I know public images should be free of charge for me, right? I know S3 would be the preferred method to store this huge amount of data, but it might be good for newcomers to aws... "
        }
      ]
    },
    {
      "id": 764782,
      "postDate": "2020-03-05T22:23:37.207Z",
      "content": "<p>Didn't received AWS credits yet.</p>",
      "rawMarkdown": "Didn't received AWS credits yet."
    },
    {
      "id": 739355,
      "postDate": "2020-02-07T18:42:24.813Z",
      "content": "<p>Hi,</p>\n\n<p>I am wondering how long it might take to receive the credit?\nThank you for the help.</p>",
      "rawMarkdown": "Hi,\n\nI am wondering how long it might take to receive the credit?\nThank you for the help.",
      "replies": [
        {
          "id": 739548,
          "postDate": "2020-02-08T00:09:34.237Z",
          "content": "<p>Please refer to this post: <a href=\"https://www.kaggle.com/c/deepfake-detection-challenge/discussion/122254\">https://www.kaggle.com/c/deepfake-detection-challenge/discussion/122254</a>. </p>",
          "rawMarkdown": "Please refer to this post: https://www.kaggle.com/c/deepfake-detection-challenge/discussion/122254. "
        }
      ]
    },
    {
      "id": 737823,
      "postDate": "2020-02-05T20:13:24.920Z",
      "content": "<p>Hi all, </p>\n\n<p>we've added a more detailed description of how to increase your Amazon SageMaker instance limits <a href=\"https://github.com/data-science-on-aws/deep-fake-detection-challenge/blob/master/quota-increase.md\">here</a>. </p>\n\n<p>You can also find a comparison and specs of the all Amazon SageMaker supported CPU- and GPU-based instances <a href=\"https://aws.amazon.com/sagemaker/pricing/instance-types/\">here</a>. You can see the GPU-based instances listed under \"Accelerated Computing – Current Generation\". </p>\n\n<p>Hope that helps! </p>",
      "rawMarkdown": "Hi all, \n\nwe've added a more detailed description of how to increase your Amazon SageMaker instance limits [here](https://github.com/data-science-on-aws/deep-fake-detection-challenge/blob/master/quota-increase.md). \n\nYou can also find a comparison and specs of the all Amazon SageMaker supported CPU- and GPU-based instances [here](https://aws.amazon.com/sagemaker/pricing/instance-types/). You can see the GPU-based instances listed under \"Accelerated Computing – Current Generation\". \n\nHope that helps! "
    },
    {
      "id": 733546,
      "postDate": "2020-01-31T09:36:32.483Z",
      "content": "<p>Hello Deep Fake Competition challengers i'm new to  this aws,  can anyone help me out about the setup of aws ?</p>\n\n<p>I've received $750 credits from aws but i don't know which  Notebook instance type is good for this challenge.</p>\n\n<p>Thanks in advance !</p>",
      "rawMarkdown": "Hello Deep Fake Competition challengers i'm new to  this aws,  can anyone help me out about the setup of aws ?\n\nI've received $750 credits from aws but i don't know which  Notebook instance type is good for this challenge.\n\nThanks in advance !",
      "replies": [
        {
          "id": 736215,
          "postDate": "2020-02-04T00:40:41.303Z",
          "content": "<p>Same question. Why I cannot use GPU?</p>",
          "rawMarkdown": "Same question. Why I cannot use GPU?",
          "isDeleted": true
        },
        {
          "id": 737085,
          "postDate": "2020-02-04T22:36:37.840Z",
          "content": "<p>You can file a request to increase your instance limit. </p>",
          "rawMarkdown": "You can file a request to increase your instance limit. "
        }
      ]
    },
    {
      "id": 720997,
      "postDate": "2020-01-16T23:26:50.130Z",
      "content": "<p>Is this the 'automl' mentioned in the notice?\n(I don't know exactly the criteria for 'automl')\nI am sorry that interpretation is not smooth using translator.</p>",
      "rawMarkdown": "Is this the 'automl' mentioned in the notice?\n(I don't know exactly the criteria for 'automl')\nI am sorry that interpretation is not smooth using translator."
    },
    {
      "id": 720824,
      "postDate": "2020-01-16T19:31:13.093Z",
      "content": "<p>Dunno if others ran into the same trouble but I got some issues:\n1. <strong>Quotas</strong>\n<em>Theory</em> - The default quotas for SageMaker should be really high: <a href=\"https://docs.aws.amazon.com/general/latest/gr/sagemaker.html#limits_sagemaker\">https://docs.aws.amazon.com/general/latest/gr/sagemaker.html#limits_sagemaker</a>\n<em>Practice</em> - It is 0 for almost all instance types except the smallest, which are fairly slow, not a single GPU instance at all.</p>\n\n<ol>\n<li><strong>Increasing limits</strong>\n<em>Theory</em> - Open the Service Quotas console at <a href=\"https://console.aws.amazon.com/servicequotas/\">https://console.aws.amazon.com/servicequotas/</a> and increase SageMaker\n<em>Practice</em> - Sagemaker / ML instances are not listed. Increasing EC2 instances take several <em>days</em>(!) and are apparently a separate quota anyway.</li>\n</ol>\n\n<p>...so, I'm basically stuck with my small slow notebook and don't know how I could use more. ¯_(ツ)_/¯</p>\n\n<p>Anyone else in this situation? Where/how exactly to increase the quota?</p>",
      "rawMarkdown": "Dunno if others ran into the same trouble but I got some issues:\n1. **Quotas**\n*Theory* - The default quotas for SageMaker should be really high: https://docs.aws.amazon.com/general/latest/gr/sagemaker.html#limits_sagemaker\n*Practice* - It is 0 for almost all instance types except the smallest, which are fairly slow, not a single GPU instance at all.\n\n2. **Increasing limits**\n*Theory* - Open the Service Quotas console at https://console.aws.amazon.com/servicequotas/ and increase SageMaker\n*Practice* - Sagemaker / ML instances are not listed. Increasing EC2 instances take several *days*(!) and are apparently a separate quota anyway.\n\n...so, I'm basically stuck with my small slow notebook and don't know how I could use more. ¯\\_(ツ)_/¯\n\nAnyone else in this situation? Where/how exactly to increase the quota?",
      "replies": [
        {
          "id": 721007,
          "postDate": "2020-01-16T23:55:46.933Z",
          "content": "<p>Mine got approved for just one day!</p>",
          "rawMarkdown": "Mine got approved for just one day!"
        },
        {
          "id": 721394,
          "postDate": "2020-01-17T10:41:13.140Z",
          "content": "<p>BTW, I think I was looking at the wrong place the whole time. I was using the \"My Service Quotas\" in the AWS Console, but apparently that is the wrong place. You have to go through the \"Support Center\" to increase the limits for SageMaker. Let's see how long it takes. </p>\n\n<p>EDIT: it was done in an hour or so. Great. Thanks</p>",
          "rawMarkdown": "BTW, I think I was looking at the wrong place the whole time. I was using the \"My Service Quotas\" in the AWS Console, but apparently that is the wrong place. You have to go through the \"Support Center\" to increase the limits for SageMaker. Let's see how long it takes. \n\nEDIT: it was done in an hour or so. Great. Thanks",
          "votes": 1
        },
        {
          "id": 736220,
          "postDate": "2020-02-04T00:48:54.783Z",
          "content": "<p>Thanks, just submitted the request to increase notebook limit on \"Support Center\". Hope it will work....</p>",
          "rawMarkdown": "Thanks, just submitted the request to increase notebook limit on \"Support Center\". Hope it will work....",
          "isDeleted": true
        },
        {
          "id": 736221,
          "postDate": "2020-02-04T00:49:27.593Z",
          "content": "<p>What notebook resources you have?</p>",
          "rawMarkdown": "What notebook resources you have?",
          "isDeleted": true
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 769155,
      "author_name": "datab",
      "author_url": "",
      "post_date": "2020-03-11T15:35:44.390000",
      "content": "<p>Is there a way to get the training video data to AWS directly?  Without having to download to my local computer, then re-upload to AWS?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 775232,
          "author_name": "Justin Güse",
          "author_url": "",
          "post_date": "2020-03-16T12:42:29.643000",
          "content": "<p><a href=\"/awsmlra\">@awsmlra</a> would it be a possibility that I create an ami &amp; make it public with the extracted videos to share? As far as I know public images should be free of charge for me, right? I know S3 would be the preferred method to store this huge amount of data, but it might be good for newcomers to aws... </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 764782,
      "author_name": "apoorv anand",
      "author_url": "",
      "post_date": "2020-03-05T22:23:37.207000",
      "content": "<p>Didn't received AWS credits yet.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 739355,
      "author_name": "TensorFrozen",
      "author_url": "",
      "post_date": "2020-02-07T18:42:24.813000",
      "content": "<p>Hi,</p>\n\n<p>I am wondering how long it might take to receive the credit?\nThank you for the help.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 739548,
          "author_name": "AWS MLRA",
          "author_url": "",
          "post_date": "2020-02-08T00:09:34.237000",
          "content": "<p>Please refer to this post: <a href=\"https://www.kaggle.com/c/deepfake-detection-challenge/discussion/122254\">https://www.kaggle.com/c/deepfake-detection-challenge/discussion/122254</a>. </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 737823,
      "author_name": "Antje",
      "author_url": "",
      "post_date": "2020-02-05T20:13:24.920000",
      "content": "<p>Hi all, </p>\n\n<p>we've added a more detailed description of how to increase your Amazon SageMaker instance limits <a href=\"https://github.com/data-science-on-aws/deep-fake-detection-challenge/blob/master/quota-increase.md\">here</a>. </p>\n\n<p>You can also find a comparison and specs of the all Amazon SageMaker supported CPU- and GPU-based instances <a href=\"https://aws.amazon.com/sagemaker/pricing/instance-types/\">here</a>. You can see the GPU-based instances listed under \"Accelerated Computing – Current Generation\". </p>\n\n<p>Hope that helps! </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 733546,
      "author_name": "Bolloju Manoj kumar",
      "author_url": "",
      "post_date": "2020-01-31T09:36:32.483000",
      "content": "<p>Hello Deep Fake Competition challengers i'm new to  this aws,  can anyone help me out about the setup of aws ?</p>\n\n<p>I've received $750 credits from aws but i don't know which  Notebook instance type is good for this challenge.</p>\n\n<p>Thanks in advance !</p>",
      "votes": 0,
      "replies": [
        {
          "id": 736215,
          "author_name": "",
          "author_url": "",
          "post_date": "2020-02-04T00:40:41.303000",
          "content": "<p>Same question. Why I cannot use GPU?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 737085,
          "author_name": "AWS MLRA",
          "author_url": "",
          "post_date": "2020-02-04T22:36:37.840000",
          "content": "<p>You can file a request to increase your instance limit. </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 720997,
      "author_name": "Donghee",
      "author_url": "",
      "post_date": "2020-01-16T23:26:50.130000",
      "content": "<p>Is this the 'automl' mentioned in the notice?\n(I don't know exactly the criteria for 'automl')\nI am sorry that interpretation is not smooth using translator.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 720824,
      "author_name": "dagnelies",
      "author_url": "",
      "post_date": "2020-01-16T19:31:13.093000",
      "content": "<p>Dunno if others ran into the same trouble but I got some issues:\n1. <strong>Quotas</strong>\n<em>Theory</em> - The default quotas for SageMaker should be really high: <a href=\"https://docs.aws.amazon.com/general/latest/gr/sagemaker.html#limits_sagemaker\">https://docs.aws.amazon.com/general/latest/gr/sagemaker.html#limits_sagemaker</a>\n<em>Practice</em> - It is 0 for almost all instance types except the smallest, which are fairly slow, not a single GPU instance at all.</p>\n\n<ol>\n<li><strong>Increasing limits</strong>\n<em>Theory</em> - Open the Service Quotas console at <a href=\"https://console.aws.amazon.com/servicequotas/\">https://console.aws.amazon.com/servicequotas/</a> and increase SageMaker\n<em>Practice</em> - Sagemaker / ML instances are not listed. Increasing EC2 instances take several <em>days</em>(!) and are apparently a separate quota anyway.</li>\n</ol>\n\n<p>...so, I'm basically stuck with my small slow notebook and don't know how I could use more. ¯_(ツ)_/¯</p>\n\n<p>Anyone else in this situation? Where/how exactly to increase the quota?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 721007,
          "author_name": "Shangqiu Li",
          "author_url": "",
          "post_date": "2020-01-16T23:55:46.933000",
          "content": "<p>Mine got approved for just one day!</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 721394,
          "author_name": "dagnelies",
          "author_url": "",
          "post_date": "2020-01-17T10:41:13.140000",
          "content": "<p>BTW, I think I was looking at the wrong place the whole time. I was using the \"My Service Quotas\" in the AWS Console, but apparently that is the wrong place. You have to go through the \"Support Center\" to increase the limits for SageMaker. Let's see how long it takes. </p>\n\n<p>EDIT: it was done in an hour or so. Great. Thanks</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 736220,
          "author_name": "",
          "author_url": "",
          "post_date": "2020-02-04T00:48:54.783000",
          "content": "<p>Thanks, just submitted the request to increase notebook limit on \"Support Center\". Hope it will work....</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 736221,
          "author_name": "",
          "author_url": "",
          "post_date": "2020-02-04T00:49:27.593000",
          "content": "<p>What notebook resources you have?</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "719966": "Dear DFDC participants, \n\nWe have put together a tutorial on using AWS for your participation in DFDC:\nhttps://github.com/data-science-on-aws/deep-fake-detection-challenge\n\nHope this information helps!",
    "769155": "Is there a way to get the training video data to AWS directly?  Without having to download to my local computer, then re-upload to AWS?",
    "764782": "Didn't received AWS credits yet.",
    "739355": "Hi,\n\nI am wondering how long it might take to receive the credit?\nThank you for the help.",
    "737823": "Hi all, \n\nwe've added a more detailed description of how to increase your Amazon SageMaker instance limits [here](https://github.com/data-science-on-aws/deep-fake-detection-challenge/blob/master/quota-increase.md). \n\nYou can also find a comparison and specs of the all Amazon SageMaker supported CPU- and GPU-based instances [here](https://aws.amazon.com/sagemaker/pricing/instance-types/). You can see the GPU-based instances listed under \"Accelerated Computing – Current Generation\". \n\nHope that helps! ",
    "733546": "Hello Deep Fake Competition challengers i'm new to  this aws,  can anyone help me out about the setup of aws ?\n\nI've received $750 credits from aws but i don't know which  Notebook instance type is good for this challenge.\n\nThanks in advance !",
    "720997": "Is this the 'automl' mentioned in the notice?\n(I don't know exactly the criteria for 'automl')\nI am sorry that interpretation is not smooth using translator.",
    "720824": "Dunno if others ran into the same trouble but I got some issues:\n1. **Quotas**\n*Theory* - The default quotas for SageMaker should be really high: https://docs.aws.amazon.com/general/latest/gr/sagemaker.html#limits_sagemaker\n*Practice* - It is 0 for almost all instance types except the smallest, which are fairly slow, not a single GPU instance at all.\n\n2. **Increasing limits**\n*Theory* - Open the Service Quotas console at https://console.aws.amazon.com/servicequotas/ and increase SageMaker\n*Practice* - Sagemaker / ML instances are not listed. Increasing EC2 instances take several *days*(!) and are apparently a separate quota anyway.\n\n...so, I'm basically stuck with my small slow notebook and don't know how I could use more. ¯\\_(ツ)_/¯\n\nAnyone else in this situation? Where/how exactly to increase the quota?"
  }
}