{
  "id": 132858,
  "title": "Questions about Private Test Set",
  "url": "/competitions/deepfake-detection-challenge/discussion/132858",
  "author_name": "",
  "post_date": "2020-02-28T08:56:46.864911Z",
  "votes": 22,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Competition description says:</p>\n\n<blockquote>\n  <p>Private Test Set: This dataset is privately held outside of Kaggle’s platform, and is used to compute the private leaderboard. It contains videos with a similar format and nature as the Training and Public Validation/Test Sets, but are real, organic videos with and without deepfakes.</p>\n</blockquote>\n\n<p>I think these 2 phrases actually opposite: \"similar format and nature as the Training and Public Validation/Test Sets\" and \"real, organic videos\". Current training set doesn't look real and organic because it was specially staged.</p>\n\n<p>So I have some questions:\n1) Were fakes generated with the same software as current training set?\n2) Was it generated just by splitting train and test actors from the same set of video sessions? Like you post for training folders 0-49 and let's say 50-69 is private test set?\n3) Does it contain some other type of videos like news reports? Youtube videos? Sport videos? Is it called 'real'? 'Organic'?\n4) Does it have the same distribution of answers, which is important for log_loss metric?</p>\n\n<p>The most important question for me is <strong>3)</strong>.</p>",
  "messages": [
    {
      "id": "758852",
      "postDate": "02/28/2020 08:56:46",
      "content": "<p>Competition description says:</p>\n\n<blockquote>\n  <p>Private Test Set: This dataset is privately held outside of Kaggle’s platform, and is used to compute the private leaderboard. It contains videos with a similar format and nature as the Training and Public Validation/Test Sets, but are real, organic videos with and without deepfakes.</p>\n</blockquote>\n\n<p>I think these 2 phrases actually opposite: \"similar format and nature as the Training and Public Validation/Test Sets\" and \"real, organic videos\". Current training set doesn't look real and organic because it was specially staged.</p>\n\n<p>So I have some questions:\n1) Were fakes generated with the same software as current training set?\n2) Was it generated just by splitting train and test actors from the same set of video sessions? Like you post for training folders 0-49 and let's say 50-69 is private test set?\n3) Does it contain some other type of videos like news reports? Youtube videos? Sport videos? Is it called 'real'? 'Organic'?\n4) Does it have the same distribution of answers, which is important for log_loss metric?</p>\n\n<p>The most important question for me is <strong>3)</strong>.</p>",
      "rawMarkdown": "Competition description says:\n&gt; Private Test Set: This dataset is privately held outside of Kaggle’s platform, and is used to compute the private leaderboard. It contains videos with a similar format and nature as the Training and Public Validation/Test Sets, but are real, organic videos with and without deepfakes.\n\nI think these 2 phrases actually opposite: \"similar format and nature as the Training and Public Validation/Test Sets\" and \"real, organic videos\". Current training set doesn't look real and organic because it was specially staged.\n\nSo I have some questions:\n1) Were fakes generated with the same software as current training set?\n2) Was it generated just by splitting train and test actors from the same set of video sessions? Like you post for training folders 0-49 and let's say 50-69 is private test set?\n3) Does it contain some other type of videos like news reports? Youtube videos? Sport videos? Is it called 'real'? 'Organic'?\n4) Does it have the same distribution of answers, which is important for log_loss metric?\n\nThe most important question for me is **3)**.",
      "votes": null
    },
    {
      "id": "758863",
      "postDate": "02/28/2020 09:10:14",
      "content": "<p>Parts of 3) and 4) were answered by the organizer in this <a href=\"https://www.kaggle.com/c/deepfake-detection-challenge/discussion/124713\">discussion</a></p>",
      "rawMarkdown": "Parts of 3) and 4) were answered by the organizer in this [discussion](https://www.kaggle.com/c/deepfake-detection-challenge/discussion/124713)",
      "votes": null
    },
    {
      "id": "758963",
      "postDate": "02/28/2020 11:57:53",
      "content": "<p>I agree that it is unclear, are organic videos in the Public Test Set as well?\nIn other words the Public Test Set is more like Training Set or Private Test Set?</p>",
      "rawMarkdown": "I agree that it is unclear, are organic videos in the Public Test Set as well?\nIn other words the Public Test Set is more like Training Set or Private Test Set?",
      "votes": null
    },
    {
      "id": "760659",
      "postDate": "03/01/2020 15:25:43",
      "content": "<p>Thanks for the pointer. This is an important point indeed!</p>",
      "rawMarkdown": "Thanks for the pointer. This is an important point indeed!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 758863,
      "author_name": "catochris",
      "author_url": "",
      "post_date": "02/28/2020 09:10:14",
      "content": "<p>Parts of 3) and 4) were answered by the organizer in this <a href=\"https://www.kaggle.com/c/deepfake-detection-challenge/discussion/124713\">discussion</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 760659,
          "author_name": "yassinealouini",
          "author_url": "",
          "post_date": "03/01/2020 15:25:43",
          "content": "<p>Thanks for the pointer. This is an important point indeed!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 758963,
      "author_name": "dthrone",
      "author_url": "",
      "post_date": "02/28/2020 11:57:53",
      "content": "<p>I agree that it is unclear, are organic videos in the Public Test Set as well?\nIn other words the Public Test Set is more like Training Set or Private Test Set?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "758852": "Competition description says:\n&gt; Private Test Set: This dataset is privately held outside of Kaggle’s platform, and is used to compute the private leaderboard. It contains videos with a similar format and nature as the Training and Public Validation/Test Sets, but are real, organic videos with and without deepfakes.\n\nI think these 2 phrases actually opposite: \"similar format and nature as the Training and Public Validation/Test Sets\" and \"real, organic videos\". Current training set doesn't look real and organic because it was specially staged.\n\nSo I have some questions:\n1) Were fakes generated with the same software as current training set?\n2) Was it generated just by splitting train and test actors from the same set of video sessions? Like you post for training folders 0-49 and let's say 50-69 is private test set?\n3) Does it contain some other type of videos like news reports? Youtube videos? Sport videos? Is it called 'real'? 'Organic'?\n4) Does it have the same distribution of answers, which is important for log_loss metric?\n\nThe most important question for me is **3)**.",
    "758863": "Parts of 3) and 4) were answered by the organizer in this [discussion](https://www.kaggle.com/c/deepfake-detection-challenge/discussion/124713)",
    "758963": "I agree that it is unclear, are organic videos in the Public Test Set as well?\nIn other words the Public Test Set is more like Training Set or Private Test Set?",
    "760659": "Thanks for the pointer. This is an important point indeed!"
  },
  "source": "meta"
}