{
  "id": 132235,
  "title": "Not displaying errors does not prevent probing.",
  "url": "/competitions/deepfake-detection-challenge/discussion/132235",
  "author_name": "",
  "post_date": "2020-02-25T03:02:32.084617300Z",
  "votes": 2,
  "comment_count": 1,
  "views": 0,
  "content": "<p>In reference to the following:  'Unfortunately, we are unable to surface any details about your error, so as to prevent error-probing. You are limited to 2 submissions per day, including submissions which error.'</p>\n\n<p>Error probing can already be done by abusing the error metric. Information on which of n cases holds is trivial to get (decide whether to set everything to 0.5, 0.6, 0.7 etc. based on some condition), and evaluating whether the nth video is Real or Fake is doable by setting the guess to 1.0, and all others to 0.5. The latter method could however be prevented if the video ordering were randomized each time. In which case, what could be gained by probing via error messages that can't already be done with error metric probing?</p>\n\n<p>The limit of 2 submissions per day is good in my opinion, but the idea of not having feedback on what goes wrong has been discouraging me from giving this contest a serious attempt.</p>",
  "messages": [
    {
      "id": "755687",
      "postDate": "02/25/2020 03:02:32",
      "content": "<p>In reference to the following:  'Unfortunately, we are unable to surface any details about your error, so as to prevent error-probing. You are limited to 2 submissions per day, including submissions which error.'</p>\n\n<p>Error probing can already be done by abusing the error metric. Information on which of n cases holds is trivial to get (decide whether to set everything to 0.5, 0.6, 0.7 etc. based on some condition), and evaluating whether the nth video is Real or Fake is doable by setting the guess to 1.0, and all others to 0.5. The latter method could however be prevented if the video ordering were randomized each time. In which case, what could be gained by probing via error messages that can't already be done with error metric probing?</p>\n\n<p>The limit of 2 submissions per day is good in my opinion, but the idea of not having feedback on what goes wrong has been discouraging me from giving this contest a serious attempt.</p>",
      "rawMarkdown": "In reference to the following:  'Unfortunately, we are unable to surface any details about your error, so as to prevent error-probing. You are limited to 2 submissions per day, including submissions which error.'\n\nError probing can already be done by abusing the error metric. Information on which of n cases holds is trivial to get (decide whether to set everything to 0.5, 0.6, 0.7 etc. based on some condition), and evaluating whether the nth video is Real or Fake is doable by setting the guess to 1.0, and all others to 0.5. The latter method could however be prevented if the video ordering were randomized each time. In which case, what could be gained by probing via error messages that can't already be done with error metric probing?\n\nThe limit of 2 submissions per day is good in my opinion, but the idea of not having feedback on what goes wrong has been discouraging me from giving this contest a serious attempt.",
      "votes": null
    },
    {
      "id": "755816",
      "postDate": "02/25/2020 06:27:24",
      "content": "<p>Kaggle want to prevent label probing as you point out. For other kinds of probing it’s impossible to prevent. </p>",
      "rawMarkdown": "Kaggle want to prevent label probing as you point out. For other kinds of probing it’s impossible to prevent.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 755816,
      "author_name": "khahuras",
      "author_url": "",
      "post_date": "02/25/2020 06:27:24",
      "content": "<p>Kaggle want to prevent label probing as you point out. For other kinds of probing it’s impossible to prevent. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "755687": "In reference to the following:  'Unfortunately, we are unable to surface any details about your error, so as to prevent error-probing. You are limited to 2 submissions per day, including submissions which error.'\n\nError probing can already be done by abusing the error metric. Information on which of n cases holds is trivial to get (decide whether to set everything to 0.5, 0.6, 0.7 etc. based on some condition), and evaluating whether the nth video is Real or Fake is doable by setting the guess to 1.0, and all others to 0.5. The latter method could however be prevented if the video ordering were randomized each time. In which case, what could be gained by probing via error messages that can't already be done with error metric probing?\n\nThe limit of 2 submissions per day is good in my opinion, but the idea of not having feedback on what goes wrong has been discouraging me from giving this contest a serious attempt.",
    "755816": "Kaggle want to prevent label probing as you point out. For other kinds of probing it’s impossible to prevent."
  },
  "source": "meta"
}