{
  "id": 8232,
  "title": "Clarification on evaluation metric",
  "url": "/competitions/seizure-detection/discussion/8232",
  "author_name": "",
  "post_date": "2014-05-20T14:40:14.277Z",
  "votes": null,
  "comment_count": 5,
  "views": 1724,
  "content": "<p>Hi, I have a question about the second part of the evaluation metric. The page states: &quot;Secondly, you must predict the probability that the clip is within the first 15 seconds its respective seizure.&quot;</p>\n<p>Does this mean we must first classify into seizure/non-seizure clips, or is the AUC only calculated over those clips which are actually seizures?</p>",
  "messages": [
    {
      "id": "44941",
      "postDate": "05/20/2014 14:40:14",
      "content": "<p>Hi, I have a question about the second part of the evaluation metric. The page states: &quot;Secondly, you must predict the probability that the clip is within the first 15 seconds its respective seizure.&quot;</p>\n<p>Does this mean we must first classify into seizure/non-seizure clips, or is the AUC only calculated over those clips which are actually seizures?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "44947",
      "postDate": "05/20/2014 14:51:28",
      "content": "<p>The AUC is calculated across all clips for both tasks. In the first half you are predicting seizure vs. not. In the second you are predicting early vs not early. &quot;not early&quot; includes both seizures and not seizures. Make sense?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "44949",
      "postDate": "05/20/2014 14:55:28",
      "content": "<p>Yes, thank you.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "44958",
      "postDate": "05/20/2014 16:11:22",
      "content": "<p>Hi</p>\n<p>I also have several questions regarding the evaluation metric.</p>\n<ol>\n<li>to be perfectly clear, is the&nbsp;rule for predicting early (latency&lt;15s) or (latency=&lt;15s) ?</li>\n<li>Since the different subjects have a different number of electrodes (and electrodes positions) and sampling frequency, the algorithms must be calibrated on each subjects separately. Therefore, the classification score may have a different dynamic for each subject. When put together for the evaluation, &nbsp;the AUC will be not representative of what you can get in a real life application where the thresholds are tailored for each subject. Am I right ?</li>\n</ol>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "46127",
      "postDate": "05/23/2014 14:07:41",
      "content": "<p>[quote=Alexandre;44958]</p>\n<ol>\n<li>to be perfectly clear, is the&nbsp;rule for predicting early (latency&lt;15s) or (latency=&lt;15s)&nbsp;</li>\n</ol>\n<p>[/quote]</p>\n<p>I would also be very interested in an answer to this question.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "46136",
      "postDate": "05/23/2014 16:17:22",
      "content": "<p>Great question. The second ROC metric is calculated on latency&lt;=15s. So you'll want to include the 15 second clips in your classification.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 44947,
      "author_name": "wcukierski",
      "author_url": "",
      "post_date": "05/20/2014 14:51:28",
      "content": "<p>The AUC is calculated across all clips for both tasks. In the first half you are predicting seizure vs. not. In the second you are predicting early vs not early. &quot;not early&quot; includes both seizures and not seizures. Make sense?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 44949,
      "author_name": "emolson",
      "author_url": "",
      "post_date": "05/20/2014 14:55:28",
      "content": "<p>Yes, thank you.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 44958,
      "author_name": "alexandrebarachant",
      "author_url": "",
      "post_date": "05/20/2014 16:11:22",
      "content": "<p>Hi</p>\n<p>I also have several questions regarding the evaluation metric.</p>\n<ol>\n<li>to be perfectly clear, is the&nbsp;rule for predicting early (latency&lt;15s) or (latency=&lt;15s) ?</li>\n<li>Since the different subjects have a different number of electrodes (and electrodes positions) and sampling frequency, the algorithms must be calibrated on each subjects separately. Therefore, the classification score may have a different dynamic for each subject. When put together for the evaluation, &nbsp;the AUC will be not representative of what you can get in a real life application where the thresholds are tailored for each subject. Am I right ?</li>\n</ol>",
      "votes": null,
      "replies": []
    },
    {
      "id": 46127,
      "author_name": "ugrossek",
      "author_url": "",
      "post_date": "05/23/2014 14:07:41",
      "content": "<p>[quote=Alexandre;44958]</p>\n<ol>\n<li>to be perfectly clear, is the&nbsp;rule for predicting early (latency&lt;15s) or (latency=&lt;15s)&nbsp;</li>\n</ol>\n<p>[/quote]</p>\n<p>I would also be very interested in an answer to this question.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 46136,
      "author_name": "bbrinkm",
      "author_url": "",
      "post_date": "05/23/2014 16:17:22",
      "content": "<p>Great question. The second ROC metric is calculated on latency&lt;=15s. So you'll want to include the 15 second clips in your classification.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "44941": "",
    "44947": "",
    "44949": "",
    "44958": "",
    "46127": "",
    "46136": ""
  },
  "source": "meta"
}