{ "id": 8299, "title": "AUC", "url": "/competitions/seizure-detection/discussion/8299", "author_name": "", "post_date": "2014-05-28T09:56:36.830Z", "votes": null, "comment_count": 7, "views": 1821, "content": "

Hi,

\n

Sorry it might be a very silly question, I am not sure about how to compute AUC between predicted and correct labels. Can someone provide a formula (matlab/python script would be great). Secondly, how would AUC translate to "These early clips are double counted because early detection is critical to successful intervention with a responsive neurostimulation device."

", "messages": [ { "id": "47167", "postDate": "05/28/2014 09:56:36", "content": "

Hi,

\n

Sorry it might be a very silly question, I am not sure about how to compute AUC between predicted and correct labels. Can someone provide a formula (matlab/python script would be great). Secondly, how would AUC translate to "These early clips are double counted because early detection is critical to successful intervention with a responsive neurostimulation device."

", "rawMarkdown": "", "votes": null }, { "id": "47169", "postDate": "05/28/2014 10:00:53", "content": "

Read: https://www.kaggle.com/wiki/AreaUnderCurve

", "rawMarkdown": "", "votes": null }, { "id": "47170", "postDate": "05/28/2014 10:08:39", "content": "

Cheers mate! :)

", "rawMarkdown": "", "votes": null }, { "id": "47173", "postDate": "05/28/2014 10:58:46", "content": "

[quote=Chaos;47167]

\n

Secondly, how would AUC translate to "These early clips are double counted because early detection is critical to successful intervention with a responsive neurostimulation device."

\n

[/quote]

\n

Hi,

\n

a precision would be nice. Does it mean that the interictal segments are not taken into account in the early detection AUC ?

", "rawMarkdown": "", "votes": null }, { "id": "47174", "postDate": "05/28/2014 11:08:22", "content": "

[quote=Alexandre;47173]

\n

[quote=Chaos;47167]

\n

Secondly, how would AUC translate to "These early clips are double counted because early detection is critical to successful intervention with a responsive neurostimulation device."

\n

[/quote]

\n

Hi,

\n

a precision would be nice. Does it mean that the interictal segments are not taken into account in the early detection AUC ?

\n

[/quote]

\n\n

exactly! thats what I was thinking if interictal segments would be accounted for in early detection. Furthermore, if early detection is double counted, does this mean:

\n

1/3(AUC seizure +2*AUC early ) 

\n

This double count is a bit ambiguous

", "rawMarkdown": "", "votes": null }, { "id": "47189", "postDate": "05/28/2014 15:52:09", "content": "

We are using the phrase "double counted" colloquially - for the first prediction you are identifying the seizure clips (which requires finding the early clips) and for the second you are identifying just the early clips (which, you guessed it, requires finding the early clips). The math is just like it is on the Evaluation page.

", "rawMarkdown": "", "votes": null }, { "id": "47191", "postDate": "05/28/2014 16:20:22", "content": "

And just to clarify, the interictal clips are taken into account in the early seizure ROC. The classification is early seizure vs. everything else (interictal plus late seizure).

", "rawMarkdown": "", "votes": null }, { "id": "48522", "postDate": "06/02/2014 21:59:36", "content": "

So the second task is to classify seizure vs. everything else. If the goal is to evaluate if an given algorithm is able to detect  the seizures as early as possible, a better solution would be to ignore the late seizure clips in the AUC esimation.

\n

The actual way to estimate the AUC makes things more complicated than they should be. First, you need to set up two different classification chain, with possibly two different set of features, parameters or classifier. This is not very practical for a clinical application. Second, the current threshold of 15s seems to be arbitrary, and there is no evidence that we can discriminate a trial at 15s from a trial at 16s for all subjects. I just want to point out that a sharp threshold may actually degrades performance for early detection.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 47169, "author_name": "abhishek", "author_url": "", "post_date": "05/28/2014 10:00:53", "content": "

Read: https://www.kaggle.com/wiki/AreaUnderCurve

", "votes": null, "replies": [] }, { "id": 47170, "author_name": "chaotic", "author_url": "", "post_date": "05/28/2014 10:08:39", "content": "

Cheers mate! :)

", "votes": null, "replies": [] }, { "id": 47173, "author_name": "alexandrebarachant", "author_url": "", "post_date": "05/28/2014 10:58:46", "content": "

[quote=Chaos;47167]

\n

Secondly, how would AUC translate to "These early clips are double counted because early detection is critical to successful intervention with a responsive neurostimulation device."

\n

[/quote]

\n

Hi,

\n

a precision would be nice. Does it mean that the interictal segments are not taken into account in the early detection AUC ?

", "votes": null, "replies": [] }, { "id": 47174, "author_name": "chaotic", "author_url": "", "post_date": "05/28/2014 11:08:22", "content": "

[quote=Alexandre;47173]

\n

[quote=Chaos;47167]

\n

Secondly, how would AUC translate to "These early clips are double counted because early detection is critical to successful intervention with a responsive neurostimulation device."

\n

[/quote]

\n

Hi,

\n

a precision would be nice. Does it mean that the interictal segments are not taken into account in the early detection AUC ?

\n

[/quote]

\n\n

exactly! thats what I was thinking if interictal segments would be accounted for in early detection. Furthermore, if early detection is double counted, does this mean:

\n

1/3(AUC seizure +2*AUC early ) 

\n

This double count is a bit ambiguous

", "votes": null, "replies": [] }, { "id": 47189, "author_name": "wcukierski", "author_url": "", "post_date": "05/28/2014 15:52:09", "content": "

We are using the phrase "double counted" colloquially - for the first prediction you are identifying the seizure clips (which requires finding the early clips) and for the second you are identifying just the early clips (which, you guessed it, requires finding the early clips). The math is just like it is on the Evaluation page.

", "votes": null, "replies": [] }, { "id": 47191, "author_name": "bbrinkm", "author_url": "", "post_date": "05/28/2014 16:20:22", "content": "

And just to clarify, the interictal clips are taken into account in the early seizure ROC. The classification is early seizure vs. everything else (interictal plus late seizure).

", "votes": null, "replies": [] }, { "id": 48522, "author_name": "alexandrebarachant", "author_url": "", "post_date": "06/02/2014 21:59:36", "content": "

So the second task is to classify seizure vs. everything else. If the goal is to evaluate if an given algorithm is able to detect  the seizures as early as possible, a better solution would be to ignore the late seizure clips in the AUC esimation.

\n

The actual way to estimate the AUC makes things more complicated than they should be. First, you need to set up two different classification chain, with possibly two different set of features, parameters or classifier. This is not very practical for a clinical application. Second, the current threshold of 15s seems to be arbitrary, and there is no evidence that we can discriminate a trial at 15s from a trial at 16s for all subjects. I just want to point out that a sharp threshold may actually degrades performance for early detection.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "47167": "", "47169": "", "47170": "", "47173": "", "47174": "", "47189": "", "47191": "", "48522": "" }, "source": "meta" }