{ "id": 10444, "title": "Binary classificator ", "url": "/competitions/seizure-prediction/discussion/10444", "author_name": "", "post_date": "2014-09-24T20:53:18.637Z", "votes": null, "comment_count": 8, "views": 2121, "content": "

Dear collegues, I'd like to know about the output expected by the administrators of this competition.   I thought they wanted a classification to the samples in "precital" or "interictal" state. Am I right?

\n\n

If so, I think "1" means that current sample is a preictal one, and "0" means a interictal sample.

\n\n

Is that true?

\n\n

Kaguiar

", "messages": [ { "id": "55185", "postDate": "09/24/2014 20:53:18", "content": "

Dear collegues, I'd like to know about the output expected by the administrators of this competition.   I thought they wanted a classification to the samples in "precital" or "interictal" state. Am I right?

\n\n

If so, I think "1" means that current sample is a preictal one, and "0" means a interictal sample.

\n\n

Is that true?

\n\n

Kaguiar

", "rawMarkdown": "", "votes": null }, { "id": "55186", "postDate": "09/24/2014 20:54:10", "content": "

* and "0" means AN interictal sample

", "rawMarkdown": "", "votes": null }, { "id": "55189", "postDate": "09/24/2014 22:50:31", "content": "

I think so but instead of 0 and 1, an output of probability (or a rank between 0 and 1) of "preictal" event is much more practical to improve roc score.

", "rawMarkdown": "", "votes": null }, { "id": "55190", "postDate": "09/24/2014 22:57:23", "content": "

Yes, indeed. 

", "rawMarkdown": "", "votes": null }, { "id": "55684", "postDate": "10/06/2014 14:15:02", "content": "

Why is outputting the probability of a preictal event (as opposed to a discrete class label 0 or 1) more practical to improve ROC score? 

", "rawMarkdown": "", "votes": null }, { "id": "55693", "postDate": "10/06/2014 17:46:04", "content": "

Because the ROC iterates over several different thresholds for the classification, its possible that your results will have a better accuracy at some threshold versus others.

", "rawMarkdown": "", "votes": null }, { "id": "55726", "postDate": "10/07/2014 11:10:34", "content": "

[quote=franklyn;55693]

\n

Because the ROC iterates over several different thresholds for the classification...

\n

[/quote]

\n

Yes, exactly.

\n

To illustrate:

\n

from sklearn.metrics import roc_auc_score

\n

y_predicted_discrete = [0.0, 0.0, 0.0, 1.0, 1.0]

\n

y_predicted_proba = [0.02, 0.01, 0.05, 0.99, 0.99]

\n

y_ground_truth = [0, 0, 1, 1, 1]

\n

print roc_auc_score(y_ground_truth, y_predicted_discrete) # 0.83

\n

print roc_auc_score(y_ground_truth, y_predicted_proba) # 1.0

\n

Even though the 3rd prediction probability was just 0.05 and the truth was 1, it still got credit for ranking it higher, causing a favorable perfect split.

", "rawMarkdown": "", "votes": null }, { "id": "55737", "postDate": "10/07/2014 14:31:29", "content": "

Gotcha. Thanks. So one can one still receive a perfect score by outputting probabilities as opposed to discrete labels? In other words, even if the third predicted discrete value was correct in your example, both sets would have received the same score under ROC?

", "rawMarkdown": "", "votes": null }, { "id": "55740", "postDate": "10/07/2014 16:41:38", "content": "

[quote=Adam Soffer;55737]

\n

Gotcha. Thanks. So one can one still receive a perfect score by outputting probabilities as opposed to discrete labels? In other words, even if the third predicted discrete value was correct in your example, both sets would have received the same score under ROC?

\n

[/quote]

\n

I take it you meant: one can still receive a perfect score by outputting discrete labels as opposed to probabilities? That answer is yes. It is possible if you yourself have found the perfect split/threshold between the positive and the negative class. AUC finds this for you by considering every threshold split and only picking the best your model has got.

\n

Since this contest is not judged on raw accuracy, it would not be taking advantage of the evaluation metric if you still submit discrete labels: Those high on the leaderboard are submitting probabilities or ranks.

\n

Other evaluation metrics call for other "tactics".

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 55186, "author_name": "kaguiar", "author_url": "", "post_date": "09/24/2014 20:54:10", "content": "

* and "0" means AN interictal sample

", "votes": null, "replies": [] }, { "id": 55189, "author_name": "jiweiliu", "author_url": "", "post_date": "09/24/2014 22:50:31", "content": "

I think so but instead of 0 and 1, an output of probability (or a rank between 0 and 1) of "preictal" event is much more practical to improve roc score.

", "votes": null, "replies": [] }, { "id": 55190, "author_name": "kaguiar", "author_url": "", "post_date": "09/24/2014 22:57:23", "content": "

Yes, indeed. 

", "votes": null, "replies": [] }, { "id": 55684, "author_name": "adamsoffer", "author_url": "", "post_date": "10/06/2014 14:15:02", "content": "

Why is outputting the probability of a preictal event (as opposed to a discrete class label 0 or 1) more practical to improve ROC score? 

", "votes": null, "replies": [] }, { "id": 55693, "author_name": "franklyn", "author_url": "", "post_date": "10/06/2014 17:46:04", "content": "

Because the ROC iterates over several different thresholds for the classification, its possible that your results will have a better accuracy at some threshold versus others.

", "votes": null, "replies": [] }, { "id": 55726, "author_name": "triskelion", "author_url": "", "post_date": "10/07/2014 11:10:34", "content": "

[quote=franklyn;55693]

\n

Because the ROC iterates over several different thresholds for the classification...

\n

[/quote]

\n

Yes, exactly.

\n

To illustrate:

\n

from sklearn.metrics import roc_auc_score

\n

y_predicted_discrete = [0.0, 0.0, 0.0, 1.0, 1.0]

\n

y_predicted_proba = [0.02, 0.01, 0.05, 0.99, 0.99]

\n

y_ground_truth = [0, 0, 1, 1, 1]

\n

print roc_auc_score(y_ground_truth, y_predicted_discrete) # 0.83

\n

print roc_auc_score(y_ground_truth, y_predicted_proba) # 1.0

\n

Even though the 3rd prediction probability was just 0.05 and the truth was 1, it still got credit for ranking it higher, causing a favorable perfect split.

", "votes": null, "replies": [] }, { "id": 55737, "author_name": "adamsoffer", "author_url": "", "post_date": "10/07/2014 14:31:29", "content": "

Gotcha. Thanks. So one can one still receive a perfect score by outputting probabilities as opposed to discrete labels? In other words, even if the third predicted discrete value was correct in your example, both sets would have received the same score under ROC?

", "votes": null, "replies": [] }, { "id": 55740, "author_name": "triskelion", "author_url": "", "post_date": "10/07/2014 16:41:38", "content": "

[quote=Adam Soffer;55737]

\n

Gotcha. Thanks. So one can one still receive a perfect score by outputting probabilities as opposed to discrete labels? In other words, even if the third predicted discrete value was correct in your example, both sets would have received the same score under ROC?

\n

[/quote]

\n

I take it you meant: one can still receive a perfect score by outputting discrete labels as opposed to probabilities? That answer is yes. It is possible if you yourself have found the perfect split/threshold between the positive and the negative class. AUC finds this for you by considering every threshold split and only picking the best your model has got.

\n

Since this contest is not judged on raw accuracy, it would not be taking advantage of the evaluation metric if you still submit discrete labels: Those high on the leaderboard are submitting probabilities or ranks.

\n

Other evaluation metrics call for other "tactics".

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "55185": "", "55186": "", "55189": "", "55190": "", "55684": "", "55693": "", "55726": "", "55737": "", "55740": "" }, "source": "meta" }