{ "id": 10528, "title": "AUC issues", "url": "/competitions/seizure-prediction/discussion/10528", "author_name": "", "post_date": "2014-10-05T08:22:07.293Z", "votes": null, "comment_count": 9, "views": 1868, "content": "

Hey Folks

\n

Looking for some insight here.  I have tried submitting files for individual subjects (with other subjects filled with zeros), to see how they do, but I am puzzled because they often return AUC < 0.5, suggesting that they are worse than random.  So I just tried submitting a set (Dog 5) which returned 0.49813, then I inverted that set (1 - values) which should in principle have an AUC > 0.5, but it didn't - it returned 0.49805.  So clearly, I am thinking too simplistically about AUC, if both the data set and its inverse return AUC < 0.5.  Anyone have any thoughts on this?  What am I missing?

", "messages": [ { "id": "55627", "postDate": "10/05/2014 08:22:07", "content": "

Hey Folks

\n

Looking for some insight here.  I have tried submitting files for individual subjects (with other subjects filled with zeros), to see how they do, but I am puzzled because they often return AUC < 0.5, suggesting that they are worse than random.  So I just tried submitting a set (Dog 5) which returned 0.49813, then I inverted that set (1 - values) which should in principle have an AUC > 0.5, but it didn't - it returned 0.49805.  So clearly, I am thinking too simplistically about AUC, if both the data set and its inverse return AUC < 0.5.  Anyone have any thoughts on this?  What am I missing?

", "rawMarkdown": "", "votes": null }, { "id": "55629", "postDate": "10/05/2014 09:12:57", "content": "

Hi there. I would suggest studying  a bit more about the ROC AUC.  The AUC for p and 1-p would sum to one if and only if the test set is exactly balanced (i.e same zeros and ones) or p=0.5 which here is not the case.

\n

Edit: I would advise when you CV to check your classifiers sensitivity and specificity alongiside the rather than just AUC

", "rawMarkdown": "", "votes": null }, { "id": "55664", "postDate": "10/06/2014 03:39:15", "content": "

Thanks for that information.  I was surprised because I have done a fair bit of reading about AUC, and e.g. this paper posted by Michael Hills  https://cours.etsmtl.ca/sys828/REFS/A1/Fawcett_PRL2006.pdf makes it clear that for 1-p you should get 1 - AUC, and that AUC is very insensitive to skew data sets.  However, I experimented again and found that if I also invert the rest of the data set (from all 0 to all 1) then I do get the expected behaviour (1 - p gives 1 - AUC), so there is some kind of interaction between the curve for the rest of the data, and the single subject I am submitting - which is probably related to the imbalance as you suggest.

", "rawMarkdown": "", "votes": null }, { "id": "55665", "postDate": "10/06/2014 03:59:57", "content": "

I too am under the impression that (1 - p) should definitely give you (1 - AUC) regardless of imbalance. I think you are seeing the issue that I was describing in that the performances between my per-patient classifiers affect each other in the leaderboard ROC AUC. It was confirmed by competition admin that the effect I was alluding to in my question was correct and I think you are seeing this same effect happening with your submissions. In fact I stumbled across the issue doing what you were doing, submitting all 0s except for 1 patient at a time... trying to figure out how well I was doing on each patient. I never did find that out.

", "rawMarkdown": "", "votes": null }, { "id": "55681", "postDate": "10/06/2014 13:58:00", "content": "

Hmm.Probably I am the one that needs to study more :)

", "rawMarkdown": "", "votes": null }, { "id": "55682", "postDate": "10/06/2014 13:59:48", "content": "

I think this is another phenomenon to do with the difference between averaging over 7 AUCs  versus calculating one AUC for the union of the 7 subjects.

", "rawMarkdown": "", "votes": null }, { "id": "55729", "postDate": "10/07/2014 11:35:24", "content": "

[quote=Jonathan Tapson;55627]

\n

Hey Folks

\n

Looking for some insight here.  I have tried submitting files for individual subjects (with other subjects filled with zeros), to see how they do, but I am puzzled because they often return AUC < 0.5, suggesting that they are worse than random.  So I just tried submitting a set (Dog 5) which returned 0.49813, then I inverted that set (1 - values) which should in principle have an AUC > 0.5, but it didn't - it returned 0.49805.  So clearly, I am thinking too simplistically about AUC, if both the data set and its inverse return AUC < 0.5.  Anyone have any thoughts on this?  What am I missing?

\n

[/quote]

\n\n

Dear Jonathan,

\n\n

I assume that you inverted the whole submission, hence submitting 1s wherever you previously submitted 0s. Am I right?

", "rawMarkdown": "", "votes": null }, { "id": "55732", "postDate": "10/07/2014 12:22:10", "content": "

Dear Jose

\n

As mentioned later in the thread, I did not, initially.  When I eventually did, I got the anticipated result (1 - p) gives (1 - AUC).  However, apart from an intuition that this is an effect of imbalanced data sets, I still don't really understand why it makes a difference.

", "rawMarkdown": "", "votes": null }, { "id": "55800", "postDate": "10/08/2014 18:25:31", "content": "

I'd like to raise another (related) point.  I'd like to ask the organizers how the AUC score is computed.  Is the final AUC score calculated over the entire set, or is it averaged over each subject's AUC score?  It's possible to get 1.0 AUCs on all of the test sets individually but get a lower score if the test examples are pooled.  This might account for the poor test results we're seeing.

", "rawMarkdown": "", "votes": null }, { "id": "55804", "postDate": "10/08/2014 20:28:30", "content": "

Tom,

\n

I think the discussion at https://www.kaggle.com/c/seizure-prediction/forums/t/10383/leaderboard-metric-roc-auc/54251 may help.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 55629, "author_name": "epinephelus", "author_url": "", "post_date": "10/05/2014 09:12:57", "content": "

Hi there. I would suggest studying  a bit more about the ROC AUC.  The AUC for p and 1-p would sum to one if and only if the test set is exactly balanced (i.e same zeros and ones) or p=0.5 which here is not the case.

\n

Edit: I would advise when you CV to check your classifiers sensitivity and specificity alongiside the rather than just AUC

", "votes": null, "replies": [] }, { "id": 55664, "author_name": "jontapson", "author_url": "", "post_date": "10/06/2014 03:39:15", "content": "

Thanks for that information.  I was surprised because I have done a fair bit of reading about AUC, and e.g. this paper posted by Michael Hills  https://cours.etsmtl.ca/sys828/REFS/A1/Fawcett_PRL2006.pdf makes it clear that for 1-p you should get 1 - AUC, and that AUC is very insensitive to skew data sets.  However, I experimented again and found that if I also invert the rest of the data set (from all 0 to all 1) then I do get the expected behaviour (1 - p gives 1 - AUC), so there is some kind of interaction between the curve for the rest of the data, and the single subject I am submitting - which is probably related to the imbalance as you suggest.

", "votes": null, "replies": [] }, { "id": 55665, "author_name": "michaelhills", "author_url": "", "post_date": "10/06/2014 03:59:57", "content": "

I too am under the impression that (1 - p) should definitely give you (1 - AUC) regardless of imbalance. I think you are seeing the issue that I was describing in that the performances between my per-patient classifiers affect each other in the leaderboard ROC AUC. It was confirmed by competition admin that the effect I was alluding to in my question was correct and I think you are seeing this same effect happening with your submissions. In fact I stumbled across the issue doing what you were doing, submitting all 0s except for 1 patient at a time... trying to figure out how well I was doing on each patient. I never did find that out.

", "votes": null, "replies": [] }, { "id": 55681, "author_name": "epinephelus", "author_url": "", "post_date": "10/06/2014 13:58:00", "content": "

Hmm.Probably I am the one that needs to study more :)

", "votes": null, "replies": [] }, { "id": 55682, "author_name": "nickau", "author_url": "", "post_date": "10/06/2014 13:59:48", "content": "

I think this is another phenomenon to do with the difference between averaging over 7 AUCs  versus calculating one AUC for the union of the 7 subjects.

", "votes": null, "replies": [] }, { "id": 55729, "author_name": "joseleiva", "author_url": "", "post_date": "10/07/2014 11:35:24", "content": "

[quote=Jonathan Tapson;55627]

\n

Hey Folks

\n

Looking for some insight here.  I have tried submitting files for individual subjects (with other subjects filled with zeros), to see how they do, but I am puzzled because they often return AUC < 0.5, suggesting that they are worse than random.  So I just tried submitting a set (Dog 5) which returned 0.49813, then I inverted that set (1 - values) which should in principle have an AUC > 0.5, but it didn't - it returned 0.49805.  So clearly, I am thinking too simplistically about AUC, if both the data set and its inverse return AUC < 0.5.  Anyone have any thoughts on this?  What am I missing?

\n

[/quote]

\n\n

Dear Jonathan,

\n\n

I assume that you inverted the whole submission, hence submitting 1s wherever you previously submitted 0s. Am I right?

", "votes": null, "replies": [] }, { "id": 55732, "author_name": "jontapson", "author_url": "", "post_date": "10/07/2014 12:22:10", "content": "

Dear Jose

\n

As mentioned later in the thread, I did not, initially.  When I eventually did, I got the anticipated result (1 - p) gives (1 - AUC).  However, apart from an intuition that this is an effect of imbalanced data sets, I still don't really understand why it makes a difference.

", "votes": null, "replies": [] }, { "id": 55800, "author_name": "", "author_url": "", "post_date": "10/08/2014 18:25:31", "content": "

I'd like to raise another (related) point.  I'd like to ask the organizers how the AUC score is computed.  Is the final AUC score calculated over the entire set, or is it averaged over each subject's AUC score?  It's possible to get 1.0 AUCs on all of the test sets individually but get a lower score if the test examples are pooled.  This might account for the poor test results we're seeing.

", "votes": null, "replies": [] }, { "id": 55804, "author_name": "trentb", "author_url": "", "post_date": "10/08/2014 20:28:30", "content": "

Tom,

\n

I think the discussion at https://www.kaggle.com/c/seizure-prediction/forums/t/10383/leaderboard-metric-roc-auc/54251 may help.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "55627": "", "55629": "", "55664": "", "55665": "", "55681": "", "55682": "", "55729": "", "55732": "", "55800": "", "55804": "" }, "source": "meta" }