\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F94532%2Ff8d7046771a43664f19a2a413ba9db35%2Froc.png?generation=1608581781774899&alt=media\")
I'm very confused with this score calculation. I made one very simple example and it give me score of 1. But obviously the 3rd classification is wrong.
\nroc_auc_score(np.asarray([0,0,1,1,0]),np.asarray([0.1,0.2,0.4,0.7,0.3]))
\ncalculated score is 1.
", "messages": [ { "id": "1121475", "postDate": "12/21/2020 17:05:58", "content": "I'm very confused with this score calculation. I made one very simple example and it give me score of 1. But obviously the 3rd classification is wrong.
\nroc_auc_score(np.asarray([0,0,1,1,0]),np.asarray([0.1,0.2,0.4,0.7,0.3]))
\ncalculated score is 1.
", "rawMarkdown": "I'm very confused with this score calculation. I made one very simple example and it give me score of 1. But obviously the 3rd classification is wrong. \n\nroc_auc_score(np.asarray([0,0,1,1,0]),np.asarray([0.1,0.2,0.4,0.7,0.3]))\n\ncalculated score is 1.", "votes": null }, { "id": "1121687", "postDate": "12/21/2020 20:19:46", "content": "There's nothing wrong with the AUC calculation. The key to understanding why is in your statement \"obviously the 3rd classification is wrong\". Why? I'm guessing you're saying that because the model gave a probility of 0.4 that the result was true (which is less than a 50-50 chance)?
\nBut what if it is more important to not misclassify something as false? Then we might lower the threshold and say that we predict something to be true as long as there's a 40% chance, for example (in which case the model would be spot-on). The AUC gives us a way of comparing models that is independent of what threshold values we might want to use.
\nI'd suggest you work through the calculation for your example by hand, and draw the ROC curve (true positive rate vs false positive rate — there's plenty of material on the web if you want a refresher).
\nThe result should be something like this (assuming I haven't screwed up somewhere =):
\n
It's also instructive to look at what happens if you were to change the 3rd prediction from 0.4 to 0.35, 0.3, and, 0.25.
", "rawMarkdown": "There's nothing wrong with the AUC calculation. The key to understanding why is in your statement \"obviously the 3rd classification is wrong\". Why? I'm guessing you're saying that because the model gave a probility of 0.4 that the result was true (which is less than a 50-50 chance)? \n\nBut what if it is more important to not misclassify something as false? Then we might lower the threshold and say that we predict something to be true as long as there's a 40% chance, for example (in which case the model would be spot-on). The AUC gives us a way of comparing models that is independent of what threshold values we might want to use.\n\nI'd suggest you work through the calculation for your example by hand, and draw the ROC curve (true positive rate vs false positive rate — there's plenty of material on the web if you want a refresher).\n\nThe result should be something like this (assuming I haven't screwed up somewhere =):\n \n\nIt's also instructive to look at what happens if you were to change the 3rd prediction from 0.4 to 0.35, 0.3, and, 0.25.", "votes": null }, { "id": "1121970", "postDate": "12/22/2020 04:25:31", "content": "Thanks for clarification. I also read some background material. So, essentially it's a relatively ranking. In my example, as long as the 3rd and 4th numbers are ranked top1 and top2, the score would be 1.
", "rawMarkdown": "Thanks for clarification. I also read some background material. So, essentially it's a relatively ranking. In my example, as long as the 3rd and 4th numbers are ranked top1 and top2, the score would be 1.", "votes": null }, { "id": "1122382", "postDate": "12/22/2020 11:53:06", "content": "AUC=1 means your model can perfectly distinguish between probability of being 0 and probability of being 1.
\nIf there any threshold that can split probabilities without any errors -> AUC=1
\nIn your case, threshold is anywhere between 0.3-0.4
That is correct. If you sort your predictions, they become 0.1, 0.2, 0.3, 0.4, 0.7 and the ground truths move into positions 0, 0, 0, 1, 1. The value of AUC is the percentage of 1's to the right of 0's which is 6 because two 1's are each to the right of three 0's. Then divide by number of all 0 and 1 pairs which in this case is 6 = 3x2. In your example, you have 6 divided by 6 equals 1.
There's nothing wrong with the AUC calculation. The key to understanding why is in your statement \"obviously the 3rd classification is wrong\". Why? I'm guessing you're saying that because the model gave a probility of 0.4 that the result was true (which is less than a 50-50 chance)?
\nBut what if it is more important to not misclassify something as false? Then we might lower the threshold and say that we predict something to be true as long as there's a 40% chance, for example (in which case the model would be spot-on). The AUC gives us a way of comparing models that is independent of what threshold values we might want to use.
\nI'd suggest you work through the calculation for your example by hand, and draw the ROC curve (true positive rate vs false positive rate — there's plenty of material on the web if you want a refresher).
\nThe result should be something like this (assuming I haven't screwed up somewhere =):
\n
It's also instructive to look at what happens if you were to change the 3rd prediction from 0.4 to 0.35, 0.3, and, 0.25.
", "votes": null, "replies": [ { "id": 1121970, "author_name": "noahxi", "author_url": "", "post_date": "12/22/2020 04:25:31", "content": "Thanks for clarification. I also read some background material. So, essentially it's a relatively ranking. In my example, as long as the 3rd and 4th numbers are ranked top1 and top2, the score would be 1.
", "votes": null, "replies": [] } ] }, { "id": 1122382, "author_name": "pavelvod", "author_url": "", "post_date": "12/22/2020 11:53:06", "content": "AUC=1 means your model can perfectly distinguish between probability of being 0 and probability of being 1.
\nIf there any threshold that can split probabilities without any errors -> AUC=1
\nIn your case, threshold is anywhere between 0.3-0.4
That is correct. If you sort your predictions, they become 0.1, 0.2, 0.3, 0.4, 0.7 and the ground truths move into positions 0, 0, 0, 1, 1. The value of AUC is the percentage of 1's to the right of 0's which is 6 because two 1's are each to the right of three 0's. Then divide by number of all 0 and 1 pairs which in this case is 6 = 3x2. In your example, you have 6 divided by 6 equals 1.