{ "id": 1545, "title": "Calculate AUC", "url": "/competitions/kddcup2012-track2/discussion/1545", "author_name": "", "post_date": "2012-03-15T11:44:00.777Z", "votes": null, "comment_count": 9, "views": 8580, "content": "

How to claculate AUC in the track.

\r\n

I don't understand the meaning of TP & FP in this track and I can't draw the ROC curve.

", "messages": [ { "id": "9349", "postDate": "03/15/2012 11:44:00", "content": "

How to claculate AUC in the track.

\r\n

I don't understand the meaning of TP & FP in this track and I can't draw the ROC curve.

", "rawMarkdown": "", "votes": null }, { "id": "9352", "postDate": "03/15/2012 12:23:15", "content": "

take a look at http://www.kddcup2012.org/c/kddcup2012-track2/forums/t/1519/auc-is-also-weighted-by-ad-impression

", "rawMarkdown": "", "votes": null }, { "id": "9386", "postDate": "03/16/2012 05:14:48", "content": "

Sorry

\r\n

Assume that there is two instance which answer are 10 clicks 40 impressions & 1 clicks 2 impressions individually, and we predict CTR are 0.24(12 clicks 50 impressions) & 0.4(2 clicks 5 impressions).

\r\n

How to calculate AUC value ?

", "rawMarkdown": "", "votes": null }, { "id": "9461", "postDate": "03/18/2012 05:57:43", "content": "

Then you wold have 55 instances to calculate AUC. 14 positive and 41 negative.

\r\n

Taking the 5 impressions as example:
\r\nSupose you predicted a CTR of 0.45, you would have (2 clicks and 5 impressions)
\r\nactual = 1, predicted = 0.45
\r\nactual = 1, predicted = 0.45
\r\nactual = 0, predicted = 0.45
\r\nactual = 0, predicted = 0.45
\r\nactual = 0, predicted = 0.45
\r\nIn this example you get an AUC of 0.5. Each impression will generate a distinct instance, and then you can calculate the AUC.

", "rawMarkdown": "", "votes": null }, { "id": "9463", "postDate": "03/18/2012 06:20:10", "content": "
$ cat case0
10 40 0.24
1 2 0.4
\r\n
$ sort -nk3 case0 | awk -f auc.awk 
0.470674
\r\n
$ cat auc.awk 
BEGIN{
f0=-1
}
{
if($3!=f0){
        auc+=(x-x0)*(y+y0);
        f0=$3
        y0=y;
        x0=x;
}
x+=$2-$1;
y+=$1;
}
END{
        auc+=(x-x0)*(y+y0);
        auc/=2*x*y;
        print auc
}
\r\n
3 points on the curve: (0,0), (30/31, 10/11), (1,1)
", "rawMarkdown": "", "votes": null }, { "id": "9523", "postDate": "03/19/2012 08:47:27", "content": "

Thanks

", "rawMarkdown": "", "votes": null }, { "id": "9800", "postDate": "03/28/2012 13:18:29", "content": "

Most papers on click prediction use perplexity metric for performance measure. Why do you choose AUC and MAE?

", "rawMarkdown": "", "votes": null }, { "id": "9954", "postDate": "04/04/2012 15:28:39", "content": "

AUC computed in the way described above seems to be almost irrelevant to the measure of algorithm performance. Given the example of two ads with statistics above even the perfect classifier that gets the CRT perfectly (0.5 and 0.25) gets AUC 0.52932, exactly\r\n the same as a very poor classifier that gives CRTs (0.1 and 0.01).
\r\nThe AUC score depends more on the data than on the algorithm itself.
\r\nThe only thing measured is whether CRTs are in the right order. But as demonstrated here two algorithms with very different quality may have identical ordering of CRTs.\r\n

", "rawMarkdown": "", "votes": null }, { "id": "9965", "postDate": "04/05/2012 02:14:20", "content": "

and that's why MAE is also used...

", "rawMarkdown": "", "votes": null }, { "id": "9969", "postDate": "04/05/2012 07:47:23", "content": "

I would like to elaborate more. I have checked it on some more examples and AUC computed in the way described above IS irrelevant for comparisons of the algorithm quality.\r\n
\r\nFor algorithm that perfectly predicts CRTs for all records the algorithm presented above gives AUC = 0.5. The algorithm with inferior quality can score both worse than that (which is not astonishing at all) and much better than that (which probably is).\r\n
\r\nSmall example:
\r\nFirst the perfect agorithm:
\r\nimp. clks rate pred FP TP interval integral
\r\n0 0
\r\n100 10 0,1 0,1 90 10 0,1 0,005
\r\n100 11 0,11 0,11 89 11 0,01 0,00105
\r\n100 12 0,12 0,12 88 12 0,01 0,00115
\r\n100 13 0,13 0,13 87 13 0,01 0,00125
\r\n100 14 0,14 0,14 86 14 0,01 0,00135
\r\n100 50 0,5 0,5 50 50 0,36 0,1152
\r\n100 1 0 100 0,5 0,375
\r\nSUM 0,5

\r\n

Then the algorithm, that gives much lower prediction for last row
\r\nimp. clks rate pred FP TP interval integral
\r\n0 0
\r\n100 10 0,1 0,1 90 10 0,1 0,005
\r\n100 11 0,11 0,11 89 11 0,01 0,00105
\r\n100 12 0,12 0,12 88 12 0,01 0,00115
\r\n100 13 0,13 0,13 87 13 0,01 0,00125
\r\n100 14 0,14 0,14 86 14 0,01 0,00135
\r\n100 50 0,5 0,15 50 50 0,01 0,0032
\r\n100 1 0 100 0,85 0,6375
\r\nSUM 0,6505

\r\n

Even when the ordering of the prodictions is wrong the AUC measure can still be better than for perfect algorithm

\r\n

imp. clks rate pred FP TP interval integral
\r\n0 0
\r\n100 50 0,5 0,1 50 50 0,1 0,025
\r\n100 11 0,11 0,11 89 11 0,01 0,00305
\r\n100 12 0,12 0,12 88 12 0,01 0,00115
\r\n100 13 0,13 0,13 87 13 0,01 0,00125
\r\n100 14 0,14 0,14 86 14 0,01 0,00135
\r\n100 15 0,15 0,15 85 15 0,01 0,00145
\r\n100 1 0 100 0,85 0,48875
\r\nSUM 0,522

\r\n

Simply this algorithm is wrong.
\r\nAnd this leaves the question open - how one should compute AUC.
\r\nThis is not obvious in any way since direct application of the algorithm mentioned in the evaluation section is not directly applicable.\r\n

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 9352, "author_name": "leustagos", "author_url": "", "post_date": "03/15/2012 12:23:15", "content": "

take a look at http://www.kddcup2012.org/c/kddcup2012-track2/forums/t/1519/auc-is-also-weighted-by-ad-impression

", "votes": null, "replies": [] }, { "id": 9386, "author_name": "helloworld34041", "author_url": "", "post_date": "03/16/2012 05:14:48", "content": "

Sorry

\r\n

Assume that there is two instance which answer are 10 clicks 40 impressions & 1 clicks 2 impressions individually, and we predict CTR are 0.24(12 clicks 50 impressions) & 0.4(2 clicks 5 impressions).

\r\n

How to calculate AUC value ?

", "votes": null, "replies": [] }, { "id": 9461, "author_name": "leustagos", "author_url": "", "post_date": "03/18/2012 05:57:43", "content": "

Then you wold have 55 instances to calculate AUC. 14 positive and 41 negative.

\r\n

Taking the 5 impressions as example:
\r\nSupose you predicted a CTR of 0.45, you would have (2 clicks and 5 impressions)
\r\nactual = 1, predicted = 0.45
\r\nactual = 1, predicted = 0.45
\r\nactual = 0, predicted = 0.45
\r\nactual = 0, predicted = 0.45
\r\nactual = 0, predicted = 0.45
\r\nIn this example you get an AUC of 0.5. Each impression will generate a distinct instance, and then you can calculate the AUC.

", "votes": null, "replies": [] }, { "id": 9463, "author_name": "wangyuantao", "author_url": "", "post_date": "03/18/2012 06:20:10", "content": "
$ cat case0
10 40 0.24
1 2 0.4
\r\n
$ sort -nk3 case0 | awk -f auc.awk 
0.470674
\r\n
$ cat auc.awk 
BEGIN{
f0=-1
}
{
if($3!=f0){
        auc+=(x-x0)*(y+y0);
        f0=$3
        y0=y;
        x0=x;
}
x+=$2-$1;
y+=$1;
}
END{
        auc+=(x-x0)*(y+y0);
        auc/=2*x*y;
        print auc
}
\r\n
3 points on the curve: (0,0), (30/31, 10/11), (1,1)
", "votes": null, "replies": [] }, { "id": 9523, "author_name": "helloworld34041", "author_url": "", "post_date": "03/19/2012 08:47:27", "content": "

Thanks

", "votes": null, "replies": [] }, { "id": 9800, "author_name": "alexeigor", "author_url": "", "post_date": "03/28/2012 13:18:29", "content": "

Most papers on click prediction use perplexity metric for performance measure. Why do you choose AUC and MAE?

", "votes": null, "replies": [] }, { "id": 9954, "author_name": "hazyforest", "author_url": "", "post_date": "04/04/2012 15:28:39", "content": "

AUC computed in the way described above seems to be almost irrelevant to the measure of algorithm performance. Given the example of two ads with statistics above even the perfect classifier that gets the CRT perfectly (0.5 and 0.25) gets AUC 0.52932, exactly\r\n the same as a very poor classifier that gives CRTs (0.1 and 0.01).
\r\nThe AUC score depends more on the data than on the algorithm itself.
\r\nThe only thing measured is whether CRTs are in the right order. But as demonstrated here two algorithms with very different quality may have identical ordering of CRTs.\r\n

", "votes": null, "replies": [] }, { "id": 9965, "author_name": "leustagos", "author_url": "", "post_date": "04/05/2012 02:14:20", "content": "

and that's why MAE is also used...

", "votes": null, "replies": [] }, { "id": 9969, "author_name": "hazyforest", "author_url": "", "post_date": "04/05/2012 07:47:23", "content": "

I would like to elaborate more. I have checked it on some more examples and AUC computed in the way described above IS irrelevant for comparisons of the algorithm quality.\r\n
\r\nFor algorithm that perfectly predicts CRTs for all records the algorithm presented above gives AUC = 0.5. The algorithm with inferior quality can score both worse than that (which is not astonishing at all) and much better than that (which probably is).\r\n
\r\nSmall example:
\r\nFirst the perfect agorithm:
\r\nimp. clks rate pred FP TP interval integral
\r\n0 0
\r\n100 10 0,1 0,1 90 10 0,1 0,005
\r\n100 11 0,11 0,11 89 11 0,01 0,00105
\r\n100 12 0,12 0,12 88 12 0,01 0,00115
\r\n100 13 0,13 0,13 87 13 0,01 0,00125
\r\n100 14 0,14 0,14 86 14 0,01 0,00135
\r\n100 50 0,5 0,5 50 50 0,36 0,1152
\r\n100 1 0 100 0,5 0,375
\r\nSUM 0,5

\r\n

Then the algorithm, that gives much lower prediction for last row
\r\nimp. clks rate pred FP TP interval integral
\r\n0 0
\r\n100 10 0,1 0,1 90 10 0,1 0,005
\r\n100 11 0,11 0,11 89 11 0,01 0,00105
\r\n100 12 0,12 0,12 88 12 0,01 0,00115
\r\n100 13 0,13 0,13 87 13 0,01 0,00125
\r\n100 14 0,14 0,14 86 14 0,01 0,00135
\r\n100 50 0,5 0,15 50 50 0,01 0,0032
\r\n100 1 0 100 0,85 0,6375
\r\nSUM 0,6505

\r\n

Even when the ordering of the prodictions is wrong the AUC measure can still be better than for perfect algorithm

\r\n

imp. clks rate pred FP TP interval integral
\r\n0 0
\r\n100 50 0,5 0,1 50 50 0,1 0,025
\r\n100 11 0,11 0,11 89 11 0,01 0,00305
\r\n100 12 0,12 0,12 88 12 0,01 0,00115
\r\n100 13 0,13 0,13 87 13 0,01 0,00125
\r\n100 14 0,14 0,14 86 14 0,01 0,00135
\r\n100 15 0,15 0,15 85 15 0,01 0,00145
\r\n100 1 0 100 0,85 0,48875
\r\nSUM 0,522

\r\n

Simply this algorithm is wrong.
\r\nAnd this leaves the question open - how one should compute AUC.
\r\nThis is not obvious in any way since direct application of the algorithm mentioned in the evaluation section is not directly applicable.\r\n

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "9349": "", "9352": "", "9386": "", "9461": "", "9463": "", "9523": "", "9800": "", "9954": "", "9965": "", "9969": "" }, "source": "meta" }