I don't understand the AUC metric in this contest. The paper referenced in the evaluation page only talks about positive and negative examples, and here we have clicks and impressions. How's the AUC applied here ?
", "messages": [ { "id": "10375", "postDate": "04/25/2012 16:55:25", "content": "I don't understand the AUC metric in this contest. The paper referenced in the evaluation page only talks about positive and negative examples, and here we have clicks and impressions. How's the AUC applied here ?
", "rawMarkdown": "", "votes": null }, { "id": "10376", "postDate": "04/25/2012 18:07:26", "content": "Each impression has a binary response - click or don't click. However, sometimes the ad is shown again to the same user at the same depth. In these cases, these test instances are aggregated together, and you don't know the # of impressions.
\r\nWhen you make a prediction (say 0.4) on a test instance, this may correspond to making a prediction of 0.4 on 100 binary responses, of which 38 are positive responses (clicks) and 62 are negative.
", "rawMarkdown": "", "votes": null }, { "id": "10417", "postDate": "04/26/2012 22:08:09", "content": "[quote=Ben Hamner;10376]When you make a prediction (say 0.4) on a test instance, this may correspond to making a prediction of 0.4 on 100 binary responses, of which 38 are positive responses (clicks) and 62 are negative.
\r\n[/quote]
\r\nYes I can see this part, but still how do you calculate AUC here ?
\r\nFor example, let's say we have:
\r\nA: 30 clicks, 70 non-clicks, prediction 0.65
\r\nB: 30 clicks, 70 non-clicks, prediction 0.75
\r\nExactly how do you calculate AUC ?
\r\n", "rawMarkdown": "", "votes": null }, { "id": "10420", "postDate": "04/26/2012 22:27:57", "content": "
From there, you treat that as making 200 predictions:
\r\n30 positive instances where you predict 0.65
\r\n70 negative instances where you predict 0.65
\r\n30 positive instances where you predict 0.75
\r\n70 negative instances where you predict 0.75,
and calculate AUC in the standard way across the 200 predictions.
\r\ndef scoreClickAUC(num_clicks, num_impressions, predicted_ctr):\r\n """\r\n Calculates the area under the ROC curve (AUC) for click rates\r\n\r\n Parameters\r\n ----------\r\n num_clicks : a list containing the number of clicks\r\n\r\n num_impressions : a list containing the number of impressions\r\n\r\n predicted_ctr : a list containing the predicted click-through rates\r\n\r\n Returns\r\n -------\r\n auc : the area under the ROC curve (AUC) for click rates\r\n """\r\n i_sorted = sorted(range(len(predicted_ctr)),key=lambda i: predicted_ctr[i],\r\n reverse=True)\r\n auc_temp = 0.0\r\n click_sum = 0.0\r\n old_click_sum = 0.0\r\n no_click = 0.0\r\n no_click_sum = 0.0\r\n\r\n # treat all instances with the same predicted_ctr as coming from the\r\n # same bucket\r\n last_ctr = predicted_ctr[i_sorted[0]] + 1.0\r\n\r\n for i in range(len(predicted_ctr)):\r\n if last_ctr != predicted_ctr[i_sorted[i]]: \r\n auc_temp += (click_sum+old_click_sum) * no_click / 2.0 \r\n old_click_sum = click_sum\r\n no_click = 0.0\r\n last_ctr = predicted_ctr[i_sorted[i]]\r\n no_click += num_impressions[i_sorted[i]] - num_clicks[i_sorted[i]]\r\n no_click_sum += num_impressions[i_sorted[i]] - num_clicks[i_sorted[i]]\r\n click_sum += num_clicks[i_sorted[i]]\r\n auc_temp += (click_sum+old_click_sum) * no_click / 2.0\r\n auc = auc_temp / (click_sum * no_click_sum)\r\n return auc\r\n", "rawMarkdown": "", "votes": null }, { "id": "10869", "postDate": "05/08/2012 15:19:29", "content": "
it may be helpful to someone (an R fast implementation of AUC, one minute to predict 20 million rows):
\r\n#y = actual response, x predicted
\r\nscore.AUC <- function(y, x) {
x1 = x[y==1]; n1 = as.numeric(length(x1))
x2 = x[y==0]; n2 = as.numeric(length(x2))
r = rank(c(x1,x2))
auc = (sum(r[1:n1]) - n1*(n1+1)/2) / (n1*n2)
return (auc)
}",
"rawMarkdown": "",
"votes": null
},
{
"id": "10994",
"postDate": "05/13/2012 11:17:19",
"content": "Hello Ben,
\r\nInstead of predicting Click & Impression for each record in test data and then the CTR, are we allowed to directly use CTR from training dataset as our target variable in regression and output CTR for test records?
", "rawMarkdown": "", "votes": null }, { "id": "11082", "postDate": "05/17/2012 16:13:42", "content": "Hey. Happened to see your name several times even from long time ago. Ensemble..
\r\nAssuming your predictions as {P_i}, your classification don't need to be I (P_i > 0.5). It could be I (P_i > T ), where T could be any number within [0,1]. Therefore, for each T, you have a type-I error and type-II error. Draw a graph of type I vs. type\r\n II error, then the area below the curve is defined as AUC, which measures the discriminant power.
\r\n\r\n
[quote=B Yang;10375]
\r\nI don't understand the AUC metric in this contest. The paper referenced in the evaluation page only talks about positive and negative examples, and here we have clicks and impressions. How's the AUC applied here ?
\r\n[/quote]
", "rawMarkdown": "", "votes": null }, { "id": "11122", "postDate": "05/19/2012 12:45:03", "content": "it is giving consistent results for me.
\r\nAnyway, it may just be an aproximation, but its running time is way lower!
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 10376, "author_name": "benhamner", "author_url": "", "post_date": "04/25/2012 18:07:26", "content": "Each impression has a binary response - click or don't click. However, sometimes the ad is shown again to the same user at the same depth. In these cases, these test instances are aggregated together, and you don't know the # of impressions.
\r\nWhen you make a prediction (say 0.4) on a test instance, this may correspond to making a prediction of 0.4 on 100 binary responses, of which 38 are positive responses (clicks) and 62 are negative.
", "votes": null, "replies": [] }, { "id": 10417, "author_name": "byang1", "author_url": "", "post_date": "04/26/2012 22:08:09", "content": "[quote=Ben Hamner;10376]When you make a prediction (say 0.4) on a test instance, this may correspond to making a prediction of 0.4 on 100 binary responses, of which 38 are positive responses (clicks) and 62 are negative.
\r\n[/quote]
\r\nYes I can see this part, but still how do you calculate AUC here ?
\r\nFor example, let's say we have:
\r\nA: 30 clicks, 70 non-clicks, prediction 0.65
\r\nB: 30 clicks, 70 non-clicks, prediction 0.75
\r\nExactly how do you calculate AUC ?
\r\n", "votes": null, "replies": [] }, { "id": 10420, "author_name": "benhamner", "author_url": "", "post_date": "04/26/2012 22:27:57", "content": "
From there, you treat that as making 200 predictions:
\r\n30 positive instances where you predict 0.65
\r\n70 negative instances where you predict 0.65
\r\n30 positive instances where you predict 0.75
\r\n70 negative instances where you predict 0.75,
and calculate AUC in the standard way across the 200 predictions.
\r\ndef scoreClickAUC(num_clicks, num_impressions, predicted_ctr):\r\n """\r\n Calculates the area under the ROC curve (AUC) for click rates\r\n\r\n Parameters\r\n ----------\r\n num_clicks : a list containing the number of clicks\r\n\r\n num_impressions : a list containing the number of impressions\r\n\r\n predicted_ctr : a list containing the predicted click-through rates\r\n\r\n Returns\r\n -------\r\n auc : the area under the ROC curve (AUC) for click rates\r\n """\r\n i_sorted = sorted(range(len(predicted_ctr)),key=lambda i: predicted_ctr[i],\r\n reverse=True)\r\n auc_temp = 0.0\r\n click_sum = 0.0\r\n old_click_sum = 0.0\r\n no_click = 0.0\r\n no_click_sum = 0.0\r\n\r\n # treat all instances with the same predicted_ctr as coming from the\r\n # same bucket\r\n last_ctr = predicted_ctr[i_sorted[0]] + 1.0\r\n\r\n for i in range(len(predicted_ctr)):\r\n if last_ctr != predicted_ctr[i_sorted[i]]: \r\n auc_temp += (click_sum+old_click_sum) * no_click / 2.0 \r\n old_click_sum = click_sum\r\n no_click = 0.0\r\n last_ctr = predicted_ctr[i_sorted[i]]\r\n no_click += num_impressions[i_sorted[i]] - num_clicks[i_sorted[i]]\r\n no_click_sum += num_impressions[i_sorted[i]] - num_clicks[i_sorted[i]]\r\n click_sum += num_clicks[i_sorted[i]]\r\n auc_temp += (click_sum+old_click_sum) * no_click / 2.0\r\n auc = auc_temp / (click_sum * no_click_sum)\r\n return auc\r\n", "votes": null, "replies": [] }, { "id": 10869, "author_name": "leustagos", "author_url": "", "post_date": "05/08/2012 15:19:29", "content": "
it may be helpful to someone (an R fast implementation of AUC, one minute to predict 20 million rows):
\r\n#y = actual response, x predicted
\r\nscore.AUC <- function(y, x) {
x1 = x[y==1]; n1 = as.numeric(length(x1))
x2 = x[y==0]; n2 = as.numeric(length(x2))
r = rank(c(x1,x2))
auc = (sum(r[1:n1]) - n1*(n1+1)/2) / (n1*n2)
return (auc)
}",
"votes": null,
"replies": []
},
{
"id": 10994,
"author_name": "sashikanthdareddy",
"author_url": "",
"post_date": "05/13/2012 11:17:19",
"content": "Hello Ben,
\r\nInstead of predicting Click & Impression for each record in test data and then the CTR, are we allowed to directly use CTR from training dataset as our target variable in regression and output CTR for test records?
", "votes": null, "replies": [] }, { "id": 11082, "author_name": "nanzhou", "author_url": "", "post_date": "05/17/2012 16:13:42", "content": "Hey. Happened to see your name several times even from long time ago. Ensemble..
\r\nAssuming your predictions as {P_i}, your classification don't need to be I (P_i > 0.5). It could be I (P_i > T ), where T could be any number within [0,1]. Therefore, for each T, you have a type-I error and type-II error. Draw a graph of type I vs. type\r\n II error, then the area below the curve is defined as AUC, which measures the discriminant power.
\r\n\r\n
[quote=B Yang;10375]
\r\nI don't understand the AUC metric in this contest. The paper referenced in the evaluation page only talks about positive and negative examples, and here we have clicks and impressions. How's the AUC applied here ?
\r\n[/quote]
", "votes": null, "replies": [] }, { "id": 11122, "author_name": "leustagos", "author_url": "", "post_date": "05/19/2012 12:45:03", "content": "it is giving consistent results for me.
\r\nAnyway, it may just be an aproximation, but its running time is way lower!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "10375": "", "10376": "", "10417": "", "10420": "", "10869": "", "10994": "", "11082": "", "11122": "" }, "source": "meta" }