I was hoping someone could correct my line of reasoning here since empirically it seems flawed given all the top performs are using logloss objectives.
\n\nHere it goes:
\n\nSo we all know using the sample click through rates of the training sets as the likelihoods for the test set gives a LB score of ~.63. Out of curiosity I ran a logistic regression with the only predictor being the inverse logistic transformation of sample clickthrough for each ad. We know that a solution of beta_0 = 0 and beta_1 = 1 would produce a CV score ~.63 since our prediction method is just applying a function and then taking the inverse. However, the logistic regression converges to an inferior solution for the Map@12 metric, producing a cv score of 0.47419. This would lead me to think I should try building a custom objective function that is better correlated with MAP@12 (since map@12 is non differentiable) but it doesn't seem like anyone is doing that, instead everyone seems very concerned with building big sparse datasets and finding efficient log loss optimization methods. What gives? Clearly im missing something. Thanks everyone!
", "messages": [ { "id": 153087, "postDate": "2016-12-29T19:33:37.457Z", "content": "I was hoping someone could correct my line of reasoning here since empirically it seems flawed given all the top performs are using logloss objectives.
\n\nHere it goes:
\n\nSo we all know using the sample click through rates of the training sets as the likelihoods for the test set gives a LB score of ~.63. Out of curiosity I ran a logistic regression with the only predictor being the inverse logistic transformation of sample clickthrough for each ad. We know that a solution of beta_0 = 0 and beta_1 = 1 would produce a CV score ~.63 since our prediction method is just applying a function and then taking the inverse. However, the logistic regression converges to an inferior solution for the Map@12 metric, producing a cv score of 0.47419. This would lead me to think I should try building a custom objective function that is better correlated with MAP@12 (since map@12 is non differentiable) but it doesn't seem like anyone is doing that, instead everyone seems very concerned with building big sparse datasets and finding efficient log loss optimization methods. What gives? Clearly im missing something. Thanks everyone!
", "rawMarkdown": "I was hoping someone could correct my line of reasoning here since empirically it seems flawed given all the top performs are using logloss objectives.\r\n\r\nHere it goes:\r\n\r\nSo we all know using the sample click through rates of the training sets as the likelihoods for the test set gives a LB score of ~.63. Out of curiosity I ran a logistic regression with the only predictor being the inverse logistic transformation of sample clickthrough for each ad. We know that a solution of beta_0 = 0 and beta_1 = 1 would produce a CV score ~.63 since our prediction method is just applying a function and then taking the inverse. However, the logistic regression converges to an inferior solution for the Map@12 metric, producing a cv score of 0.47419. This would lead me to think I should try building a custom objective function that is better correlated with MAP@12 (since map@12 is non differentiable) but it doesn't seem like anyone is doing that, instead everyone seems very concerned with building big sparse datasets and finding efficient log loss optimization methods. What gives? Clearly im missing something. Thanks everyone!" } ], "comments": [], "raw_markdown_by_id": { "153087": "I was hoping someone could correct my line of reasoning here since empirically it seems flawed given all the top performs are using logloss objectives.\r\n\r\nHere it goes:\r\n\r\nSo we all know using the sample click through rates of the training sets as the likelihoods for the test set gives a LB score of ~.63. Out of curiosity I ran a logistic regression with the only predictor being the inverse logistic transformation of sample clickthrough for each ad. We know that a solution of beta_0 = 0 and beta_1 = 1 would produce a CV score ~.63 since our prediction method is just applying a function and then taking the inverse. However, the logistic regression converges to an inferior solution for the Map@12 metric, producing a cv score of 0.47419. This would lead me to think I should try building a custom objective function that is better correlated with MAP@12 (since map@12 is non differentiable) but it doesn't seem like anyone is doing that, instead everyone seems very concerned with building big sparse datasets and finding efficient log loss optimization methods. What gives? Clearly im missing something. Thanks everyone!" } }