I'm confused about the submission and MAP5.
\n\nBasically, in our training data, we are given one hotel cluster per row.\nYet for each row in the test data, we will have to submit 5 guesses per row.
\n\nIs the best we can hope for is for one of the 5 hotel clusters we guessed matches with the correct true value? Or are there 5 actual true values per row and we need to get all 5?
\n\nI hope my question makes sense.
", "messages": [ { "id": "118748", "postDate": "05/05/2016 05:27:11", "content": "I'm confused about the submission and MAP5.
\n\nBasically, in our training data, we are given one hotel cluster per row.\nYet for each row in the test data, we will have to submit 5 guesses per row.
\n\nIs the best we can hope for is for one of the 5 hotel clusters we guessed matches with the correct true value? Or are there 5 actual true values per row and we need to get all 5?
\n\nI hope my question makes sense.
", "rawMarkdown": "I'm confused about the submission and MAP5.\r\n\r\nBasically, in our training data, we are given one hotel cluster per row.\r\nYet for each row in the test data, we will have to submit 5 guesses per row.\r\n\r\nIs the best we can hope for is for one of the 5 hotel clusters we guessed matches with the correct true value? Or are there 5 actual true values per row and we need to get all 5?\r\n\r\nI hope my question makes sense.", "votes": null }, { "id": "118756", "postDate": "05/05/2016 06:44:39", "content": "There is only one true answer for each row and you can submit up to five guesses. If your first guess is right you get a 1 for this row, if the second 1/2, if the third then 1/3 and so on. If the true answer does not appear in your guesses then you get a 0. An average of these values over all rows will be your final score.
\n\nTo do well you should put the most probable predictions first. And if your approach results in less than five predictions for some rows, don't hesitate to fill the rest with something - it can't make things worse =).
", "rawMarkdown": "There is only one true answer for each row and you can submit up to five guesses. If your first guess is right you get a 1 for this row, if the second 1/2, if the third then 1/3 and so on. If the true answer does not appear in your guesses then you get a 0. An average of these values over all rows will be your final score. \r\n\r\nTo do well you should put the most probable predictions first. And if your approach results in less than five predictions for some rows, don't hesitate to fill the rest with something - it can't make things worse =).", "votes": null }, { "id": "118757", "postDate": "05/05/2016 06:45:45", "content": "More on the topic is here.
", "rawMarkdown": "* We don't know how many predictions are true in row\r\n* We can recommend at most 5 for each user\r\n* Bad guesses are not penalized! \r\n* If there is incorrect prediction, order matters. If all predictions are correct, order doesn't matter. It is better to submit more certain recommendations first. AP@5 score reflects this. \r\n\r\nMore on the topic is [here][1]. \r\n\r\n\r\n [1]: https://www.kaggle.com/c/FacebookRecruiting/forums/t/2002/alternate-explanation-of-mean-average-precision", "votes": null }, { "id": "119198", "postDate": "05/08/2016 00:51:36", "content": "[quote=dune_dweller;118756]
\n\nThere is only one true answer for each row and you can submit up to five guesses. If your first guess is right you get a 1 for this row, if the second 1/2, if the third then 1/3 and so on. If the true answer does not appear in your guesses then you get a 0. An average of these values over all rows will be your final score.
\n\nTo do well you should put the most probable predictions first. And if your approach results in less than five predictions for some rows, don't hesitate to fill the rest with something - it can't make things worse =).
\n\n[/quote]\nCan you explain why you think the position on the row matters. As far as I can see the evaluation formula, it is a summation of the probability of each guess.
", "rawMarkdown": "[quote=dune_dweller;118756]\r\n\r\nThere is only one true answer for each row and you can submit up to five guesses. If your first guess is right you get a 1 for this row, if the second 1/2, if the third then 1/3 and so on. If the true answer does not appear in your guesses then you get a 0. An average of these values over all rows will be your final score. \r\n\r\nTo do well you should put the most probable predictions first. And if your approach results in less than five predictions for some rows, don't hesitate to fill the rest with something - it can't make things worse =).\r\n\r\n[/quote]\r\nCan you explain why you think the position on the row matters. As far as I can see the evaluation formula, it is a summation of the probability of each guess.", "votes": null }, { "id": "119220", "postDate": "05/08/2016 07:20:18", "content": "Hi,
\n\nhave a close look at the formula again! If you hit the right cluster in the first place you get 1. If you only manage to predict the right cluster in the fifth place then you get 1/5!
\n\nGerhard
", "rawMarkdown": "Hi,\r\n\r\nhave a close look at the formula again! If you hit the right cluster in the first place you get 1. If you only manage to predict the right cluster in the fifth place then you get 1/5!\r\n\r\nGerhard", "votes": null }, { "id": "119230", "postDate": "05/08/2016 09:19:42", "content": "[quote=Ashwin Murthy;119198]
\n\nCan you explain why you think the position on the row matters. As far as I can see the evaluation formula, it is a summation of the probability of each guess.
\n\n[/quote]
\n\nP(k) in the formula is not probability, it's precision at k. We don't even submit any probabilities. =)
\n\nRead the link that @Gennady Khvorykh provided and try to figure out how it simplifies in case of single correct answer per row.
", "rawMarkdown": "[quote=Ashwin Murthy;119198]\r\n\r\nCan you explain why you think the position on the row matters. As far as I can see the evaluation formula, it is a summation of the probability of each guess. \r\n\r\n\r\n[/quote]\r\n\r\nP(k) in the formula is not probability, it's precision at k. We don't even submit any probabilities. =)\r\n\r\nRead the link that @Gennady Khvorykh provided and try to figure out how it simplifies in case of single correct answer per row.", "votes": null }, { "id": "119311", "postDate": "05/09/2016 04:52:50", "content": "About why we are permitted to select up to 5 predictions - my explanation has been that, on the Expedia website, on a search results page, for the "average" computer monitor size, about 5 hotels would appear above the fold. So even if the eventual hotel (target in the training data) a user would be booking appears in the #5 spot, that is still somewhat powerful in terms of getting the users' eyes on it. How this exactly translates between hotels <=> hotel clusters I'm not sure, but the notion of wanting to predict for the top 5 makes sense in regards to what users would see on a web page. There may be other reasons, but that's my rationale on it.
", "rawMarkdown": "About why we are permitted to select up to 5 predictions - my explanation has been that, on the Expedia website, on a search results page, for the \"average\" computer monitor size, about 5 hotels would appear above the fold. So even if the eventual hotel (target in the training data) a user would be booking appears in the #5 spot, that is still somewhat powerful in terms of getting the users' eyes on it. How this exactly translates between hotels <=> hotel clusters I'm not sure, but the notion of wanting to predict for the top 5 makes sense in regards to what users would see on a web page. There may be other reasons, but that's my rationale on it.", "votes": null }, { "id": "119345", "postDate": "05/09/2016 10:34:40", "content": "[quote=MightyBird;119220]
\n\nHi,
\n\nhave a close look at the formula again! If you hit the right cluster in the first place you get 1. If you only manage to predict the right cluster in the fifth place then you get 1/5!
\n\nGerhard
\n\n[/quote]
", "rawMarkdown": "[quote=MightyBird;119220]\r\n\r\nHi,\r\n\r\nhave a close look at the formula again! If you hit the right cluster in the first place you get 1. If you only manage to predict the right cluster in the fifth place then you get 1/5!\r\n\r\nGerhard\r\n\r\n\r\n[/quote]", "votes": null }, { "id": "119349", "postDate": "05/09/2016 11:43:49", "content": "https://www.kaggle.com/wendykan/expedia-hotel-recommendations/map-k-demo
\n\nObviusly, order matter.
", "rawMarkdown": "https://www.kaggle.com/wendykan/expedia-hotel-recommendations/map-k-demo\r\n\r\nObviusly, order matter.", "votes": null }, { "id": "119350", "postDate": "05/09/2016 11:59:46", "content": "I have read the formula over and over again, but I don't see how it says what you and dune_dweller claim it says. If your fifth prediction is the correct one, you indeed get 1/5. However, the "average precision" (the sum of P's) according to this formula is not 1 if the first prediction is correct, and neither is it 0.5 if the second prediction is correct, etc.
\n\nInstead, what the formula says is that you have to sum the precisions from k = 1 up to k = 5 (assuming that you submitted 5 predictions), which means that if you get the first prediction right, the AP score for that row is:
\n\nP(1) + P(2) + P(3) + P(4) + P(5) = 1 + 0.5 + 0.3333 + 0.25 + 0.2 = 2.2833 (or 137/60, to be exact)
\n\nSimilarly, if the second prediction is correct, you get 0 + 0.5 + 0.3333 + 0.25 + 0.2 = 1.2833
\n\nAnd so on.
\n\nThis formula in fact puts a heavy penalty on your prediction if you submit less than 5 predicted clusters. For example, if you are completely sure that you have the right answer on your first try and submit only 1 prediction, n in the formula will be 1, so you only sum from k = 1 up to k = 1, and therefore your AP score will be equal to P(1) = 1, which is much less than the score you got if you had added four further predictions that you knew to be incorrect, which doesn't really make any sense.
\n\nThis is all assuming that you don't list the same cluster several times in one row. If you listed your correct prediction 5 times, your score would be the maximum attainable value of 5 according to the formula.
\n\nThe interpretation of the formula that you suggest would be correct if n were defined not as the number of predicted hotel clusters (as is the case) but as the rank of the first correct prediction among the clusters that you listed. Alternatively and more generally (in a way also applicable to situations where there is not just one correct target value), you would also get this result if P(k) were defined as the precision at cutoff k if the element at k is correct (i.e. its precision by itself is 1) and does not also appear at any position j < k, and 0 otherwise.
\n\nThis is actually what the C# production implementation of the MAP algorithm published by Kaggle does: It only sums over those P(k) values where the item in the kth position in your submitted list is in fact a correct prediction.
\n\nIf I am wrong, my apologies, and please correct me.
", "rawMarkdown": "I have read the formula over and over again, but I don't see how it says what you and dune_dweller claim it says. If your fifth prediction is the correct one, you indeed get 1/5. However, the \"average precision\" (the sum of *P*'s) according to this formula is not 1 if the first prediction is correct, and neither is it 0.5 if the second prediction is correct, etc.\r\n\r\nInstead, what the formula says is that you have to **sum** the precisions from *k* = 1 up to *k* = 5 (assuming that you submitted 5 predictions), which means that if you get the **first** prediction right, the AP score for that row is:\r\n\r\n*P*(1) + *P*(2) + *P*(3) + *P*(4) + *P*(5) = 1 + 0.5 + 0.3333 + 0.25 + 0.2 = 2.2833 (or 137/60, to be exact)\r\n\r\nSimilarly, if the **second** prediction is correct, you get 0 + 0.5 + 0.3333 + 0.25 + 0.2 = 1.2833\r\n\r\nAnd so on.\r\n\r\nThis formula in fact puts a heavy penalty on your prediction if you submit less than 5 predicted clusters. For example, if you are completely sure that you have the right answer on your first try and submit only 1 prediction, *n* in the formula will be 1, so you only sum from *k* = 1 up to *k* = 1, and therefore your AP score will be equal to *P*(1) = 1, which is much less than the score you got if you had added four further predictions that you knew to be incorrect, which doesn't really make any sense.\r\n\r\nThis is all assuming that you don't list the same cluster several times in one row. If you listed your correct prediction 5 times, your score would be the maximum attainable value of 5 according to the formula.\r\n\r\nThe interpretation of the formula that you suggest would be correct if n were defined not as the number of predicted hotel clusters (as is the case) but as **the rank of the first correct prediction** among the clusters that you listed. Alternatively and more generally (in a way also applicable to situations where there is not just one correct target value), you would also get this result if *P*(*k*) were defined as the precision at cutoff *k* **if the element at *k* is correct (i.e. its precision by itself is 1) and does not also appear at any position j < k, and 0 otherwise**.\r\n\r\nThis is actually what the C# production implementation of the MAP algorithm published by Kaggle does: It only sums over those *P*(*k*) values where the item in the kth position in your submitted list is in fact a correct prediction.\r\n\r\nIf I am wrong, my apologies, and please correct me.", "votes": null }, { "id": "119352", "postDate": "05/09/2016 12:29:54", "content": "Just took a look at the formula used in R. You don't get more points if you repeat a valid cluster.
\n\napk <- function(k, actual, predicted)\n{\n score <- 0.0\n cnt <- 0.0\n for (i in 1:min(k,length(predicted)))\n {\n if (predicted[i] %in% actual && !(predicted[i] %in% predicted[0:(i-1)]))\n {\n cnt <- cnt + 1\n score <- score + cnt/i \n }\n }\n score <- score / min(length(actual), k)\n score\n}\nThe relevant line being:
\n\nif (predicted[i] %in% actual && !(predicted[i] %in% predicted[0:(i-1)]))\nSince "actual" is allways of length 1 the inner if statements are only executed once per instance/test row.
\n\nHere is the corresponding C code
\n\nif (solTokens.Contains(subTokens[i], _StringComparer) && !alreadyRecommended.Contains(subTokens[i], _StringComparer)) {\nCheers
\n\nGerhard
", "rawMarkdown": "Just took a look at the formula used in R. You don't get more points if you repeat a valid cluster.\r\n\r\n apk <- function(k, actual, predicted)\r\n {\r\n score <- 0.0\r\n cnt <- 0.0\r\n for (i in 1:min(k,length(predicted)))\r\n {\r\n if (predicted[i] %in% actual && !(predicted[i] %in% predicted[0:(i-1)]))\r\n {\r\n cnt <- cnt + 1\r\n score <- score + cnt/i \r\n }\r\n }\r\n score <- score / min(length(actual), k)\r\n score\r\n }\r\n\r\nThe relevant line being:\r\n\r\n if (predicted[i] %in% actual && !(predicted[i] %in% predicted[0:(i-1)]))\r\n\r\nSince \"actual\" is allways of length 1 the inner if statements are only executed once per instance/test row.\r\n\r\nHere is the corresponding C code\r\n\r\n if (solTokens.Contains(subTokens[i], _StringComparer) && !alreadyRecommended.Contains(subTokens[i], _StringComparer)) {\r\n\r\nCheers\r\n\r\nGerhard", "votes": null }, { "id": "119396", "postDate": "05/09/2016 21:21:57", "content": "@Gergely Petho,
\n\nIf you look at the wiki page for MAP metric, you'll see that
\n\n\n\n\nP(k) means the precision at cut-off k in the item list, i.e., the ratio of number of users followed up to the position k over the number k, and P(k) equals 0 when the k-th item is not followed upon recommendation...
\n
It talks about users and not hotel clusters, but it's clear that for each row you only take precision values from the places where you have a correct prediction. In this competition we always have a single correct answer per row. So you get P(i) if the correct answer is in place i of your submission, or 0 if no guesses are correct.
", "rawMarkdown": "Gergely Petho, \r\n\r\nIf you look at the [wiki page for MAP metric](https://www.kaggle.com/wiki/MeanAveragePrecision), you'll see that\r\n\r\n> P(k) means the precision at cut-off k in the item list, i.e., the ratio of number of users followed up to the position k over the number k, and P(k) equals **0** when the k-th item is not followed upon recommendation...\r\n\r\nIt talks about users and not hotel clusters, but it's clear that for each row you only take precision values from the places where you have a correct prediction. In this competition we always have a single correct answer per row. So you get P(i) if the correct answer is in place i of your submission, or 0 if no guesses are correct.", "votes": null }, { "id": "119413", "postDate": "05/10/2016 03:13:18", "content": "Hi, @dune_dweller
\n\nHow do you know their single correct answer per row (ground truth in the test data)? Is it confirmed by Admin?
\n\nThank you
", "rawMarkdown": "Hi, @dune_dweller \r\n\r\nHow do you know their single correct answer per row (ground truth in the test data)? Is it confirmed by Admin?\r\n\r\nThank you", "votes": null }, { "id": "119414", "postDate": "05/10/2016 03:44:45", "content": "", "rawMarkdown": "[It is](https://www.kaggle.com/c/expedia-hotel-recommendations/forums/t/20228/what-is-the-prediction-goal/117936#post117936).", "votes": null } ], "comments": [ { "id": 118756, "author_name": "dvasyukova", "author_url": "", "post_date": "05/05/2016 06:44:39", "content": "There is only one true answer for each row and you can submit up to five guesses. If your first guess is right you get a 1 for this row, if the second 1/2, if the third then 1/3 and so on. If the true answer does not appear in your guesses then you get a 0. An average of these values over all rows will be your final score.
\n\nTo do well you should put the most probable predictions first. And if your approach results in less than five predictions for some rows, don't hesitate to fill the rest with something - it can't make things worse =).
", "votes": null, "replies": [] }, { "id": 118757, "author_name": "followbigdata", "author_url": "", "post_date": "05/05/2016 06:45:45", "content": "More on the topic is here.
", "votes": null, "replies": [] }, { "id": 119198, "author_name": "ashwinmurthy", "author_url": "", "post_date": "05/08/2016 00:51:36", "content": "[quote=dune_dweller;118756]
\n\nThere is only one true answer for each row and you can submit up to five guesses. If your first guess is right you get a 1 for this row, if the second 1/2, if the third then 1/3 and so on. If the true answer does not appear in your guesses then you get a 0. An average of these values over all rows will be your final score.
\n\nTo do well you should put the most probable predictions first. And if your approach results in less than five predictions for some rows, don't hesitate to fill the rest with something - it can't make things worse =).
\n\n[/quote]\nCan you explain why you think the position on the row matters. As far as I can see the evaluation formula, it is a summation of the probability of each guess.
", "votes": null, "replies": [] }, { "id": 119220, "author_name": "mightybird", "author_url": "", "post_date": "05/08/2016 07:20:18", "content": "Hi,
\n\nhave a close look at the formula again! If you hit the right cluster in the first place you get 1. If you only manage to predict the right cluster in the fifth place then you get 1/5!
\n\nGerhard
", "votes": null, "replies": [] }, { "id": 119230, "author_name": "dvasyukova", "author_url": "", "post_date": "05/08/2016 09:19:42", "content": "[quote=Ashwin Murthy;119198]
\n\nCan you explain why you think the position on the row matters. As far as I can see the evaluation formula, it is a summation of the probability of each guess.
\n\n[/quote]
\n\nP(k) in the formula is not probability, it's precision at k. We don't even submit any probabilities. =)
\n\nRead the link that @Gennady Khvorykh provided and try to figure out how it simplifies in case of single correct answer per row.
", "votes": null, "replies": [] }, { "id": 119311, "author_name": "siliconvalley", "author_url": "", "post_date": "05/09/2016 04:52:50", "content": "About why we are permitted to select up to 5 predictions - my explanation has been that, on the Expedia website, on a search results page, for the "average" computer monitor size, about 5 hotels would appear above the fold. So even if the eventual hotel (target in the training data) a user would be booking appears in the #5 spot, that is still somewhat powerful in terms of getting the users' eyes on it. How this exactly translates between hotels <=> hotel clusters I'm not sure, but the notion of wanting to predict for the top 5 makes sense in regards to what users would see on a web page. There may be other reasons, but that's my rationale on it.
", "votes": null, "replies": [] }, { "id": 119345, "author_name": "pethog", "author_url": "", "post_date": "05/09/2016 10:34:40", "content": "[quote=MightyBird;119220]
\n\nHi,
\n\nhave a close look at the formula again! If you hit the right cluster in the first place you get 1. If you only manage to predict the right cluster in the fifth place then you get 1/5!
\n\nGerhard
\n\n[/quote]
", "votes": null, "replies": [] }, { "id": 119349, "author_name": "khaoticmind", "author_url": "", "post_date": "05/09/2016 11:43:49", "content": "https://www.kaggle.com/wendykan/expedia-hotel-recommendations/map-k-demo
\n\nObviusly, order matter.
", "votes": null, "replies": [] }, { "id": 119350, "author_name": "pethog", "author_url": "", "post_date": "05/09/2016 11:59:46", "content": "I have read the formula over and over again, but I don't see how it says what you and dune_dweller claim it says. If your fifth prediction is the correct one, you indeed get 1/5. However, the "average precision" (the sum of P's) according to this formula is not 1 if the first prediction is correct, and neither is it 0.5 if the second prediction is correct, etc.
\n\nInstead, what the formula says is that you have to sum the precisions from k = 1 up to k = 5 (assuming that you submitted 5 predictions), which means that if you get the first prediction right, the AP score for that row is:
\n\nP(1) + P(2) + P(3) + P(4) + P(5) = 1 + 0.5 + 0.3333 + 0.25 + 0.2 = 2.2833 (or 137/60, to be exact)
\n\nSimilarly, if the second prediction is correct, you get 0 + 0.5 + 0.3333 + 0.25 + 0.2 = 1.2833
\n\nAnd so on.
\n\nThis formula in fact puts a heavy penalty on your prediction if you submit less than 5 predicted clusters. For example, if you are completely sure that you have the right answer on your first try and submit only 1 prediction, n in the formula will be 1, so you only sum from k = 1 up to k = 1, and therefore your AP score will be equal to P(1) = 1, which is much less than the score you got if you had added four further predictions that you knew to be incorrect, which doesn't really make any sense.
\n\nThis is all assuming that you don't list the same cluster several times in one row. If you listed your correct prediction 5 times, your score would be the maximum attainable value of 5 according to the formula.
\n\nThe interpretation of the formula that you suggest would be correct if n were defined not as the number of predicted hotel clusters (as is the case) but as the rank of the first correct prediction among the clusters that you listed. Alternatively and more generally (in a way also applicable to situations where there is not just one correct target value), you would also get this result if P(k) were defined as the precision at cutoff k if the element at k is correct (i.e. its precision by itself is 1) and does not also appear at any position j < k, and 0 otherwise.
\n\nThis is actually what the C# production implementation of the MAP algorithm published by Kaggle does: It only sums over those P(k) values where the item in the kth position in your submitted list is in fact a correct prediction.
\n\nIf I am wrong, my apologies, and please correct me.
", "votes": null, "replies": [] }, { "id": 119352, "author_name": "mightybird", "author_url": "", "post_date": "05/09/2016 12:29:54", "content": "Just took a look at the formula used in R. You don't get more points if you repeat a valid cluster.
\n\napk <- function(k, actual, predicted)\n{\n score <- 0.0\n cnt <- 0.0\n for (i in 1:min(k,length(predicted)))\n {\n if (predicted[i] %in% actual && !(predicted[i] %in% predicted[0:(i-1)]))\n {\n cnt <- cnt + 1\n score <- score + cnt/i \n }\n }\n score <- score / min(length(actual), k)\n score\n}\nThe relevant line being:
\n\nif (predicted[i] %in% actual && !(predicted[i] %in% predicted[0:(i-1)]))\nSince "actual" is allways of length 1 the inner if statements are only executed once per instance/test row.
\n\nHere is the corresponding C code
\n\nif (solTokens.Contains(subTokens[i], _StringComparer) && !alreadyRecommended.Contains(subTokens[i], _StringComparer)) {\nCheers
\n\nGerhard
", "votes": null, "replies": [] }, { "id": 119396, "author_name": "dvasyukova", "author_url": "", "post_date": "05/09/2016 21:21:57", "content": "@Gergely Petho,
\n\nIf you look at the wiki page for MAP metric, you'll see that
\n\n\n\n\nP(k) means the precision at cut-off k in the item list, i.e., the ratio of number of users followed up to the position k over the number k, and P(k) equals 0 when the k-th item is not followed upon recommendation...
\n
It talks about users and not hotel clusters, but it's clear that for each row you only take precision values from the places where you have a correct prediction. In this competition we always have a single correct answer per row. So you get P(i) if the correct answer is in place i of your submission, or 0 if no guesses are correct.
", "votes": null, "replies": [] }, { "id": 119413, "author_name": "rmtogether", "author_url": "", "post_date": "05/10/2016 03:13:18", "content": "Hi, @dune_dweller
\n\nHow do you know their single correct answer per row (ground truth in the test data)? Is it confirmed by Admin?
\n\nThank you
", "votes": null, "replies": [] }, { "id": 119414, "author_name": "dvasyukova", "author_url": "", "post_date": "05/10/2016 03:44:45", "content": "", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "118748": "I'm confused about the submission and MAP5.\r\n\r\nBasically, in our training data, we are given one hotel cluster per row.\r\nYet for each row in the test data, we will have to submit 5 guesses per row.\r\n\r\nIs the best we can hope for is for one of the 5 hotel clusters we guessed matches with the correct true value? Or are there 5 actual true values per row and we need to get all 5?\r\n\r\nI hope my question makes sense.", "118756": "There is only one true answer for each row and you can submit up to five guesses. If your first guess is right you get a 1 for this row, if the second 1/2, if the third then 1/3 and so on. If the true answer does not appear in your guesses then you get a 0. An average of these values over all rows will be your final score. \r\n\r\nTo do well you should put the most probable predictions first. And if your approach results in less than five predictions for some rows, don't hesitate to fill the rest with something - it can't make things worse =).", "118757": "* We don't know how many predictions are true in row\r\n* We can recommend at most 5 for each user\r\n* Bad guesses are not penalized! \r\n* If there is incorrect prediction, order matters. If all predictions are correct, order doesn't matter. It is better to submit more certain recommendations first. AP@5 score reflects this. \r\n\r\nMore on the topic is [here][1]. \r\n\r\n\r\n [1]: https://www.kaggle.com/c/FacebookRecruiting/forums/t/2002/alternate-explanation-of-mean-average-precision", "119198": "[quote=dune_dweller;118756]\r\n\r\nThere is only one true answer for each row and you can submit up to five guesses. If your first guess is right you get a 1 for this row, if the second 1/2, if the third then 1/3 and so on. If the true answer does not appear in your guesses then you get a 0. An average of these values over all rows will be your final score. \r\n\r\nTo do well you should put the most probable predictions first. And if your approach results in less than five predictions for some rows, don't hesitate to fill the rest with something - it can't make things worse =).\r\n\r\n[/quote]\r\nCan you explain why you think the position on the row matters. As far as I can see the evaluation formula, it is a summation of the probability of each guess.", "119220": "Hi,\r\n\r\nhave a close look at the formula again! If you hit the right cluster in the first place you get 1. If you only manage to predict the right cluster in the fifth place then you get 1/5!\r\n\r\nGerhard", "119230": "[quote=Ashwin Murthy;119198]\r\n\r\nCan you explain why you think the position on the row matters. As far as I can see the evaluation formula, it is a summation of the probability of each guess. \r\n\r\n\r\n[/quote]\r\n\r\nP(k) in the formula is not probability, it's precision at k. We don't even submit any probabilities. =)\r\n\r\nRead the link that @Gennady Khvorykh provided and try to figure out how it simplifies in case of single correct answer per row.", "119311": "About why we are permitted to select up to 5 predictions - my explanation has been that, on the Expedia website, on a search results page, for the \"average\" computer monitor size, about 5 hotels would appear above the fold. So even if the eventual hotel (target in the training data) a user would be booking appears in the #5 spot, that is still somewhat powerful in terms of getting the users' eyes on it. How this exactly translates between hotels <=> hotel clusters I'm not sure, but the notion of wanting to predict for the top 5 makes sense in regards to what users would see on a web page. There may be other reasons, but that's my rationale on it.", "119345": "[quote=MightyBird;119220]\r\n\r\nHi,\r\n\r\nhave a close look at the formula again! If you hit the right cluster in the first place you get 1. If you only manage to predict the right cluster in the fifth place then you get 1/5!\r\n\r\nGerhard\r\n\r\n\r\n[/quote]", "119349": "https://www.kaggle.com/wendykan/expedia-hotel-recommendations/map-k-demo\r\n\r\nObviusly, order matter.", "119350": "I have read the formula over and over again, but I don't see how it says what you and dune_dweller claim it says. If your fifth prediction is the correct one, you indeed get 1/5. However, the \"average precision\" (the sum of *P*'s) according to this formula is not 1 if the first prediction is correct, and neither is it 0.5 if the second prediction is correct, etc.\r\n\r\nInstead, what the formula says is that you have to **sum** the precisions from *k* = 1 up to *k* = 5 (assuming that you submitted 5 predictions), which means that if you get the **first** prediction right, the AP score for that row is:\r\n\r\n*P*(1) + *P*(2) + *P*(3) + *P*(4) + *P*(5) = 1 + 0.5 + 0.3333 + 0.25 + 0.2 = 2.2833 (or 137/60, to be exact)\r\n\r\nSimilarly, if the **second** prediction is correct, you get 0 + 0.5 + 0.3333 + 0.25 + 0.2 = 1.2833\r\n\r\nAnd so on.\r\n\r\nThis formula in fact puts a heavy penalty on your prediction if you submit less than 5 predicted clusters. For example, if you are completely sure that you have the right answer on your first try and submit only 1 prediction, *n* in the formula will be 1, so you only sum from *k* = 1 up to *k* = 1, and therefore your AP score will be equal to *P*(1) = 1, which is much less than the score you got if you had added four further predictions that you knew to be incorrect, which doesn't really make any sense.\r\n\r\nThis is all assuming that you don't list the same cluster several times in one row. If you listed your correct prediction 5 times, your score would be the maximum attainable value of 5 according to the formula.\r\n\r\nThe interpretation of the formula that you suggest would be correct if n were defined not as the number of predicted hotel clusters (as is the case) but as **the rank of the first correct prediction** among the clusters that you listed. Alternatively and more generally (in a way also applicable to situations where there is not just one correct target value), you would also get this result if *P*(*k*) were defined as the precision at cutoff *k* **if the element at *k* is correct (i.e. its precision by itself is 1) and does not also appear at any position j < k, and 0 otherwise**.\r\n\r\nThis is actually what the C# production implementation of the MAP algorithm published by Kaggle does: It only sums over those *P*(*k*) values where the item in the kth position in your submitted list is in fact a correct prediction.\r\n\r\nIf I am wrong, my apologies, and please correct me.", "119352": "Just took a look at the formula used in R. You don't get more points if you repeat a valid cluster.\r\n\r\n apk <- function(k, actual, predicted)\r\n {\r\n score <- 0.0\r\n cnt <- 0.0\r\n for (i in 1:min(k,length(predicted)))\r\n {\r\n if (predicted[i] %in% actual && !(predicted[i] %in% predicted[0:(i-1)]))\r\n {\r\n cnt <- cnt + 1\r\n score <- score + cnt/i \r\n }\r\n }\r\n score <- score / min(length(actual), k)\r\n score\r\n }\r\n\r\nThe relevant line being:\r\n\r\n if (predicted[i] %in% actual && !(predicted[i] %in% predicted[0:(i-1)]))\r\n\r\nSince \"actual\" is allways of length 1 the inner if statements are only executed once per instance/test row.\r\n\r\nHere is the corresponding C code\r\n\r\n if (solTokens.Contains(subTokens[i], _StringComparer) && !alreadyRecommended.Contains(subTokens[i], _StringComparer)) {\r\n\r\nCheers\r\n\r\nGerhard", "119396": "Gergely Petho, \r\n\r\nIf you look at the [wiki page for MAP metric](https://www.kaggle.com/wiki/MeanAveragePrecision), you'll see that\r\n\r\n> P(k) means the precision at cut-off k in the item list, i.e., the ratio of number of users followed up to the position k over the number k, and P(k) equals **0** when the k-th item is not followed upon recommendation...\r\n\r\nIt talks about users and not hotel clusters, but it's clear that for each row you only take precision values from the places where you have a correct prediction. In this competition we always have a single correct answer per row. So you get P(i) if the correct answer is in place i of your submission, or 0 if no guesses are correct.", "119413": "Hi, @dune_dweller \r\n\r\nHow do you know their single correct answer per row (ground truth in the test data)? Is it confirmed by Admin?\r\n\r\nThank you", "119414": "[It is](https://www.kaggle.com/c/expedia-hotel-recommendations/forums/t/20228/what-is-the-prediction-goal/117936#post117936)." }, "source": "meta" }