From what I learned, the sequence of items in prediction vs answer matters a lot in mAPk metric.
\nSo, how does the organizer order the final answer for each customer? Are they sorted by purchase time, or are they sorted alphabetically by article_id, or some other ways?
\nThanks!
", "messages": [ { "id": "1728089", "postDate": "03/18/2022 15:20:20", "content": "From what I learned, the sequence of items in prediction vs answer matters a lot in mAPk metric.
\nSo, how does the organizer order the final answer for each customer? Are they sorted by purchase time, or are they sorted alphabetically by article_id, or some other ways?
\nThanks!
", "rawMarkdown": "From what I learned, the sequence of items in prediction vs answer matters a lot in mAPk metric.\n\nSo, how does the organizer order the final answer for each customer? Are they sorted by purchase time, or are they sorted alphabetically by article_id, or some other ways?\n\nThanks!", "votes": null }, { "id": "1728224", "postDate": "03/18/2022 17:21:07", "content": "Hi @reaverlee, my understanding is that the groundtruth for a customer can be viewed as an unordered set. The ordering only matters for prediction sequence because the precision at cutoff k (i.e., P(k)) will drop if the relevant item appears later in the sequence. Hence, we hope those relevant can be predicted at top in the prediction sequence. Hope this helps.
\nIf there's any misunderstanding, please correct me, thanks a lot.
Correct. Ground truth sequence does not matter. And prediction sequence does matter.
", "rawMarkdown": "Correct. Ground truth sequence **does not** matter. And prediction sequence **does** matter.", "votes": null }, { "id": "1729055", "postDate": "03/19/2022 15:22:43", "content": "Thank you! I got it wrong when I tested mapk function with [1,2,3,4,5] vs [5,4,3,2,1]. I should've tested them for apk function since it's only for one sample.
", "rawMarkdown": "Thank you! I got it wrong when I tested mapk function with [1,2,3,4,5] vs [5,4,3,2,1]. I should've tested them for apk function since it's only for one sample.", "votes": null }, { "id": "1729075", "postDate": "03/19/2022 15:47:34", "content": "Hi @cdeotte, thanks a lot for the confirmation and explanation.😊
", "rawMarkdown": "Hi @cdeotte, thanks a lot for the confirmation and explanation.😊", "votes": null }, { "id": "1736765", "postDate": "03/27/2022 17:39:34", "content": "A quick follow-up question RE: 'can be viewed as an (unordered) set'. Would be grateful if someone's able to clarify.
\nShould I take this to mean that it is enough to be a member of the ground truth set in order to qualify as a true positive in the precision calculation? What about rel(k) - is it enough for the item at rank k to be a member of the ground truth set for rel(k) be 1, or is that truly matching ranks?
\ne.g. If the ground truth for a customer was [A, B, C] and we were to predict [C, B, A, …], then P(3) = 1, and rel(3) = 1 too?
", "rawMarkdown": "A quick follow-up question RE: 'can be viewed as an (unordered) set'. Would be grateful if someone's able to clarify.\n\nShould I take this to mean that it is enough to be a member of the ground truth set in order to qualify as a true positive in the precision calculation? What about *rel(k)* - is it enough for the item at rank *k* to be a member of the ground truth set for *rel(k)* be 1, or is that truly matching ranks?\n\ne.g. If the ground truth for a customer was [A, B, C] and we were to predict [C, B, A, ...], then P(3) = 1, and rel(3) = 1 too?", "votes": null }, { "id": "1736870", "postDate": "03/27/2022 20:26:06", "content": "\n\ne.g. If the ground truth for a customer was [A, B, C] and we were to predict [C, B, A, …], then P(3) = 1, and rel(3) = 1 too?
\n
Yes, exactly. I made a simple base line prediction notebook that shows the calculation of the MAP@12 in context of a four-fold validation that matches quite well the leaderboard score: 0.0073 vs. 0.0071
\nhttps://www.kaggle.com/code/tbierhance/basic-best-seller-prediction-w-cross-validation
", "rawMarkdown": "> e.g. If the ground truth for a customer was [A, B, C] and we were to predict [C, B, A, …], then P(3) = 1, and rel(3) = 1 too?\n\nYes, exactly. I made a simple base line prediction notebook that shows the calculation of the MAP@12 in context of a four-fold validation that matches quite well the leaderboard score: 0.0073 vs. 0.0071\n\nhttps://www.kaggle.com/code/tbierhance/basic-best-seller-prediction-w-cross-validation", "votes": null }, { "id": "1736887", "postDate": "03/27/2022 20:54:27", "content": "Nice, thanks for that @tbierhance, I'll check it out.
", "rawMarkdown": "Nice, thanks for that @tbierhance, I'll check it out.", "votes": null }, { "id": "1737541", "postDate": "03/28/2022 14:11:12", "content": "thanks for your share
", "rawMarkdown": "thanks for your share", "votes": null }, { "id": "1737608", "postDate": "03/28/2022 14:57:56", "content": "\n\n\nit is enough to be a member of the ground truth
\n
Yes, but it pays to be right in first n predictions
\n\nAlso, in your example, the mapk would $$ \\frac{1}{3} ( (1/1) \\times 1 + (2/2) \\times 1 +(3/3)\\times 1 )= 1 $$
\nrel(k) is a mask. For this competition, it is 1 or 0 so you are not penalized for making a wrong prediction.
\nSo if, in your case, the prediction was [D,C,B,A], then the calculation would change.
", "rawMarkdown": "jagofc \n\n> it is enough to be a member of the ground truth \n\nYes, but it pays to be right in first n predictions\n\nsee this - https://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/discussion/314192#1733955\n\nAlso, in your example, the mapk would $$ \\frac{1}{3} ( (1/1) \\times 1 + (2/2) \\times 1 +(3/3)\\times 1 )= 1 $$\n\nrel(k) is a mask. For this competition, it is 1 or 0 so you are not penalized for making a wrong prediction. \n\nSo if, in your case, the prediction was [D,C,B,A], then the calculation would change.", "votes": null }, { "id": "1737626", "postDate": "03/28/2022 15:09:43", "content": "Great, got it, thanks. Linking in the code for the competition metric which makes all of this clear (featured in a couple of other discussions).
", "rawMarkdown": "Great, got it, thanks. Linking in the [code for the competition metric](https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py) which makes all of this clear (featured in a couple of other discussions).", "votes": null } ], "comments": [ { "id": 1728224, "author_name": "abaojiang", "author_url": "", "post_date": "03/18/2022 17:21:07", "content": "Hi @reaverlee, my understanding is that the groundtruth for a customer can be viewed as an unordered set. The ordering only matters for prediction sequence because the precision at cutoff k (i.e., P(k)) will drop if the relevant item appears later in the sequence. Hence, we hope those relevant can be predicted at top in the prediction sequence. Hope this helps.
\nIf there's any misunderstanding, please correct me, thanks a lot.
Correct. Ground truth sequence does not matter. And prediction sequence does matter.
", "votes": null, "replies": [] }, { "id": 1729055, "author_name": "reaverlee", "author_url": "", "post_date": "03/19/2022 15:22:43", "content": "Thank you! I got it wrong when I tested mapk function with [1,2,3,4,5] vs [5,4,3,2,1]. I should've tested them for apk function since it's only for one sample.
", "votes": null, "replies": [] }, { "id": 1729075, "author_name": "abaojiang", "author_url": "", "post_date": "03/19/2022 15:47:34", "content": "Hi @cdeotte, thanks a lot for the confirmation and explanation.😊
", "votes": null, "replies": [] }, { "id": 1736765, "author_name": "jagofc", "author_url": "", "post_date": "03/27/2022 17:39:34", "content": "A quick follow-up question RE: 'can be viewed as an (unordered) set'. Would be grateful if someone's able to clarify.
\nShould I take this to mean that it is enough to be a member of the ground truth set in order to qualify as a true positive in the precision calculation? What about rel(k) - is it enough for the item at rank k to be a member of the ground truth set for rel(k) be 1, or is that truly matching ranks?
\ne.g. If the ground truth for a customer was [A, B, C] and we were to predict [C, B, A, …], then P(3) = 1, and rel(3) = 1 too?
", "votes": null, "replies": [] }, { "id": 1736870, "author_name": "tbierhance", "author_url": "", "post_date": "03/27/2022 20:26:06", "content": "\n\ne.g. If the ground truth for a customer was [A, B, C] and we were to predict [C, B, A, …], then P(3) = 1, and rel(3) = 1 too?
\n
Yes, exactly. I made a simple base line prediction notebook that shows the calculation of the MAP@12 in context of a four-fold validation that matches quite well the leaderboard score: 0.0073 vs. 0.0071
\nhttps://www.kaggle.com/code/tbierhance/basic-best-seller-prediction-w-cross-validation
", "votes": null, "replies": [] }, { "id": 1736887, "author_name": "jagofc", "author_url": "", "post_date": "03/27/2022 20:54:27", "content": "Nice, thanks for that @tbierhance, I'll check it out.
", "votes": null, "replies": [] }, { "id": 1737608, "author_name": "atulverma", "author_url": "", "post_date": "03/28/2022 14:57:56", "content": "\n\n\nit is enough to be a member of the ground truth
\n
Yes, but it pays to be right in first n predictions
\n\nAlso, in your example, the mapk would $$ \\frac{1}{3} ( (1/1) \\times 1 + (2/2) \\times 1 +(3/3)\\times 1 )= 1 $$
\nrel(k) is a mask. For this competition, it is 1 or 0 so you are not penalized for making a wrong prediction.
\nSo if, in your case, the prediction was [D,C,B,A], then the calculation would change.
", "votes": null, "replies": [] }, { "id": 1737626, "author_name": "jagofc", "author_url": "", "post_date": "03/28/2022 15:09:43", "content": "Great, got it, thanks. Linking in the code for the competition metric which makes all of this clear (featured in a couple of other discussions).
", "votes": null, "replies": [] } ] }, { "id": 1737541, "author_name": "deepkun1995", "author_url": "", "post_date": "03/28/2022 14:11:12", "content": "thanks for your share
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1728089": "From what I learned, the sequence of items in prediction vs answer matters a lot in mAPk metric.\n\nSo, how does the organizer order the final answer for each customer? Are they sorted by purchase time, or are they sorted alphabetically by article_id, or some other ways?\n\nThanks!", "1728224": "Hi @reaverlee, my understanding is that the groundtruth for a customer can be viewed as an **unordered set**. The ordering only matters for **prediction sequence** because the precision at cutoff k (*i.e.*, `P(k)`) will drop if the relevant item appears **later** in the sequence. Hence, we hope those relevant can be predicted at top in the prediction sequence. Hope this helps.