{
  "id": 309152,
  "title": "Doubts in the error metric",
  "url": "/competitions/h-and-m-personalized-fashion-recommendations/discussion/309152",
  "author_name": "Shahnawaz Ahmad",
  "post_date": "2022-02-22T05:27:41.422000",
  "votes": 5,
  "comment_count": 12,
  "views": 0,
  "content": "<p>In the error metric i.e.:</p>\n<p>$$MAP@k = \\frac{1}{U}\\sum_{i=1}^{U}\\sum_{k=1}^{min(n,k)}P(k)*rel(k)$$</p>\n<p>Let,</p>\n<table>\n<thead>\n<tr>\n<th>Cust_ID</th>\n<th>Y_true</th>\n<th>Y_pred</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>[1,2,3,4,5,6,7,8,9,10,11,12,15,13]</td>\n<td>[1,2,3,4,5,6,7,8,9,10,11,12,15,13]</td>\n</tr>\n<tr>\n<td>2</td>\n<td>[2,5,4,6,7,9,10,11,121,234,122,121]</td>\n<td>[2,5,4,6,7,9,10,11,121,234,122,121]</td>\n</tr>\n</tbody>\n</table>\n<p>Here prediction string is the same as the ground truth.</p>\n<p>So, P(i) and rel(i) terms for the above cases are: </p>\n<table>\n<thead>\n<tr>\n<th>Cust_ID</th>\n<th>P(1),rel(1)</th>\n<th>P(2),rel(2)</th>\n<th>P(3),rel(3)</th>\n<th>P(4),rel(4)</th>\n<th>P(5),rel(5)</th>\n<th>P(6),rel(6)</th>\n<th>P(7),rel(7)</th>\n<th>P(8),rel(8)</th>\n<th>P(9),rel(9)</th>\n<th>P(10),rel(10)</th>\n<th>P(11),rel(11)</th>\n<th>P(12),rel(12)</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n</tr>\n<tr>\n<td>2</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n</tr>\n</tbody>\n</table>\n<p>Here all the rel(i) terms are equal to 1 as all the ith strings are relevant.<br>\nApplying the formula:</p>\n<table>\n<thead>\n<tr>\n<th>Cust_ID</th>\n<th>$$\\sum_{k=1}^{12}P(k)*rel(k)$$</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>12</td>\n</tr>\n<tr>\n<td>2</td>\n<td>12</td>\n</tr>\n</tbody>\n</table>\n<p>So, MAP@12 = (12+12)/2 = 12</p>\n<p>But, I think that the value should be in the range [0,1]. If there is an error in the above steps, please indicate.<br>\nI have doubts about the rel(k) term, if you have any resources for understanding this metric please share those.</p>\n<p>TIA</p>",
  "messages": [
    {
      "id": 1700589,
      "postDate": "2022-02-22T05:27:41.423Z",
      "content": "<p>In the error metric i.e.:</p>\n<p>$$MAP@k = \\frac{1}{U}\\sum_{i=1}^{U}\\sum_{k=1}^{min(n,k)}P(k)*rel(k)$$</p>\n<p>Let,</p>\n<table>\n<thead>\n<tr>\n<th>Cust_ID</th>\n<th>Y_true</th>\n<th>Y_pred</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>[1,2,3,4,5,6,7,8,9,10,11,12,15,13]</td>\n<td>[1,2,3,4,5,6,7,8,9,10,11,12,15,13]</td>\n</tr>\n<tr>\n<td>2</td>\n<td>[2,5,4,6,7,9,10,11,121,234,122,121]</td>\n<td>[2,5,4,6,7,9,10,11,121,234,122,121]</td>\n</tr>\n</tbody>\n</table>\n<p>Here prediction string is the same as the ground truth.</p>\n<p>So, P(i) and rel(i) terms for the above cases are: </p>\n<table>\n<thead>\n<tr>\n<th>Cust_ID</th>\n<th>P(1),rel(1)</th>\n<th>P(2),rel(2)</th>\n<th>P(3),rel(3)</th>\n<th>P(4),rel(4)</th>\n<th>P(5),rel(5)</th>\n<th>P(6),rel(6)</th>\n<th>P(7),rel(7)</th>\n<th>P(8),rel(8)</th>\n<th>P(9),rel(9)</th>\n<th>P(10),rel(10)</th>\n<th>P(11),rel(11)</th>\n<th>P(12),rel(12)</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n</tr>\n<tr>\n<td>2</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n<td>1,1</td>\n</tr>\n</tbody>\n</table>\n<p>Here all the rel(i) terms are equal to 1 as all the ith strings are relevant.<br>\nApplying the formula:</p>\n<table>\n<thead>\n<tr>\n<th>Cust_ID</th>\n<th>$$\\sum_{k=1}^{12}P(k)*rel(k)$$</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>12</td>\n</tr>\n<tr>\n<td>2</td>\n<td>12</td>\n</tr>\n</tbody>\n</table>\n<p>So, MAP@12 = (12+12)/2 = 12</p>\n<p>But, I think that the value should be in the range [0,1]. If there is an error in the above steps, please indicate.<br>\nI have doubts about the rel(k) term, if you have any resources for understanding this metric please share those.</p>\n<p>TIA</p>",
      "rawMarkdown": "In the error metric i.e.:\n\n$$MAP@k = \\frac{1}{U}\\sum_{i=1}^{U}\\sum_{k=1}^{min(n,k)}P(k)*rel(k)$$\n\nLet,\n\n| Cust_ID |                        Y_true                     |                     Y_pred                       |\n|---------|----------------------------------|----------------------------------|\n| 1            | [1,2,3,4,5,6,7,8,9,10,11,12,15,13]     | [1,2,3,4,5,6,7,8,9,10,11,12,15,13]     |\n| 2           | [2,5,4,6,7,9,10,11,121,234,122,121] | [2,5,4,6,7,9,10,11,121,234,122,121] |\n\nHere prediction string is the same as the ground truth.\n\nSo, P(i) and rel(i) terms for the above cases are: \n\n| Cust_ID | P(1),rel(1) | P(2),rel(2) | P(3),rel(3) | P(4),rel(4) | P(5),rel(5) | P(6),rel(6) | P(7),rel(7) | P(8),rel(8) | P(9),rel(9) | P(10),rel(10) | P(11),rel(11) | P(12),rel(12) |\n|---------|------|------|------|------|------|------|------|------|------|-------|------|-------|\n| 1            | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 |\n| 2           | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 |\n\nHere all the rel(i) terms are equal to 1 as all the ith strings are relevant.\nApplying the formula:\n\n|Cust_ID|$$\\sum_{k=1}^{12}P(k)*rel(k)$$|\n|--------|-------------------|\n|1           |              12              |\n|2           |               12             |\n\nSo, MAP@12 = (12+12)/2 = 12\n\nBut, I think that the value should be in the range [0,1]. If there is an error in the above steps, please indicate.\nI have doubts about the rel(k) term, if you have any resources for understanding this metric please share those.\n\nTIA",
      "votes": 5
    },
    {
      "id": 1711194,
      "postDate": "2022-03-03T17:41:56.967Z",
      "content": "<p>The correct formula is</p>\n<p>$$ MAP@12 = \\frac{1}{U} \\sum_{u=1}^{U} \\left( \\frac{1}{min(m,12)} \\sum_{k=1}^{min(n,12)}{P(k) \\times rel(k)} \\right) $$</p>\n<p>where <code>m</code> is the number of ground truths per customer, <code>n</code> is the number of predictions per customer, and <code>U</code> is the number of customers. <code>P(k)</code> is the precision at cutoff <code>k</code>, and <code>rel(k)</code> is an indicator function equaling 1 if the item at rank <code>k</code> is a relevant (correct) label, zero otherwise.</p>",
      "rawMarkdown": "The correct formula is\n\n$$ MAP@12 = \\frac{1}{U} \\sum_{u=1}^{U} \\left( \\frac{1}{min(m,12)} \\sum_{k=1}^{min(n,12)}{P(k) \\times rel(k)} \\right) $$\n\nwhere `m` is the number of ground truths per customer, `n` is the number of predictions per customer, and `U` is the number of customers. `P(k)` is the precision at cutoff `k`, and `rel(k)` is an indicator function equaling 1 if the item at rank `k` is a relevant (correct) label, zero otherwise.",
      "votes": 3,
      "replies": [
        {
          "id": 1722777,
          "postDate": "2022-03-14T21:40:44.450Z",
          "content": "<p><a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> You are absolutely correct. The missing term was dropped due to a copy-paste \"bug\". Since many of our competitions with MAP@K have a single ground truth, that term often drops out. I've updated the Evaluation page.</p>",
          "rawMarkdown": "@cdeotte You are absolutely correct. The missing term was dropped due to a copy-paste \"bug\". Since many of our competitions with MAP@K have a single ground truth, that term often drops out. I've updated the Evaluation page.",
          "votes": 1
        },
        {
          "id": 1722807,
          "postDate": "2022-03-14T21:57:21.493Z",
          "content": "<p>Yey! 🤛 We were right!</p>",
          "rawMarkdown": "Yey! 🤛 We were right!",
          "votes": 1
        }
      ]
    },
    {
      "id": 1708692,
      "postDate": "2022-03-01T16:18:57.723Z",
      "content": "<p>Hmm, i agree. It appears that the formula is missing the fraction  UPDATE <code>(1/min(m,k))</code> where <code>m</code> is number of ground truths per customer . </p>\n<p>Also the your formula (not Kaggle formula) uses the variable <code>k</code> twice to mean two different things. (It is both the <code>@k</code> and the index variable under summation).  They should use two different variables. (i.e. j=1 to min(n,k) )</p>",
      "rawMarkdown": "Hmm, i agree. It appears that the formula is missing the fraction ~~`(1 / min(n,k))`~~ UPDATE `(1/min(m,k))` where `m` is number of ground truths per customer . \n\nAlso the your formula (not Kaggle formula) uses the variable `k` twice to mean two different things. (It is both the `@k` and the index variable under summation).  They should use two different variables. (i.e. j=1 to min(n,k) )",
      "votes": 2,
      "replies": [
        {
          "id": 1711174,
          "postDate": "2022-03-03T17:28:33.773Z",
          "content": "<p>Actually the formula is missing a <code>(1 / min(m,k) )</code> term where <code>m</code> is the number of ground truths for customer. The variable <code>n</code> is the number of predictions for customer. I posted another comment with the entire latex formula.</p>",
          "rawMarkdown": "Actually the formula is missing a `(1 / min(m,k) )` term where `m` is the number of ground truths for customer. The variable `n` is the number of predictions for customer. I posted another comment with the entire latex formula."
        }
      ]
    },
    {
      "id": 1709138,
      "postDate": "2022-03-02T02:20:14.060Z",
      "content": "<p><a href=\"https://www.kaggle.com/fridarimark\" target=\"_blank\">@fridarimark</a> </p>\n<p>Do we have a genuine issue here or are we just missing something and need it to learn it?</p>",
      "rawMarkdown": "@fridarimark \n\nDo we have a genuine issue here or are we just missing something and need it to learn it?",
      "replies": [
        {
          "id": 1709144,
          "postDate": "2022-03-02T02:28:53.023Z",
          "content": "<p>We can ignore the formula. Kaggle posted the code <a href=\"https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/306007#1680513\" target=\"_blank\">here</a>. And the metric is demonstrated in public notebook <a href=\"https://www.kaggle.com/kaerunantoka/h-m-how-to-calculate-map-12\" target=\"_blank\">here</a>.</p>",
          "rawMarkdown": "We can ignore the formula. Kaggle posted the code [here][1]. And the metric is demonstrated in public notebook [here][2].\n\n[1]: https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/306007#1680513\n[2]: https://www.kaggle.com/kaerunantoka/h-m-how-to-calculate-map-12"
        }
      ]
    },
    {
      "id": 1709029,
      "postDate": "2022-03-01T22:26:43.577Z",
      "content": "<p><a href=\"https://www.kaggle.com/onodera\" target=\"_blank\">@onodera</a>   had something to say too here</p>\n<p><a href=\"https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/305952#1690119\" target=\"_blank\">https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/305952#1690119</a></p>\n<p>Did they just update the metric or my eyes are playing tricks?</p>",
      "rawMarkdown": "@onodera   had something to say too here\n\nhttps://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/305952#1690119\n\nDid they just update the metric or my eyes are playing tricks?"
    },
    {
      "id": 1708199,
      "postDate": "2022-03-01T07:49:50.513Z",
      "content": "<p><a href=\"https://www.kaggle.com/shahnawazahmad461\" target=\"_blank\">@shahnawazahmad461</a> </p>\n<p>Are you sure the denominator would be 2? If yes, then MAP could go as high as numbers of purchases.</p>",
      "rawMarkdown": "@shahnawazahmad461 \n\nAre you sure the denominator would be 2? If yes, then MAP could go as high as numbers of purchases.",
      "replies": [
        {
          "id": 1708841,
          "postDate": "2022-03-01T18:32:24.140Z",
          "content": "<p>I have just applied the formula which was given in the evaluation metric, and since I considered just two customers, so for calculating the mean, the denominator was taken as 2.</p>\n<p>But that formula is probably missing the fraction \\(1 / min(n,k))\\) term which is why the MAP is turning out to be greater than 1.</p>",
          "rawMarkdown": "I have just applied the formula which was given in the evaluation metric, and since I considered just two customers, so for calculating the mean, the denominator was taken as 2.\n\nBut that formula is probably missing the fraction \\\\(1 / min(n,k))\\\\) term which is why the MAP is turning out to be greater than 1."
        },
        {
          "id": 1709017,
          "postDate": "2022-03-01T22:11:37.007Z",
          "content": "<p><a href=\"https://www.kaggle.com/shahnawazahmad461\" target=\"_blank\">@shahnawazahmad461</a> </p>\n<p>Which is what I was wondering in this thread</p>\n<p><a href=\"https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/310363\" target=\"_blank\">https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/310363</a></p>",
          "rawMarkdown": "@shahnawazahmad461 \n\nWhich is what I was wondering in this thread\n\nhttps://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/310363"
        }
      ]
    },
    {
      "id": 1709021,
      "postDate": "2022-03-01T22:16:42.730Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 1711194,
      "author_name": "Chris Deotte",
      "author_url": "",
      "post_date": "2022-03-03T17:41:56.967000",
      "content": "<p>The correct formula is</p>\n<p>$$ MAP@12 = \\frac{1}{U} \\sum_{u=1}^{U} \\left( \\frac{1}{min(m,12)} \\sum_{k=1}^{min(n,12)}{P(k) \\times rel(k)} \\right) $$</p>\n<p>where <code>m</code> is the number of ground truths per customer, <code>n</code> is the number of predictions per customer, and <code>U</code> is the number of customers. <code>P(k)</code> is the precision at cutoff <code>k</code>, and <code>rel(k)</code> is an indicator function equaling 1 if the item at rank <code>k</code> is a relevant (correct) label, zero otherwise.</p>",
      "votes": 3,
      "replies": [
        {
          "id": 1722777,
          "author_name": "inversion",
          "author_url": "",
          "post_date": "2022-03-14T21:40:44.450000",
          "content": "<p><a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> You are absolutely correct. The missing term was dropped due to a copy-paste \"bug\". Since many of our competitions with MAP@K have a single ground truth, that term often drops out. I've updated the Evaluation page.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1722807,
          "author_name": "AtulVerma",
          "author_url": "",
          "post_date": "2022-03-14T21:57:21.493000",
          "content": "<p>Yey! 🤛 We were right!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1708692,
      "author_name": "Chris Deotte",
      "author_url": "",
      "post_date": "2022-03-01T16:18:57.723000",
      "content": "<p>Hmm, i agree. It appears that the formula is missing the fraction  UPDATE <code>(1/min(m,k))</code> where <code>m</code> is number of ground truths per customer . </p>\n<p>Also the your formula (not Kaggle formula) uses the variable <code>k</code> twice to mean two different things. (It is both the <code>@k</code> and the index variable under summation).  They should use two different variables. (i.e. j=1 to min(n,k) )</p>",
      "votes": 2,
      "replies": [
        {
          "id": 1711174,
          "author_name": "Chris Deotte",
          "author_url": "",
          "post_date": "2022-03-03T17:28:33.773000",
          "content": "<p>Actually the formula is missing a <code>(1 / min(m,k) )</code> term where <code>m</code> is the number of ground truths for customer. The variable <code>n</code> is the number of predictions for customer. I posted another comment with the entire latex formula.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1709138,
      "author_name": "AtulVerma",
      "author_url": "",
      "post_date": "2022-03-02T02:20:14.060000",
      "content": "<p><a href=\"https://www.kaggle.com/fridarimark\" target=\"_blank\">@fridarimark</a> </p>\n<p>Do we have a genuine issue here or are we just missing something and need it to learn it?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1709144,
          "author_name": "Chris Deotte",
          "author_url": "",
          "post_date": "2022-03-02T02:28:53.023000",
          "content": "<p>We can ignore the formula. Kaggle posted the code <a href=\"https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/306007#1680513\" target=\"_blank\">here</a>. And the metric is demonstrated in public notebook <a href=\"https://www.kaggle.com/kaerunantoka/h-m-how-to-calculate-map-12\" target=\"_blank\">here</a>.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1709029,
      "author_name": "AtulVerma",
      "author_url": "",
      "post_date": "2022-03-01T22:26:43.577000",
      "content": "<p><a href=\"https://www.kaggle.com/onodera\" target=\"_blank\">@onodera</a>   had something to say too here</p>\n<p><a href=\"https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/305952#1690119\" target=\"_blank\">https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/305952#1690119</a></p>\n<p>Did they just update the metric or my eyes are playing tricks?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1708199,
      "author_name": "AtulVerma",
      "author_url": "",
      "post_date": "2022-03-01T07:49:50.513000",
      "content": "<p><a href=\"https://www.kaggle.com/shahnawazahmad461\" target=\"_blank\">@shahnawazahmad461</a> </p>\n<p>Are you sure the denominator would be 2? If yes, then MAP could go as high as numbers of purchases.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1708841,
          "author_name": "Shahnawaz Ahmad",
          "author_url": "",
          "post_date": "2022-03-01T18:32:24.140000",
          "content": "<p>I have just applied the formula which was given in the evaluation metric, and since I considered just two customers, so for calculating the mean, the denominator was taken as 2.</p>\n<p>But that formula is probably missing the fraction \\(1 / min(n,k))\\) term which is why the MAP is turning out to be greater than 1.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1709017,
          "author_name": "AtulVerma",
          "author_url": "",
          "post_date": "2022-03-01T22:11:37.007000",
          "content": "<p><a href=\"https://www.kaggle.com/shahnawazahmad461\" target=\"_blank\">@shahnawazahmad461</a> </p>\n<p>Which is what I was wondering in this thread</p>\n<p><a href=\"https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/310363\" target=\"_blank\">https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/310363</a></p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1709021,
      "author_name": "",
      "author_url": "",
      "post_date": "2022-03-01T22:16:42.730000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1700589": "In the error metric i.e.:\n\n$$MAP@k = \\frac{1}{U}\\sum_{i=1}^{U}\\sum_{k=1}^{min(n,k)}P(k)*rel(k)$$\n\nLet,\n\n| Cust_ID |                        Y_true                     |                     Y_pred                       |\n|---------|----------------------------------|----------------------------------|\n| 1            | [1,2,3,4,5,6,7,8,9,10,11,12,15,13]     | [1,2,3,4,5,6,7,8,9,10,11,12,15,13]     |\n| 2           | [2,5,4,6,7,9,10,11,121,234,122,121] | [2,5,4,6,7,9,10,11,121,234,122,121] |\n\nHere prediction string is the same as the ground truth.\n\nSo, P(i) and rel(i) terms for the above cases are: \n\n| Cust_ID | P(1),rel(1) | P(2),rel(2) | P(3),rel(3) | P(4),rel(4) | P(5),rel(5) | P(6),rel(6) | P(7),rel(7) | P(8),rel(8) | P(9),rel(9) | P(10),rel(10) | P(11),rel(11) | P(12),rel(12) |\n|---------|------|------|------|------|------|------|------|------|------|-------|------|-------|\n| 1            | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 |\n| 2           | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 | 1,1 |\n\nHere all the rel(i) terms are equal to 1 as all the ith strings are relevant.\nApplying the formula:\n\n|Cust_ID|$$\\sum_{k=1}^{12}P(k)*rel(k)$$|\n|--------|-------------------|\n|1           |              12              |\n|2           |               12             |\n\nSo, MAP@12 = (12+12)/2 = 12\n\nBut, I think that the value should be in the range [0,1]. If there is an error in the above steps, please indicate.\nI have doubts about the rel(k) term, if you have any resources for understanding this metric please share those.\n\nTIA",
    "1711194": "The correct formula is\n\n$$ MAP@12 = \\frac{1}{U} \\sum_{u=1}^{U} \\left( \\frac{1}{min(m,12)} \\sum_{k=1}^{min(n,12)}{P(k) \\times rel(k)} \\right) $$\n\nwhere `m` is the number of ground truths per customer, `n` is the number of predictions per customer, and `U` is the number of customers. `P(k)` is the precision at cutoff `k`, and `rel(k)` is an indicator function equaling 1 if the item at rank `k` is a relevant (correct) label, zero otherwise.",
    "1708692": "Hmm, i agree. It appears that the formula is missing the fraction ~~`(1 / min(n,k))`~~ UPDATE `(1/min(m,k))` where `m` is number of ground truths per customer . \n\nAlso the your formula (not Kaggle formula) uses the variable `k` twice to mean two different things. (It is both the `@k` and the index variable under summation).  They should use two different variables. (i.e. j=1 to min(n,k) )",
    "1709138": "@fridarimark \n\nDo we have a genuine issue here or are we just missing something and need it to learn it?",
    "1709029": "@onodera   had something to say too here\n\nhttps://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/305952#1690119\n\nDid they just update the metric or my eyes are playing tricks?",
    "1708199": "@shahnawazahmad461 \n\nAre you sure the denominator would be 2? If yes, then MAP could go as high as numbers of purchases.",
    "1709021": ""
  }
}