{
  "id": 73303,
  "title": "Wrong MAP@5 formula in the competition description",
  "url": "/competitions/humpback-whale-identification/discussion/73303",
  "author_name": "",
  "post_date": "2018-12-01T17:30:28.891548300Z",
  "votes": 10,
  "comment_count": 5,
  "views": 0,
  "content": "<p>The formula \n$MAP@5 = \\frac{1}{U} \\sum_{u=1}^{U}  \\sum_{k=1}^{min(n,5)} P(k)$ \nin <a href=\"https://www.kaggle.com/c/humpback-whale-identification#evaluation\">https://www.kaggle.com/c/humpback-whale-identification#evaluation</a>\nlooks wrong to me.\nLet's assume we have a perfect classifier, than the first predicted identity will alway be the same as ground truth. <br>\nSince there is only one correct identity for every test image, <br>\nmaximal value of P(1) is 1, <br>\nmaximal value of P(2) is 1/2, <br>\nmaximal value of P(3) is 1/3, <br>\nmaximal value of P(4) is 1/4, <br>\nmaximal value of P(5) is 1/5. <br>\nSo the maximal value of MAP@5 according to this formula is 1 + 1/2 + 1/3 + 1/4 + 1/5 =  2.2833.\nBut it should not  be like this.</p>\n\n<p>Tell me if I'm wrong.</p>",
  "messages": [
    {
      "id": "431160",
      "postDate": "12/01/2018 17:30:28",
      "content": "<p>The formula \n$MAP@5 = \\frac{1}{U} \\sum_{u=1}^{U}  \\sum_{k=1}^{min(n,5)} P(k)$ \nin <a href=\"https://www.kaggle.com/c/humpback-whale-identification#evaluation\">https://www.kaggle.com/c/humpback-whale-identification#evaluation</a>\nlooks wrong to me.\nLet's assume we have a perfect classifier, than the first predicted identity will alway be the same as ground truth. <br>\nSince there is only one correct identity for every test image, <br>\nmaximal value of P(1) is 1, <br>\nmaximal value of P(2) is 1/2, <br>\nmaximal value of P(3) is 1/3, <br>\nmaximal value of P(4) is 1/4, <br>\nmaximal value of P(5) is 1/5. <br>\nSo the maximal value of MAP@5 according to this formula is 1 + 1/2 + 1/3 + 1/4 + 1/5 =  2.2833.\nBut it should not  be like this.</p>\n\n<p>Tell me if I'm wrong.</p>",
      "rawMarkdown": "The formula \n$MAP@5 = \\frac{1}{U} \\sum_{u=1}^{U}  \\sum_{k=1}^{min(n,5)} P(k)$ \nin https://www.kaggle.com/c/humpback-whale-identification#evaluation\nlooks wrong to me.\nLet's assume we have a perfect classifier, than the first predicted identity will alway be the same as ground truth.   \nSince there is only one correct identity for every test image,  \nmaximal value of P(1) is 1,   \nmaximal value of P(2) is 1/2,  \nmaximal value of P(3) is 1/3,  \nmaximal value of P(4) is 1/4,  \nmaximal value of P(5) is 1/5.  \nSo the maximal value of MAP@5 according to this formula is 1 + 1/2 + 1/3 + 1/4 + 1/5 =  2.2833.\nBut it should not  be like this.\n\nTell me if I'm wrong.",
      "votes": null
    },
    {
      "id": "431164",
      "postDate": "12/01/2018 17:35:30",
      "content": "<p>In this case, the calculation would stop after the first occurrence of the correct whale, so  P(1) = 1. So, a prediction that is <code>correct</code> <code>incorrect</code> <code>incorrect</code> <code>incorrect</code> <code>incorrect</code> also scores 1.</p>",
      "rawMarkdown": "In this case, the calculation would stop after the first occurrence of the correct whale, so  P(1) = 1. So, a prediction that is `correct` `incorrect` `incorrect` `incorrect` `incorrect` also scores 1.",
      "votes": null
    },
    {
      "id": "431177",
      "postDate": "12/01/2018 18:07:18",
      "content": "<p>Anyway, formally the definition \n$MAP@5 = \\frac{1}{U} \\sum{u=1}^{U} \\sum{k=1}^{min(n,5)} P(k)$  is wrong   </p>\n\n<p>the right one is \n$MAP@5 = \\frac{1}{U} \\sum{u=1}^{U} AP@5$  </p>",
      "rawMarkdown": "Anyway, formally the definition \n$MAP@5 = \\frac{1}{U} \\sum{u=1}^{U} \\sum{k=1}^{min(n,5)} P(k)$  is wrong   \n\nthe right one is \n$MAP@5 = \\frac{1}{U} \\sum{u=1}^{U} AP@5$",
      "votes": null
    },
    {
      "id": "431445",
      "postDate": "12/02/2018 09:30:24",
      "content": "<p>I was thinking exactly the same... Can you change the equation or make this clarification to avoid misunderstandings? </p>",
      "rawMarkdown": "I was thinking exactly the same... Can you change the equation or make this clarification to avoid misunderstandings?",
      "votes": null
    },
    {
      "id": "431454",
      "postDate": "12/02/2018 09:50:50",
      "content": "<p>Just to check my understanding: we should never submit less than 5 predictions for a row, right? If our first prediction is correct, we're not going to be penalized for the subsequent ones.</p>",
      "rawMarkdown": "Just to check my understanding: we should never submit less than 5 predictions for a row, right? If our first prediction is correct, we're not going to be penalized for the subsequent ones.",
      "votes": null
    },
    {
      "id": "441495",
      "postDate": "12/18/2018 17:55:17",
      "content": "<p>Thanks everyone for the input. I've addressed this here:</p>\n\n<p><a href=\"https://www.kaggle.com/c/humpback-whale-identification/discussion/75127\">https://www.kaggle.com/c/humpback-whale-identification/discussion/75127</a></p>",
      "rawMarkdown": "Thanks everyone for the input. I've addressed this here:\n\nhttps://www.kaggle.com/c/humpback-whale-identification/discussion/75127",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 431164,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "12/01/2018 17:35:30",
      "content": "<p>In this case, the calculation would stop after the first occurrence of the correct whale, so  P(1) = 1. So, a prediction that is <code>correct</code> <code>incorrect</code> <code>incorrect</code> <code>incorrect</code> <code>incorrect</code> also scores 1.</p>",
      "votes": null,
      "replies": [
        {
          "id": 431177,
          "author_name": "asanakoev",
          "author_url": "",
          "post_date": "12/01/2018 18:07:18",
          "content": "<p>Anyway, formally the definition \n$MAP@5 = \\frac{1}{U} \\sum{u=1}^{U} \\sum{k=1}^{min(n,5)} P(k)$  is wrong   </p>\n\n<p>the right one is \n$MAP@5 = \\frac{1}{U} \\sum{u=1}^{U} AP@5$  </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 431445,
          "author_name": "frsanchez",
          "author_url": "",
          "post_date": "12/02/2018 09:30:24",
          "content": "<p>I was thinking exactly the same... Can you change the equation or make this clarification to avoid misunderstandings? </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 431454,
      "author_name": "kk1694",
      "author_url": "",
      "post_date": "12/02/2018 09:50:50",
      "content": "<p>Just to check my understanding: we should never submit less than 5 predictions for a row, right? If our first prediction is correct, we're not going to be penalized for the subsequent ones.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 441495,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "12/18/2018 17:55:17",
      "content": "<p>Thanks everyone for the input. I've addressed this here:</p>\n\n<p><a href=\"https://www.kaggle.com/c/humpback-whale-identification/discussion/75127\">https://www.kaggle.com/c/humpback-whale-identification/discussion/75127</a></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "431160": "The formula \n$MAP@5 = \\frac{1}{U} \\sum_{u=1}^{U}  \\sum_{k=1}^{min(n,5)} P(k)$ \nin https://www.kaggle.com/c/humpback-whale-identification#evaluation\nlooks wrong to me.\nLet's assume we have a perfect classifier, than the first predicted identity will alway be the same as ground truth.   \nSince there is only one correct identity for every test image,  \nmaximal value of P(1) is 1,   \nmaximal value of P(2) is 1/2,  \nmaximal value of P(3) is 1/3,  \nmaximal value of P(4) is 1/4,  \nmaximal value of P(5) is 1/5.  \nSo the maximal value of MAP@5 according to this formula is 1 + 1/2 + 1/3 + 1/4 + 1/5 =  2.2833.\nBut it should not  be like this.\n\nTell me if I'm wrong.",
    "431164": "In this case, the calculation would stop after the first occurrence of the correct whale, so  P(1) = 1. So, a prediction that is `correct` `incorrect` `incorrect` `incorrect` `incorrect` also scores 1.",
    "431177": "Anyway, formally the definition \n$MAP@5 = \\frac{1}{U} \\sum{u=1}^{U} \\sum{k=1}^{min(n,5)} P(k)$  is wrong   \n\nthe right one is \n$MAP@5 = \\frac{1}{U} \\sum{u=1}^{U} AP@5$",
    "431445": "I was thinking exactly the same... Can you change the equation or make this clarification to avoid misunderstandings?",
    "431454": "Just to check my understanding: we should never submit less than 5 predictions for a row, right? If our first prediction is correct, we're not going to be penalized for the subsequent ones.",
    "441495": "Thanks everyone for the input. I've addressed this here:\n\nhttps://www.kaggle.com/c/humpback-whale-identification/discussion/75127"
  },
  "source": "meta"
}