I'm not super sure about the equation shown in the evaluation section. Is it correct ? Does it mean for each testing audio, the score is P(1) + P(2) + P(3). So for a prediction [correct, wrong, wrong], the score would be 1 +0.5+0.33 = 1.83. So if I submit an all correct prediction, my MAP@3 would be 1.83? I think the equation does match the explanation link they give. The explanation link means the score for each testing audio should be one of [1, 0.5, 0.33, 0].
", "messages": [ { "id": "359675", "postDate": "07/20/2018 15:12:36", "content": "I'm not super sure about the equation shown in the evaluation section. Is it correct ? Does it mean for each testing audio, the score is P(1) + P(2) + P(3). So for a prediction [correct, wrong, wrong], the score would be 1 +0.5+0.33 = 1.83. So if I submit an all correct prediction, my MAP@3 would be 1.83? I think the equation does match the explanation link they give. The explanation link means the score for each testing audio should be one of [1, 0.5, 0.33, 0].
", "rawMarkdown": "I'm not super sure about the equation shown in the evaluation section. Is it correct ? Does it mean for each testing audio, the score is P(1) + P(2) + P(3). So for a prediction [correct, wrong, wrong], the score would be 1 +0.5+0.33 = 1.83. So if I submit an all correct prediction, my MAP@3 would be 1.83? I think the equation does match the explanation link they give. The explanation link means the score for each testing audio should be one of [1, 0.5, 0.33, 0].", "votes": null }, { "id": "359865", "postDate": "07/21/2018 00:03:29", "content": "Hi there!
\n\nOnly the correct prediction gives you scores (as long as it is predicted in 1st, or 2nd or 3rd place). For a prediction [correct, wrong, wrong], the score would be 1 + 0 + 0 =1. If you submit an all correct prediction, the mAP@3 would be 1.0.
\n\nThanks!
", "rawMarkdown": "Hi there!\n\nOnly the **correct** prediction gives you scores (as long as it is predicted in 1st, **or** 2nd **or** 3rd place). For a prediction [correct, wrong, wrong], the score would be 1 + 0 + 0 =1. If you submit an all correct prediction, the mAP@3 would be 1.0.\n\nThanks!", "votes": null }, { "id": "359998", "postDate": "07/21/2018 08:18:08", "content": "Hi, Thanks a lot! \nBut does this mean the equation is wrong? Shouldn't the sum of P(k) in the equation (https://www.kaggle.com/c/freesound-audio-tagging#evaluation) be max(P(1), P(2), P(3))?
", "rawMarkdown": "Hi, Thanks a lot! \nBut does this mean the equation is wrong? Shouldn't the sum of P(k) in the equation (https://www.kaggle.com/c/freesound-audio-tagging#evaluation) be max(P(1), P(2), P(3))?", "votes": null }, { "id": "360856", "postDate": "07/23/2018 11:07:13", "content": "Hi, maybe it is just a matter of interpretation. \nA single label must be assigned to each file in the test set. However, up to three labels (MAP@3) can be predicted for each file in such a way that partial credit can be obtained for the correct prediction in second or third place.
\n\nUsing your notation, only one term of {P(1), P(2), P(3)} will be greater than 0. Hence there is no need for the max.
\n\nhope this helps!
", "rawMarkdown": "Hi, maybe it is just a matter of interpretation. \nA single label must be assigned to each file in the test set. However, up to three labels (MAP@3) can be predicted for each file in such a way that partial credit can be obtained for the correct prediction in second or third place.\n\nUsing your notation, only one term of {P(1), P(2), P(3)} will be greater than 0. Hence there is no need for the *max*.\n\nhope this helps!", "votes": null }, { "id": "360869", "postDate": "07/23/2018 11:43:25", "content": "Great thanks again! And sorry for being so obsessed with the equations.
\n\nHere(https://www.kaggle.com/c/freesound-audio-tagging#evaluation) says 'P(k) is the precision at cutoff k'. So what does 'the precision at cutoff k' mean?
\n\nFor example, if the predictions is [correct, wrong1, wrong2]. Wouldn't the P(1) be 1, P(2) be 0.5, P(3) be 0.33? So all of the term {P(1), P(2), P(3)} would be greater than 0. So sum would be 1.83, and max of them should be 1. Then the sum and max is different!
\n\n(Maybe I understand the 'precision at cutoff k' totally wrong..... but i could not find any detailed explanation on this concept.)
", "rawMarkdown": "Great thanks again! And sorry for being so obsessed with the equations.\n\nHere(https://www.kaggle.com/c/freesound-audio-tagging#evaluation) says 'P(k) is the precision at cutoff k'. So what does 'the precision at cutoff k' mean? \n\nFor example, if the predictions is [correct, wrong1, wrong2]. Wouldn't the P(1) be 1, P(2) be 0.5, P(3) be 0.33? So all of the term {P(1), P(2), P(3)} would be greater than 0. So sum would be 1.83, and max of them should be 1. Then the sum and max is different!\n\n(Maybe I understand the 'precision at cutoff k' totally wrong..... but i could not find any detailed explanation on this concept.)", "votes": null }, { "id": "360960", "postDate": "07/23/2018 15:27:59", "content": "It's worth stepping through the code here if the metric is unclear. You'll see how, if the correct answer is in the first position it doesn't matter what is after it.
\n\nhttps://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py
", "rawMarkdown": "It's worth stepping through the code here if the metric is unclear. You'll see how, if the correct answer is in the first position it doesn't matter what is after it.\n\nhttps://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py", "votes": null } ], "comments": [ { "id": 359865, "author_name": "eduardofonseca", "author_url": "", "post_date": "07/21/2018 00:03:29", "content": "Hi there!
\n\nOnly the correct prediction gives you scores (as long as it is predicted in 1st, or 2nd or 3rd place). For a prediction [correct, wrong, wrong], the score would be 1 + 0 + 0 =1. If you submit an all correct prediction, the mAP@3 would be 1.0.
\n\nThanks!
", "votes": null, "replies": [ { "id": 359998, "author_name": "zhicun", "author_url": "", "post_date": "07/21/2018 08:18:08", "content": "Hi, Thanks a lot! \nBut does this mean the equation is wrong? Shouldn't the sum of P(k) in the equation (https://www.kaggle.com/c/freesound-audio-tagging#evaluation) be max(P(1), P(2), P(3))?
", "votes": null, "replies": [] }, { "id": 360856, "author_name": "eduardofonseca", "author_url": "", "post_date": "07/23/2018 11:07:13", "content": "Hi, maybe it is just a matter of interpretation. \nA single label must be assigned to each file in the test set. However, up to three labels (MAP@3) can be predicted for each file in such a way that partial credit can be obtained for the correct prediction in second or third place.
\n\nUsing your notation, only one term of {P(1), P(2), P(3)} will be greater than 0. Hence there is no need for the max.
\n\nhope this helps!
", "votes": null, "replies": [] }, { "id": 360869, "author_name": "zhicun", "author_url": "", "post_date": "07/23/2018 11:43:25", "content": "Great thanks again! And sorry for being so obsessed with the equations.
\n\nHere(https://www.kaggle.com/c/freesound-audio-tagging#evaluation) says 'P(k) is the precision at cutoff k'. So what does 'the precision at cutoff k' mean?
\n\nFor example, if the predictions is [correct, wrong1, wrong2]. Wouldn't the P(1) be 1, P(2) be 0.5, P(3) be 0.33? So all of the term {P(1), P(2), P(3)} would be greater than 0. So sum would be 1.83, and max of them should be 1. Then the sum and max is different!
\n\n(Maybe I understand the 'precision at cutoff k' totally wrong..... but i could not find any detailed explanation on this concept.)
", "votes": null, "replies": [] }, { "id": 360960, "author_name": "inversion", "author_url": "", "post_date": "07/23/2018 15:27:59", "content": "It's worth stepping through the code here if the metric is unclear. You'll see how, if the correct answer is in the first position it doesn't matter what is after it.
\n\nhttps://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "359675": "I'm not super sure about the equation shown in the evaluation section. Is it correct ? Does it mean for each testing audio, the score is P(1) + P(2) + P(3). So for a prediction [correct, wrong, wrong], the score would be 1 +0.5+0.33 = 1.83. So if I submit an all correct prediction, my MAP@3 would be 1.83? I think the equation does match the explanation link they give. The explanation link means the score for each testing audio should be one of [1, 0.5, 0.33, 0].", "359865": "Hi there!\n\nOnly the **correct** prediction gives you scores (as long as it is predicted in 1st, **or** 2nd **or** 3rd place). For a prediction [correct, wrong, wrong], the score would be 1 + 0 + 0 =1. If you submit an all correct prediction, the mAP@3 would be 1.0.\n\nThanks!", "359998": "Hi, Thanks a lot! \nBut does this mean the equation is wrong? Shouldn't the sum of P(k) in the equation (https://www.kaggle.com/c/freesound-audio-tagging#evaluation) be max(P(1), P(2), P(3))?", "360856": "Hi, maybe it is just a matter of interpretation. \nA single label must be assigned to each file in the test set. However, up to three labels (MAP@3) can be predicted for each file in such a way that partial credit can be obtained for the correct prediction in second or third place.\n\nUsing your notation, only one term of {P(1), P(2), P(3)} will be greater than 0. Hence there is no need for the *max*.\n\nhope this helps!", "360869": "Great thanks again! And sorry for being so obsessed with the equations.\n\nHere(https://www.kaggle.com/c/freesound-audio-tagging#evaluation) says 'P(k) is the precision at cutoff k'. So what does 'the precision at cutoff k' mean? \n\nFor example, if the predictions is [correct, wrong1, wrong2]. Wouldn't the P(1) be 1, P(2) be 0.5, P(3) be 0.33? So all of the term {P(1), P(2), P(3)} would be greater than 0. So sum would be 1.83, and max of them should be 1. Then the sum and max is different!\n\n(Maybe I understand the 'precision at cutoff k' totally wrong..... but i could not find any detailed explanation on this concept.)", "360960": "It's worth stepping through the code here if the metric is unclear. You'll see how, if the correct answer is in the first position it doesn't matter what is after it.\n\nhttps://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py" }, "source": "meta" }