I am not sure if I understand the metric correctly so I wanted to check my understanding
\nSuppose I have ground_truth = [a, b, c, d, e] and preds = [b, c, a, d, f] and suppose I want to calculate AP@4
P@1 = 0/1, rel@1 = 0\nP@2 = 1/2, rel@2 = 0\nP@3 = 3/3, rel@3 = 1\nP@4 = 4/4, rel@4 = 1\nAP@4 = 1/4 x [(3/3) * 1 + (4/4) * 1] = 1/2
\nP@1 = 1/1, rel@1 = 1\nP@2 = 2/2, rel@2 = 1\nP@3 = 3/3, rel@3 = 1\nP@4 = 4/4, rel@4 = 1\nAP@4 = 1/4 x [(1/1) * 1 + (2/2) * 1 + (3/3) * 1 + (4/4) * 1] = 1
", "messages": [ { "id": 1680511, "postDate": "2022-02-07T21:37:58.377Z", "content": "I am not sure if I understand the metric correctly so I wanted to check my understanding
\nSuppose I have ground_truth = [a, b, c, d, e] and preds = [b, c, a, d, f] and suppose I want to calculate AP@4
P@1 = 0/1, rel@1 = 0\nP@2 = 1/2, rel@2 = 0\nP@3 = 3/3, rel@3 = 1\nP@4 = 4/4, rel@4 = 1\nAP@4 = 1/4 x [(3/3) * 1 + (4/4) * 1] = 1/2
\nP@1 = 1/1, rel@1 = 1\nP@2 = 2/2, rel@2 = 1\nP@3 = 3/3, rel@3 = 1\nP@4 = 4/4, rel@4 = 1\nAP@4 = 1/4 x [(1/1) * 1 + (2/2) * 1 + (3/3) * 1 + (4/4) * 1] = 1
", "rawMarkdown": "I am not sure if I understand the metric correctly so I wanted to check my understanding\nSuppose I have `ground_truth = [a, b, c, d, e]` and `preds = [b, c, a, d, f]` and suppose I want to calculate `AP@4`\n```\nP@1 = 0/1, rel@1 = 0\nP@2 = 1/2, rel@2 = 0\nP@3 = 3/3, rel@3 = 1\nP@4 = 4/4, rel@4 = 1\n```\n\nAP@4 = 1/4 x [(3/3) * 1 + (4/4) * 1] = 1/2\n\n@inversion Should the formula be revised at the evaluation page to this
\n$$MAP@12 = \\frac{1}{U} \\sum_{u=1}^{U} \\frac{1}{min(n, 12)} \\sum_{k=1}^{min(n,12)} P(k) \\times rel(k)$$
\ninstead of this
\n$$MAP@12 = \\frac{1}{U} \\sum_{u=1}^{U} \\sum_{k=1}^{min(n,12)} P(k) \\times rel(k)$$
I agree with @debarshichanda
\nbecause of this line: https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py#L39
@inversion @maggiemd Can anyone confirm this?
", "rawMarkdown": "@inversion @maggiemd Can anyone confirm this?", "votes": 1 }, { "id": 1722781, "postDate": "2022-03-14T21:41:33.110Z", "content": "Sorry for missing this! The metric page has been updated. Please see this thread.
\nhttps://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/309152#1711194
", "rawMarkdown": "Sorry for missing this! The metric page has been updated. Please see this thread.\n\nhttps://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/309152#1711194" }, { "id": 1722804, "postDate": "2022-03-14T21:54:54.517Z", "content": "Clicking the link says - \"No Access\"
", "rawMarkdown": "Clicking the link says - \"No Access\"" }, { "id": 1722826, "postDate": "2022-03-14T22:27:34.230Z", "content": "@atulverma Should work now. :-)
", "rawMarkdown": "@atulverma Should work now. :-)" }, { "id": 1722879, "postDate": "2022-03-15T00:55:29.020Z", "content": "Sorry but if @cdeotte below (https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/306007#1693573) is right, then formula on evaluation page doesn't mirror that
", "rawMarkdown": "Sorry but if @cdeotte below (https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/306007#1693573) is right, then formula on evaluation page doesn't mirror that " }, { "id": 1722890, "postDate": "2022-03-15T01:03:07.670Z", "content": "\n\nSorry but if @cdeotte below (https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/306007#1693573) is right, then formula doesn't mirror that
\n
Which formula is incorrect. The formula that Debarshi proposes is different than mine and the competition's metric. Debarshi uses min(n,12) twice when it should be min(m,12) and min(n,12). Where one refers to number of predictions and one refers to number of ground truths. (Also note that 12 is the k in @k).
Both.. but i was referring to your formula here https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/309152#1711194 which has been reflected in the evaluation page now.
\nI think
\n$$MAP@12 = \\frac{1}{U} \\sum_{u=1}^{U} \\frac{1}{min(m, 12)} \\sum_{k=1}^{min(n,12)} \\frac{1}{min(k, n)} rel(k)$$
\nOr
\n$$MAP@12 = \\frac{1}{U} \\sum_{u=1}^{U} \\frac{1}{min(m, 12)} \\sum_{k=1}^{min(n,12)} P(k)$$
\nWhere
\n$$P(k) = \\frac{1}{min(k, n)} rel(k)$$
", "rawMarkdown": "Both.. but i was referring to your formula here https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/309152#1711194 which has been reflected in the evaluation page now.\n\n\nI think\n\n$$MAP@12 = \\frac{1}{U} \\sum_{u=1}^{U} \\frac{1}{min(m, 12)} \\sum_{k=1}^{min(n,12)} \\frac{1}{min(k, n)} rel(k)$$\n\nOr\n\n$$MAP@12 = \\frac{1}{U} \\sum_{u=1}^{U} \\frac{1}{min(m, 12)} \\sum_{k=1}^{min(n,12)} P(k)$$ \n\nWhere\n\n$$P(k) = \\frac{1}{min(k, n)} rel(k)$$" }, { "id": 1722957, "postDate": "2022-03-15T02:08:52.780Z", "content": "Or
\n$$MAP@12 = \\frac{1}{U} \\sum_{u=1}^{U} \\frac{1}{min(n, 12)} \\sum_{k=1}^{min(n,12)} (\\sum_{k=1}^{min(n,12)} \\frac{1}{k} rel(k)) \\times rel(k) $$
\nOr
\n$$MAP@12 = \\frac{1}{U} \\sum_{u=1}^{U} \\frac{1}{min(n, 12)} \\sum_{k=1}^{min(n,12)} P(k) \\times rel(k) $$
\nWhere
\n$$P(k) = \\sum_{k=1}^{min(n,12)} rel(k) \\frac{1}{k} $$
", "rawMarkdown": "Or\n\n$$MAP@12 = \\frac{1}{U} \\sum_{u=1}^{U} \\frac{1}{min(n, 12)} \\sum_{k=1}^{min(n,12)} (\\sum_{k=1}^{min(n,12)} \\frac{1}{k} rel(k)) \\times rel(k) $$\n\nOr\n\n$$MAP@12 = \\frac{1}{U} \\sum_{u=1}^{U} \\frac{1}{min(n, 12)} \\sum_{k=1}^{min(n,12)} P(k) \\times rel(k) $$\n\nWhere\n\n$$P(k) = \\sum_{k=1}^{min(n,12)} rel(k) \\frac{1}{k} $$" }, { "id": 1722959, "postDate": "2022-03-15T02:12:34.737Z", "content": "Why do you think it is wrong? Can you provide an example?
\n(Note that your two formulas don't include precision, they only include rel(k). Since the metric is mAP i.e. mean average precision, we need to include the computation of precision)
Below you say this
\n\n\nPrecision@1 = 0/1, recall doesn't increase
\n
\n Precision@2 = 1/2, recall increase
\n Precision@3 = 2/3, recall increase
\n Precision@4 = 3/4, recall increase
numerator is sum of rel(k).. one summation is missing in the current formula
\ndenominator is min(k,n) where k is 1 to min(n,12) .. this division is also missing in the current formula.
Both the formulas in this post https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/306007#1722957 include Precision as defined in your post below
", "rawMarkdown": "Below you say this\n\n> Precision@1 = 0/1, recall doesn't increase\n> Precision@2 = 1/2, recall increase\n> Precision@3 = 2/3, recall increase\n> Precision@4 = 3/4, recall increase\n\nnumerator is sum of rel(k).. one summation is missing in the current formula\ndenominator is min(k,n) where k is 1 to min(n,12) .. this division is also missing in the current formula.\n\nBoth the formulas in this post https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/306007#1722957 include Precision as defined in your post below" }, { "id": 1723024, "postDate": "2022-03-15T04:01:26.970Z", "content": "The new kaggle formula is now correct. In my example, we have
\n(0/1 * 0) + (1/2 * 1) + (2/3 * 1) + (3/4 * 1) where this is (p(0) * rel(0)) + (p(1) * rel(1)) + etc etc
This is the summation of p(k) * rel(k) for k in [1, 2, ..., n] where n is the number of predictions. After computing this summation, we divide by m where m is the number of ground truths. (And we use 12 in place of n and m if it is lower than either).
In your formula, you need to multiply p(k) times rel(k). The function rel(k) is a mask that only considers certain p(k)
\n\nIn your formula, you need to multiply
\np(k)timesrel(k). The functionrel(k)is a mask that only considers certainp(k)
Yes, I missed that.. but the current formula doesn't describe P(k) and P(k), it seems also uses rel(k) which is what I am trying above (Updated)
\nIt seems P(k) is summation of rel(k) divided by k.
", "rawMarkdown": "> In your formula, you need to multiply `p(k)` times `rel(k)`. The function `rel(k)` is a mask that only considers certain `p(k)`\n\nYes, I missed that.. but the current formula doesn't describe P(k) and P(k), it seems also uses rel(k) which is what I am trying above (Updated)\n\nIt seems P(k) is summation of rel(k) divided by k." } ] }, { "id": 1723041, "postDate": "2022-03-15T04:33:51.977Z", "content": "p(k) is precision. It is TP/ (TP+FP) when only considering your first k predictions. It is completely different than rel(k). The term rel(k) is 1 if you newest prediction (i.e. prediction at place k) improves your recall. And 0 if your newest prediction does not improve your recall.
I agree it is very weird. It makes more sense if you look at precision recall plots (where rel(k) moves the plot line to the right whereas p(k) only moves plot line up and down)
You can also walk through this code:
\nhttps://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py
", "rawMarkdown": "You can also walk through this code:\n\nhttps://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py", "votes": 8, "replies": [ { "id": 1680546, "postDate": "2022-02-07T22:15:26.457Z", "content": "Thanks for this but I am not sure why the order doesn't matter
\nIf ground_truth = [a, b, c, d, e] then the above code gives the same answer for both [d, c, b, a, f] and [a, b, c, d, e] i.e. 1 for AP@4
There is no order to the ground truth, it only matters whether your predictions are found in the ground truth. For both of your predictions, the first 4 labels are found in the ground truth. Since you are looking at AP@4, they both score 1.0.
Thank you so much! Understood my mistake
", "rawMarkdown": "Thank you so much! Understood my mistake", "votes": 1 }, { "id": 1687848, "postDate": "2022-02-13T07:49:20.147Z", "content": "\n\nYou can also walk through this code:
\n
\n https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py
@inversion that code won't handle well a case when y_true=[] i.e. a costumer doesn't buy anything on the test-data timeframe.
Example:
\n{'Customer01': {\n 'y_true': [], \n 'y_pred': [1,2,3],\n}},\n{'Customer02': {\n 'y_true': [1,2,3], \n 'y_pred': [1,2,3],\n}},\n\nObtained Score = 0.5\nExpected Score = 1.0\nCiting from Evaluation section in the competition overview:
\n\n\n\n
\n- Customer that did not make any purchase during test period are excluded from the scoring.
\n- There is never a penalty for using the full 12 predictions for a customer that ordered fewer than 12 items; thus, it's advantageous to make 12 predictions for each customer.
\n
Cheers,
", "rawMarkdown": "> You can also walk through this code:\nhttps://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py\n\n@inversion that code won't handle well a case when `y_true=[]` i.e. a costumer doesn't buy anything on the test-data timeframe.\n\nExample:\n```\n{'Customer01': {\n 'y_true': [], \n 'y_pred': [1,2,3],\n}},\n{'Customer02': {\n 'y_true': [1,2,3], \n 'y_pred': [1,2,3],\n}},\n\nObtained Score = 0.5\nExpected Score = 1.0\n```\n\nCiting from Evaluation section in the competition overview:\n> - Customer that did not make any purchase during test period are excluded from the scoring.\n- There is never a penalty for using the full 12 predictions for a customer that ordered fewer than 12 items; thus, it's advantageous to make 12 predictions for each customer.\n\nCheers," }, { "id": 1688869, "postDate": "2022-02-13T21:33:32.150Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1693397, "postDate": "2022-02-16T16:43:41.740Z", "content": "@debarshichanda
\nhow can AP@4 would be same for the two predictions as ordering effect average precision.
The order of your predictions matters but order of ground truth does not matter. In your original question ground_truth = [a, b, c, d, e] and preds = [b, c, a, d, f], the answer is AP@4 = 1.
(Note that the competition metric is mAP@12 not mAP@4 but we can do mAP@4 in example below, we must just stop after using the first 4 predictions)
If your predictions are [f, b, c, a, d], then your AP@4 is less than 1. Note that we don't use the last prediction of d here since we are doing AP@4.
Precision@1 = 0/1, recall doesn't increase
\nPrecision@2 = 1/2, recall increase
\nPrecision@3 = 2/3, recall increase
\nPrecision@4 = 3/4, recall increase
AP@4 = sum of \"recall increase\" / min(4,len(true)) = (1/2 + 2/3 + 3/4)/4 = 23/48 = 0.479
@inversion Should the formula be revised at the evaluation page to this
\n$$MAP@12 = \\frac{1}{U} \\sum_{u=1}^{U} \\frac{1}{min(n, 12)} \\sum_{k=1}^{min(n,12)} P(k) \\times rel(k)$$
\ninstead of this
\n$$MAP@12 = \\frac{1}{U} \\sum_{u=1}^{U} \\sum_{k=1}^{min(n,12)} P(k) \\times rel(k)$$
I agree with @debarshichanda
\nbecause of this line: https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py#L39
@inversion @maggiemd Can anyone confirm this?
", "votes": 1, "replies": [] }, { "id": 1722781, "author_name": "inversion", "author_url": "", "post_date": "2022-03-14T21:41:33.110000", "content": "Sorry for missing this! The metric page has been updated. Please see this thread.
\nhttps://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/309152#1711194
", "votes": 0, "replies": [] }, { "id": 1722804, "author_name": "AtulVerma", "author_url": "", "post_date": "2022-03-14T21:54:54.517000", "content": "Clicking the link says - \"No Access\"
", "votes": 0, "replies": [] }, { "id": 1722826, "author_name": "inversion", "author_url": "", "post_date": "2022-03-14T22:27:34.230000", "content": "@atulverma Should work now. :-)
", "votes": 0, "replies": [] }, { "id": 1722879, "author_name": "AtulVerma", "author_url": "", "post_date": "2022-03-15T00:55:29.020000", "content": "Sorry but if @cdeotte below (https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/306007#1693573) is right, then formula on evaluation page doesn't mirror that
", "votes": 0, "replies": [] }, { "id": 1722890, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-03-15T01:03:07.670000", "content": "\n\nSorry but if @cdeotte below (https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/306007#1693573) is right, then formula doesn't mirror that
\n
Which formula is incorrect. The formula that Debarshi proposes is different than mine and the competition's metric. Debarshi uses min(n,12) twice when it should be min(m,12) and min(n,12). Where one refers to number of predictions and one refers to number of ground truths. (Also note that 12 is the k in @k).
Both.. but i was referring to your formula here https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/309152#1711194 which has been reflected in the evaluation page now.
\nI think
\n$$MAP@12 = \\frac{1}{U} \\sum_{u=1}^{U} \\frac{1}{min(m, 12)} \\sum_{k=1}^{min(n,12)} \\frac{1}{min(k, n)} rel(k)$$
\nOr
\n$$MAP@12 = \\frac{1}{U} \\sum_{u=1}^{U} \\frac{1}{min(m, 12)} \\sum_{k=1}^{min(n,12)} P(k)$$
\nWhere
\n$$P(k) = \\frac{1}{min(k, n)} rel(k)$$
", "votes": 0, "replies": [] }, { "id": 1722957, "author_name": "AtulVerma", "author_url": "", "post_date": "2022-03-15T02:08:52.780000", "content": "Or
\n$$MAP@12 = \\frac{1}{U} \\sum_{u=1}^{U} \\frac{1}{min(n, 12)} \\sum_{k=1}^{min(n,12)} (\\sum_{k=1}^{min(n,12)} \\frac{1}{k} rel(k)) \\times rel(k) $$
\nOr
\n$$MAP@12 = \\frac{1}{U} \\sum_{u=1}^{U} \\frac{1}{min(n, 12)} \\sum_{k=1}^{min(n,12)} P(k) \\times rel(k) $$
\nWhere
\n$$P(k) = \\sum_{k=1}^{min(n,12)} rel(k) \\frac{1}{k} $$
", "votes": 0, "replies": [] }, { "id": 1722959, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-03-15T02:12:34.737000", "content": "Why do you think it is wrong? Can you provide an example?
\n(Note that your two formulas don't include precision, they only include rel(k). Since the metric is mAP i.e. mean average precision, we need to include the computation of precision)
Below you say this
\n\n\nPrecision@1 = 0/1, recall doesn't increase
\n
\n Precision@2 = 1/2, recall increase
\n Precision@3 = 2/3, recall increase
\n Precision@4 = 3/4, recall increase
numerator is sum of rel(k).. one summation is missing in the current formula
\ndenominator is min(k,n) where k is 1 to min(n,12) .. this division is also missing in the current formula.
Both the formulas in this post https://www.kaggle.com/c/h-and-m-personalized-fashion-recommendations/discussion/306007#1722957 include Precision as defined in your post below
", "votes": 0, "replies": [] }, { "id": 1723024, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-03-15T04:01:26.970000", "content": "The new kaggle formula is now correct. In my example, we have
\n(0/1 * 0) + (1/2 * 1) + (2/3 * 1) + (3/4 * 1) where this is (p(0) * rel(0)) + (p(1) * rel(1)) + etc etc
This is the summation of p(k) * rel(k) for k in [1, 2, ..., n] where n is the number of predictions. After computing this summation, we divide by m where m is the number of ground truths. (And we use 12 in place of n and m if it is lower than either).
In your formula, you need to multiply p(k) times rel(k). The function rel(k) is a mask that only considers certain p(k)
\n\nIn your formula, you need to multiply
\np(k)timesrel(k). The functionrel(k)is a mask that only considers certainp(k)
Yes, I missed that.. but the current formula doesn't describe P(k) and P(k), it seems also uses rel(k) which is what I am trying above (Updated)
\nIt seems P(k) is summation of rel(k) divided by k.
", "votes": 0, "replies": [] } ] }, { "id": 1723041, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-03-15T04:33:51.977000", "content": "p(k) is precision. It is TP/ (TP+FP) when only considering your first k predictions. It is completely different than rel(k). The term rel(k) is 1 if you newest prediction (i.e. prediction at place k) improves your recall. And 0 if your newest prediction does not improve your recall.
I agree it is very weird. It makes more sense if you look at precision recall plots (where rel(k) moves the plot line to the right whereas p(k) only moves plot line up and down)
You can also walk through this code:
\nhttps://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py
", "votes": 8, "replies": [ { "id": 1680546, "author_name": "Debarshi Chanda", "author_url": "", "post_date": "2022-02-07T22:15:26.457000", "content": "Thanks for this but I am not sure why the order doesn't matter
\nIf ground_truth = [a, b, c, d, e] then the above code gives the same answer for both [d, c, b, a, f] and [a, b, c, d, e] i.e. 1 for AP@4
There is no order to the ground truth, it only matters whether your predictions are found in the ground truth. For both of your predictions, the first 4 labels are found in the ground truth. Since you are looking at AP@4, they both score 1.0.
Thank you so much! Understood my mistake
", "votes": 1, "replies": [] }, { "id": 1687848, "author_name": "Thariq Nugrohotomo", "author_url": "", "post_date": "2022-02-13T07:49:20.147000", "content": "\n\nYou can also walk through this code:
\n
\n https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py
@inversion that code won't handle well a case when y_true=[] i.e. a costumer doesn't buy anything on the test-data timeframe.
Example:
\n{'Customer01': {\n 'y_true': [], \n 'y_pred': [1,2,3],\n}},\n{'Customer02': {\n 'y_true': [1,2,3], \n 'y_pred': [1,2,3],\n}},\n\nObtained Score = 0.5\nExpected Score = 1.0\nCiting from Evaluation section in the competition overview:
\n\n\n\n
\n- Customer that did not make any purchase during test period are excluded from the scoring.
\n- There is never a penalty for using the full 12 predictions for a customer that ordered fewer than 12 items; thus, it's advantageous to make 12 predictions for each customer.
\n
Cheers,
", "votes": 0, "replies": [] }, { "id": 1688869, "author_name": "", "author_url": "", "post_date": "2022-02-13T21:33:32.150000", "content": "", "votes": 0, "replies": [] }, { "id": 1693397, "author_name": "ayush", "author_url": "", "post_date": "2022-02-16T16:43:41.740000", "content": "@debarshichanda
\nhow can AP@4 would be same for the two predictions as ordering effect average precision.
The order of your predictions matters but order of ground truth does not matter. In your original question ground_truth = [a, b, c, d, e] and preds = [b, c, a, d, f], the answer is AP@4 = 1.
(Note that the competition metric is mAP@12 not mAP@4 but we can do mAP@4 in example below, we must just stop after using the first 4 predictions)
If your predictions are [f, b, c, a, d], then your AP@4 is less than 1. Note that we don't use the last prediction of d here since we are doing AP@4.
Precision@1 = 0/1, recall doesn't increase
\nPrecision@2 = 1/2, recall increase
\nPrecision@3 = 2/3, recall increase
\nPrecision@4 = 3/4, recall increase
AP@4 = sum of \"recall increase\" / min(4,len(true)) = (1/2 + 2/3 + 3/4)/4 = 23/48 = 0.479