{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### このノートブックは，[Understanding the Evaluation Metric ](https://www.kaggle.com/code/debarshichanda/understanding-mean-average-precision)を日本語にしたものです．\n\n### H&Mコンペの評価尺度である，MAPの理解になればと思います．  \n\n##### ※分かりやすくするために，多少表現を変えたり，追記している箇所があります．","metadata":{}},{"cell_type":"markdown","source":"----","metadata":{}},{"cell_type":"markdown","source":"## 評価尺度を理解する\nこのノートブックは，[this great notebook](https://www.kaggle.com/pestipeti/explanation-of-map5-scoring-metric)を参考にしています．","metadata":{"execution":{"iopub.execute_input":"2022-04-11T08:39:27.342384Z","iopub.status.busy":"2022-04-11T08:39:27.342017Z","iopub.status.idle":"2022-04-11T08:39:27.37411Z","shell.execute_reply":"2022-04-11T08:39:27.372971Z","shell.execute_reply.started":"2022-04-11T08:39:27.342292Z"}}},{"cell_type":"markdown","source":"# Mean Average Precision(MAP)\n\nSubmissionは，Mean Average Precision@12（MAP@12）で評価されます．\n\n$$MAP@12 = {1 \\over U} \\sum_{u=1}^{U} \\sum_{k=1}^{min(n,12)}P(k) \\times rel(k)$$\n\n`U`はユーザーの数，`P(k)`は，k位までの精度です．`rel(k)`は`k`位が正しければ1を，誤っていれば０を返す関数です．また，nは予測数です（今回の場合は推薦する商品数）．\n\n最終的なMAP算出関数に向けて，一歩ずつ組み立ていきましょう．","metadata":{}},{"cell_type":"code","source":"import numpy as np\n\n# 真の値（ground_truth）\ngt = np.array(['a', 'b', 'c', 'd', 'e'])\n\n# 予測値\npreds1 = np.array(['b', 'c', 'a', 'd', 'e'])\npreds2 = np.array(['a', 'b', 'c', 'd', 'e'])\npreds3 = np.array(['f', 'b', 'c', 'd', 'e'])\npreds4 = np.array(['a', 'f', 'e', 'g', 'b'])\npreds5 = np.array(['a', 'f', 'c', 'g', 'b'])\npreds6 = np.array(['d', 'c', 'b', 'a', 'e'])","metadata":{"execution":{"iopub.status.busy":"2022-04-12T02:15:35.361106Z","iopub.execute_input":"2022-04-12T02:15:35.36151Z","iopub.status.idle":"2022-04-12T02:15:35.38669Z","shell.execute_reply.started":"2022-04-12T02:15:35.361411Z","shell.execute_reply":"2022-04-12T02:15:35.386035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Precision","metadata":{}},{"cell_type":"markdown","source":"Precisionは，真陽性率のことです．分類問題の際には，モデルから算出されるtotalの陽性率を，真陽性数で割ることで得られます．\n\n$$ P = { \\#\\ of\\ correct\\ predictions\\over \\#\\ of\\ all\\ predictions  } = {TP \\over (TP + FP)}$$\n\n<br>\n\n情報検索の分野では，Precisionは適合した文書の割合を表します．\n\n$${\\displaystyle {\\text{P}}={\\frac {|\\{{\\text{relevant documents}}\\}\\cap \\{{\\text{retrieved documents}}\\}|}{|\\{{\\text{retrieved documents}}\\}|}}}$$","metadata":{"execution":{"iopub.execute_input":"2022-04-11T09:39:04.826226Z","iopub.status.busy":"2022-04-11T09:39:04.825754Z","iopub.status.idle":"2022-04-11T09:39:04.860308Z","shell.execute_reply":"2022-04-11T09:39:04.858814Z","shell.execute_reply.started":"2022-04-11T09:39:04.82611Z"}}},{"cell_type":"markdown","source":"# Precision at K","metadata":{}},{"cell_type":"markdown","source":"k位で区切った精度である，`P(k)`は，1位からk位までの集合を考えて求めることができる精度です．  \nこの値を計算することで，k位までのレコメンデーションを取ることができ，正解データをもとにした精度を考えることができます． <br>\n<br>\nExample 1:\n`gt=[a,b,c,d,e]` と `pred=[b,c,a,d,e]` について`P@1` を考えます．`pred`からの1つのレコメンデーションとして`b`を取り，`gt`との`precision`を計算すると以下のようになります．<br>\n<br>\n$${\\displaystyle {\\text{P}}={\\frac {|\\{{\\text{gt}}\\}\\cap \\{{\\text{pred[:1]}}\\}|}{|\\{{\\text{pred[:1]}}\\}|}}={\\frac {\\text{1}}{\\text{1}}}}$$ \n<br>\nExample 2:\n`gt=[a,b,c,d,e]` と `pred=[f,b,c,d,e]` について`P@1`　を考えます．`pred`からの1つのレコメンデーションとして，`f`を取り，`gt`との`precision`を計算すると以下のようになります．<br>\n<br>\n$${\\displaystyle {\\text{P}}={\\frac {|\\{{\\text{gt}}\\}\\cap \\{{\\text{pred[:1]}}\\}|}{|\\{{\\text{pred[:1]}}\\}|}}={\\frac {\\text{0}}{\\text{1}}}}$$\n<br>\nExample 3:\n`gt=[a,b,c,d,e]` と `pred=[a,f,e,g,b]` について`P@2` を考えます．`pred`からの2つのレコメンデーションとして，`[a,f]`を取り，`gt`との`precision`を計算します．`gt`に現れるのは`a`たった１つであるため，共通集合数は`１`になります．<br>\n<br>\n$${\\displaystyle {\\text{P}}={\\frac {|\\{{\\text{gt}}\\}\\cap \\{{\\text{pred[:2]}}\\}|}{|\\{{\\text{pred[:2]}}\\}|}}={\\frac {\\text{1}}{\\text{2}}}}$$\n<br>\n\n具体例は以下のようになります．\n\n| true  | predicted   | k  | P(k) |\n|:-:|:-:|:-:|:-:|\n| [a, b, c, d, e]  | [b, c, a, d, e]   | 1  | 1.0  |\n| [a, b, c, d, e]  | [a, b, c, d, e]   | 1  | 1.0  |\n| [a, b, c, d, e]  | [f, b, c, d, e]   | 1  | 0.0  |\n| [a, b, c, d, e]  | [a, f, e, g, b]   | 2  | $$1\\over2$$  |\n| [a, b, c, d, e]  | [a, f, c, g, b]   | 3  | $$2\\over3$$  |\n| [a, b, c, d, e]  | [d, c, b, a, e]   | 3  | $$3\\over3$$  |","metadata":{"execution":{"iopub.execute_input":"2022-04-11T10:21:34.451919Z","iopub.status.busy":"2022-04-11T10:21:34.45144Z","iopub.status.idle":"2022-04-11T10:21:34.487539Z","shell.execute_reply":"2022-04-11T10:21:34.485891Z","shell.execute_reply.started":"2022-04-11T10:21:34.451798Z"}}},{"cell_type":"code","source":"def precision_at_k(y_true, y_pred, k=12):\n    \"\"\" Computes Precision at k for one sample\n    \n    Parameters\n    __________\n    y_true: np.array\n            Array of correct recommendations (Order doesn't matter)\n    y_pred: np.array\n            Array of predicted recommendations (Order does matter)\n    k: int, optional\n       Maximum number of predicted recommendations\n            \n    Returns\n    _______\n    score: double\n           Precision at k\n    \"\"\"\n    intersection = np.intersect1d(y_true, y_pred[:k])\n    return len(intersection) / k","metadata":{"execution":{"iopub.status.busy":"2022-04-12T02:15:35.400829Z","iopub.execute_input":"2022-04-12T02:15:35.401254Z","iopub.status.idle":"2022-04-12T02:15:35.406349Z","shell.execute_reply.started":"2022-04-12T02:15:35.401223Z","shell.execute_reply":"2022-04-12T02:15:35.405443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"assert precision_at_k(gt, preds1, k=1) == 1.0\nassert precision_at_k(gt, preds2, k=1) == 1.0\nassert precision_at_k(gt, preds3, k=1) == 0.0\nassert precision_at_k(gt, preds4, k=2) == 1/2\nassert precision_at_k(gt, preds5, k=3) == 2/3\nassert precision_at_k(gt, preds6, k=3) == 3/3","metadata":{"execution":{"iopub.status.busy":"2022-04-12T02:15:35.411183Z","iopub.execute_input":"2022-04-12T02:15:35.412073Z","iopub.status.idle":"2022-04-12T02:15:35.420431Z","shell.execute_reply.started":"2022-04-12T02:15:35.412016Z","shell.execute_reply":"2022-04-12T02:15:35.419663Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Rel at K","metadata":{}},{"cell_type":"markdown","source":"`Rel(k)` は，k位のアイテムが適合（正しい）場合に１を，そうでない場合は0を返します<br>\nExample 1:\n`gt=[a,b,c,d,e]` と `pred=[b,c,a,d,e]` について `rel@1` を考えます． `pred`からの1つのレコメンデーションとして，`b`を取り，　`gt`との`precision`を計算すると以下のようになります．<br>\n$${\\displaystyle {\\text{rel(k)}}=1.0}$$\n<br>\nExample 2:\n`gt=[a,b,c,d,e]` と `pred=[f,b,c,d,e]` について `rel@1` を考えます． `pred`からの１つのレコメンデーションとして，`f`を取り，`gt`との`precision`を計算すると以下のようになります．<br>\n$${\\displaystyle {\\text{rel(k)}}=0.0}$$\n<br>\nExample 3:\n`gt=[a,b,c,d,e]` と `pred=[a,f,e,g,b]` について `rel@2` を考えます． `pred`からの１つのレコメンデーションとして，`f`を取り，`gt`との`precision`を計算すると以下のようになります．\n$${\\displaystyle {\\text{rel(k)}}=0.0}$$\n<br>\n\n具体例は以下のようになります．\n\n| true  | predicted   | k  | rel(k) |\n|:-:|:-:|:-:|:-:|\n| [a, b, c, d, e]  | [b, c, a, d, e]   | 1  | 1.0  |\n| [a, b, c, d, e]  | [a, b, c, d, e]   | 1  | 1.0  |\n| [a, b, c, d, e]  | [f, b, c, d, e]   | 1  | 0.0  |\n| [a, b, c, d, e]  | [a, f, e, g, b]   | 2  | 0.0  |\n| [a, b, c, d, e]  | [a, f, c, g, b]   | 3  | 1.0  |\n| [a, b, c, d, e]  | [d, c, b, a, e]   | 3  | 1.0  |","metadata":{}},{"cell_type":"code","source":"def rel_at_k(y_true, y_pred, k=12):\n    \"\"\" Computes Relevance at k for one sample\n    \n    Parameters\n    __________\n    y_true: np.array\n            Array of correct recommendations (Order doesn't matter)\n    y_pred: np.array\n            Array of predicted recommendations (Order does matter)\n    k: int, optional\n       Maximum number of predicted recommendations\n            \n    Returns\n    _______\n    score: double\n           Relevance at k\n    \"\"\"\n    if y_pred[k-1] in y_true:\n        return 1\n    else:\n        return 0","metadata":{"execution":{"iopub.status.busy":"2022-04-12T02:15:35.425869Z","iopub.execute_input":"2022-04-12T02:15:35.426155Z","iopub.status.idle":"2022-04-12T02:15:35.432695Z","shell.execute_reply.started":"2022-04-12T02:15:35.426121Z","shell.execute_reply":"2022-04-12T02:15:35.431713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"assert rel_at_k(gt, preds1, k=1) == 1.0\nassert rel_at_k(gt, preds2, k=1) == 1.0\nassert rel_at_k(gt, preds3, k=1) == 0.0\nassert rel_at_k(gt, preds4, k=2) == 0.0\nassert rel_at_k(gt, preds5, k=3) == 1.0\nassert rel_at_k(gt, preds6, k=3) == 1.0","metadata":{"execution":{"iopub.status.busy":"2022-04-12T02:15:35.434038Z","iopub.execute_input":"2022-04-12T02:15:35.434926Z","iopub.status.idle":"2022-04-12T02:15:35.445164Z","shell.execute_reply.started":"2022-04-12T02:15:35.434869Z","shell.execute_reply":"2022-04-12T02:15:35.444263Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Average Precision at K","metadata":{}},{"cell_type":"markdown","source":"全ての`k`についての`P@k`と`rel(k)`から，全製品について単純な平均を取ったものです．\n\n$${1\\over{{min(n,12)}}} {\\sum_{k=1}^{min(n,12)}P(k) \\times rel(k)}$$","metadata":{}},{"cell_type":"code","source":"def average_precision_at_k(y_true, y_pred, k=12):\n    \"\"\" Computes Average Precision at k for one sample\n    \n    Parameters\n    __________\n    y_true: np.array\n            Array of correct recommendations (Order doesn't matter)\n    y_pred: np.array\n            Array of predicted recommendations (Order does matter)\n    k: int, optional\n       Maximum number of predicted recommendations\n            \n    Returns\n    _______\n    score: double\n           Average Precision at k\n    \"\"\"\n    ap = 0.0\n    for i in range(1, k+1):\n        ap += precision_at_k(y_true, y_pred, i) * rel_at_k(y_true, y_pred, i)\n        \n    return ap / min(k, len(y_true))","metadata":{"execution":{"iopub.status.busy":"2022-04-12T02:15:35.448912Z","iopub.execute_input":"2022-04-12T02:15:35.449194Z","iopub.status.idle":"2022-04-12T02:15:35.457956Z","shell.execute_reply.started":"2022-04-12T02:15:35.449161Z","shell.execute_reply":"2022-04-12T02:15:35.457129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"assert average_precision_at_k(gt, preds1, k=1) == 1.0\nassert average_precision_at_k(gt, preds2, k=1) == 1.0\nassert average_precision_at_k(gt, preds3, k=1) == 0.0\nassert average_precision_at_k(gt, preds4, k=2) == 0.5\nassert average_precision_at_k(gt, preds5, k=3) == 0.5555555555555555\nassert average_precision_at_k(gt, preds6, k=3) == 1.0","metadata":{"execution":{"iopub.status.busy":"2022-04-12T02:15:35.459796Z","iopub.execute_input":"2022-04-12T02:15:35.460051Z","iopub.status.idle":"2022-04-12T02:15:35.475872Z","shell.execute_reply.started":"2022-04-12T02:15:35.460022Z","shell.execute_reply":"2022-04-12T02:15:35.475181Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Mean Average Precision at K","metadata":{}},{"cell_type":"markdown","source":"全ユーザーについてAP（Average Precision）の平均を取ったものです．","metadata":{}},{"cell_type":"code","source":"def mean_average_precision(y_true, y_pred, k=12):\n    \"\"\" Computes MAP at k\n    \n    Parameters\n    __________\n    y_true: np.array\n            2D Array of correct recommendations (Order doesn't matter)\n    y_pred: np.array\n            2D Array of predicted recommendations (Order does matter)\n    k: int, optional\n       Maximum number of predicted recommendations\n            \n    Returns\n    _______\n    score: double\n           MAP at k\n    \"\"\"\n    return np.mean([average_precision_at_k(gt, pred, k) \\\n                    for gt, pred in zip(y_true, y_pred)])","metadata":{"execution":{"iopub.status.busy":"2022-04-12T02:15:35.477889Z","iopub.execute_input":"2022-04-12T02:15:35.478482Z","iopub.status.idle":"2022-04-12T02:15:35.486611Z","shell.execute_reply.started":"2022-04-12T02:15:35.478428Z","shell.execute_reply":"2022-04-12T02:15:35.48591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_true = np.array([gt, gt, gt, gt, gt, gt])\ny_pred = np.array([preds1, preds2, preds3, preds4, preds5, preds6])","metadata":{"execution":{"iopub.status.busy":"2022-04-12T02:15:35.48789Z","iopub.execute_input":"2022-04-12T02:15:35.488737Z","iopub.status.idle":"2022-04-12T02:15:35.498304Z","shell.execute_reply.started":"2022-04-12T02:15:35.488683Z","shell.execute_reply":"2022-04-12T02:15:35.497532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(average_precision_at_k(gt, preds1, k=4))\nprint(average_precision_at_k(gt, preds2, k=4))\nprint(average_precision_at_k(gt, preds3, k=4))\nprint(average_precision_at_k(gt, preds4, k=4))\nprint(average_precision_at_k(gt, preds5, k=4))\nprint(average_precision_at_k(gt, preds6, k=4))","metadata":{"execution":{"iopub.status.busy":"2022-04-12T02:15:35.500272Z","iopub.execute_input":"2022-04-12T02:15:35.50083Z","iopub.status.idle":"2022-04-12T02:15:35.514695Z","shell.execute_reply.started":"2022-04-12T02:15:35.500794Z","shell.execute_reply":"2022-04-12T02:15:35.513985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mean_average_precision(y_true, y_pred, k=4)","metadata":{"execution":{"iopub.status.busy":"2022-04-12T02:15:35.516208Z","iopub.execute_input":"2022-04-12T02:15:35.516683Z","iopub.status.idle":"2022-04-12T02:15:35.529954Z","shell.execute_reply.started":"2022-04-12T02:15:35.516647Z","shell.execute_reply":"2022-04-12T02:15:35.529247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 読んで頂きありがとうございました．","metadata":{}}]}