{"cells":[{"metadata":{},"cell_type":"markdown","source":"This Notebook is copied from https://www.kaggle.com/osciiart/understanding-lwlrap-sorry-it-s-in-japanese\n\nI just translated the text with google translator."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# Library loading\nimport numpy as np\nimport sklearn.metrics","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"# LwLRAP Calculation function\n# from official code https://colab.research.google.com/drive/1AgPdhSp7ttY18O3fEoHOQKlt_3HJDLi8#scrollTo=cRCaCIb9oguU\ndef _one_sample_positive_class_precisions(scores, truth):\n    \"\"\"Calculate precisions for each true class for a single sample.\n\n    Args:\n      scores: np.array of (num_classes,) giving the individual classifier scores.\n      truth: np.array of (num_classes,) bools indicating which classes are true.\n\n    Returns:\n      pos_class_indices: np.array of indices of the true classes for this sample.\n      pos_class_precisions: np.array of precisions corresponding to each of those\n        classes.\n    \"\"\"\n    num_classes = scores.shape[0]\n    pos_class_indices = np.flatnonzero(truth > 0)\n    # Only calculate precisions if there are some true classes.\n    if not len(pos_class_indices):\n        return pos_class_indices, np.zeros(0)\n    # Retrieval list of classes for this sample.\n    retrieved_classes = np.argsort(scores)[::-1]\n    # class_rankings[top_scoring_class_index] == 0 etc.\n    class_rankings = np.zeros(num_classes, dtype=np.int)\n    class_rankings[retrieved_classes] = range(num_classes)\n    # Which of these is a true label?\n    retrieved_class_true = np.zeros(num_classes, dtype=np.bool)\n    retrieved_class_true[class_rankings[pos_class_indices]] = True\n    # Num hits for every truncated retrieval list.\n    retrieved_cumulative_hits = np.cumsum(retrieved_class_true)\n    # Precision of retrieval list truncated at each hit, in order of pos_labels.\n    precision_at_hits = (\n            retrieved_cumulative_hits[class_rankings[pos_class_indices]] /\n            (1 + class_rankings[pos_class_indices].astype(np.float)))\n    return pos_class_indices, precision_at_hits\n\n\ndef calculate_per_class_lwlrap(truth, scores):\n    \"\"\"Calculate label-weighted label-ranking average precision.\n\n    Arguments:\n      truth: np.array of (num_samples, num_classes) giving boolean ground-truth\n        of presence of that class in that sample.\n      scores: np.array of (num_samples, num_classes) giving the classifier-under-\n        test's real-valued score for each class for each sample.\n\n    Returns:\n      per_class_lwlrap: np.array of (num_classes,) giving the lwlrap for each\n        class.\n      weight_per_class: np.array of (num_classes,) giving the prior of each\n        class within the truth labels.  Then the overall unbalanced lwlrap is\n        simply np.sum(per_class_lwlrap * weight_per_class)\n    \"\"\"\n    assert truth.shape == scores.shape\n    num_samples, num_classes = scores.shape\n    # Space to store a distinct precision value for each class on each sample.\n    # Only the classes that are true for each sample will be filled in.\n    precisions_for_samples_by_classes = np.zeros((num_samples, num_classes))\n    for sample_num in range(num_samples):\n        pos_class_indices, precision_at_hits = (\n            _one_sample_positive_class_precisions(scores[sample_num, :],\n                                                  truth[sample_num, :]))\n        precisions_for_samples_by_classes[sample_num, pos_class_indices] = (\n            precision_at_hits)\n    labels_per_class = np.sum(truth > 0, axis=0)\n    weight_per_class = labels_per_class / float(np.sum(labels_per_class))\n    # Form average of each column, i.e. all the precisions assigned to labels in\n    # a particular class.\n    per_class_lwlrap = (np.sum(precisions_for_samples_by_classes, axis=0) /\n                        np.maximum(1, labels_per_class))\n    # overall_lwlrap = simple average of all the actual per-class, per-sample precisions\n    #                = np.sum(precisions_for_samples_by_classes) / np.sum(precisions_for_samples_by_classes > 0)\n    #           also = weighted mean of per-class lwlraps, weighted by class label prior across samples\n    #                = np.sum(per_class_lwlrap * weight_per_class)\n    return per_class_lwlrap, weight_per_class\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 1 sample Calculation per  \nIf there is one correct answer label.<br>\nAssume that the class is A, B, C.<br>\nConsider the case where the correct answer label = A, prediction = (A: 0.7, B: 0.1, 0.2) as an example.<br>\nFirst, rank the predictions (swing numbers in order of decreasing value).<br>\n-> Forecast = (A: 1, B: 3, C: 2)<br>\nScore = 1 <br>\nCalculated as the number of correct answers up to the rank of the correct answer label / rank of the correct answer label.<br>\nin this case, Correct label rank = 1<br>\nNumber of correct answers from 1 to the rank of correct label => number of correct answers from rank 1 to 1 = 1<br>\nSo,<br>\nScore = 1/1 = 1.0.<br>"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's actually calculate.\ny_true = np.array([1, 0, 0,])\ny_score = np.array([0.7, 0.1, 0.2])\npos_class_indices, precision_at_hits = _one_sample_positive_class_precisions(y_score, y_true)\nprint(\"Correct answer label\", pos_class_indices)\nprint(\"Score\", precision_at_hits)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Another example<br>\nIf the correct answer label = A, prediction = (A: 0.1, B: 0.7, 0.2).<br>\nRanked prediction = (A: 3, B: 1, C: 2)<br>\nCorrect answer label rank = 3<br>\nNumber of correct answers from 1 to the rank of correct label = number of correct answers from rank 1 to 3 = 1<br>\nSo,Score = 1/3 = 0.33.<br>"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's actually calculate.\ny_true = np.array([1, 0, 0,])\ny_score = np.array([0.1, 0.7, 0.2])\npos_class_indices, precision_at_hits = _one_sample_positive_class_precisions(y_score, y_true)\nprint(\"Correct answer label\", pos_class_indices)\nprint(\"Score\", precision_at_hits)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If there are multiple correct answer labels.<br>\nAssume that the class is A, B, C.<br>\nConsider the case where the correct answer label = A, C prediction = (A: 0.7, B: 0.1, 0.2) as an example.<br>\nScore is calculated for each correct answer label.<br>\nFirst, when calculating the score for the correct answer label A,<br>\nRanked prediction = (A: 1, B: 3, C: 2)<br>\nCorrect label rank = 1<br>\nNumber of correct answers from 1 to the rank of correct label = number of correct answers from rank 1 to 1 = 1<br>\nSo,Score = 1/1 = 1.0.<br>\n<br>\nNext, when calculating the score for the correct answer label C,<br>\nRanked prediction = (A: 1, B: 3, C: 2)<br>\nCorrect answer label rank = 2<br>\nNumber of correct answers from 1 to the rank of correct label = number of correct answers from rank 1 to 2 = 2<br>\nSo,Score = 2/2 = 1.0<br>"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's actually calculate.\ny_true = np.array([1, 0, 1,])\ny_score = np.array([0.7, 0.1, 0.2])\npos_class_indices, precision_at_hits = _one_sample_positive_class_precisions(y_score, y_true)\nprint(\"Correct answer label\", pos_class_indices)\nprint(\"Score\", precision_at_hits)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Another example  <br>\n<br>\nConsider the case where the correct answer label = A, C prediction = (A: 0.1, B: 0.7, 0.2) as an example.<br>\nScore is calculated for each correct answer label.<br>\nFirst, when calculating the score for the correct answer label A,<br>\nRanked prediction = (A: 3, B: 1, C: 2)<br>\nCorrect answer label rank = 3<br>\nNumber of correct answers from 1 to the rank of correct label = number of correct answers from rank 1 to 3 = 2<br>\nSo,Score = 2/3 = 0.67.<br>\n<br>\nNext, when calculating the score for the correct answer label C,<br>\nRanked prediction = (A: 3, B: 1, C: 2)<br>\nCorrect answer label rank = 2<br>\nNumber of correct answers from 1 to the rank of correct label = number of correct answers from rank 1 to 2 = 1<br>\nSo,Score = 1/2 = 0.5. <br>"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's actually calculate.\ny_true = np.array([1, 0, 1,])\ny_score = np.array([0.1, 0.7, 0.2])\npos_class_indices, precision_at_hits = _one_sample_positive_class_precisions(y_score, y_true)\nprint(\"Correct answer label\", pos_class_indices)\nprint(\"Score\", precision_at_hits)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Calculation for all samples<br>\n<br>\nSample 1: Correct answer label = A, C prediction = (A: 0.1, B: 0.7, 0.2)<br>\nSample 2: Correct answer label = B, C prediction = (A: 0.1, B: 0.7, 0.2)<br>\nConsider the case.<br>\nFirst, calculate the score for each class.<br>\nScore = total score for a class / number of correct labels for a class<br>\nin this case,<br>\nSample 1 score = A: 0.6667, C: 0.5<br>\nSample 2 score = B: 1.0, C: 1.0<br>\nSo,<br>\nClass A score = 0.6667 / 1 = 0.6667<br>\nClass B score = 1.0 / 1 = 1.0<br>\nClass C score = (0.5 + 1.0) / 2 = 0.75<br>\nIt becomes.<br>\n<br>\nWhen calculating scores for all classes, averaging the scores of each class does not take into account the bias in the number of correct labels for each class.<br>\nFrequent classes have less impact on the final score of one label,<br>\nInfrequently occurring classes have a greater effect on the final score of one label.<br>\nTherefore, a weighted average is taken with the number of occurrences of each class as a weight.<br>\nCorrect label occurrence number = (A: 1. B: 1, C: 2)<br>\nWeight = Number of correct label occurrence / Total number of correct labels = (A: 1. B: 1, C: 2) / 4 = (A: 0.25, B: 0.25. C: 0.5)<br>\nScore = Sum of (score weight for each class) = A: 0.6667 0.25 + B: 1.0 0.25 + C: 0.75 0.5 = 0.7917<br>\nThis is ultimately equal to the average of the scores for each label.<br>\nAverage score for each label = (0.6667 + 0.5 + 1.0 + 1.0) / 4 = 0.7917 <br>"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's actually calculate.\ny_true = np.array([[1, 0, 1,], [0, 1, 1]])\ny_score = np.array([[0.1, 0.7, 0.2], [0.1, 0.7, 0.2]])\n_, precision_at_hits1 = _one_sample_positive_class_precisions(y_score[0], y_true[0])\nprint(\"sample 1 Score\", precision_at_hits1)\n_, precision_at_hits2 = _one_sample_positive_class_precisions(y_score[1], y_true[1])\nprint(\"sample 2 Score\", precision_at_hits2)\nscore, weight = calculate_per_class_lwlrap(y_true, y_score)\nprint(\"Each class score\", score)\nprint(\"Weight of each class\", weight)\nLwLRAP = (score*weight).sum()\nprint(\"LwLRAP\", LwLRAP)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}