{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **Understand the Metric: Row-Wise Micro Averaged F1**","metadata":{}},{"cell_type":"markdown","source":"### Kernels \n\n- notebook by [@ihelon](https://www.kaggle.com/ihelon) :: [row-wise-micro-averaged-f1-score-metric](https://www.kaggle.com/ihelon/row-wise-micro-averaged-f1-score-metric)\n\n- notebook by [@shonenkov](https://www.kaggle.com/shonenkov) :: [competition-metrics](https://www.kaggle.com/shonenkov/competition-metrics)\n\n- https://github.com/yisaienkov/evaluations   \n- https://evaluations.readthedocs.io/en/latest/","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true}},{"cell_type":"markdown","source":"### References\n\n- https://machinelearningmastery.com/fbeta-measure-for-machine-learning/\n- https://medium.com/@douglaspsteen/beyond-the-f-1-score-a-look-at-the-f-beta-score-3743ac2ef6e3","metadata":{}},{"cell_type":"code","source":"# Internet ON.\n\n!pip install -U pip\n!pip install evaluations -q\n\n# Internet OFF. You need to add evaluations dataset (see input folders).\n# !pip install --no-deps '../input/evaluations/'","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from evaluations.kaggle_2020 import row_wise_micro_averaged_f1_score\nimport numpy as np\nimport pandas as pd\nimport os, sys, gc\nimport warnings\nwarnings.filterwarnings(action='ignore')","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Introduction \n\nSince the F-score (and Fbeta) is calculated using precision and recall let's refresh first these definitions:\n\n- `Precision`:  Precision is a metric that quantifies the number of correct positive predictions made.\n\n    It is calculated as the ratio of correctly predicted positive examples divided by the total number of positive examples that were predicted: \n\n    `Precision = TruePositives / (TruePositives + FalsePositives)`\n\n    The result is a value between 0.0 for no precision and 1.0 for full or perfect precision. The intuition for precision is that it is not concerned with false negatives and it minimizes false positives. \n\n- `Recall`: calculates the percentage of correct predictions for the positive class out of all positive predictions that could be made. It is calculated as the ratio of correctly predicted positive examples divided by the total number of positive examples that could be predicted.\n\n    `Recall = TruePositives / (TruePositives + FalseNegatives)`\n\n    The result is a value between 0.0 for no recall and 1.0 for full or perfect recall. The intuition for recall is that it is not concerned with false positives and it minimizes false negatives.\n\nMaximizing precision will minimize the false-positive errors, whereas maximizing recall will minimize the false-negative errors.\n\n- `F1 Score`: calculated as the harmonic mean of precision and recall, giving each the same weighting. The F-1 score can be represented by the following equation:\n\n    `F1 = 2 x Precision x Recall / (Precision + Recall)`\n    \n \n- The `Fbeta score` is a generalization of the F-measure that adds a configuration parameter called beta. A default beta value is 1.0, which is the same as the F-measure. A smaller beta value, such as 0.5, gives more weight to precision and less to recall, whereas a larger beta value, such as 2.0, gives less weight to precision and more weight to recall in the calculation of the score.\n\nIt is a useful metric to use when both precision and recall are important but slightly more attention is needed on one or the other, such as when false negatives are more important than false positives, or the reverse.\n","metadata":{}},{"cell_type":"markdown","source":"# Competition metric\n\n> From description: Submissions will be evaluated based on their `row-wise micro averaged F1 score`. For each row_id/time window, you need to provide a space delimited list of the set of unique birds that made a call beginning or ending in that time window. If there are no bird calls in a time window, use the code nocall.\n\n\n\n\n\n- `Row-wise`: means that TP, FN, FP is calculated using every value (bird) in row\n\n- `Micro averaged`: means that F1 is caluclated by counting the total TP, FN and FP in one row (!), after F1 for all rows are used as average","metadata":{}},{"cell_type":"markdown","source":"# Implementation \n\nIf you are interested on the implementation unfold below to see the code produced by [@shonenkov](https://www.kaggle.com/shonenkov/) ","metadata":{}},{"cell_type":"code","source":"## see original nb: https://www.kaggle.com/shonenkov/competition-metrics\n\nimport numpy as np\n\ndef row_wise_f1_score_micro(y_true, y_pred):\n    \"\"\" author @shonenkov \"\"\"\n    F1 = []\n    for preds, trues in zip(y_pred, y_true):\n        TP, FN, FP = 0, 0, 0\n        preds = preds.split()\n        trues = trues.split()\n        for true in trues:\n            if true in preds:\n                TP += 1\n            else:\n                FN += 1\n        for pred in preds:\n            if pred not in trues:\n                FP += 1\n        F1.append(2*TP / (2*TP + FN + FP))\n    return np.mean(F1)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.read_csv('../input/birdclef-2021/train_metadata.csv',)\ntrain_csv = pd.read_csv(\"../input/birdclef-2021/train_soundscape_labels.csv\")\n# test_csv = pd.read_csv(\"../input/birdclef-2021/test.csv\")\n# sample_sub= pd.read_csv(\"../input/birdclef-2021/sample_submission.csv\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# len(train_csv.birds.unique()), train_csv.birds.unique()\n\n# 'nocall', 'rubwre1', 'obnthr1', 'brnjay', 'brnjay sthwoo1',\n# 'rucwar', 'grekis rucwar', 'rucwar runwre1', 'rtlhum rucwar',\n#  'hofwoo1', 'hofwoo1 rucwar', 'hofwoo1 rucwar runwre1', 'runwre1',\n#  'grekis', 'grekis runwre1', 'clcrob rucwar', 'clcrob',\n#  'runwre1 yehcar1', 'rucwar runwre1 yehcar1', 'melbla1', 'crfpar',\n#  'crfpar rucwar', 'rucwar whcpar', 'whcpar', 'hofwoo1 whcpar',\n#  'crfpar runwre1', 'bobfly1', 'bobfly1 rucwar', 'grhcha1',\n#  'plawre1', 'bobfly1 plawre1', 'orcpar', 'bobfly1 orfpar',","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Simple examples","metadata":{}},{"cell_type":"code","source":"# intuition for precision\nfrom sklearn.metrics import precision_score\n\n# no precision\ny_true = [0, 0, 0, 0, 0, 1, 1, 1, 1, 1]\ny_pred = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\nscore = precision_score(y_true, y_pred)\nprint('No Precision: %.3f' % score)\n\n# some false positives\ny_true = [0, 0, 0, 0, 0, 1, 1, 1, 1, 1]\ny_pred = [0, 0, 0, 1, 1, 1, 1, 1, 1, 1]\nscore = precision_score(y_true, y_pred)\nprint('Some False Positives: %.3f' % score)\n\n# some false negatives\ny_true = [0, 0, 0, 0, 0, 1, 1, 1, 1, 1]\ny_pred = [0, 0, 0, 0, 0, 0, 0, 1, 1, 1]\nscore = precision_score(y_true, y_pred)\nprint('Some False Negatives: %.3f' % score)\n\n# perfect precision\ny_true = [0, 0, 0, 0, 0, 1, 1, 1, 1, 1]\ny_pred = [0, 0, 0, 0, 0, 1, 1, 1, 1, 1]\nscore = precision_score(y_true, y_pred)\nprint('Perfect Precision: %.3f' % score)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## some simple tests\n\nprint('Competition metric - simple tests\\n')\n\n\ny_true = [\n   'acafly', \n    'acowoo', \n    'aldfly',\n    'nocall',\n]\n\ny_pred = [\n    'acafly', \n    'acowoo', \n    'aldfly',\n    'nocall',\n]\n\ny_pred1 = [\n    'acafly', \n    'acowoo', \n    'aldfly',\n#     'nocall',\n]\n\n# print('single sample scores\\n')\nprint(f'F1-score with all correct:', row_wise_micro_averaged_f1_score(y_true, y_pred) )\nprint(f'F1-score with 3 correct:', row_wise_micro_averaged_f1_score(y_true, y_pred1) )","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Competition metric - more tests\\n')\n\n\nprint('[all equal]:', row_wise_micro_averaged_f1_score(\n    y_true=['nocall', 'ameavo'], \n    y_pred=['nocall', 'ameavo'],\n))\n\nprint()\nprint('[nothing]:', row_wise_micro_averaged_f1_score(\n    y_true=['nocall', 'ameavo'], \n    y_pred=['amebit', 'amebit'],\n))\n\nprint()\nprint('[1 correct]:', row_wise_micro_averaged_f1_score(\n    y_true=['nocall', 'ameavo'], \n    y_pred=['nocall', 'amebit'],\n))\n\nprint()\nprint('[double prediction]:', row_wise_micro_averaged_f1_score(\n    y_true=['nocall', 'ameavo amebit'], \n    y_pred=['nocall', 'ameavo amebit'],\n))\n\nprint()\nprint('[double prediction with permutation]:', row_wise_micro_averaged_f1_score(\n    y_true=['nocall', 'ameavo amebit'], \n    y_pred=['nocall', 'amebit ameavo'],\n))\n\nprint()\nprint('[semi prediction]:', row_wise_micro_averaged_f1_score(\n    y_true=['nocall', 'ameavo amebit'], \n    y_pred=['nocall', 'ameavo'],\n))\n\nprint()\nprint('[semi prediction with odd]:', row_wise_micro_averaged_f1_score(\n    y_true=['nocall', 'ameavo'], \n    y_pred=['nocall', 'ameavo amebit'],\n))\n\nprint()\nprint('[semi prediction with double odd]:', row_wise_micro_averaged_f1_score(\n    y_true=['nocall', 'ameavo'], \n    y_pred=['nocall', 'ameavo amebit amecro'],\n))\n\nprint()\nprint('[semi prediction of triple with odd]:', row_wise_micro_averaged_f1_score(\n    y_true=['nocall', 'ameavo amecro'], \n    y_pred=['nocall', 'ameavo amebit amecro'],\n))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Please, if I'm missing anything or you have any feedback leave a comment bellow\n\n#### Fork it and start experimenting to be familiar with the metric before start the heavy job :-)","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}