{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Competition Metrics\n\n\nHi everyone!\n\nI would like to share with you my metrics implementations for this competition. \n\nI think correct calculating metrics is very important! \n\n### If you find any imprecision in calculating metrics, please, let me know!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# CHANGELOG\n\n- v1, initial\n- v2, fix global average from micro to macro (see [here](https://www.kaggle.com/c/birdsong-recognition/discussion/160320) in discussion), fix explanation `Micro averaged`\n- v3, fix mistake with swapping names FN/FP  (thanks for noticing [@nandhuelan](https://www.kaggle.com/nandhuelan))\n- v4, fix sorting in numpy method\n- v5, added example with usage of ready method from [sklearn](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html) (thanks for kindly noticing [@Maxwell](https://www.kaggle.com/maxwell110))\n\n<img src=\"https://i.stack.imgur.com/VxiS5.png\" width=\"500\" align=\"left\"/>","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Explanation\n\nIn description:\n> Submissions will be evaluated based on their row-wise micro averaged F1 score.\n\n- The F1 score is the harmonic mean of the precision and recall (more information [here](https://en.wikipedia.org/wiki/F1_score)). Equation:\n\n$ F_1 = {2 * precision * recall \\over precision + recall} = {2 * TP \\over 2*TP + FN + FP} $\n\n- Row-wise means that TP, FN, FP is calculated using every value (bird) in row (thanks a lot [@dhananjay3](https://www.kaggle.com/dhananjay3) for explanation [here](https://www.kaggle.com/c/birdsong-recognition/discussion/159968#893120) for me)\n\n- `Micro averaged` means that F1 is caluclated by counting the total TP, FN and FP in one row (!), after F1 for all rows are used as average (thanks a lot [@dhananjay3](https://www.kaggle.com/dhananjay3) and [@carriesmi](https://www.kaggle.com/carriesmi) for explanation and experiment [here](https://www.kaggle.com/c/birdsong-recognition/discussion/160320))","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Using Birds\n\nimplementation with using string birds","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\n\ndef row_wise_f1_score_micro(y_true, y_pred):\n    \"\"\" author @shonenkov \"\"\"\n    F1 = []\n    for preds, trues in zip(y_pred, y_true):\n        TP, FN, FP = 0, 0, 0\n        preds = preds.split()\n        trues = trues.split()\n        for true in trues:\n            if true in preds:\n                TP += 1\n            else:\n                FN += 1\n        for pred in preds:\n            if pred not in trues:\n                FP += 1\n        F1.append(2*TP / (2*TP + FN + FP))\n    return np.mean(F1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### tests:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print('[all equal]:', row_wise_f1_score_micro(\n    y_true=['nocall', 'ameavo'], \n    y_pred=['nocall', 'ameavo'],\n))\n\nprint('[nothing]:', row_wise_f1_score_micro(\n    y_true=['nocall', 'ameavo'], \n    y_pred=['amebit', 'amebit'],\n))\n\nprint('[1 correct]:', row_wise_f1_score_micro(\n    y_true=['nocall', 'ameavo'], \n    y_pred=['nocall', 'amebit'],\n))\n\nprint('[double prediction]:', row_wise_f1_score_micro(\n    y_true=['nocall', 'ameavo amebit'], \n    y_pred=['nocall', 'ameavo amebit'],\n))\n\nprint('[double prediction with permutation]:', row_wise_f1_score_micro(\n    y_true=['nocall', 'ameavo amebit'], \n    y_pred=['nocall', 'amebit ameavo'],\n))\n\n\nprint('[semi prediction]:', row_wise_f1_score_micro(\n    y_true=['nocall', 'ameavo amebit'], \n    y_pred=['nocall', 'ameavo'],\n))\n\nprint('[semi prediction with odd]:', row_wise_f1_score_micro(\n    y_true=['nocall', 'ameavo'], \n    y_pred=['nocall', 'ameavo amebit'],\n))\n\nprint('[semi prediction with double odd]:', row_wise_f1_score_micro(\n    y_true=['nocall', 'ameavo'], \n    y_pred=['nocall', 'ameavo amebit amecro'],\n))\n\nprint('[semi prediction of triple with odd]:', row_wise_f1_score_micro(\n    y_true=['nocall', 'ameavo amecro'], \n    y_pred=['nocall', 'ameavo amebit amecro'],\n))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Using Numpy Vectors\n\nFor example during evaluation of model. ","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"def row_wise_f1_score_micro_numpy(y_true, y_pred, threshold=0.5, count=5):\n    \"\"\" \n    @author shonenkov \n    \n    y_true - 2d npy vector with gt\n    y_pred - 2d npy vector with prediction\n    threshold - for round labels\n    count - number of preds (used sorting by confidence)\n    \"\"\"\n    def meth_agn_v2(x, threshold):\n        idx, = np.where(x > threshold)\n        return idx[np.argsort(x[idx])[::-1]]\n\n    F1 = []\n    for preds, trues in zip(y_pred, y_true):\n        TP, FN, FP = 0, 0, 0\n        preds = meth_agn_v2(preds, threshold)[:count]\n        trues = meth_agn_v2(trues, threshold)\n        for true in trues:\n            if true in preds:\n                TP += 1\n            else:\n                FN += 1\n        for pred in preds:\n            if pred not in trues:\n                FP += 1\n        F1.append(2*TP / (2*TP + FN + FP))\n    return np.mean(F1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### test","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pred = np.array([\n    [0.4,0.6,0.9],\n    [0.1,0.9,0.8],\n    [0.1,0.4,0.2],\n    [0.9,0.9,0.9],\n])\n\ny_true = np.array([\n    [0,0,1],\n    [0,1,1],\n    [1,0,1],\n    [1,1,1],\n])\n\nthreshold = 0.5\n\nrow_wise_f1_score_micro_numpy(y_true, y_pred, threshold=threshold)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pred = np.array([\n    [0.4,0.6,0.9],\n    [0.1,0.9,0.8],\n    [0.1,0.4,0.2],\n    [0.9,0.9,0.9],\n])\n\ny_true = np.array([\n    [0,0,1],\n    [0,1,1],\n    [1,0,1],\n    [1,1,1],\n])\n\nthreshold = 0.6\n\nrow_wise_f1_score_micro_numpy(y_true, y_pred, threshold=threshold)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Using Sklearn\n\nThanks a lot [@Maxwell](https://www.kaggle.com/maxwell110) for noticing simpler implementation in [sklearn](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html) with average \"samples\".\n\n\nP.S.\nThis method doesnt have param \"count\", but it is very good method for usage also! Param \"count\" provides to restrict prediction count using sorting by confidence (if count = 3, that means \"no more than 3\") ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import f1_score","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pred = np.array([\n    [0.4,0.6,0.9],\n    [0.1,0.9,0.8],\n    [0.1,0.4,0.2],\n    [0.9,0.9,0.9],\n])\n\ny_true = np.array([\n    [0,0,1],\n    [0,1,1],\n    [1,0,1],\n    [1,1,1],\n])\n\nthreshold = 0.5\n\nf1_score(y_true, np.where(y_pred > threshold, 1, 0), average='samples')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_pred = np.array([\n    [0.4,0.6,0.9],\n    [0.1,0.9,0.8],\n    [0.1,0.4,0.2],\n    [0.9,0.9,0.9],\n])\n\ny_true = np.array([\n    [0,0,1],\n    [0,1,1],\n    [1,0,1],\n    [1,1,1],\n])\n\nthreshold = 0.6\n\nf1_score(y_true, np.where(y_pred > threshold, 1, 0), average='samples')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Thank you for attention!\n\nDon't forget to read my kernel about sample submission using custom check phase (it helps to find and avoid bugs before using button submission): \n\n- [[Sample Submission] Using Custom Check](https://www.kaggle.com/shonenkov/sample-submission-using-custom-check)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### If you find any imprecision in calculating metrics, please, let me know!","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}