{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"This notebook is based on below discussion:\nhttps://www.kaggle.com/competitions/birdclef-2022/discussion/321883#1772162","metadata":{}},{"cell_type":"code","source":"!pip install nb_black > /dev/null","metadata":{"execution":{"iopub.status.busy":"2022-05-18T20:44:59.742225Z","iopub.execute_input":"2022-05-18T20:44:59.742913Z","iopub.status.idle":"2022-05-18T20:45:12.9716Z","shell.execute_reply.started":"2022-05-18T20:44:59.742883Z","shell.execute_reply":"2022-05-18T20:45:12.970634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nplt.style.use(\"ggplot\")\n\n%load_ext lab_black","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-05-18T20:45:12.973853Z","iopub.execute_input":"2022-05-18T20:45:12.974204Z","iopub.status.idle":"2022-05-18T20:45:14.134997Z","shell.execute_reply.started":"2022-05-18T20:45:12.974162Z","shell.execute_reply":"2022-05-18T20:45:14.134264Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Evaluation metrics","metadata":{}},{"cell_type":"markdown","source":"## Balanced Accuracy\n\n$$\n\\text{BA} = \\frac{\\text{TPR} + \\text{TNR}}{2}\n$$\n\nhttps://scikit-learn.org/stable/modules/model_evaluation.html#balanced-accuracy-score","metadata":{}},{"cell_type":"code","source":"def balanced_accuracy(pred, target, eps=1e-6):\n    tp = (pred * target).sum(axis=-1)\n    fn = ((1 - pred) * target).sum(axis=-1)\n    fp = (pred * (1 - target)).sum(axis=-1)\n    tn = ((1 - pred) * (1 - target)).sum(axis=-1)\n    tpr = tp / (tp + fn + eps)\n    tnr = tn / (tn + fp + eps)\n    return 0.5 * (tpr + tnr)","metadata":{"execution":{"iopub.status.busy":"2022-05-18T20:45:14.136084Z","iopub.execute_input":"2022-05-18T20:45:14.1363Z","iopub.status.idle":"2022-05-18T20:45:14.154749Z","shell.execute_reply.started":"2022-05-18T20:45:14.136276Z","shell.execute_reply":"2022-05-18T20:45:14.15392Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Harmonic Mean of TPR and TNR\n\nWe call this \"balanced F1\" for convenience:\n\n$$\n\\text{BF1} = \\frac{2\\text{TPR} \\times \\text{TNR}}{\\text{TPR} + \\text{TNR}}\n$$","metadata":{}},{"cell_type":"code","source":"def balanced_f1(pred, target, eps=1e-6):\n    tp = (pred * target).sum(axis=-1)\n    fn = ((1 - pred) * target).sum(axis=-1)\n    fp = (pred * (1 - target)).sum(axis=-1)\n    tn = ((1 - pred) * (1 - target)).sum(axis=-1)\n    tpr = tp / (tp + fn + eps)\n    tnr = tn / (tn + fp + eps)\n    return 2 * (tpr * tnr) / (tpr + tnr + eps)","metadata":{"execution":{"iopub.status.busy":"2022-05-18T20:45:14.156176Z","iopub.execute_input":"2022-05-18T20:45:14.156764Z","iopub.status.idle":"2022-05-18T20:45:14.176379Z","shell.execute_reply.started":"2022-05-18T20:45:14.156721Z","shell.execute_reply":"2022-05-18T20:45:14.175583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Mean F1 and inverted F1 (MFIF)\n\nThis metrics is suggested by [Enrique Gurdiel](https://www.kaggle.com/gurdiel) in [this post](https://www.kaggle.com/competitions/birdclef-2022/discussion/321883#1772071).\n\nThis score calculates the mean of F1 and F1 for inverted both prediction and target.\nWe call this score MFIF for simplicity.\n\n```\nMFIF Score = 0.5 * (F1(pred, target) + F1(1 - pred, 1 - target))\n```","metadata":{}},{"cell_type":"code","source":"def f1(pred, target, eps=1e-6):\n    tp = (pred * target).sum(axis=-1)\n    fn = ((1 - pred) * target).sum(axis=-1)\n    fp = (pred * (1 - target)).sum(axis=-1)\n    tn = ((1 - pred) * (1 - target)).sum(axis=-1)\n    precision = tp / (tp + fp + eps)\n    recall = tp / (tp + fn + eps)\n    return 2 * precision * recall / (precision + recall + eps)\n\n\ndef mean_f1_and_inv_f1(pred, target):\n    return 0.5 * (f1(pred, target) + f1(1 - pred, 1 - target))","metadata":{"execution":{"iopub.status.busy":"2022-05-18T20:45:14.177711Z","iopub.execute_input":"2022-05-18T20:45:14.178006Z","iopub.status.idle":"2022-05-18T20:45:14.200648Z","shell.execute_reply.started":"2022-05-18T20:45:14.177968Z","shell.execute_reply":"2022-05-18T20:45:14.199911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Numerical Simulation\n\n## Condistion of simulation\n\n* Here, there are 10,560 samples (which is identical to the number of samples of public test data) with negative and positive probability is $p_\\text{pos}$ and $p_\\text{neg}$ respectively.\n* The probability of correct prediction of the simulated model is constant: the probability of correct answer is $p_\\text{correct}$ for both positive and negative targets.","metadata":{}},{"cell_type":"markdown","source":"## Prepare target and simulated model prediction","metadata":{}},{"cell_type":"markdown","source":"## Experiment","metadata":{}},{"cell_type":"code","source":"class TargetGenerator:\n    def __init__(self, p_pos=0.1):\n        self.p_pos = p_pos\n\n    def generate(self, n_samples, N):\n        return np.random.choice(2, (N, n_samples), p=[1 - self.p_pos, self.p_pos])\n\n\nclass PredictorGenerator:\n    def __init__(self, p_correct):\n        self.p_correct = p_correct\n\n    def generate(self, n_samples, target, N):\n        base = np.random.choice(\n            2, (N, n_samples), p=[1 - self.p_correct, self.p_correct]\n        )\n        pred = target * base + (1 - target) * (1 - base)\n        return pred","metadata":{"execution":{"iopub.status.busy":"2022-05-18T20:47:27.191575Z","iopub.execute_input":"2022-05-18T20:47:27.192113Z","iopub.status.idle":"2022-05-18T20:47:27.213213Z","shell.execute_reply.started":"2022-05-18T20:47:27.192079Z","shell.execute_reply":"2022-05-18T20:47:27.212292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def simulation(\n    target_generator,\n    predictor_generator,\n    n_samples=10_560,\n    N=1_000,\n    n_disp=50,\n    n_bins=30,\n    stat=\"percent\",\n):\n    \"\"\"\n    n_samples = 5500 * 12 * 0.16 = 10560\n    \"\"\"\n    print(\"=\" * 14 + \" Experiment \" + \"=\" * 14)\n    print(f\"- {target_generator.p_pos} of positive samples\")\n    if hasattr(predictor_generator, \"p_correct\"):\n        print(f\"- {predictor_generator.p_correct} of correct prediction\")\n    print(f\"- {n_samples} samples\")\n    print(f\"- {N} trials\")\n    print(\"=\" * 40)\n\n    target = target_generator.generate(n_samples, N)\n    pred = predictor_generator.generate(n_samples, target, N)\n\n    print(f\"target (first {n_disp} samples): {target[0, :n_disp].tolist()}\")\n    print(\"\")\n    print(f\"prediction (first {n_disp} samples): {pred[0, :n_disp].tolist()}\")\n    print(\"\")\n    print(\"** Calculated Score **\")\n    ba = balanced_accuracy(pred, target)\n    mfif = mean_f1_and_inv_f1(pred, target)\n    f1_ = f1(pred, target)\n    bf1 = balanced_f1(pred, target)\n\n    print(f\"F1: {f1_.mean(axis=0):.4f} (std={f1_.std(axis=0):.4f})\")\n    print(f\"MFIF: {mfif.mean(axis=0):.4f} (std={mfif.std(axis=0):.4f})\")\n    print(f\"balanced accuracy: {ba.mean(axis=0):.4f} (std={ba.std(axis=0):.4f})\")\n    print(f\"balanced F1: {bf1.mean(axis=0):.4f} (std={bf1.std(axis=0):.4f})\")\n\n    _, ax = plt.subplots(figsize=(8, 5))\n    sns.histplot(\n        f1_, label=\"F1\", color=\"blue\", ax=ax, stat=stat, bins=n_bins, alpha=0.5\n    )\n    sns.histplot(\n        mfif, label=\"MFIF\", color=\"green\", ax=ax, stat=stat, bins=n_bins, alpha=0.5\n    )\n    sns.histplot(\n        ba,\n        label=\"balanced accuracy\",\n        color=\"red\",\n        ax=ax,\n        stat=stat,\n        bins=n_bins,\n        alpha=0.5,\n    )\n    sns.histplot(\n        bf1,\n        label=\"balanced F1\",\n        color=\"orange\",\n        ax=ax,\n        stat=stat,\n        bins=n_bins,\n        alpha=0.5,\n    )\n    ax.set(title=f\"Distribution of calculated score (N={N})\", xlabel=\"Score\")\n    ax.legend()\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-18T20:48:13.320857Z","iopub.execute_input":"2022-05-18T20:48:13.321151Z","iopub.status.idle":"2022-05-18T20:48:13.402722Z","shell.execute_reply.started":"2022-05-18T20:48:13.321122Z","shell.execute_reply":"2022-05-18T20:48:13.401998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target_gen = TargetGenerator()\npred_gen = PredictorGenerator(0.7)\nsimulation(target_gen, pred_gen)","metadata":{"execution":{"iopub.status.busy":"2022-05-18T20:48:13.687366Z","iopub.execute_input":"2022-05-18T20:48:13.687649Z","iopub.status.idle":"2022-05-18T20:48:15.99939Z","shell.execute_reply.started":"2022-05-18T20:48:13.687622Z","shell.execute_reply":"2022-05-18T20:48:15.998438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target_gen = TargetGenerator()\npred_gen = PredictorGenerator(0.8)\nsimulation(target_gen, pred_gen)","metadata":{"execution":{"iopub.status.busy":"2022-05-18T20:47:35.125618Z","iopub.execute_input":"2022-05-18T20:47:35.125918Z","iopub.status.idle":"2022-05-18T20:47:37.622823Z","shell.execute_reply.started":"2022-05-18T20:47:35.125886Z","shell.execute_reply":"2022-05-18T20:47:37.621931Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target_gen = TargetGenerator(p_pos=0.01)\npred_gen = PredictorGenerator(0.7)\nsimulation(target_gen, pred_gen)","metadata":{"execution":{"iopub.status.busy":"2022-05-18T20:47:37.624727Z","iopub.execute_input":"2022-05-18T20:47:37.625165Z","iopub.status.idle":"2022-05-18T20:47:39.994001Z","shell.execute_reply.started":"2022-05-18T20:47:37.625119Z","shell.execute_reply":"2022-05-18T20:47:39.993152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target_gen = TargetGenerator(p_pos=0.001)\npred_gen = PredictorGenerator(0.7)\nsimulation(target_gen, pred_gen)","metadata":{"execution":{"iopub.status.busy":"2022-05-18T20:47:43.372698Z","iopub.execute_input":"2022-05-18T20:47:43.373537Z","iopub.status.idle":"2022-05-18T20:47:45.7287Z","shell.execute_reply.started":"2022-05-18T20:47:43.373496Z","shell.execute_reply":"2022-05-18T20:47:45.728011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Discussion: features of balanced accuracy\n\n### Tolerance for False Positives\n\nIf there is a class imbalance where there are fewer positive examples than negative examples, balanced accuracy will be more tolerant of false positives. For example, if there are 100 samples, 10 positive examples and 90 negative examples, 10 false positives and 1 false negative will have the same impact on the score.\n\n### Invariant with respect to the ratio of positive examples\n\nIf the model's probability of correct prediction is constant, the balanced accuracy is constant regardless of the proportion of positive examples. This makes it difficult to estimate the proportion of positive and negative cases in the private test data by LB probing.\n\nOn the other hand, the smaller the proportion of positive examples, the larger the variance of the balanced accuracy score. Since balanced accuracy treats the percentage of positive and negative examples equally, the smaller the number of positive examples, the greater the impact on the score when one positive example is answered correctly or incorrectly.","metadata":{}},{"cell_type":"markdown","source":"# Conclusion\n\n* balanced accuracy is more tolerant of false positives than mfif. This explains our observation well. ","metadata":{}},{"cell_type":"markdown","source":"# Appendix: Simulations of naive prediction","metadata":{}},{"cell_type":"markdown","source":"## A1: All true prediction","metadata":{}},{"cell_type":"code","source":"class PositivePredictorGenerator:\n    def generate(self, n_samples, target, N):\n        return np.ones((N, n_samples))\n\n\ntarget_gen = TargetGenerator()\npred_gen = PositivePredictorGenerator()\nsimulation(target_gen, pred_gen, N=1_000)","metadata":{"execution":{"iopub.status.busy":"2022-05-18T20:49:20.954138Z","iopub.execute_input":"2022-05-18T20:49:20.95476Z","iopub.status.idle":"2022-05-18T20:49:23.006004Z","shell.execute_reply.started":"2022-05-18T20:49:20.954712Z","shell.execute_reply":"2022-05-18T20:49:23.00504Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## A2. all zero prediction","metadata":{}},{"cell_type":"code","source":"class FalsePredictorGenerator:\n    def generate(self, n_samples, target, N):\n        return np.zeros((N, n_samples))\n\n\ntarget_gen = TargetGenerator()\npred_gen = FalsePredictorGenerator()\nsimulation(target_gen, pred_gen, N=1_000)","metadata":{"execution":{"iopub.status.busy":"2022-05-18T20:49:23.007424Z","iopub.execute_input":"2022-05-18T20:49:23.007639Z","iopub.status.idle":"2022-05-18T20:49:24.931531Z","shell.execute_reply.started":"2022-05-18T20:49:23.007611Z","shell.execute_reply":"2022-05-18T20:49:24.930639Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## A3. Random prediction","metadata":{}},{"cell_type":"code","source":"class RandomPredictorGenerator:\n    def generate(self, n_samples, target, N):\n        return np.random.choice(2, (N, n_samples))\n\n\ntarget_gen = TargetGenerator()\npred_gen = RandomPredictorGenerator()\nsimulation(target_gen, pred_gen)","metadata":{"execution":{"iopub.status.busy":"2022-05-18T20:49:24.933116Z","iopub.execute_input":"2022-05-18T20:49:24.934036Z","iopub.status.idle":"2022-05-18T20:49:27.154585Z","shell.execute_reply.started":"2022-05-18T20:49:24.933977Z","shell.execute_reply":"2022-05-18T20:49:27.15361Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Conclusion\n\nWhen the evaluation metric is balanced accuracy, these simulation results fit well with observations from naive submissions.","metadata":{}}]}