{"cells":[{"metadata":{},"cell_type":"markdown","source":"## An implementation of the competition metric\n\nThis kernel is a clean-up of **Max Jeblick**'s evaluation metric implementation into an easy to use function: https://www.kaggle.com/maxjeblick/alaska2-efficientnet-on-tpus-competition-metric\n\nSee the [evaluation page](https://www.kaggle.com/c/alaska2-image-steganalysis/overview/evaluation) for more info about the metric!\n\n**EDIT:** Changed to reflect the update in evaluation metric of the competition!"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn import metrics\nimport numpy as np\n\ndef alaska_weighted_auc(y_true, y_valid):\n    tpr_thresholds = [0.0, 0.4, 1.0]\n    weights =        [       2,   1]\n    \n    fpr, tpr, thresholds = metrics.roc_curve(y_true, y_valid, pos_label=1)\n    \n    # size of subsets\n    areas = np.array(tpr_thresholds[1:]) - np.array(tpr_thresholds[:-1])\n    \n    # The total area is normalized by the sum of weights such that the final weighted AUC is between 0 and 1.\n    normalization = np.dot(areas, weights)\n    \n    competition_metric = 0\n    for idx, weight in enumerate(weights):\n        y_min = tpr_thresholds[idx]\n        y_max = tpr_thresholds[idx + 1]\n        mask = (y_min < tpr) & (tpr < y_max)\n\n        x_padding = np.linspace(fpr[mask][-1], 1, 100)\n\n        x = np.concatenate([fpr[mask], x_padding])\n        y = np.concatenate([tpr[mask], [y_max] * len(x_padding)])\n        y = y - y_min # normalize such that curve starts at y=0\n        score = metrics.auc(x, y)\n        submetric = score * weight\n        best_subscore = (y_max - y_min) * weight\n        competition_metric += submetric\n        \n    return competition_metric / normalization","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## A toy example\n\nAnd a comparison to other evaluation metrics.  \nWe can see here that Weighted AUC gives  more favourable scores than both accuracy and standard AUC (before the metric update, this was even more pronounced! 0.64 AUC become 0.96 weighted AUC)"},{"metadata":{"trusted":true},"cell_type":"code","source":"np.random.seed(42)\n\nsignal = np.random.random(1000)\nlabels = (signal > 0.5).astype(int)\npreds = np.random.random(1000) + (signal - 0.5) * 0.3","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"auc = metrics.roc_auc_score(labels, preds)\nweighted_auc = alaska_weighted_auc(labels, preds)\naccuracy = (labels == (preds > 0.5)).mean()\n\nprint(f'   Accuracy = {round(accuracy, 5)}')\nprint(f'        AUC = {round(auc, 5)}')\nprint(f'WeightedAUC = {round(weighted_auc, 5)}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nfpr, tpr, thresholds = metrics.roc_curve(labels, preds, pos_label=1)\n\nplt.figure(figsize=(6, 6))\nplt.plot(fpr, tpr)\nplt.xlabel('FPR')\nplt.ylabel('TPR')\nplt.title('Example ROC Curve (AUC=0.64, WeightedAUC=0.72)')\nplt.xlim(0, 1)\nplt.ylim(0, 1)\nplt.plot([0, 1], [0, 1], alpha=0.1)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Random guessing gives ~0.585 Weighted AUC\nSo this is the score to beat with your models"},{"metadata":{"trusted":true},"cell_type":"code","source":"alaska_weighted_auc(np.random.random(1000000) > 0.5, np.random.random(1000000))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":4}