{
  "id": 203375,
  "title": "Question about AUC metric",
  "url": "/competitions/ranzcr-clip-catheter-line-classification/discussion/203375",
  "author_name": "",
  "post_date": "2020-12-15T01:20:12.915650200Z",
  "votes": 24,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Is there a reason why the AUC metric is calculated by flattening all the labels and predictions versus averaging the label-wise AUCs? </p>\n<p>The latter seems more intuitive and relevant to me, but I could be formulating the problem in my head incorrectly. </p>",
  "messages": [
    {
      "id": "1112886",
      "postDate": "12/15/2020 01:20:12",
      "content": "<p>Is there a reason why the AUC metric is calculated by flattening all the labels and predictions versus averaging the label-wise AUCs? </p>\n<p>The latter seems more intuitive and relevant to me, but I could be formulating the problem in my head incorrectly. </p>",
      "rawMarkdown": "Is there a reason why the AUC metric is calculated by flattening all the labels and predictions versus averaging the label-wise AUCs? \n\nThe latter seems more intuitive and relevant to me, but I could be formulating the problem in my head incorrectly.",
      "votes": null
    },
    {
      "id": "1112951",
      "postDate": "12/15/2020 03:39:26",
      "content": "<p>That's pretty interesting. I tried computing both the flattened and mean-column AUC using <a href=\"https://www.kaggle.com/titericz\" target=\"_blank\">@titericz</a>'s mean baseline (experiments in <a href=\"https://www.kaggle.com/xhlulu/comparing-column-wise-auc-vs-flattened-auc/\" target=\"_blank\">this notebook</a>). As a result, I got 0.85 when it's flattened and 0.5 when it is mean over individual columns.</p>",
      "rawMarkdown": "That's pretty interesting. I tried computing both the flattened and mean-column AUC using @titericz's mean baseline (experiments in [this notebook](https://www.kaggle.com/xhlulu/comparing-column-wise-auc-vs-flattened-auc/)). As a result, I got 0.85 when it's flattened and 0.5 when it is mean over individual columns.",
      "votes": null
    },
    {
      "id": "1112958",
      "postDate": "12/15/2020 04:02:56",
      "content": "<p>For me AUC averaging column wise is more intuitive also. See comments <a href=\"https://www.kaggle.com/titericz/baseline-mean-average\" target=\"_blank\">here</a></p>",
      "rawMarkdown": "For me AUC averaging column wise is more intuitive also. See comments [here](https://www.kaggle.com/titericz/baseline-mean-average)",
      "votes": null
    },
    {
      "id": "1113925",
      "postDate": "12/15/2020 20:08:24",
      "content": "<p>Hi! Thanks Ian - yes, it looks like the metric was mistakenly set to a micro-averaged AUC. We're switching it to macro-averaged AUC (as you posit, and as planned), and we'll re-run all existing submissions - might take some time for that to happen. Good catch, and thanks for reporting.</p>",
      "rawMarkdown": "Hi! Thanks Ian - yes, it looks like the metric was mistakenly set to a micro-averaged AUC. We're switching it to macro-averaged AUC (as you posit, and as planned), and we'll re-run all existing submissions - might take some time for that to happen. Good catch, and thanks for reporting.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1112951,
      "author_name": "xhlulu",
      "author_url": "",
      "post_date": "12/15/2020 03:39:26",
      "content": "<p>That's pretty interesting. I tried computing both the flattened and mean-column AUC using <a href=\"https://www.kaggle.com/titericz\" target=\"_blank\">@titericz</a>'s mean baseline (experiments in <a href=\"https://www.kaggle.com/xhlulu/comparing-column-wise-auc-vs-flattened-auc/\" target=\"_blank\">this notebook</a>). As a result, I got 0.85 when it's flattened and 0.5 when it is mean over individual columns.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1112958,
      "author_name": "titericz",
      "author_url": "",
      "post_date": "12/15/2020 04:02:56",
      "content": "<p>For me AUC averaging column wise is more intuitive also. See comments <a href=\"https://www.kaggle.com/titericz/baseline-mean-average\" target=\"_blank\">here</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1113925,
      "author_name": "philculliton",
      "author_url": "",
      "post_date": "12/15/2020 20:08:24",
      "content": "<p>Hi! Thanks Ian - yes, it looks like the metric was mistakenly set to a micro-averaged AUC. We're switching it to macro-averaged AUC (as you posit, and as planned), and we'll re-run all existing submissions - might take some time for that to happen. Good catch, and thanks for reporting.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1112886": "Is there a reason why the AUC metric is calculated by flattening all the labels and predictions versus averaging the label-wise AUCs? \n\nThe latter seems more intuitive and relevant to me, but I could be formulating the problem in my head incorrectly.",
    "1112951": "That's pretty interesting. I tried computing both the flattened and mean-column AUC using @titericz's mean baseline (experiments in [this notebook](https://www.kaggle.com/xhlulu/comparing-column-wise-auc-vs-flattened-auc/)). As a result, I got 0.85 when it's flattened and 0.5 when it is mean over individual columns.",
    "1112958": "For me AUC averaging column wise is more intuitive also. See comments [here](https://www.kaggle.com/titericz/baseline-mean-average)",
    "1113925": "Hi! Thanks Ian - yes, it looks like the metric was mistakenly set to a micro-averaged AUC. We're switching it to macro-averaged AUC (as you posit, and as planned), and we'll re-run all existing submissions - might take some time for that to happen. Good catch, and thanks for reporting."
  },
  "source": "meta"
}