{ "id": 222672, "title": "An error when calculating AUC score", "url": "/competitions/ranzcr-clip-catheter-line-classification/discussion/222672", "author_name": "tsujino", "post_date": "2021-02-28T15:43:43.424000", "votes": 4, "comment_count": 6, "views": 0, "content": "

When I try to calculate score by using sklearn.metrics.roc_auc_score , I encountered an error below.

\n
ValueError: Only one class present in y_true. ROC AUC score is not defined in that case.\n
\n

I think this is beacuse data is imbalanced.
\nI want to calculate AUC score per epochs, but I can't bacause of this error.

\n

How to calculating AUC score without error?
\nDo I have no other choice than to divide the data so that it is not imbalanced (e.g. using StratifiedKFold, etc.)?

\n

Thanks!

", "messages": [ { "id": 1221063, "postDate": "2021-02-28T16:36:01.443Z", "content": "

When you call sklearn.metrics.roc_auc_score , make sure you put the true labels first and your predictions second. As in roc_auc_score(true,pred). Next make sure that you stratify the targets over your folds. The target ETT - Abnormal only has 79 positive targets out of 30083 images. If any of your folds have zero of these positive targets then you cannot compute AUC for that fold (because AUC is undefined when you don't have any positive true targets).

", "rawMarkdown": "When you call `sklearn.metrics.roc_auc_score` , make sure you put the true labels first and your predictions second. As in `roc_auc_score(true,pred)`. Next make sure that you stratify the targets over your folds. The target `ETT - Abnormal` only has 79 positive targets out of 30083 images. If any of your folds have **zero** of these positive targets then you cannot compute AUC for that fold (because AUC is undefined when you don't have any positive true targets).", "votes": 4, "replies": [ { "id": 1221476, "postDate": "2021-03-01T02:55:21.407Z", "content": "

Very helpful advice! Thanks!

", "rawMarkdown": "Very helpful advice! Thanks!" } ] }, { "id": 1221009, "postDate": "2021-02-28T15:43:43.423Z", "content": "

When I try to calculate score by using sklearn.metrics.roc_auc_score , I encountered an error below.

\n
ValueError: Only one class present in y_true. ROC AUC score is not defined in that case.\n
\n

I think this is beacuse data is imbalanced.
\nI want to calculate AUC score per epochs, but I can't bacause of this error.

\n

How to calculating AUC score without error?
\nDo I have no other choice than to divide the data so that it is not imbalanced (e.g. using StratifiedKFold, etc.)?

\n

Thanks!

", "rawMarkdown": "When I try to calculate score by using sklearn.metrics.roc_auc_score , I encountered an error below.\n```\nValueError: Only one class present in y_true. ROC AUC score is not defined in that case.\n```\nI think this is beacuse data is imbalanced.\nI want to calculate AUC score per epochs, but I can't bacause of this error.\n\nHow to calculating AUC score without error?\nDo I have no other choice than to divide the data so that it is not imbalanced (e.g. using StratifiedKFold, etc.)?\n\nThanks!", "votes": 4 }, { "id": 1221324, "postDate": "2021-02-28T22:19:51.767Z", "content": "

I recommend accumulating predictions and labels and calculating after all batches.

\n
preds = []\nlabels = []\n\nfor batch in loader:\n    ...\n\n    preds.extend(probabilities.cpu().numpy())\n    labels.extend(batch[\"labels\"].cpu().numpy())\n\nroc_auc_score(labels, preds, average='macro')\n
", "rawMarkdown": "I recommend accumulating predictions and labels and calculating after all batches.\n```\npreds = []\nlabels = []\n\nfor batch in loader:\n ...\n\n preds.extend(probabilities.cpu().numpy())\n labels.extend(batch[\"labels\"].cpu().numpy())\n\nroc_auc_score(labels, preds, average='macro')\n```", "votes": 1, "replies": [ { "id": 1221477, "postDate": "2021-03-01T02:55:26.577Z", "content": "

Thanks very much! I'll try it!

", "rawMarkdown": "Thanks very much! I'll try it!" } ] }, { "id": 1222636, "postDate": "2021-03-02T01:45:04.577Z", "content": "

you might using only part of the datas?when you compute auc score,
\nchange it to \"

\n

try:
\n score = roc_auc_score(y_true[:,i], y_pred[:,i])
\n scores.append(score)
\n except ValueError:
\n scores.append(0)

", "rawMarkdown": "you might using only part of the datas?when you compute auc score,\nchange it to \"\n\n\n try:\n score = roc_auc_score(y_true[:,i], y_pred[:,i])\n scores.append(score)\n except ValueError:\n scores.append(0)" }, { "id": 1222628, "postDate": "2021-03-02T01:42:20.517Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1221063, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-02-28T16:36:01.443000", "content": "

When you call sklearn.metrics.roc_auc_score , make sure you put the true labels first and your predictions second. As in roc_auc_score(true,pred). Next make sure that you stratify the targets over your folds. The target ETT - Abnormal only has 79 positive targets out of 30083 images. If any of your folds have zero of these positive targets then you cannot compute AUC for that fold (because AUC is undefined when you don't have any positive true targets).

", "votes": 4, "replies": [ { "id": 1221476, "author_name": "tsujino", "author_url": "", "post_date": "2021-03-01T02:55:21.407000", "content": "

Very helpful advice! Thanks!

", "votes": 0, "replies": [] } ] }, { "id": 1221324, "author_name": "rp_prodan", "author_url": "", "post_date": "2021-02-28T22:19:51.767000", "content": "

I recommend accumulating predictions and labels and calculating after all batches.

\n
preds = []\nlabels = []\n\nfor batch in loader:\n    ...\n\n    preds.extend(probabilities.cpu().numpy())\n    labels.extend(batch[\"labels\"].cpu().numpy())\n\nroc_auc_score(labels, preds, average='macro')\n
", "votes": 1, "replies": [ { "id": 1221477, "author_name": "tsujino", "author_url": "", "post_date": "2021-03-01T02:55:26.577000", "content": "

Thanks very much! I'll try it!

", "votes": 0, "replies": [] } ] }, { "id": 1222636, "author_name": "FatLiuyun", "author_url": "", "post_date": "2021-03-02T01:45:04.577000", "content": "

you might using only part of the datas?when you compute auc score,
\nchange it to \"

\n

try:
\n score = roc_auc_score(y_true[:,i], y_pred[:,i])
\n scores.append(score)
\n except ValueError:
\n scores.append(0)

", "votes": 0, "replies": [] }, { "id": 1222628, "author_name": "", "author_url": "", "post_date": "2021-03-02T01:42:20.517000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1221063": "When you call `sklearn.metrics.roc_auc_score` , make sure you put the true labels first and your predictions second. As in `roc_auc_score(true,pred)`. Next make sure that you stratify the targets over your folds. The target `ETT - Abnormal` only has 79 positive targets out of 30083 images. If any of your folds have **zero** of these positive targets then you cannot compute AUC for that fold (because AUC is undefined when you don't have any positive true targets).", "1221009": "When I try to calculate score by using sklearn.metrics.roc_auc_score , I encountered an error below.\n```\nValueError: Only one class present in y_true. ROC AUC score is not defined in that case.\n```\nI think this is beacuse data is imbalanced.\nI want to calculate AUC score per epochs, but I can't bacause of this error.\n\nHow to calculating AUC score without error?\nDo I have no other choice than to divide the data so that it is not imbalanced (e.g. using StratifiedKFold, etc.)?\n\nThanks!", "1221324": "I recommend accumulating predictions and labels and calculating after all batches.\n```\npreds = []\nlabels = []\n\nfor batch in loader:\n ...\n\n preds.extend(probabilities.cpu().numpy())\n labels.extend(batch[\"labels\"].cpu().numpy())\n\nroc_auc_score(labels, preds, average='macro')\n```", "1222636": "you might using only part of the datas?when you compute auc score,\nchange it to \"\n\n\n try:\n score = roc_auc_score(y_true[:,i], y_pred[:,i])\n scores.append(score)\n except ValueError:\n scores.append(0)", "1222628": "" } }