{
  "id": 191369,
  "title": "ROC AUC Evaluation Metric: worked example - what is the value?",
  "url": "/competitions/riiid-test-answer-prediction/discussion/191369",
  "author_name": "",
  "post_date": "2020-10-16T06:06:19.522169200Z",
  "votes": 2,
  "comment_count": 2,
  "views": 0,
  "content": "<p>There is a great example of the ROC AUC computation with an Excel spreadsheet at <a href=\"https://www.real-statistics.com/descriptive-statistics/roc-curve-classification-table/roc-curve/\" target=\"_blank\">https://www.real-statistics.com/descriptive-statistics/roc-curve-classification-table/roc-curve/</a></p>\n<p>Let's suppose we have 4 observations and their predictions:<br>\n1) Observed 1 - predicted 0.8<br>\n2) Observed 1 - predicted 0.4<br>\n3) Observed 0 - predicted 0.1<br>\n4) Observed 0 - predicted 0.4</p>\n<p>What is the ROC AUC? Step-by-step computation please ….</p>",
  "messages": [
    {
      "id": "1051068",
      "postDate": "10/16/2020 06:06:19",
      "content": "<p>There is a great example of the ROC AUC computation with an Excel spreadsheet at <a href=\"https://www.real-statistics.com/descriptive-statistics/roc-curve-classification-table/roc-curve/\" target=\"_blank\">https://www.real-statistics.com/descriptive-statistics/roc-curve-classification-table/roc-curve/</a></p>\n<p>Let's suppose we have 4 observations and their predictions:<br>\n1) Observed 1 - predicted 0.8<br>\n2) Observed 1 - predicted 0.4<br>\n3) Observed 0 - predicted 0.1<br>\n4) Observed 0 - predicted 0.4</p>\n<p>What is the ROC AUC? Step-by-step computation please ….</p>",
      "rawMarkdown": "There is a great example of the ROC AUC computation with an Excel spreadsheet at https://www.real-statistics.com/descriptive-statistics/roc-curve-classification-table/roc-curve/\n\nLet's suppose we have 4 observations and their predictions:\n1) Observed 1 - predicted 0.8\n2) Observed 1 - predicted 0.4\n3) Observed 0 - predicted 0.1\n4) Observed 0 - predicted 0.4\n\nWhat is the ROC AUC? Step-by-step computation please ....",
      "votes": null
    },
    {
      "id": "1052495",
      "postDate": "10/17/2020 19:47:17",
      "content": "<p>ROC AUC curve gives the trade off between the true positives(TP) and the false positives(FP).<br>\nThis is clearly based on the true positive rate(TPR) and the false positive rate(FPR) you achieve based on the threshold value.<br>\nSuppose I take threshold value = 0.8 , following are true positives and false positives -</p>\n<p>1) Observed 1 - predicted 0.8 -&gt; 1<br>\n2) Observed 1 - predicted 0.4 -&gt; 0 <br>\n3) Observed 0 - predicted 0.1 -&gt; 0 <br>\n4) Observed 0 - predicted 0.4 -&gt; 0 </p>\n<p>hence TPR = TP/(TP+FN) = 1 and FPR = FP/(FP+TN) = 0<br>\nhere FN is False Negative and TN is True Negative</p>\n<p>then,</p>\n<p>Suppose I take threshold value = 0.4 , following are true positives and false positives -</p>\n<p>1) Observed 1 - predicted 0.8 -&gt; 1<br>\n2) Observed 1 - predicted 0.4 -&gt; 1 <br>\n3) Observed 0 - predicted 0.1 -&gt; 0 <br>\n4) Observed 0 - predicted 0.4 -&gt; 1</p>\n<p>hence TPR = TP/(TP+FN) = 1 and FPR = FP/(FP+TN) = 0.5</p>\n<p>and then,</p>\n<p>Suppose I take threshold value = 0.1 , following are true positives and false positives -</p>\n<p>1) Observed 1 - predicted 0.8 -&gt; 1<br>\n2) Observed 1 - predicted 0.4 -&gt; 1 <br>\n3) Observed 0 - predicted 0.1 -&gt; 0 <br>\n4) Observed 0 - predicted 0.4 -&gt; 1</p>\n<p>hence TPR = TP/(TP+FN) = 1 and FPR = FP/(FP+TN) = 0.5</p>\n<p>Similarly for your multiple observed and predicted values, you continue and plot the ROC for True Positive Pate (TPR) against the False Positive Rate (FPR) at various thresholds to obtain the Area Under the Curve(AUC).</p>",
      "rawMarkdown": "ROC AUC curve gives the trade off between the true positives(TP) and the false positives(FP).\nThis is clearly based on the true positive rate(TPR) and the false positive rate(FPR) you achieve based on the threshold value.\nSuppose I take threshold value = 0.8 , following are true positives and false positives -\n\n1) Observed 1 - predicted 0.8 -> 1\n2) Observed 1 - predicted 0.4 -> 0 \n3) Observed 0 - predicted 0.1 -> 0 \n4) Observed 0 - predicted 0.4 -> 0 \n\nhence TPR = TP/(TP+FN) = 1 and FPR = FP/(FP+TN) = 0\nhere FN is False Negative and TN is True Negative\n\nthen,\n\nSuppose I take threshold value = 0.4 , following are true positives and false positives -\n\n1) Observed 1 - predicted 0.8 -> 1\n2) Observed 1 - predicted 0.4 -> 1 \n3) Observed 0 - predicted 0.1 -> 0 \n4) Observed 0 - predicted 0.4 -> 1\n\nhence TPR = TP/(TP+FN) = 1 and FPR = FP/(FP+TN) = 0.5\n\nand then,\n\nSuppose I take threshold value = 0.1 , following are true positives and false positives -\n\n1) Observed 1 - predicted 0.8 -> 1\n2) Observed 1 - predicted 0.4 -> 1 \n3) Observed 0 - predicted 0.1 -> 0 \n4) Observed 0 - predicted 0.4 -> 1\n\nhence TPR = TP/(TP+FN) = 1 and FPR = FP/(FP+TN) = 0.5\n\nSimilarly for your multiple observed and predicted values, you continue and plot the ROC for True Positive Pate (TPR) against the False Positive Rate (FPR) at various thresholds to obtain the Area Under the Curve(AUC).",
      "votes": null
    },
    {
      "id": "1052565",
      "postDate": "10/17/2020 23:34:14",
      "content": "<p>Thanks, Pooja. </p>\n<p>Between Thresholds 0.79 and 0.41, we have TP, FN, TN, TN:  TPR = 0.5 and FPR = 0</p>\n<p>This gives us two quadrilaterals under the ROC with upper points: (0,0.5),(0.5,0.5) and (0.5, 0.5), (1, 1)</p>\n<p>So ROC AUC = (0.5*0.5) + (0.5*(0.5+1)*0.5) = 0.625</p>\n<p>Is this correct?</p>",
      "rawMarkdown": "Thanks, Pooja. \n\nBetween Thresholds 0.79 and 0.41, we have TP, FN, TN, TN:  TPR = 0.5 and FPR = 0\n\nThis gives us two quadrilaterals under the ROC with upper points: (0,0.5),(0.5,0.5) and (0.5, 0.5), (1, 1)\n\nSo ROC AUC = (0.5\\*0.5) + (0.5\\*(0.5+1)*0.5) = 0.625\n\nIs this correct?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1052495,
      "author_name": "jainpooja",
      "author_url": "",
      "post_date": "10/17/2020 19:47:17",
      "content": "<p>ROC AUC curve gives the trade off between the true positives(TP) and the false positives(FP).<br>\nThis is clearly based on the true positive rate(TPR) and the false positive rate(FPR) you achieve based on the threshold value.<br>\nSuppose I take threshold value = 0.8 , following are true positives and false positives -</p>\n<p>1) Observed 1 - predicted 0.8 -&gt; 1<br>\n2) Observed 1 - predicted 0.4 -&gt; 0 <br>\n3) Observed 0 - predicted 0.1 -&gt; 0 <br>\n4) Observed 0 - predicted 0.4 -&gt; 0 </p>\n<p>hence TPR = TP/(TP+FN) = 1 and FPR = FP/(FP+TN) = 0<br>\nhere FN is False Negative and TN is True Negative</p>\n<p>then,</p>\n<p>Suppose I take threshold value = 0.4 , following are true positives and false positives -</p>\n<p>1) Observed 1 - predicted 0.8 -&gt; 1<br>\n2) Observed 1 - predicted 0.4 -&gt; 1 <br>\n3) Observed 0 - predicted 0.1 -&gt; 0 <br>\n4) Observed 0 - predicted 0.4 -&gt; 1</p>\n<p>hence TPR = TP/(TP+FN) = 1 and FPR = FP/(FP+TN) = 0.5</p>\n<p>and then,</p>\n<p>Suppose I take threshold value = 0.1 , following are true positives and false positives -</p>\n<p>1) Observed 1 - predicted 0.8 -&gt; 1<br>\n2) Observed 1 - predicted 0.4 -&gt; 1 <br>\n3) Observed 0 - predicted 0.1 -&gt; 0 <br>\n4) Observed 0 - predicted 0.4 -&gt; 1</p>\n<p>hence TPR = TP/(TP+FN) = 1 and FPR = FP/(FP+TN) = 0.5</p>\n<p>Similarly for your multiple observed and predicted values, you continue and plot the ROC for True Positive Pate (TPR) against the False Positive Rate (FPR) at various thresholds to obtain the Area Under the Curve(AUC).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1052565,
      "author_name": "mikel1",
      "author_url": "",
      "post_date": "10/17/2020 23:34:14",
      "content": "<p>Thanks, Pooja. </p>\n<p>Between Thresholds 0.79 and 0.41, we have TP, FN, TN, TN:  TPR = 0.5 and FPR = 0</p>\n<p>This gives us two quadrilaterals under the ROC with upper points: (0,0.5),(0.5,0.5) and (0.5, 0.5), (1, 1)</p>\n<p>So ROC AUC = (0.5*0.5) + (0.5*(0.5+1)*0.5) = 0.625</p>\n<p>Is this correct?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1051068": "There is a great example of the ROC AUC computation with an Excel spreadsheet at https://www.real-statistics.com/descriptive-statistics/roc-curve-classification-table/roc-curve/\n\nLet's suppose we have 4 observations and their predictions:\n1) Observed 1 - predicted 0.8\n2) Observed 1 - predicted 0.4\n3) Observed 0 - predicted 0.1\n4) Observed 0 - predicted 0.4\n\nWhat is the ROC AUC? Step-by-step computation please ....",
    "1052495": "ROC AUC curve gives the trade off between the true positives(TP) and the false positives(FP).\nThis is clearly based on the true positive rate(TPR) and the false positive rate(FPR) you achieve based on the threshold value.\nSuppose I take threshold value = 0.8 , following are true positives and false positives -\n\n1) Observed 1 - predicted 0.8 -> 1\n2) Observed 1 - predicted 0.4 -> 0 \n3) Observed 0 - predicted 0.1 -> 0 \n4) Observed 0 - predicted 0.4 -> 0 \n\nhence TPR = TP/(TP+FN) = 1 and FPR = FP/(FP+TN) = 0\nhere FN is False Negative and TN is True Negative\n\nthen,\n\nSuppose I take threshold value = 0.4 , following are true positives and false positives -\n\n1) Observed 1 - predicted 0.8 -> 1\n2) Observed 1 - predicted 0.4 -> 1 \n3) Observed 0 - predicted 0.1 -> 0 \n4) Observed 0 - predicted 0.4 -> 1\n\nhence TPR = TP/(TP+FN) = 1 and FPR = FP/(FP+TN) = 0.5\n\nand then,\n\nSuppose I take threshold value = 0.1 , following are true positives and false positives -\n\n1) Observed 1 - predicted 0.8 -> 1\n2) Observed 1 - predicted 0.4 -> 1 \n3) Observed 0 - predicted 0.1 -> 0 \n4) Observed 0 - predicted 0.4 -> 1\n\nhence TPR = TP/(TP+FN) = 1 and FPR = FP/(FP+TN) = 0.5\n\nSimilarly for your multiple observed and predicted values, you continue and plot the ROC for True Positive Pate (TPR) against the False Positive Rate (FPR) at various thresholds to obtain the Area Under the Curve(AUC).",
    "1052565": "Thanks, Pooja. \n\nBetween Thresholds 0.79 and 0.41, we have TP, FN, TN, TN:  TPR = 0.5 and FPR = 0\n\nThis gives us two quadrilaterals under the ROC with upper points: (0,0.5),(0.5,0.5) and (0.5, 0.5), (1, 1)\n\nSo ROC AUC = (0.5\\*0.5) + (0.5\\*(0.5+1)*0.5) = 0.625\n\nIs this correct?"
  },
  "source": "meta"
}