{
  "id": 168458,
  "title": "How to interpret the AUC during training?",
  "url": "/competitions/siim-isic-melanoma-classification/discussion/168458",
  "author_name": "",
  "post_date": "2020-07-20T17:47:37.403012800Z",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I'm using the AUC metric from Keras. I noticed a huge difference between the metrics during training and the final AUC score.</p>\n\n<p>The auc also seems to stabilize at 0.52 during training on both val and train. </p>\n\n<p>Does this show a problem in the model? It gets 0.7 auc on the LB</p>",
  "messages": [
    {
      "id": "937073",
      "postDate": "07/20/2020 17:47:37",
      "content": "<p>I'm using the AUC metric from Keras. I noticed a huge difference between the metrics during training and the final AUC score.</p>\n\n<p>The auc also seems to stabilize at 0.52 during training on both val and train. </p>\n\n<p>Does this show a problem in the model? It gets 0.7 auc on the LB</p>",
      "rawMarkdown": "I'm using the AUC metric from Keras. I noticed a huge difference between the metrics during training and the final AUC score.\n\nThe auc also seems to stabilize at 0.52 during training on both val and train. \n\nDoes this show a problem in the model? It gets 0.7 auc on the LB",
      "votes": null
    },
    {
      "id": "937094",
      "postDate": "07/20/2020 18:06:33",
      "content": "<p>The value of AUC of 0.52 indicates that there is some problem with your model or training pipeline. This value of AUC is very close to what you get if you just assign your labels randomly (AUC = 0.5). You should expect your training, validation and LB AUC's to be around 0.9+/-0.1.</p>",
      "rawMarkdown": "The value of AUC of 0.52 indicates that there is some problem with your model or training pipeline. This value of AUC is very close to what you get if you just assign your labels randomly (AUC = 0.5). You should expect your training, validation and LB AUC's to be around 0.9+/-0.1.",
      "votes": null
    },
    {
      "id": "937115",
      "postDate": "07/20/2020 18:22:45",
      "content": "<p>Need to re-evaluate the model then, looks fishy...</p>",
      "rawMarkdown": "Need to re-evaluate the model then, looks fishy...",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 937094,
      "author_name": "graf10a",
      "author_url": "",
      "post_date": "07/20/2020 18:06:33",
      "content": "<p>The value of AUC of 0.52 indicates that there is some problem with your model or training pipeline. This value of AUC is very close to what you get if you just assign your labels randomly (AUC = 0.5). You should expect your training, validation and LB AUC's to be around 0.9+/-0.1.</p>",
      "votes": null,
      "replies": [
        {
          "id": 937115,
          "author_name": "amneves",
          "author_url": "",
          "post_date": "07/20/2020 18:22:45",
          "content": "<p>Need to re-evaluate the model then, looks fishy...</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "937073": "I'm using the AUC metric from Keras. I noticed a huge difference between the metrics during training and the final AUC score.\n\nThe auc also seems to stabilize at 0.52 during training on both val and train. \n\nDoes this show a problem in the model? It gets 0.7 auc on the LB",
    "937094": "The value of AUC of 0.52 indicates that there is some problem with your model or training pipeline. This value of AUC is very close to what you get if you just assign your labels randomly (AUC = 0.5). You should expect your training, validation and LB AUC's to be around 0.9+/-0.1.",
    "937115": "Need to re-evaluate the model then, looks fishy..."
  },
  "source": "meta"
}