{
  "id": 161788,
  "title": "AUC monitoring in TF2",
  "url": "/competitions/siim-isic-melanoma-classification/discussion/161788",
  "author_name": "Bayartsogt Yadamsuren",
  "post_date": "2020-06-26T07:05:32.425000",
  "votes": 0,
  "comment_count": 8,
  "views": 0,
  "content": "<p>Seems like <code>tf.keras.metrics.AUC</code> is not working accurate enough.\nhow do I monitor the AUC during <code>.fit</code> function?</p>",
  "messages": [
    {
      "id": 905770,
      "postDate": "2020-06-28T19:18:44.527Z",
      "content": "<p>It happened to me ... and I was shuffling the augmentations without keeping proper track of the labels... just in case...</p>",
      "rawMarkdown": "It happened to me ... and I was shuffling the augmentations without keeping proper track of the labels... just in case...",
      "votes": 1
    },
    {
      "id": 902666,
      "postDate": "2020-06-26T09:41:45.213Z",
      "content": "<p>I am using tf.keras.metrics.AUC with TF 2.2 and I do not have this issue. I use ModelCheckpoint to store the weights of the model with best val_auc and load the weights after training. The roc_auc_score of sklearn.metrics give approximately the same results for me.</p>",
      "rawMarkdown": "I am using tf.keras.metrics.AUC with TF 2.2 and I do not have this issue. I use ModelCheckpoint to store the weights of the model with best val_auc and load the weights after training. The roc_auc_score of sklearn.metrics give approximately the same results for me.",
      "votes": 2,
      "replies": [
        {
          "id": 905763,
          "postDate": "2020-06-28T19:12:51.503Z",
          "rawMarkdown": "",
          "isDeleted": true
        }
      ]
    },
    {
      "id": 902496,
      "postDate": "2020-06-26T07:28:43.637Z",
      "content": "<p>Just curious: what's wrong with the <code>tf.keras.metrics.AUC</code> function? What makes you think that it is not accurate?</p>",
      "rawMarkdown": "Just curious: what's wrong with the `tf.keras.metrics.AUC` function? What makes you think that it is not accurate?",
      "replies": [
        {
          "id": 902519,
          "postDate": "2020-06-26T07:43:22.500Z",
          "content": "<p>It is working different from <code>sklearn.metrics.roc_auc_score</code> for me.\nDuring training validation auc was <code>val_auc = 0.91</code>, but I tried to used <code>sklearn</code> one, score was so low.</p>\n\n<p>I tried like this:\n```\nmodel.compile(\n            optimizer = \"adam\",\n            loss      = loss,\n            metrics   = [tf.keras.metrics.AUC()])</p>\n\n<p>model.fit(...) # val_auc=0.91</p>\n\n<p>y_pred = model.predict(ds_valid)\nprint('val_auc=', sklearn.metrics.roc_auc_score(y_valid, y_pred)) # val_auc = 0.51\n```</p>\n\n<p>So started looking at discussions and found this (which is not working with TPU setup)\n<a href=\"https://www.kaggle.com/c/santander-customer-transaction-prediction/discussion/80807\">https://www.kaggle.com/c/santander-customer-transaction-prediction/discussion/80807</a></p>\n\n<p>[edit] maybe it is because we set number of thresholds manually. But not sure really.</p>",
          "rawMarkdown": "It is working different from `sklearn.metrics.roc_auc_score` for me.\nDuring training validation auc was `val_auc = 0.91`, but I tried to used `sklearn` one, score was so low.\n\nI tried like this:\n```\nmodel.compile(\n            optimizer = \"adam\",\n            loss      = loss,\n            metrics   = [tf.keras.metrics.AUC()])\n\nmodel.fit(...) # val_auc=0.91\n\ny_pred = model.predict(ds_valid)\nprint('val_auc=', sklearn.metrics.roc_auc_score(y_valid, y_pred)) # val_auc = 0.51\n```\n\nSo started looking at discussions and found this (which is not working with TPU setup)\nhttps://www.kaggle.com/c/santander-customer-transaction-prediction/discussion/80807\n\n[edit] maybe it is because we set number of thresholds manually. But not sure really."
        },
        {
          "id": 903055,
          "postDate": "2020-06-26T14:47:40.470Z",
          "content": "<p>Your validation AUC is very close to the one you get by a random guess. If I were you, I would check your validation pipeline -- there might be some issues with it. I am using the same AUC metric in my training algorithm and I have never observed any discrepancies like this.</p>",
          "rawMarkdown": "Your validation AUC is very close to the one you get by a random guess. If I were you, I would check your validation pipeline -- there might be some issues with it. I am using the same AUC metric in my training algorithm and I have never observed any discrepancies like this.",
          "votes": 2
        },
        {
          "id": 905359,
          "postDate": "2020-06-28T13:44:03.440Z",
          "content": "<p>Thank you for your advice! Working on it!</p>",
          "rawMarkdown": "Thank you for your advice! Working on it!"
        },
        {
          "id": 905532,
          "postDate": "2020-06-28T16:07:59.777Z",
          "content": "<p>In this data set I have also hit the 0.50 auc when my learning rate is too large.</p>",
          "rawMarkdown": "In this data set I have also hit the 0.50 auc when my learning rate is too large.",
          "votes": 1
        }
      ]
    },
    {
      "id": 902471,
      "postDate": "2020-06-26T07:05:32.427Z",
      "content": "<p>Seems like <code>tf.keras.metrics.AUC</code> is not working accurate enough.\nhow do I monitor the AUC during <code>.fit</code> function?</p>",
      "rawMarkdown": "Seems like `tf.keras.metrics.AUC` is not working accurate enough.\nhow do I monitor the AUC during `.fit` function?"
    }
  ],
  "comments": [
    {
      "id": 905770,
      "author_name": "Marcelo Kittlein",
      "author_url": "",
      "post_date": "2020-06-28T19:18:44.527000",
      "content": "<p>It happened to me ... and I was shuffling the augmentations without keeping proper track of the labels... just in case...</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 902666,
      "author_name": "Helgi",
      "author_url": "",
      "post_date": "2020-06-26T09:41:45.213000",
      "content": "<p>I am using tf.keras.metrics.AUC with TF 2.2 and I do not have this issue. I use ModelCheckpoint to store the weights of the model with best val_auc and load the weights after training. The roc_auc_score of sklearn.metrics give approximately the same results for me.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 905763,
          "author_name": "",
          "author_url": "",
          "post_date": "2020-06-28T19:12:51.503000",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 902496,
      "author_name": "Alexey Pronin",
      "author_url": "",
      "post_date": "2020-06-26T07:28:43.637000",
      "content": "<p>Just curious: what's wrong with the <code>tf.keras.metrics.AUC</code> function? What makes you think that it is not accurate?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 902519,
          "author_name": "Bayartsogt Yadamsuren",
          "author_url": "",
          "post_date": "2020-06-26T07:43:22.500000",
          "content": "<p>It is working different from <code>sklearn.metrics.roc_auc_score</code> for me.\nDuring training validation auc was <code>val_auc = 0.91</code>, but I tried to used <code>sklearn</code> one, score was so low.</p>\n\n<p>I tried like this:\n```\nmodel.compile(\n            optimizer = \"adam\",\n            loss      = loss,\n            metrics   = [tf.keras.metrics.AUC()])</p>\n\n<p>model.fit(...) # val_auc=0.91</p>\n\n<p>y_pred = model.predict(ds_valid)\nprint('val_auc=', sklearn.metrics.roc_auc_score(y_valid, y_pred)) # val_auc = 0.51\n```</p>\n\n<p>So started looking at discussions and found this (which is not working with TPU setup)\n<a href=\"https://www.kaggle.com/c/santander-customer-transaction-prediction/discussion/80807\">https://www.kaggle.com/c/santander-customer-transaction-prediction/discussion/80807</a></p>\n\n<p>[edit] maybe it is because we set number of thresholds manually. But not sure really.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 903055,
          "author_name": "Alexey Pronin",
          "author_url": "",
          "post_date": "2020-06-26T14:47:40.470000",
          "content": "<p>Your validation AUC is very close to the one you get by a random guess. If I were you, I would check your validation pipeline -- there might be some issues with it. I am using the same AUC metric in my training algorithm and I have never observed any discrepancies like this.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 905359,
          "author_name": "Bayartsogt Yadamsuren",
          "author_url": "",
          "post_date": "2020-06-28T13:44:03.440000",
          "content": "<p>Thank you for your advice! Working on it!</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 905532,
          "author_name": "PC Jimmmy",
          "author_url": "",
          "post_date": "2020-06-28T16:07:59.777000",
          "content": "<p>In this data set I have also hit the 0.50 auc when my learning rate is too large.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "905770": "It happened to me ... and I was shuffling the augmentations without keeping proper track of the labels... just in case...",
    "902666": "I am using tf.keras.metrics.AUC with TF 2.2 and I do not have this issue. I use ModelCheckpoint to store the weights of the model with best val_auc and load the weights after training. The roc_auc_score of sklearn.metrics give approximately the same results for me.",
    "902496": "Just curious: what's wrong with the `tf.keras.metrics.AUC` function? What makes you think that it is not accurate?",
    "902471": "Seems like `tf.keras.metrics.AUC` is not working accurate enough.\nhow do I monitor the AUC during `.fit` function?"
  }
}