{
  "id": 159777,
  "title": "Magical val_accuracy level: what is it?",
  "url": "/competitions/siim-isic-melanoma-classification/discussion/159777",
  "author_name": "",
  "post_date": "2020-06-18T17:08:26.358902100Z",
  "votes": null,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hi,\nI am using a copy of one of standard efficient net notebooks. Everything works fine, but...\nTake a look at the listing. Pay attention at val_accuracy. It stops at 0.9837. It stops there even if I change the network, say, b7 to b7 or b0.\nHow can that be possible?</p>\n\n<blockquote>\n  <p>Epoch 00001: LearningRateScheduler reducing learning rate to 1e-05.\n  Epoch 1/5\n  420/420 [==============================] - 523s 1s/step - accuracy: 0.9450 - loss: 0.2126 - val_accuracy: 0.9836 - val_loss: 0.1243 - lr: 1.0000e-05</p>\n</blockquote>\n\n<p>Epoch 00002: LearningRateScheduler reducing learning rate to 4.95e-05.\nEpoch 2/5\n420/420 [==============================] - 489s 1s/step - accuracy: 0.9821 - loss: 0.0819 - val_accuracy: 0.9837 - val_loss: 0.0687 - lr: 4.9500e-05</p>\n\n<p>Epoch 00003: LearningRateScheduler reducing learning rate to 8.9e-05.\nEpoch 3/5\n420/420 [==============================] - 489s 1s/step - accuracy: 0.9820 - loss: 0.0743 - val_accuracy: 0.9837 - val_loss: 0.0711 - lr: 8.9000e-05</p>\n\n<p>Epoch 00004: LearningRateScheduler reducing learning rate to 0.0001285.\nEpoch 4/5\n420/420 [==============================] - 490s 1s/step - accuracy: 0.9818 - loss: 0.0740 - val_accuracy: 0.9837 - val_loss: 0.0680 - lr: 1.2850e-04</p>\n\n<p>Epoch 00005: LearningRateScheduler reducing learning rate to 0.000168.\nEpoch 5/5\n420/420 [==============================] - 488s 1s/step - accuracy: 0.9825 - loss: 0.0711 - val_accuracy: 0.9836 - val_loss: 0.0621 - lr: 1.6800e-04</p>",
  "messages": [
    {
      "id": "892115",
      "postDate": "06/18/2020 17:08:26",
      "content": "<p>Hi,\nI am using a copy of one of standard efficient net notebooks. Everything works fine, but...\nTake a look at the listing. Pay attention at val_accuracy. It stops at 0.9837. It stops there even if I change the network, say, b7 to b7 or b0.\nHow can that be possible?</p>\n\n<blockquote>\n  <p>Epoch 00001: LearningRateScheduler reducing learning rate to 1e-05.\n  Epoch 1/5\n  420/420 [==============================] - 523s 1s/step - accuracy: 0.9450 - loss: 0.2126 - val_accuracy: 0.9836 - val_loss: 0.1243 - lr: 1.0000e-05</p>\n</blockquote>\n\n<p>Epoch 00002: LearningRateScheduler reducing learning rate to 4.95e-05.\nEpoch 2/5\n420/420 [==============================] - 489s 1s/step - accuracy: 0.9821 - loss: 0.0819 - val_accuracy: 0.9837 - val_loss: 0.0687 - lr: 4.9500e-05</p>\n\n<p>Epoch 00003: LearningRateScheduler reducing learning rate to 8.9e-05.\nEpoch 3/5\n420/420 [==============================] - 489s 1s/step - accuracy: 0.9820 - loss: 0.0743 - val_accuracy: 0.9837 - val_loss: 0.0711 - lr: 8.9000e-05</p>\n\n<p>Epoch 00004: LearningRateScheduler reducing learning rate to 0.0001285.\nEpoch 4/5\n420/420 [==============================] - 490s 1s/step - accuracy: 0.9818 - loss: 0.0740 - val_accuracy: 0.9837 - val_loss: 0.0680 - lr: 1.2850e-04</p>\n\n<p>Epoch 00005: LearningRateScheduler reducing learning rate to 0.000168.\nEpoch 5/5\n420/420 [==============================] - 488s 1s/step - accuracy: 0.9825 - loss: 0.0711 - val_accuracy: 0.9836 - val_loss: 0.0621 - lr: 1.6800e-04</p>",
      "rawMarkdown": "Hi,\nI am using a copy of one of standard efficient net notebooks. Everything works fine, but...\nTake a look at the listing. Pay attention at val_accuracy. It stops at 0.9837. It stops there even if I change the network, say, b7 to b7 or b0.\nHow can that be possible?\n\n&gt; \nEpoch 00001: LearningRateScheduler reducing learning rate to 1e-05.\nEpoch 1/5\n420/420 [==============================] - 523s 1s/step - accuracy: 0.9450 - loss: 0.2126 - val_accuracy: 0.9836 - val_loss: 0.1243 - lr: 1.0000e-05\n\nEpoch 00002: LearningRateScheduler reducing learning rate to 4.95e-05.\nEpoch 2/5\n420/420 [==============================] - 489s 1s/step - accuracy: 0.9821 - loss: 0.0819 - val_accuracy: 0.9837 - val_loss: 0.0687 - lr: 4.9500e-05\n\nEpoch 00003: LearningRateScheduler reducing learning rate to 8.9e-05.\nEpoch 3/5\n420/420 [==============================] - 489s 1s/step - accuracy: 0.9820 - loss: 0.0743 - val_accuracy: 0.9837 - val_loss: 0.0711 - lr: 8.9000e-05\n\nEpoch 00004: LearningRateScheduler reducing learning rate to 0.0001285.\nEpoch 4/5\n420/420 [==============================] - 490s 1s/step - accuracy: 0.9818 - loss: 0.0740 - val_accuracy: 0.9837 - val_loss: 0.0680 - lr: 1.2850e-04\n\nEpoch 00005: LearningRateScheduler reducing learning rate to 0.000168.\nEpoch 5/5\n420/420 [==============================] - 488s 1s/step - accuracy: 0.9825 - loss: 0.0711 - val_accuracy: 0.9836 - val_loss: 0.0621 - lr: 1.6800e-04",
      "votes": null
    },
    {
      "id": "892222",
      "postDate": "06/18/2020 18:09:17",
      "content": "<p>This could be the model always predicting benign. Since that's the majority class it explains why that yields an accuracy of ~98. Could be something else, but I would print the model predictions before debugging other causes.</p>",
      "rawMarkdown": "This could be the model always predicting benign. Since that's the majority class it explains why that yields an accuracy of ~98. Could be something else, but I would print the model predictions before debugging other causes.",
      "votes": null
    },
    {
      "id": "892237",
      "postDate": "06/18/2020 18:27:17",
      "content": "<p>For this imbalanced dataset, it's useful to watch AUC after each epoch, since that handles the imbalanced data better. Still, the validation AUC can jump around a lot and might not be useful for early stopping.</p>",
      "rawMarkdown": "For this imbalanced dataset, it's useful to watch AUC after each epoch, since that handles the imbalanced data better. Still, the validation AUC can jump around a lot and might not be useful for early stopping.",
      "votes": null
    },
    {
      "id": "892297",
      "postDate": "06/18/2020 19:26:00",
      "content": "<p>I don't think it always predicts benign - it rates 0.86\nYet, the level is just too precise...</p>",
      "rawMarkdown": "I don't think it always predicts benign - it rates 0.86\nYet, the level is just too precise...",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 892222,
      "author_name": "chriscareaga",
      "author_url": "",
      "post_date": "06/18/2020 18:09:17",
      "content": "<p>This could be the model always predicting benign. Since that's the majority class it explains why that yields an accuracy of ~98. Could be something else, but I would print the model predictions before debugging other causes.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 892237,
      "author_name": "richardepstein",
      "author_url": "",
      "post_date": "06/18/2020 18:27:17",
      "content": "<p>For this imbalanced dataset, it's useful to watch AUC after each epoch, since that handles the imbalanced data better. Still, the validation AUC can jump around a lot and might not be useful for early stopping.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 892297,
      "author_name": "fizpok",
      "author_url": "",
      "post_date": "06/18/2020 19:26:00",
      "content": "<p>I don't think it always predicts benign - it rates 0.86\nYet, the level is just too precise...</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "892115": "Hi,\nI am using a copy of one of standard efficient net notebooks. Everything works fine, but...\nTake a look at the listing. Pay attention at val_accuracy. It stops at 0.9837. It stops there even if I change the network, say, b7 to b7 or b0.\nHow can that be possible?\n\n&gt; \nEpoch 00001: LearningRateScheduler reducing learning rate to 1e-05.\nEpoch 1/5\n420/420 [==============================] - 523s 1s/step - accuracy: 0.9450 - loss: 0.2126 - val_accuracy: 0.9836 - val_loss: 0.1243 - lr: 1.0000e-05\n\nEpoch 00002: LearningRateScheduler reducing learning rate to 4.95e-05.\nEpoch 2/5\n420/420 [==============================] - 489s 1s/step - accuracy: 0.9821 - loss: 0.0819 - val_accuracy: 0.9837 - val_loss: 0.0687 - lr: 4.9500e-05\n\nEpoch 00003: LearningRateScheduler reducing learning rate to 8.9e-05.\nEpoch 3/5\n420/420 [==============================] - 489s 1s/step - accuracy: 0.9820 - loss: 0.0743 - val_accuracy: 0.9837 - val_loss: 0.0711 - lr: 8.9000e-05\n\nEpoch 00004: LearningRateScheduler reducing learning rate to 0.0001285.\nEpoch 4/5\n420/420 [==============================] - 490s 1s/step - accuracy: 0.9818 - loss: 0.0740 - val_accuracy: 0.9837 - val_loss: 0.0680 - lr: 1.2850e-04\n\nEpoch 00005: LearningRateScheduler reducing learning rate to 0.000168.\nEpoch 5/5\n420/420 [==============================] - 488s 1s/step - accuracy: 0.9825 - loss: 0.0711 - val_accuracy: 0.9836 - val_loss: 0.0621 - lr: 1.6800e-04",
    "892222": "This could be the model always predicting benign. Since that's the majority class it explains why that yields an accuracy of ~98. Could be something else, but I would print the model predictions before debugging other causes.",
    "892237": "For this imbalanced dataset, it's useful to watch AUC after each epoch, since that handles the imbalanced data better. Still, the validation AUC can jump around a lot and might not be useful for early stopping.",
    "892297": "I don't think it always predicts benign - it rates 0.86\nYet, the level is just too precise..."
  },
  "source": "meta"
}