{
  "id": 173620,
  "title": "ROC - Predictions - Percentage Chance of Melanoma",
  "url": "/competitions/siim-isic-melanoma-classification/discussion/173620",
  "author_name": "",
  "post_date": "2020-08-10T03:18:01.423659500Z",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I've noticed that a lot of prediction sets predict almost no images at a greater than 50% chance of the image being melanoma.</p>\n<p>I know that there are probably very few melanoma images in the test dataset but this seems pretty crazy to me and makes me wonder whether my model is really learning much about what separates melanoma images from benign images.</p>\n<p>I would expect to get a few images with a high percentage of being melanoma, a couple more with percentages in between and then almost all the other images with a near 0 percent chance of being melanoma.</p>\n<p>What is your take on this? Am I thinking about it in the wrong way?</p>",
  "messages": [
    {
      "id": "964598",
      "postDate": "08/10/2020 03:18:01",
      "content": "<p>I've noticed that a lot of prediction sets predict almost no images at a greater than 50% chance of the image being melanoma.</p>\n<p>I know that there are probably very few melanoma images in the test dataset but this seems pretty crazy to me and makes me wonder whether my model is really learning much about what separates melanoma images from benign images.</p>\n<p>I would expect to get a few images with a high percentage of being melanoma, a couple more with percentages in between and then almost all the other images with a near 0 percent chance of being melanoma.</p>\n<p>What is your take on this? Am I thinking about it in the wrong way?</p>",
      "rawMarkdown": "I've noticed that a lot of prediction sets predict almost no images at a greater than 50% chance of the image being melanoma.\n\nI know that there are probably very few melanoma images in the test dataset but this seems pretty crazy to me and makes me wonder whether my model is really learning much about what separates melanoma images from benign images.\n\nI would expect to get a few images with a high percentage of being melanoma, a couple more with percentages in between and then almost all the other images with a near 0 percent chance of being melanoma.\n\nWhat is your take on this? Am I thinking about it in the wrong way?",
      "votes": null
    },
    {
      "id": "964779",
      "postDate": "08/10/2020 06:55:02",
      "content": "<p>Your predictions are just not calibrated, due to the loss functions being used. As long as the malignant predictions are higher than the benign, you should not worry. You can always apply calibration as a post-processing step in order to better stretch the predictions between 0 and 1.</p>\n\n<p>If your AUC is high, then your model definitely separates melanoma images from benign images. See <a href=\"https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/173020\">this post</a></p>",
      "rawMarkdown": "Your predictions are just not calibrated, due to the loss functions being used. As long as the malignant predictions are higher than the benign, you should not worry. You can always apply calibration as a post-processing step in order to better stretch the predictions between 0 and 1.\n\nIf your AUC is high, then your model definitely separates melanoma images from benign images. See [this post](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/173020)",
      "votes": null
    },
    {
      "id": "965105",
      "postDate": "08/10/2020 11:42:42",
      "content": "<p>Thank you! That was a great post.</p>",
      "rawMarkdown": "Thank you! That was a great post.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 964779,
      "author_name": "group16",
      "author_url": "",
      "post_date": "08/10/2020 06:55:02",
      "content": "<p>Your predictions are just not calibrated, due to the loss functions being used. As long as the malignant predictions are higher than the benign, you should not worry. You can always apply calibration as a post-processing step in order to better stretch the predictions between 0 and 1.</p>\n\n<p>If your AUC is high, then your model definitely separates melanoma images from benign images. See <a href=\"https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/173020\">this post</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 965105,
          "author_name": "campbellhutcheson",
          "author_url": "",
          "post_date": "08/10/2020 11:42:42",
          "content": "<p>Thank you! That was a great post.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "964598": "I've noticed that a lot of prediction sets predict almost no images at a greater than 50% chance of the image being melanoma.\n\nI know that there are probably very few melanoma images in the test dataset but this seems pretty crazy to me and makes me wonder whether my model is really learning much about what separates melanoma images from benign images.\n\nI would expect to get a few images with a high percentage of being melanoma, a couple more with percentages in between and then almost all the other images with a near 0 percent chance of being melanoma.\n\nWhat is your take on this? Am I thinking about it in the wrong way?",
    "964779": "Your predictions are just not calibrated, due to the loss functions being used. As long as the malignant predictions are higher than the benign, you should not worry. You can always apply calibration as a post-processing step in order to better stretch the predictions between 0 and 1.\n\nIf your AUC is high, then your model definitely separates melanoma images from benign images. See [this post](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/173020)",
    "965105": "Thank you! That was a great post."
  },
  "source": "meta"
}