{
  "id": 197787,
  "title": "What is meant by prediction confidence ",
  "url": "/competitions/osic-pulmonary-fibrosis-progression/discussion/197787",
  "author_name": "",
  "post_date": "2020-11-18T04:25:09.936164Z",
  "votes": 3,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Hi all,<br>\nI am attempting this after the fact, so I hope some are still around.</p>\n<p>I am confused what is meant by \"prediction  confidence\" and how to go about evaluating the confidence of a prediction with out being an interval. </p>\n<p>Any input would be appreciated. </p>",
  "messages": [
    {
      "id": "1082608",
      "postDate": "11/18/2020 04:25:09",
      "content": "<p>Hi all,<br>\nI am attempting this after the fact, so I hope some are still around.</p>\n<p>I am confused what is meant by \"prediction  confidence\" and how to go about evaluating the confidence of a prediction with out being an interval. </p>\n<p>Any input would be appreciated. </p>",
      "rawMarkdown": "Hi all,\nI am attempting this after the fact, so I hope some are still around.\n\nI am confused what is meant by \"prediction  confidence\" and how to go about evaluating the confidence of a prediction with out being an interval. \n\nAny input would be appreciated.",
      "votes": null
    },
    {
      "id": "1082819",
      "postDate": "11/18/2020 09:03:08",
      "content": "<p>There isnt a closed answer for this. Machine learning in general arent very good to predict confidence. A good estimate is the standad deviation of your prediction ( that u can get by many ways, running several batches, etc). </p>",
      "rawMarkdown": "There isnt a closed answer for this. Machine learning in general arent very good to predict confidence. A good estimate is the standad deviation of your prediction ( that u can get by many ways, running several batches, etc).",
      "votes": null
    },
    {
      "id": "1083205",
      "postDate": "11/18/2020 17:55:33",
      "content": "<p>As <a href=\"https://www.kaggle.com/rpsantosakaggle\" target=\"_blank\">@rpsantosakaggle</a> mentioned standard deviation is a good measure of prediction uncertainty.<br>\nTo calculate the standard deviation for a NN model, </p>\n<ul>\n<li>Add dropout layer during prediction</li>\n<li>Make multiple prediction for the same input</li>\n<li>Calculate the standard deviation of the predictions</li>\n</ul>",
      "rawMarkdown": "As @rpsantosakaggle mentioned standard deviation is a good measure of prediction uncertainty.\nTo calculate the standard deviation for a NN model, \n- Add dropout layer during prediction\n- Make multiple prediction for the same input\n- Calculate the standard deviation of the predictions",
      "votes": null
    },
    {
      "id": "1083218",
      "postDate": "11/18/2020 18:10:34",
      "content": "<p><a href=\"https://www.kaggle.com/umeyrkl\" target=\"_blank\">@umeyrkl</a> interesting, thanks. When you suggest making multiple predictions from the same input, are you suggesting from the same model? i.e. the dropout layer will add some variance?</p>",
      "rawMarkdown": "umeyrkl interesting, thanks. When you suggest making multiple predictions from the same input, are you suggesting from the same model? i.e. the dropout layer will add some variance?",
      "votes": null
    },
    {
      "id": "1083307",
      "postDate": "11/18/2020 20:29:08",
      "content": "<p>Relying on your model, repeat predicitons doenst apply. You need to figure out one that fits your solution. i got something like MSE over each pacient. But if u use Hierarquical Bayesian Network, you get that directly.</p>",
      "rawMarkdown": "Relying on your model, repeat predicitons doenst apply. You need to figure out one that fits your solution. i got something like MSE over each pacient. But if u use Hierarquical Bayesian Network, you get that directly.",
      "votes": null
    },
    {
      "id": "1086953",
      "postDate": "11/22/2020 07:59:31",
      "content": "<p>Yep. The same model with the dropout layer for variance.</p>",
      "rawMarkdown": "Yep. The same model with the dropout layer for variance.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1082819,
      "author_name": "rpsantosakaggle",
      "author_url": "",
      "post_date": "11/18/2020 09:03:08",
      "content": "<p>There isnt a closed answer for this. Machine learning in general arent very good to predict confidence. A good estimate is the standad deviation of your prediction ( that u can get by many ways, running several batches, etc). </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1083205,
      "author_name": "umeyrkl",
      "author_url": "",
      "post_date": "11/18/2020 17:55:33",
      "content": "<p>As <a href=\"https://www.kaggle.com/rpsantosakaggle\" target=\"_blank\">@rpsantosakaggle</a> mentioned standard deviation is a good measure of prediction uncertainty.<br>\nTo calculate the standard deviation for a NN model, </p>\n<ul>\n<li>Add dropout layer during prediction</li>\n<li>Make multiple prediction for the same input</li>\n<li>Calculate the standard deviation of the predictions</li>\n</ul>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1083218,
      "author_name": "devpool",
      "author_url": "",
      "post_date": "11/18/2020 18:10:34",
      "content": "<p><a href=\"https://www.kaggle.com/umeyrkl\" target=\"_blank\">@umeyrkl</a> interesting, thanks. When you suggest making multiple predictions from the same input, are you suggesting from the same model? i.e. the dropout layer will add some variance?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1086953,
          "author_name": "umeyrkl",
          "author_url": "",
          "post_date": "11/22/2020 07:59:31",
          "content": "<p>Yep. The same model with the dropout layer for variance.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1083307,
      "author_name": "rpsantosakaggle",
      "author_url": "",
      "post_date": "11/18/2020 20:29:08",
      "content": "<p>Relying on your model, repeat predicitons doenst apply. You need to figure out one that fits your solution. i got something like MSE over each pacient. But if u use Hierarquical Bayesian Network, you get that directly.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1082608": "Hi all,\nI am attempting this after the fact, so I hope some are still around.\n\nI am confused what is meant by \"prediction  confidence\" and how to go about evaluating the confidence of a prediction with out being an interval. \n\nAny input would be appreciated.",
    "1082819": "There isnt a closed answer for this. Machine learning in general arent very good to predict confidence. A good estimate is the standad deviation of your prediction ( that u can get by many ways, running several batches, etc).",
    "1083205": "As @rpsantosakaggle mentioned standard deviation is a good measure of prediction uncertainty.\nTo calculate the standard deviation for a NN model, \n- Add dropout layer during prediction\n- Make multiple prediction for the same input\n- Calculate the standard deviation of the predictions",
    "1083218": "umeyrkl interesting, thanks. When you suggest making multiple predictions from the same input, are you suggesting from the same model? i.e. the dropout layer will add some variance?",
    "1083307": "Relying on your model, repeat predicitons doenst apply. You need to figure out one that fits your solution. i got something like MSE over each pacient. But if u use Hierarquical Bayesian Network, you get that directly.",
    "1086953": "Yep. The same model with the dropout layer for variance."
  },
  "source": "meta"
}