The description says that we also need to calculate the confidence value of our prediction and it has a unit of ml.
\nCan anyone explain what confidence is exactly and how to calculate it
The description says that we also need to calculate the confidence value of our prediction and it has a unit of ml.
\nCan anyone explain what confidence is exactly and how to calculate it
The uncertainty that is used in the metric calculation is basically the uncertainty of the model predictions in terms of its standard deviation. For ex: if a model says the FVC at this point is 2000, how confident is the model in this prediction? If it is quite confident, its sigma is low, if the model is not confident then sigma is high. Probabilistic models like (Bayesian linear regression for example) allow you to get such uncertainty as it outputs a distribution. Conversely, deep learning models output point estimates, so you can't get uncertainty here. There are many methods that try to get such uncertainty from deep learning models (for example by approximation, ensembles, etc). You will also find several approaches here in the forum.
\nSince we have a reasonable understanding of the inherent measurement error in FVC (around 70ml) we need to take that into consideration. In a clinical setting, it is useful if the machine gives a reasonable estimate of the confidence in its prediction -- otherwise, the machine might very confidently predict a wildly inaccurate FVC value. In general, uncertainty allows for a better interpretation and judgment of a machine learning models' prediction.
\nI hope that makes sense and good luck!!
", "rawMarkdown": "The uncertainty that is used in the metric calculation is basically the uncertainty of the model predictions in terms of its standard deviation. For ex: if a model says the FVC at this point is 2000, how confident is the model in this prediction? If it is quite confident, its sigma is low, if the model is not confident then sigma is high. Probabilistic models like (Bayesian linear regression for example) allow you to get such uncertainty as it outputs a distribution. Conversely, deep learning models output point estimates, so you can't get uncertainty here. There are many methods that try to get such uncertainty from deep learning models (for example by approximation, ensembles, etc). You will also find several approaches here in the forum.\n\nSince we have a reasonable understanding of the inherent measurement error in FVC (around 70ml) we need to take that into consideration. In a clinical setting, it is useful if the machine gives a reasonable estimate of the confidence in its prediction -- otherwise, the machine might very confidently predict a wildly inaccurate FVC value. In general, uncertainty allows for a better interpretation and judgment of a machine learning models' prediction.\n\nI hope that makes sense and good luck!!", "votes": null }, { "id": "998152", "postDate": "09/04/2020 14:29:10", "content": "Ok
\nThanks for explaining
I don't know how accurate this would be, but I started thinking of the confidence as instead of predicting say 2000 as the output, you predict 2000 ± 100. Here, 100 would be your confidence.
", "rawMarkdown": "I don't know how accurate this would be, but I started thinking of the confidence as instead of predicting say 2000 as the output, you predict 2000 ± 100. Here, 100 would be your confidence.", "votes": null }, { "id": "998409", "postDate": "09/04/2020 17:55:26", "content": "So comprehensive, thanks! I haven' known before that 70 sigma restriction corresponds with a level of errors.
", "rawMarkdown": "So comprehensive, thanks! I haven' known before that 70 sigma restriction corresponds with a level of errors.", "votes": null }, { "id": "998415", "postDate": "09/04/2020 17:57:04", "content": "I think it's impossible to use such raw value of confidence equals 100 without any postprocessing.
", "rawMarkdown": "I think it's impossible to use such raw value of confidence equals 100 without any postprocessing.", "votes": null } ], "comments": [ { "id": 998105, "author_name": "ahmedhshahin", "author_url": "", "post_date": "09/04/2020 13:58:23", "content": "The uncertainty that is used in the metric calculation is basically the uncertainty of the model predictions in terms of its standard deviation. For ex: if a model says the FVC at this point is 2000, how confident is the model in this prediction? If it is quite confident, its sigma is low, if the model is not confident then sigma is high. Probabilistic models like (Bayesian linear regression for example) allow you to get such uncertainty as it outputs a distribution. Conversely, deep learning models output point estimates, so you can't get uncertainty here. There are many methods that try to get such uncertainty from deep learning models (for example by approximation, ensembles, etc). You will also find several approaches here in the forum.
\nSince we have a reasonable understanding of the inherent measurement error in FVC (around 70ml) we need to take that into consideration. In a clinical setting, it is useful if the machine gives a reasonable estimate of the confidence in its prediction -- otherwise, the machine might very confidently predict a wildly inaccurate FVC value. In general, uncertainty allows for a better interpretation and judgment of a machine learning models' prediction.
\nI hope that makes sense and good luck!!
", "votes": null, "replies": [ { "id": 998152, "author_name": "ask5898", "author_url": "", "post_date": "09/04/2020 14:29:10", "content": "Ok
\nThanks for explaining
I don't know how accurate this would be, but I started thinking of the confidence as instead of predicting say 2000 as the output, you predict 2000 ± 100. Here, 100 would be your confidence.
", "votes": null, "replies": [] }, { "id": 998409, "author_name": "koza4ukdmitrij", "author_url": "", "post_date": "09/04/2020 17:55:26", "content": "So comprehensive, thanks! I haven' known before that 70 sigma restriction corresponds with a level of errors.
", "votes": null, "replies": [] }, { "id": 998415, "author_name": "koza4ukdmitrij", "author_url": "", "post_date": "09/04/2020 17:57:04", "content": "I think it's impossible to use such raw value of confidence equals 100 without any postprocessing.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "998027": "The description says that we also need to calculate the confidence value of our prediction and it has a unit of ml.\nCan anyone explain what confidence is exactly and how to calculate it", "998105": "The uncertainty that is used in the metric calculation is basically the uncertainty of the model predictions in terms of its standard deviation. For ex: if a model says the FVC at this point is 2000, how confident is the model in this prediction? If it is quite confident, its sigma is low, if the model is not confident then sigma is high. Probabilistic models like (Bayesian linear regression for example) allow you to get such uncertainty as it outputs a distribution. Conversely, deep learning models output point estimates, so you can't get uncertainty here. There are many methods that try to get such uncertainty from deep learning models (for example by approximation, ensembles, etc). You will also find several approaches here in the forum.\n\nSince we have a reasonable understanding of the inherent measurement error in FVC (around 70ml) we need to take that into consideration. In a clinical setting, it is useful if the machine gives a reasonable estimate of the confidence in its prediction -- otherwise, the machine might very confidently predict a wildly inaccurate FVC value. In general, uncertainty allows for a better interpretation and judgment of a machine learning models' prediction.\n\nI hope that makes sense and good luck!!", "998152": "Ok\nThanks for explaining", "998282": "I don't know how accurate this would be, but I started thinking of the confidence as instead of predicting say 2000 as the output, you predict 2000 ± 100. Here, 100 would be your confidence.", "998409": "So comprehensive, thanks! I haven' known before that 70 sigma restriction corresponds with a level of errors.", "998415": "I think it's impossible to use such raw value of confidence equals 100 without any postprocessing." }, "source": "meta" }