{ "id": 181632, "title": "What was your approach in predicting confidence?", "url": "/competitions/osic-pulmonary-fibrosis-progression/discussion/181632", "author_name": "", "post_date": "2020-09-09T15:20:41.567071500Z", "votes": 11, "comment_count": 7, "views": 0, "content": "

Hi there,

\n

I would just like to know how did u predict your confidence values and please explain it here in detail.

\n

Thank you!

", "messages": [ { "id": "1004278", "postDate": "09/09/2020 15:20:41", "content": "

Hi there,

\n

I would just like to know how did u predict your confidence values and please explain it here in detail.

\n

Thank you!

", "rawMarkdown": "Hi there,\n\nI would just like to know how did u predict your confidence values and please explain it here in detail.\n\nThank you!", "votes": null }, { "id": "1004446", "postDate": "09/09/2020 18:12:11", "content": "

I had the same doubt

", "rawMarkdown": "I had the same doubt", "votes": null }, { "id": "1004582", "postDate": "09/09/2020 20:15:58", "content": "

I haven't had the time to code it, but the way I would approach it ideally is to calculate predictions numerous times (preferably using multiple models), and use the range of these predictions for each patient and each week to derive the confidence level. In other words, if all predictions are very close to one another, I have high confidence (= a low number) in my prediction, and vice versa.

", "rawMarkdown": "I haven't had the time to code it, but the way I would approach it ideally is to calculate predictions numerous times (preferably using multiple models), and use the range of these predictions for each patient and each week to derive the confidence level. In other words, if all predictions are very close to one another, I have high confidence (= a low number) in my prediction, and vice versa.", "votes": null }, { "id": "1004608", "postDate": "09/09/2020 21:02:50", "content": "

Take a look into quantile Regression topic this might give you an Idea of how to get confidence attribute.https://towardsdatascience.com/quantile-regression-ff2343c4a03#:~:text=Quantile%20regression%20is%20an%20extension,the%20mean%20takes%20the%20form

", "rawMarkdown": "Take a look into quantile Regression topic this might give you an Idea of how to get confidence attribute.[https://towardsdatascience.com/quantile-regression-ff2343c4a03#:~:text=Quantile%20regression%20is%20an%20extension,the%20mean%20takes%20the%20form](url)", "votes": null }, { "id": "1004612", "postDate": "09/09/2020 21:10:48", "content": "

There are a bunch of options I have seen, so far:

\n
    \n
  1. quantile regression (setting confidence e.g. to be the difference between the estimates for two suitably chosen quantiles - one low and one high such as 0.15 and 0.85, or 0.2 and 0.8) and/or losses that target quantile regression (such as pinball loss - see e.g. this notebook or this one)
    2. \n
  2. using a loss function that allows you to directly estimate a point estimate and the confidence (e.g. like in this notebook - but it is hard to directly replicate the competition metric, see e.g. this discussion)
    3. \n
  3. Some kind of Bayesian approach, of which there are multiple flavors - such as this notebook - you can obviously do the same thing with many other packages like pystan in python, rstan, rstanarm or brms in R etc.) that give you a predictive distribution for the quantity of interest. Alternatives may include Bayesian deep learning.
    4. \n
  4. Attempts for a two-stage estimation (first get an estimate), then develop a model that predicts the confidence. That's not trivial though (for a start, make sure you don't build the second model on the same data on which you buildt your estimates - i.e. you want to work with out-of-fold predictions).
    5. \n
  5. The suggestion that was already mentioned in the thread of having a large variety of models (this is also closely related to some forms of Bayesian Deep Learning - see e.g. the work of Yarin Gal). I have tried that, but perhaps my models were not varied enough, because I have so far always got too small a value for the confidence (and did not find an obvious way of scaling it upwards appropriately).
    6. \n
\n

The first 3 have definitely been demonstrated to work okay on this challenge, but I have not noticed a really good example of the 4th and 5th approach, but both seem kind of plausible to me.

\n

Whatever you do, also make sure you try a really simple baseline and make sure you do better than that. The most obvious baseline is to always set the confidence to the same value such as 250 (or some value you e.g. determine via the performance on your out-of-fold predictions). If your approach does worse than that, then you need to re-think your approach.

", "rawMarkdown": "There are a bunch of options I have seen, so far:\n1. quantile regression (setting confidence e.g. to be the difference between the estimates for two suitably chosen quantiles - one low and one high such as 0.15 and 0.85, or 0.2 and 0.8) and/or losses that target quantile regression (such as pinball loss - see e.g. [this notebook](https://www.kaggle.com/ulrich07/osic-multiple-quantile-regression-starter) or [this one](https://www.kaggle.com/leoisleo1/efficientnets-quantile-regression-inference))\n2. using a loss function that allows you to directly estimate a point estimate and the confidence (e.g. like in [this notebook](https://www.kaggle.com/ttahara/osic-baseline-lgbm-with-custom-metric) - but it is hard to directly replicate the competition metric, see e.g. [this discussion](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/181505))\n3. Some kind of Bayesian approach, of which there are multiple flavors - such as [this notebook](https://www.kaggle.com/carlossouza/bayesian-experiments) - you can obviously do the same thing with many other packages like pystan in python, rstan, rstanarm or brms in R etc.) that give you a predictive distribution for the quantity of interest. Alternatives may include Bayesian deep learning.\n4. Attempts for a two-stage estimation (first get an estimate), then develop a model that predicts the confidence. That's not trivial though (for a start, make sure you don't build the second model on the same data on which you buildt your estimates - i.e. you want to work with out-of-fold predictions).\n5. The suggestion that was already mentioned in the thread of having a large variety of models (this is also closely related to some forms of Bayesian Deep Learning - see e.g. the work of Yarin Gal). I have tried that, but perhaps my models were not varied enough, because I have so far always got too small a value for the confidence (and did not find an obvious way of scaling it upwards appropriately).\n\nThe first 3 have definitely been demonstrated to work okay on this challenge, but I have not noticed a really good example of the 4th and 5th approach, but both seem kind of plausible to me.\n\nWhatever you do, also make sure you try a really simple baseline and make sure you do better than that. The most obvious baseline is to always set the confidence to the same value such as 250 (or some value you e.g. determine via the performance on your out-of-fold predictions). If your approach does worse than that, then you need to re-think your approach.", "votes": null }, { "id": "1004616", "postDate": "09/09/2020 21:14:15", "content": "

Makes sense, but I suspect the challenge is to get enough variety in the models to really capture the uncertainty. Oh, and greetings to another participant based near Basel. 👍

", "rawMarkdown": "Makes sense, but I suspect the challenge is to get enough variety in the models to really capture the uncertainty. Oh, and greetings to another participant based near Basel. 👍", "votes": null }, { "id": "1004805", "postDate": "09/10/2020 03:43:32", "content": "

Thanks for listing these out! Appreciate your efforts in finding these options out :)

", "rawMarkdown": "Thanks for listing these out! Appreciate your efforts in finding these options out :)", "votes": null }, { "id": "1004906", "postDate": "09/10/2020 06:02:13", "content": "

Greetings! Or should I say, \"sali\" :)
\nOn the issue of confidence, if all models converge to the same prediction, then from an information theory perspective it's because the data is so informative of a patient's future status that widely different models all converge to the same forecast - which in turn would give an analyst high confidence (= low dispersion of forecast) on those predictions.

\n

But for me, the challenge has been finding the time to set up the models, so at this point I am still in the phase of forking peoples' notebooks (also to learn more python, since I am a \"native R speaker\").

", "rawMarkdown": "Greetings! Or should I say, \"sali\" :)\nOn the issue of confidence, if all models converge to the same prediction, then from an information theory perspective it's because the data is so informative of a patient's future status that widely different models all converge to the same forecast - which in turn would give an analyst high confidence (= low dispersion of forecast) on those predictions.\n\nBut for me, the challenge has been finding the time to set up the models, so at this point I am still in the phase of forking peoples' notebooks (also to learn more python, since I am a \"native R speaker\").", "votes": null } ], "comments": [ { "id": 1004446, "author_name": "theforcecoder", "author_url": "", "post_date": "09/09/2020 18:12:11", "content": "

I had the same doubt

", "votes": null, "replies": [] }, { "id": 1004608, "author_name": "raviet", "author_url": "", "post_date": "09/09/2020 21:02:50", "content": "

Take a look into quantile Regression topic this might give you an Idea of how to get confidence attribute.https://towardsdatascience.com/quantile-regression-ff2343c4a03#:~:text=Quantile%20regression%20is%20an%20extension,the%20mean%20takes%20the%20form

", "votes": null, "replies": [] }, { "id": 1004612, "author_name": "bjoernholzhauer", "author_url": "", "post_date": "09/09/2020 21:10:48", "content": "

There are a bunch of options I have seen, so far:

\n
    \n
  1. quantile regression (setting confidence e.g. to be the difference between the estimates for two suitably chosen quantiles - one low and one high such as 0.15 and 0.85, or 0.2 and 0.8) and/or losses that target quantile regression (such as pinball loss - see e.g. this notebook or this one)
    2. \n
  2. using a loss function that allows you to directly estimate a point estimate and the confidence (e.g. like in this notebook - but it is hard to directly replicate the competition metric, see e.g. this discussion)
    3. \n
  3. Some kind of Bayesian approach, of which there are multiple flavors - such as this notebook - you can obviously do the same thing with many other packages like pystan in python, rstan, rstanarm or brms in R etc.) that give you a predictive distribution for the quantity of interest. Alternatives may include Bayesian deep learning.
    4. \n
  4. Attempts for a two-stage estimation (first get an estimate), then develop a model that predicts the confidence. That's not trivial though (for a start, make sure you don't build the second model on the same data on which you buildt your estimates - i.e. you want to work with out-of-fold predictions).
    5. \n
  5. The suggestion that was already mentioned in the thread of having a large variety of models (this is also closely related to some forms of Bayesian Deep Learning - see e.g. the work of Yarin Gal). I have tried that, but perhaps my models were not varied enough, because I have so far always got too small a value for the confidence (and did not find an obvious way of scaling it upwards appropriately).
    6. \n
\n

The first 3 have definitely been demonstrated to work okay on this challenge, but I have not noticed a really good example of the 4th and 5th approach, but both seem kind of plausible to me.

\n

Whatever you do, also make sure you try a really simple baseline and make sure you do better than that. The most obvious baseline is to always set the confidence to the same value such as 250 (or some value you e.g. determine via the performance on your out-of-fold predictions). If your approach does worse than that, then you need to re-think your approach.

", "votes": null, "replies": [ { "id": 1004805, "author_name": "aadhavvignesh", "author_url": "", "post_date": "09/10/2020 03:43:32", "content": "

Thanks for listing these out! Appreciate your efforts in finding these options out :)

", "votes": null, "replies": [] } ] }, { "id": 1004582, "author_name": "douglaskgaraujo", "author_url": "", "post_date": "09/09/2020 20:15:58", "content": "

I haven't had the time to code it, but the way I would approach it ideally is to calculate predictions numerous times (preferably using multiple models), and use the range of these predictions for each patient and each week to derive the confidence level. In other words, if all predictions are very close to one another, I have high confidence (= a low number) in my prediction, and vice versa.

", "votes": null, "replies": [ { "id": 1004616, "author_name": "bjoernholzhauer", "author_url": "", "post_date": "09/09/2020 21:14:15", "content": "

Makes sense, but I suspect the challenge is to get enough variety in the models to really capture the uncertainty. Oh, and greetings to another participant based near Basel. 👍

", "votes": null, "replies": [] }, { "id": 1004906, "author_name": "douglaskgaraujo", "author_url": "", "post_date": "09/10/2020 06:02:13", "content": "

Greetings! Or should I say, \"sali\" :)
\nOn the issue of confidence, if all models converge to the same prediction, then from an information theory perspective it's because the data is so informative of a patient's future status that widely different models all converge to the same forecast - which in turn would give an analyst high confidence (= low dispersion of forecast) on those predictions.

\n

But for me, the challenge has been finding the time to set up the models, so at this point I am still in the phase of forking peoples' notebooks (also to learn more python, since I am a \"native R speaker\").

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1004278": "Hi there,\n\nI would just like to know how did u predict your confidence values and please explain it here in detail.\n\nThank you!", "1004446": "I had the same doubt", "1004582": "I haven't had the time to code it, but the way I would approach it ideally is to calculate predictions numerous times (preferably using multiple models), and use the range of these predictions for each patient and each week to derive the confidence level. In other words, if all predictions are very close to one another, I have high confidence (= a low number) in my prediction, and vice versa.", "1004608": "Take a look into quantile Regression topic this might give you an Idea of how to get confidence attribute.[https://towardsdatascience.com/quantile-regression-ff2343c4a03#:~:text=Quantile%20regression%20is%20an%20extension,the%20mean%20takes%20the%20form](url)", "1004612": "There are a bunch of options I have seen, so far:\n1. quantile regression (setting confidence e.g. to be the difference between the estimates for two suitably chosen quantiles - one low and one high such as 0.15 and 0.85, or 0.2 and 0.8) and/or losses that target quantile regression (such as pinball loss - see e.g. [this notebook](https://www.kaggle.com/ulrich07/osic-multiple-quantile-regression-starter) or [this one](https://www.kaggle.com/leoisleo1/efficientnets-quantile-regression-inference))\n2. using a loss function that allows you to directly estimate a point estimate and the confidence (e.g. like in [this notebook](https://www.kaggle.com/ttahara/osic-baseline-lgbm-with-custom-metric) - but it is hard to directly replicate the competition metric, see e.g. [this discussion](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/181505))\n3. Some kind of Bayesian approach, of which there are multiple flavors - such as [this notebook](https://www.kaggle.com/carlossouza/bayesian-experiments) - you can obviously do the same thing with many other packages like pystan in python, rstan, rstanarm or brms in R etc.) that give you a predictive distribution for the quantity of interest. Alternatives may include Bayesian deep learning.\n4. Attempts for a two-stage estimation (first get an estimate), then develop a model that predicts the confidence. That's not trivial though (for a start, make sure you don't build the second model on the same data on which you buildt your estimates - i.e. you want to work with out-of-fold predictions).\n5. The suggestion that was already mentioned in the thread of having a large variety of models (this is also closely related to some forms of Bayesian Deep Learning - see e.g. the work of Yarin Gal). I have tried that, but perhaps my models were not varied enough, because I have so far always got too small a value for the confidence (and did not find an obvious way of scaling it upwards appropriately).\n\nThe first 3 have definitely been demonstrated to work okay on this challenge, but I have not noticed a really good example of the 4th and 5th approach, but both seem kind of plausible to me.\n\nWhatever you do, also make sure you try a really simple baseline and make sure you do better than that. The most obvious baseline is to always set the confidence to the same value such as 250 (or some value you e.g. determine via the performance on your out-of-fold predictions). If your approach does worse than that, then you need to re-think your approach.", "1004616": "Makes sense, but I suspect the challenge is to get enough variety in the models to really capture the uncertainty. Oh, and greetings to another participant based near Basel. 👍", "1004805": "Thanks for listing these out! Appreciate your efforts in finding these options out :)", "1004906": "Greetings! Or should I say, \"sali\" :)\nOn the issue of confidence, if all models converge to the same prediction, then from an information theory perspective it's because the data is so informative of a patient's future status that widely different models all converge to the same forecast - which in turn would give an analyst high confidence (= low dispersion of forecast) on those predictions.\n\nBut for me, the challenge has been finding the time to set up the models, so at this point I am still in the phase of forking peoples' notebooks (also to learn more python, since I am a \"native R speaker\")." }, "source": "meta" }