I know it is a pretty simple question but it is an important one. In the \"OSIC Pulmonary Fibrosis Progression\" competition, it is asked to calculate the \"confidence value.\" Is this value the \"percentage of the confidence interval.\" Please clarify.
", "messages": [ { "id": "945940", "postDate": "07/26/2020 08:24:53", "content": "I know it is a pretty simple question but it is an important one. In the \"OSIC Pulmonary Fibrosis Progression\" competition, it is asked to calculate the \"confidence value.\" Is this value the \"percentage of the confidence interval.\" Please clarify.
", "rawMarkdown": "I know it is a pretty simple question but it is an important one. In the \"OSIC Pulmonary Fibrosis Progression\" competition, it is asked to calculate the \"confidence value.\" Is this value the \"percentage of the confidence interval.\" Please clarify.", "votes": null }, { "id": "946653", "postDate": "07/26/2020 17:52:45", "content": "It's a number of units (and can be more than 100). It should be inversely proportional to how confident your model is about a given prediction. If you look at the scoring formula, you see the following dynamic: the larger \"confidence\" value you have, the less you will be penalized for the absolute error on a prediction. However, you are also penalized for large \"confidence\" values, so you don't want to exaggerate in that direction either!
", "rawMarkdown": "It's a number of units (and can be more than 100). It should be inversely proportional to how confident your model is about a given prediction. If you look at the scoring formula, you see the following dynamic: the larger \"confidence\" value you have, the less you will be penalized for the absolute error on a prediction. However, you are also penalized for large \"confidence\" values, so you don't want to exaggerate in that direction either!", "votes": null }, { "id": "947922", "postDate": "07/27/2020 14:58:25", "content": "Standard deviation is needed to evaluate the model confidence in its predictions and is used in the evaluation metric.
", "rawMarkdown": "Standard deviation is needed to evaluate the model confidence in its predictions and is used in the evaluation metric.", "votes": null }, { "id": "960317", "postDate": "08/06/2020 10:00:16", "content": "@archimedus \ncould u please help me understand how is confidence calculated in this notebook\nhttps://www.kaggle.com/yasufuminakama/osic-lgb-baseline/output?select=submission.csv\nI see that target is passed as confidence in this noteook for light gbm.. quite confused.
", "rawMarkdown": "archimedus \ncould u please help me understand how is confidence calculated in this notebook\nhttps://www.kaggle.com/yasufuminakama/osic-lgb-baseline/output?select=submission.csv\nI see that target is passed as confidence in this noteook for light gbm.. quite confused.", "votes": null }, { "id": "960803", "postDate": "08/06/2020 17:21:11", "content": "@jaideepvalani It's a bit confusing yes, not a lot of comments..! It looks like the author computes an optimal Confidence value for each training prediction (by maximizing the score function on each line with the scipy.optimize library), and then feeds these training values to a LightGBM model which then predicts expected optimal Confidence values for the submission.
", "rawMarkdown": "jaideepvalani It's a bit confusing yes, not a lot of comments..! It looks like the author computes an optimal Confidence value for each training prediction (by maximizing the score function on each line with the scipy.optimize library), and then feeds these training values to a LightGBM model which then predicts expected optimal Confidence values for the submission.", "votes": null } ], "comments": [ { "id": 946653, "author_name": "archimedus", "author_url": "", "post_date": "07/26/2020 17:52:45", "content": "It's a number of units (and can be more than 100). It should be inversely proportional to how confident your model is about a given prediction. If you look at the scoring formula, you see the following dynamic: the larger \"confidence\" value you have, the less you will be penalized for the absolute error on a prediction. However, you are also penalized for large \"confidence\" values, so you don't want to exaggerate in that direction either!
", "votes": null, "replies": [ { "id": 960317, "author_name": "jaideepvalani", "author_url": "", "post_date": "08/06/2020 10:00:16", "content": "@archimedus \ncould u please help me understand how is confidence calculated in this notebook\nhttps://www.kaggle.com/yasufuminakama/osic-lgb-baseline/output?select=submission.csv\nI see that target is passed as confidence in this noteook for light gbm.. quite confused.
", "votes": null, "replies": [] }, { "id": 960803, "author_name": "archimedus", "author_url": "", "post_date": "08/06/2020 17:21:11", "content": "@jaideepvalani It's a bit confusing yes, not a lot of comments..! It looks like the author computes an optimal Confidence value for each training prediction (by maximizing the score function on each line with the scipy.optimize library), and then feeds these training values to a LightGBM model which then predicts expected optimal Confidence values for the submission.
", "votes": null, "replies": [] } ] }, { "id": 947922, "author_name": "ahmedhshahin", "author_url": "", "post_date": "07/27/2020 14:58:25", "content": "Standard deviation is needed to evaluate the model confidence in its predictions and is used in the evaluation metric.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "945940": "I know it is a pretty simple question but it is an important one. In the \"OSIC Pulmonary Fibrosis Progression\" competition, it is asked to calculate the \"confidence value.\" Is this value the \"percentage of the confidence interval.\" Please clarify.", "946653": "It's a number of units (and can be more than 100). It should be inversely proportional to how confident your model is about a given prediction. If you look at the scoring formula, you see the following dynamic: the larger \"confidence\" value you have, the less you will be penalized for the absolute error on a prediction. However, you are also penalized for large \"confidence\" values, so you don't want to exaggerate in that direction either!", "947922": "Standard deviation is needed to evaluate the model confidence in its predictions and is used in the evaluation metric.", "960317": "archimedus \ncould u please help me understand how is confidence calculated in this notebook\nhttps://www.kaggle.com/yasufuminakama/osic-lgb-baseline/output?select=submission.csv\nI see that target is passed as confidence in this noteook for light gbm.. quite confused.", "960803": "jaideepvalani It's a bit confusing yes, not a lot of comments..! It looks like the author computes an optimal Confidence value for each training prediction (by maximizing the score function on each line with the scipy.optimize library), and then feeds these training values to a LightGBM model which then predicts expected optimal Confidence values for the submission." }, "source": "meta" }