{ "id": 422172, "title": "The Impact of Confidence Threshold on Results", "url": "/competitions/hubmap-hacking-the-human-vasculature/discussion/422172", "author_name": "", "post_date": "2023-07-08T14:42:21.921382500Z", "votes": 2, "comment_count": 3, "views": 0, "content": "

In the process of using the Yolo series model, I found that the confidence threshold has an impact on the final result. When I set it to 0.001, the result will be higher than 0.20. What is the reason? Although models with low thresholds predict more masks, shouldn't masks with low confidence scores be inaccurate?

", "messages": [ { "id": "2335432", "postDate": "07/08/2023 14:42:21", "content": "

In the process of using the Yolo series model, I found that the confidence threshold has an impact on the final result. When I set it to 0.001, the result will be higher than 0.20. What is the reason? Although models with low thresholds predict more masks, shouldn't masks with low confidence scores be inaccurate?

", "rawMarkdown": "In the process of using the Yolo series model, I found that the confidence threshold has an impact on the final result. When I set it to 0.001, the result will be higher than 0.20. What is the reason? Although models with low thresholds predict more masks, shouldn't masks with low confidence scores be inaccurate?", "votes": null }, { "id": "2335895", "postDate": "07/08/2023 23:35:14", "content": "

As far as I understand the metric, it is AUC precision/recall. Thus, incorrect predictions at the tail of the list (low score) do not affect the final score in any way. But a mixture of right and wrong is better than nothing.

", "rawMarkdown": "As far as I understand the metric, it is AUC precision/recall. Thus, incorrect predictions at the tail of the list (low score) do not affect the final score in any way. But a mixture of right and wrong is better than nothing.", "votes": null }, { "id": "2336000", "postDate": "07/09/2023 03:01:34", "content": "

it is about AUC of the precision/recall curve. if you set higer threshold, you are trucating the curve, i.e. less area

\n

the trick is the rank the predicted instance correctly.
\ne.g.

\n
tp:  posistive\nfp:  posistive\n\nrank\ntp,tp,tp tp, fp, fp ,fp,fp\nscore = , ..              ,  ..\n
\n

it is actually better to learn a ranking function as post processing.
\nsee my post https://www.kaggle.com/competitions/hubmap-hacking-the-human-vasculature/discussion/422250

\n

this is used in previous kaggle competitions

", "rawMarkdown": "it is about AUC of the precision/recall curve. if you set higer threshold, you are trucating the curve, i.e. less area\n\nthe trick is the rank the predicted instance correctly.\ne.g.\n\n```\ntp: true posistive\nfp: false posistive\n\nrank\ntp,tp,tp ...........................tp, fp, .................................fp ,fp,fp\nscore = 1.0, ..... 0.2, 0.19 .........................................0.0.\n\n```\n\nit is actually better to learn a ranking function as post processing.\nsee my post https://www.kaggle.com/competitions/hubmap-hacking-the-human-vasculature/discussion/422250\n\nthis is used in previous kaggle competitions", "votes": null }, { "id": "2336014", "postDate": "07/09/2023 03:11:46", "content": "

Your explanation has enlightened me. Thank you

", "rawMarkdown": "Your explanation has enlightened me. Thank you", "votes": null } ], "comments": [ { "id": 2335895, "author_name": "tsobolev", "author_url": "", "post_date": "07/08/2023 23:35:14", "content": "

As far as I understand the metric, it is AUC precision/recall. Thus, incorrect predictions at the tail of the list (low score) do not affect the final score in any way. But a mixture of right and wrong is better than nothing.

", "votes": null, "replies": [] }, { "id": 2336000, "author_name": "hengck23", "author_url": "", "post_date": "07/09/2023 03:01:34", "content": "

it is about AUC of the precision/recall curve. if you set higer threshold, you are trucating the curve, i.e. less area

\n

the trick is the rank the predicted instance correctly.
\ne.g.

\n
tp:  posistive\nfp:  posistive\n\nrank\ntp,tp,tp tp, fp, fp ,fp,fp\nscore = , ..              ,  ..\n
\n

it is actually better to learn a ranking function as post processing.
\nsee my post https://www.kaggle.com/competitions/hubmap-hacking-the-human-vasculature/discussion/422250

\n

this is used in previous kaggle competitions

", "votes": null, "replies": [ { "id": 2336014, "author_name": "bent1e", "author_url": "", "post_date": "07/09/2023 03:11:46", "content": "

Your explanation has enlightened me. Thank you

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2335432": "In the process of using the Yolo series model, I found that the confidence threshold has an impact on the final result. When I set it to 0.001, the result will be higher than 0.20. What is the reason? Although models with low thresholds predict more masks, shouldn't masks with low confidence scores be inaccurate?", "2335895": "As far as I understand the metric, it is AUC precision/recall. Thus, incorrect predictions at the tail of the list (low score) do not affect the final score in any way. But a mixture of right and wrong is better than nothing.", "2336000": "it is about AUC of the precision/recall curve. if you set higer threshold, you are trucating the curve, i.e. less area\n\nthe trick is the rank the predicted instance correctly.\ne.g.\n\n```\ntp: true posistive\nfp: false posistive\n\nrank\ntp,tp,tp ...........................tp, fp, .................................fp ,fp,fp\nscore = 1.0, ..... 0.2, 0.19 .........................................0.0.\n\n```\n\nit is actually better to learn a ranking function as post processing.\nsee my post https://www.kaggle.com/competitions/hubmap-hacking-the-human-vasculature/discussion/422250\n\nthis is used in previous kaggle competitions", "2336014": "Your explanation has enlightened me. Thank you" }, "source": "meta" }