AUC is a ranking metric and therefore does not need any probability calibration.
\n\nHowever the following can be read in the data section :
\n\n\nYou are predicting a binary target for each image. Your model should predict the probability (floating point) that the lesion in the image is malignant (the target). In the training data, train.csv, the value 0 denotes benign, and 1 indicates malignant. Predictions should be floating point values between 0.0 and 1.0, with 0.5 as a binary decision threshold.\n
Yes : Predictions should be floating point values between 0.0 and 1.0, with 0.5 as a binary decision threshold, which suggests calibration unless dataset is balanced.
\n\nFrom train.csv I can see that target mean is 0.0176xx so most models won't output probabilities that work with a 0.5 threshold...
", "messages": [ { "id": 865006, "postDate": "2020-05-28T10:27:03.520Z", "content": "AUC is a ranking metric and therefore does not need any probability calibration.
\n\nHowever the following can be read in the data section :
\n\n\nYou are predicting a binary target for each image. Your model should predict the probability (floating point) that the lesion in the image is malignant (the target). In the training data, train.csv, the value 0 denotes benign, and 1 indicates malignant. Predictions should be floating point values between 0.0 and 1.0, with 0.5 as a binary decision threshold.\n
Yes : Predictions should be floating point values between 0.0 and 1.0, with 0.5 as a binary decision threshold, which suggests calibration unless dataset is balanced.
\n\nFrom train.csv I can see that target mean is 0.0176xx so most models won't output probabilities that work with a 0.5 threshold...
", "rawMarkdown": "AUC is a ranking metric and therefore does not need any probability calibration.\n\nHowever the following can be read in the data section : \n\n```\nYou are predicting a binary target for each image. Your model should predict the probability (floating point) that the lesion in the image is malignant (the target). In the training data, train.csv, the value 0 denotes benign, and 1 indicates malignant. Predictions should be floating point values between 0.0 and 1.0, with 0.5 as a binary decision threshold.\n```\n\nYes : Predictions should be floating point values between 0.0 and 1.0, **with 0.5 as a binary decision threshold**, which suggests calibration unless dataset is balanced.\n\nFrom train.csv I can see that target mean is 0.0176xx so most models won't output probabilities that work with a 0.5 threshold...", "votes": 8 }, { "id": 865061, "postDate": "2020-05-28T11:09:01.217Z", "content": "I can't see how calibration will matter because the evaluation is only on AUC right?
", "rawMarkdown": "I can't see how calibration will matter because the evaluation is only on AUC right?", "votes": 1, "replies": [ { "id": 865343, "postDate": "2020-05-28T15:16:14.443Z", "content": "Calibration does not matter for AUC, that is why I'm surprised to see this sentence in the data section.
", "rawMarkdown": "Calibration does not matter for AUC, that is why I'm surprised to see this sentence in the data section.", "votes": 1 } ] }, { "id": 865974, "postDate": "2020-05-29T02:36:48.643Z", "content": "It now says this:\nYou are predicting a binary target for each image. Your model should predict the probability (floating point) between 0.0 and 1.0 that the lesion in the image is malignant (the target). In the training data, train.csv, the value 0 denotes benign, and 1 indicates malignant.\nThey removed the line about threshold.
So it was certainly a typo then. Thanks.
", "rawMarkdown": "So it was certainly a typo then. Thanks.", "votes": 1 } ] }, { "id": 865344, "postDate": "2020-05-28T15:17:50.320Z", "content": "Here is the link : https://www.kaggle.com/c/siim-isic-melanoma-classification/data
", "rawMarkdown": "Here is the link : https://www.kaggle.com/c/siim-isic-melanoma-classification/data", "replies": [ { "id": 865762, "postDate": "2020-05-28T21:32:28.980Z", "content": "wait. 0.9x AUC in medicine on the first day... That never happens. I guess the metric is wrong
", "rawMarkdown": "wait. 0.9x AUC in medicine on the first day... That never happens. I guess the metric is wrong" }, { "id": 867096, "postDate": "2020-05-30T02:05:42.807Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 865170, "postDate": "2020-05-28T12:47:34.060Z", "content": "I'm not saying you cannot calibrate, I'm saying it seems you have to despite the fact it's a ranking metric.
", "rawMarkdown": "I'm not saying you cannot calibrate, I'm saying it seems you have to despite the fact it's a ranking metric. " }, { "id": 865105, "postDate": "2020-05-28T11:45:26.237Z", "content": "Why you can't tune threshold of 0.5 based on validation?
", "rawMarkdown": "Why you can't tune threshold of 0.5 based on validation?" } ], "comments": [ { "id": 865061, "author_name": "Josh Myers", "author_url": "", "post_date": "2020-05-28T11:09:01.217000", "content": "I can't see how calibration will matter because the evaluation is only on AUC right?
", "votes": 1, "replies": [ { "id": 865343, "author_name": "olivier", "author_url": "", "post_date": "2020-05-28T15:16:14.443000", "content": "Calibration does not matter for AUC, that is why I'm surprised to see this sentence in the data section.
", "votes": 1, "replies": [] } ] }, { "id": 865974, "author_name": "Abhay Pawar", "author_url": "", "post_date": "2020-05-29T02:36:48.643000", "content": "It now says this:\nYou are predicting a binary target for each image. Your model should predict the probability (floating point) between 0.0 and 1.0 that the lesion in the image is malignant (the target). In the training data, train.csv, the value 0 denotes benign, and 1 indicates malignant.\nThey removed the line about threshold.
So it was certainly a typo then. Thanks.
", "votes": 1, "replies": [] } ] }, { "id": 865344, "author_name": "olivier", "author_url": "", "post_date": "2020-05-28T15:17:50.320000", "content": "Here is the link : https://www.kaggle.com/c/siim-isic-melanoma-classification/data
", "votes": 0, "replies": [ { "id": 865762, "author_name": "Blonde", "author_url": "", "post_date": "2020-05-28T21:32:28.980000", "content": "wait. 0.9x AUC in medicine on the first day... That never happens. I guess the metric is wrong
", "votes": 0, "replies": [] }, { "id": 867096, "author_name": "", "author_url": "", "post_date": "2020-05-30T02:05:42.807000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 865170, "author_name": "olivier", "author_url": "", "post_date": "2020-05-28T12:47:34.060000", "content": "I'm not saying you cannot calibrate, I'm saying it seems you have to despite the fact it's a ranking metric.
", "votes": 0, "replies": [] }, { "id": 865105, "author_name": "Blonde", "author_url": "", "post_date": "2020-05-28T11:45:26.237000", "content": "Why you can't tune threshold of 0.5 based on validation?
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "865006": "AUC is a ranking metric and therefore does not need any probability calibration.\n\nHowever the following can be read in the data section : \n\n```\nYou are predicting a binary target for each image. Your model should predict the probability (floating point) that the lesion in the image is malignant (the target). In the training data, train.csv, the value 0 denotes benign, and 1 indicates malignant. Predictions should be floating point values between 0.0 and 1.0, with 0.5 as a binary decision threshold.\n```\n\nYes : Predictions should be floating point values between 0.0 and 1.0, **with 0.5 as a binary decision threshold**, which suggests calibration unless dataset is balanced.\n\nFrom train.csv I can see that target mean is 0.0176xx so most models won't output probabilities that work with a 0.5 threshold...", "865061": "I can't see how calibration will matter because the evaluation is only on AUC right?", "865974": "It now says this:\n``You are predicting a binary target for each image. Your model should predict the probability (floating point) between 0.0 and 1.0 that the lesion in the image is malignant (the target). In the training data, train.csv, the value 0 denotes benign, and 1 indicates malignant.``\nThey removed the line about threshold.", "865344": "Here is the link : https://www.kaggle.com/c/siim-isic-melanoma-classification/data", "865170": "I'm not saying you cannot calibrate, I'm saying it seems you have to despite the fact it's a ranking metric. ", "865105": "Why you can't tune threshold of 0.5 based on validation?" } }