I was training cnns over a long time in this competition throughout the run i noticed that the loss often does not correlate with the auc like when loss decreases the auc does too and same for increasing . Continuing like this means that model tries to reduce the loss and thus also decreasing the AUC . Any solution or thoughts ?
", "messages": [ { "id": 1518522, "postDate": "2021-09-20T17:57:34.033Z", "content": "Depends on the loss you are using. If this is CE without any calibration, it penalizes predictions distant from 0 or 1. Auc-roc can be equal to 1 even if the probabilities are spread among some little distance, like for example between 0.5 and 0.6. What matters here is just the class separability. There are also loss functions that directly optimize auc-roc - you can see a better correlation between the loss and validation auc using them!
", "rawMarkdown": "Depends on the loss you are using. If this is CE without any calibration, it penalizes predictions distant from 0 or 1. Auc-roc can be equal to 1 even if the probabilities are spread among some little distance, like for example between 0.5 and 0.6. What matters here is just the class separability. There are also loss functions that directly optimize auc-roc - you can see a better correlation between the loss and validation auc using them! ", "votes": 5, "replies": [ { "id": 1518787, "postDate": "2021-09-21T04:33:44.600Z", "content": "i implemented something like shifting the predictions to mean of 0.5 and increase the variance , it seems to work better now
", "rawMarkdown": "i implemented something like shifting the predictions to mean of 0.5 and increase the variance , it seems to work better now" }, { "id": 1529291, "postDate": "2021-09-30T09:16:56.630Z", "content": "Could you tell me which loss functions can directly optimize arc-roc?
", "rawMarkdown": "Could you tell me which loss functions can directly optimize arc-roc?" }, { "id": 1529816, "postDate": "2021-09-30T17:09:03.800Z", "content": "There are several implementations, for example:
\nPytorch roc-star
\nTensorflow Tflearn
There is also a library libAUC
\nIts source code is not published yet (however, it is possible to easily extract it from the wheel available in this dataset)
I was training cnns over a long time in this competition throughout the run i noticed that the loss often does not correlate with the auc like when loss decreases the auc does too and same for increasing . Continuing like this means that model tries to reduce the loss and thus also decreasing the AUC . Any solution or thoughts ?
", "rawMarkdown": "I was training cnns over a long time in this competition throughout the run i noticed that the loss often does not correlate with the auc like when loss decreases the auc does too and same for increasing . Continuing like this means that model tries to reduce the loss and thus also decreasing the AUC . Any solution or thoughts ?", "votes": 1 }, { "id": 1518390, "postDate": "2021-09-20T15:51:39.777Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 1518401, "postDate": "2021-09-20T15:58:32.833Z", "content": "hi there , i am not talking about cv lb here , it has nothing to do with the topic
", "rawMarkdown": "hi there , i am not talking about cv lb here , it has nothing to do with the topic\n" }, { "id": 1518434, "postDate": "2021-09-20T16:28:07.077Z", "content": "sorry i wrote lb by mistake
", "rawMarkdown": "sorry i wrote lb by mistake" } ] } ], "comments": [ { "id": 1518522, "author_name": "Michał Choiński", "author_url": "", "post_date": "2021-09-20T17:57:34.033000", "content": "Depends on the loss you are using. If this is CE without any calibration, it penalizes predictions distant from 0 or 1. Auc-roc can be equal to 1 even if the probabilities are spread among some little distance, like for example between 0.5 and 0.6. What matters here is just the class separability. There are also loss functions that directly optimize auc-roc - you can see a better correlation between the loss and validation auc using them!
", "votes": 5, "replies": [ { "id": 1518787, "author_name": "Sayantan Mazumdar", "author_url": "", "post_date": "2021-09-21T04:33:44.600000", "content": "i implemented something like shifting the predictions to mean of 0.5 and increase the variance , it seems to work better now
", "votes": 0, "replies": [] }, { "id": 1529291, "author_name": "Nhan Ho", "author_url": "", "post_date": "2021-09-30T09:16:56.630000", "content": "Could you tell me which loss functions can directly optimize arc-roc?
", "votes": 0, "replies": [] }, { "id": 1529816, "author_name": "Michał Choiński", "author_url": "", "post_date": "2021-09-30T17:09:03.800000", "content": "There are several implementations, for example:
\nPytorch roc-star
\nTensorflow Tflearn
There is also a library libAUC
\nIts source code is not published yet (however, it is possible to easily extract it from the wheel available in this dataset)
hi there , i am not talking about cv lb here , it has nothing to do with the topic
", "votes": 0, "replies": [] }, { "id": 1518434, "author_name": "Sayantan Mazumdar", "author_url": "", "post_date": "2021-09-20T16:28:07.077000", "content": "sorry i wrote lb by mistake
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1518522": "Depends on the loss you are using. If this is CE without any calibration, it penalizes predictions distant from 0 or 1. Auc-roc can be equal to 1 even if the probabilities are spread among some little distance, like for example between 0.5 and 0.6. What matters here is just the class separability. There are also loss functions that directly optimize auc-roc - you can see a better correlation between the loss and validation auc using them! ", "1518313": "I was training cnns over a long time in this competition throughout the run i noticed that the loss often does not correlate with the auc like when loss decreases the auc does too and same for increasing . Continuing like this means that model tries to reduce the loss and thus also decreasing the AUC . Any solution or thoughts ?", "1518390": "" } }