Hi guys, this has been bugging me for a while and I'm not sure what the answer is.
\nWhat do you use to determine when to early stop & checkpoint your models?
\nSometimes you can get a strange situation where valid loss increases (i.e. overfitting) but AUC is also increasing (noise maybe??). If you stop/checkpoint using valid loss, you could potentially be leaving a few AUC points on the table, but minimising the risk of overfitting and vice versa if you use AUC. What are your thoughts?
", "messages": [ { "id": "1221897", "postDate": "03/01/2021 11:41:07", "content": "Hi guys, this has been bugging me for a while and I'm not sure what the answer is.
\nWhat do you use to determine when to early stop & checkpoint your models?
\nSometimes you can get a strange situation where valid loss increases (i.e. overfitting) but AUC is also increasing (noise maybe??). If you stop/checkpoint using valid loss, you could potentially be leaving a few AUC points on the table, but minimising the risk of overfitting and vice versa if you use AUC. What are your thoughts?
", "rawMarkdown": "Hi guys, this has been bugging me for a while and I'm not sure what the answer is.\n\nWhat do you use to determine when to early stop & checkpoint your models?\n1. Validation loss (minimise)\n2. AUC (maximise)\n\nSometimes you can get a strange situation where valid loss increases (i.e. overfitting) but AUC is also increasing (noise maybe??). If you stop/checkpoint using valid loss, you could potentially be leaving a few AUC points on the table, but minimising the risk of overfitting and vice versa if you use AUC. What are your thoughts?", "votes": null }, { "id": "1221916", "postDate": "03/01/2021 12:01:56", "content": "If you've doubt, why not use both and average their predictions during inference ?
\nIn Pytorch it's easy to add in the training process, in TF you can create custom callback if you use model.fit or add it like in Pytorch if you use custom training loop.
or use SWA and avoid minima from a single epoch!?
", "rawMarkdown": "or use SWA and avoid minima from a single epoch!?", "votes": null }, { "id": "1222023", "postDate": "03/01/2021 13:33:29", "content": "I asked a similar question before, I chose val loss. Then according to my observation, if val loss and val auc rise at the same time, the model is usually not very good and the LB score is not high.
\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/217145
", "rawMarkdown": "I asked a similar question before, I chose val loss. Then according to my observation, if val loss and val auc rise at the same time, the model is usually not very good and the LB score is not high.\n\n[https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/217145](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/217145)", "votes": null }, { "id": "1222048", "postDate": "03/01/2021 14:01:50", "content": "SWA/EMA is good alternative
", "rawMarkdown": "SWA/EMA is good alternative", "votes": null }, { "id": "1222537", "postDate": "03/01/2021 21:50:48", "content": "It is a bit of a difficult question because auc is rather noisy in comparison. ETT - Abnormal only has a handful of positive samples so a single poorly classified sample can cause large variation, but at the same time loss does not perfectly correlate with auc. it is possible to improve loss sometimes simply by calibrating predictions up or down, but gaining nothing in terms of auc. I am tending to focus more on loss because it is much smoother and does generally map to auc.
", "rawMarkdown": "It is a bit of a difficult question because auc is rather noisy in comparison. ETT - Abnormal only has a handful of positive samples so a single poorly classified sample can cause large variation, but at the same time loss does not perfectly correlate with auc. it is possible to improve loss sometimes simply by calibrating predictions up or down, but gaining nothing in terms of auc. I am tending to focus more on loss because it is much smoother and does generally map to auc.", "votes": null }, { "id": "1222538", "postDate": "03/01/2021 21:50:54", "content": "Also tried averaging checkpoints for the best auc and best loss and found no real performance gain there
", "rawMarkdown": "Also tried averaging checkpoints for the best auc and best loss and found no real performance gain there", "votes": null }, { "id": "1224440", "postDate": "03/02/2021 18:14:15", "content": "Yes, I had the same question too so I did an experiment by submitting lowest val loss and highest val auc (same fold but different epoch). lowest val loss always give me better LB result, so i'm not sure which one to trust (People always say trust your CV).
\nI also noticed in @yasufuminakama notebooks. He saved both best loss and best accuracy epochs but in the training he picked best loss epoch for each stage.
", "rawMarkdown": "Yes, I had the same question too so I did an experiment by submitting lowest val loss and highest val auc (same fold but different epoch). lowest val loss always give me better LB result, so i'm not sure which one to trust (People always say trust your CV).\n\nI also noticed in @yasufuminakama notebooks. He saved both best loss and best accuracy epochs but in the training he picked best loss epoch for each stage.", "votes": null } ], "comments": [ { "id": 1221916, "author_name": "serigne", "author_url": "", "post_date": "03/01/2021 12:01:56", "content": "If you've doubt, why not use both and average their predictions during inference ?
\nIn Pytorch it's easy to add in the training process, in TF you can create custom callback if you use model.fit or add it like in Pytorch if you use custom training loop.
or use SWA and avoid minima from a single epoch!?
", "votes": null, "replies": [] }, { "id": 1222048, "author_name": "serigne", "author_url": "", "post_date": "03/01/2021 14:01:50", "content": "SWA/EMA is good alternative
", "votes": null, "replies": [] } ] }, { "id": 1222023, "author_name": "h053473666", "author_url": "", "post_date": "03/01/2021 13:33:29", "content": "I asked a similar question before, I chose val loss. Then according to my observation, if val loss and val auc rise at the same time, the model is usually not very good and the LB score is not high.
\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/217145
", "votes": null, "replies": [] }, { "id": 1222537, "author_name": "ryches", "author_url": "", "post_date": "03/01/2021 21:50:48", "content": "It is a bit of a difficult question because auc is rather noisy in comparison. ETT - Abnormal only has a handful of positive samples so a single poorly classified sample can cause large variation, but at the same time loss does not perfectly correlate with auc. it is possible to improve loss sometimes simply by calibrating predictions up or down, but gaining nothing in terms of auc. I am tending to focus more on loss because it is much smoother and does generally map to auc.
", "votes": null, "replies": [ { "id": 1222538, "author_name": "ryches", "author_url": "", "post_date": "03/01/2021 21:50:54", "content": "Also tried averaging checkpoints for the best auc and best loss and found no real performance gain there
", "votes": null, "replies": [] } ] }, { "id": 1224440, "author_name": "tom88jerry", "author_url": "", "post_date": "03/02/2021 18:14:15", "content": "Yes, I had the same question too so I did an experiment by submitting lowest val loss and highest val auc (same fold but different epoch). lowest val loss always give me better LB result, so i'm not sure which one to trust (People always say trust your CV).
\nI also noticed in @yasufuminakama notebooks. He saved both best loss and best accuracy epochs but in the training he picked best loss epoch for each stage.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1221897": "Hi guys, this has been bugging me for a while and I'm not sure what the answer is.\n\nWhat do you use to determine when to early stop & checkpoint your models?\n1. Validation loss (minimise)\n2. AUC (maximise)\n\nSometimes you can get a strange situation where valid loss increases (i.e. overfitting) but AUC is also increasing (noise maybe??). If you stop/checkpoint using valid loss, you could potentially be leaving a few AUC points on the table, but minimising the risk of overfitting and vice versa if you use AUC. What are your thoughts?", "1221916": "If you've doubt, why not use both and average their predictions during inference ? \n\nIn Pytorch it's easy to add in the training process, in TF you can create custom callback if you use `model.fit` or add it like in Pytorch if you use custom training loop.", "1222001": "or use SWA and avoid minima from a single epoch!?", "1222023": "I asked a similar question before, I chose val loss. Then according to my observation, if val loss and val auc rise at the same time, the model is usually not very good and the LB score is not high.\n\n[https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/217145](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/217145)", "1222048": "SWA/EMA is good alternative", "1222537": "It is a bit of a difficult question because auc is rather noisy in comparison. ETT - Abnormal only has a handful of positive samples so a single poorly classified sample can cause large variation, but at the same time loss does not perfectly correlate with auc. it is possible to improve loss sometimes simply by calibrating predictions up or down, but gaining nothing in terms of auc. I am tending to focus more on loss because it is much smoother and does generally map to auc.", "1222538": "Also tried averaging checkpoints for the best auc and best loss and found no real performance gain there", "1224440": "Yes, I had the same question too so I did an experiment by submitting lowest val loss and highest val auc (same fold but different epoch). lowest val loss always give me better LB result, so i'm not sure which one to trust (People always say trust your CV).\n\nI also noticed in @yasufuminakama notebooks. He saved both best loss and best accuracy epochs but in the training he picked best loss epoch for each stage." }, "source": "meta" }