While training , i found that , there were many times that the val_loss decreased , but the val_acc also decreased instead of increasing and in some cases , the val_acc increases where val_loss also decreases.
\nCan anyone help me in understanding this.
\nI am a bit confused on this.
While training , i found that , there were many times that the val_loss decreased , but the val_acc also decreased instead of increasing and in some cases , the val_acc increases where val_loss also decreases.
\nCan anyone help me in understanding this.
\nI am a bit confused on this.
The metric is accuracy, why do you care about \"val_loss\" ?
", "rawMarkdown": "The metric is accuracy, why do you care about \"val_loss\" ?", "votes": null }, { "id": "1165907", "postDate": "01/23/2021 10:03:37", "content": "the reason i care about this is that ,
\nlower val_loss can perform much better on the private dataset .
\nI am not sure about this , i just think so.
\nwhat are your views on it ?
Couldn't you technically get a high val_loss but also high val_acc by pure chance with a random model that just happens to guess the labels correctly? Whereas a smaller val_loss would imply the model is more \"confident\" in its prediction?
", "rawMarkdown": "Couldn't you technically get a high val_loss but also high val_acc by pure chance with a random model that just happens to guess the labels correctly? Whereas a smaller val_loss would imply the model is more \"confident\" in its prediction?", "votes": null }, { "id": "1166204", "postDate": "01/23/2021 13:01:01", "content": "Hi,
\nWhile I was training a vgg16 model and analysing the val_loss and val_acc, I found the minimum val_loss without model getting overfitted as the best moment to stop the training. The prediction from these models was a bit more promising compared to the one with more val_acc. Maybe the link below would help you to get a notion on the interpretation of loss.
\nshorturl.at/lyAP3
Validation loss refers to how sure your model is about its predictions. Validation accuracy refers to how accurate it is. If you have val acc and val loss decreasing, it means that your model is becoming more sure of wrong predictions.
\nTLDR; Val acc > val loss
", "rawMarkdown": "Validation loss refers to how sure your model is about its predictions. Validation accuracy refers to how accurate it is. If you have val acc and val loss decreasing, it means that your model is becoming more sure of wrong predictions.\n\nTLDR; Val acc > val loss", "votes": null }, { "id": "1166616", "postDate": "01/23/2021 18:05:48", "content": "This is the same reason i created this topic , i was too getting similar results in some previous competition and hence wanted to confirm from some masters here.
\nThank You for sharing .
", "rawMarkdown": "This is the same reason i created this topic , i was too getting similar results in some previous competition and hence wanted to confirm from some masters here.\n\nThank You for sharing .", "votes": null }, { "id": "1166618", "postDate": "01/23/2021 18:07:58", "content": "Thank You for this explanation @junyingsg .
", "rawMarkdown": "Thank You for this explanation @junyingsg .", "votes": null }, { "id": "1166645", "postDate": "01/23/2021 18:26:11", "content": "If even a high score on public LB can't guarantee to perform similarly on private data, then how can you expect your val_acc (or low val_loss) to reflect on private data?
", "rawMarkdown": "If even a high score on public LB can't guarantee to perform similarly on private data, then how can you expect your val_acc (or low val_loss) to reflect on private data?", "votes": null }, { "id": "1166649", "postDate": "01/23/2021 18:29:39", "content": "i can't say about val_acc to perform better or worse.
\nBut lower val_loss can provide a kind of confidence , that yes I am going to work better with predictions.
@junyingsg Is right. Another thing to think about is that accuracy will change only when a discreet change in predictions happens, but loss reflects continuous changes in model parameters. The small model parameter changes may switch a couple predictions the other way, in effect having a big effect on accuracy. But that could just be a temporary model state during the training process.
", "rawMarkdown": "junyingsg Is right. Another thing to think about is that accuracy will change only when a discreet change in predictions happens, but loss reflects continuous changes in model parameters. The small model parameter changes may switch a couple predictions the other way, in effect having a big effect on accuracy. But that could just be a temporary model state during the training process.", "votes": null }, { "id": "1167307", "postDate": "01/24/2021 07:04:36", "content": "The competition metric is of course accuracy. So, one night be tempted to rely solely on validation accuracy. However, it's a very noisy low information metric (just 0 or 1), where small changes can just be by chance. On the other hand, I believe there's supposed to be a phenomenon where you overfit just a little bit and it can make accuracy better, but not binary cross-entropy. Some other loss functions might help there (e.g. label smoothed cross entropy, focal loss), while also helping with noisy labels.
", "rawMarkdown": "The competition metric is of course accuracy. So, one night be tempted to rely solely on validation accuracy. However, it's a very noisy low information metric (just 0 or 1), where small changes can just be by chance. On the other hand, I believe there's supposed to be a phenomenon where you overfit just a little bit and it can make accuracy better, but not binary cross-entropy. Some other loss functions might help there (e.g. label smoothed cross entropy, focal loss), while also helping with noisy labels.", "votes": null }, { "id": "1167399", "postDate": "01/24/2021 08:27:57", "content": "I agree.
\nbut will it provide any kind of confidence or estimate that how our model is going to perform on private leaderboard ?
Did you tried it with focal loss ?
\nI don't think it would be beneficial, although its just my point of view , how does it performs is still based on experiment.
I've not seen a real difference between focal loss and label smoothed cross entropy once I had tweaked the parameters of both, but for me both seemed to lead to better CV accuracy and slightly more closely correlated CV loss/CV accuracy than binary cross entropy. Not a huge effect, but I'm inclined to think it's a real effect, because it makes sense on theoretical grounds.
", "rawMarkdown": "I've not seen a real difference between focal loss and label smoothed cross entropy once I had tweaked the parameters of both, but for me both seemed to lead to better CV accuracy and slightly more closely correlated CV loss/CV accuracy than binary cross entropy. Not a huge effect, but I'm inclined to think it's a real effect, because it makes sense on theoretical grounds.", "votes": null }, { "id": "1167790", "postDate": "01/24/2021 13:34:21", "content": "Nicely experimented.
\nThank You for sharing this.
Also keep in mind that our dataset is imbalanced and accuracy is a very bad metric for imbalanced datasets.
", "rawMarkdown": "Also keep in mind that our dataset is imbalanced and accuracy is a very bad metric for imbalanced datasets.", "votes": null }, { "id": "1168835", "postDate": "01/25/2021 07:40:41", "content": "Yeah that correct. so instead of val_acc should we give more focus to any other metric also ? like f1_score …etc.
", "rawMarkdown": "Yeah that correct. so instead of val_acc should we give more focus to any other metric also ? like f1_score ...etc.", "votes": null }, { "id": "1168915", "postDate": "01/25/2021 08:49:44", "content": "While it's a bad metric in a sense for an imbalanced dataset, it is the competition metric. Some of the ways in which it is a bad metric really don't matter for the purpose of the competition. E.g. we all know you can do things like predict the majority class when there's one class that has way more data and get a high accuracy. You can consider that misleading, but that's not a problem here (we really don't care about that) and metrics that overcome that (e.g. F1-score, log-loss) may not all correlate well with the private LB accuracy.
\nIn the end, the private LB accuracy is what counts, the question really is what validation metric you can assess will correlate with it the most reliably. So, the question is whether accuracy is too unstable/too noisy (being a crude 0 vs. 1 dichotomization that can be heavily influenced by one example having a tiny bit higher or lower predicted probability) and whether some other metrics are more stably predicting private LB accuracy despite being a different metric.
", "rawMarkdown": "While it's a bad metric in a sense for an imbalanced dataset, it is the competition metric. Some of the ways in which it is a bad metric really don't matter for the purpose of the competition. E.g. we all know you can do things like predict the majority class when there's one class that has way more data and get a high accuracy. You can consider that misleading, but that's not a problem here (we really don't care about that) and metrics that overcome that (e.g. F1-score, log-loss) may not all correlate well with the private LB accuracy. \n\nIn the end, the private LB accuracy is what counts, the question really is what validation metric you can assess will correlate with it the most reliably. So, the question is whether accuracy is too unstable/too noisy (being a crude 0 vs. 1 dichotomization that can be heavily influenced by one example having a tiny bit higher or lower predicted probability) and whether some other metrics are more stably predicting private LB accuracy despite being a different metric.", "votes": null }, { "id": "1170362", "postDate": "01/26/2021 07:04:21", "content": "what have you found more useful for this competition?
\ncan you please share , if you don't have any problem.
The metric is accuracy, why do you care about \"val_loss\" ?
", "votes": null, "replies": [ { "id": 1165907, "author_name": "prashantarorat", "author_url": "", "post_date": "01/23/2021 10:03:37", "content": "the reason i care about this is that ,
\nlower val_loss can perform much better on the private dataset .
\nI am not sure about this , i just think so.
\nwhat are your views on it ?
If even a high score on public LB can't guarantee to perform similarly on private data, then how can you expect your val_acc (or low val_loss) to reflect on private data?
", "votes": null, "replies": [] }, { "id": 1166649, "author_name": "prashantarorat", "author_url": "", "post_date": "01/23/2021 18:29:39", "content": "i can't say about val_acc to perform better or worse.
\nBut lower val_loss can provide a kind of confidence , that yes I am going to work better with predictions.
Couldn't you technically get a high val_loss but also high val_acc by pure chance with a random model that just happens to guess the labels correctly? Whereas a smaller val_loss would imply the model is more \"confident\" in its prediction?
", "votes": null, "replies": [ { "id": 1166204, "author_name": "prashantarorat", "author_url": "", "post_date": "01/23/2021 13:01:01", "content": "Hi,
\nWhile I was training a vgg16 model and analysing the val_loss and val_acc, I found the minimum val_loss without model getting overfitted as the best moment to stop the training. The prediction from these models was a bit more promising compared to the one with more val_acc. Maybe the link below would help you to get a notion on the interpretation of loss.
\nshorturl.at/lyAP3
This is the same reason i created this topic , i was too getting similar results in some previous competition and hence wanted to confirm from some masters here.
\nThank You for sharing .
", "votes": null, "replies": [] } ] }, { "id": 1166559, "author_name": "junyingsg", "author_url": "", "post_date": "01/23/2021 17:24:17", "content": "Validation loss refers to how sure your model is about its predictions. Validation accuracy refers to how accurate it is. If you have val acc and val loss decreasing, it means that your model is becoming more sure of wrong predictions.
\nTLDR; Val acc > val loss
", "votes": null, "replies": [ { "id": 1166618, "author_name": "prashantarorat", "author_url": "", "post_date": "01/23/2021 18:07:58", "content": "Thank You for this explanation @junyingsg .
", "votes": null, "replies": [] }, { "id": 1166772, "author_name": "gurharkhalsa", "author_url": "", "post_date": "01/23/2021 20:19:36", "content": "@junyingsg Is right. Another thing to think about is that accuracy will change only when a discreet change in predictions happens, but loss reflects continuous changes in model parameters. The small model parameter changes may switch a couple predictions the other way, in effect having a big effect on accuracy. But that could just be a temporary model state during the training process.
", "votes": null, "replies": [] }, { "id": 1167399, "author_name": "prashantarorat", "author_url": "", "post_date": "01/24/2021 08:27:57", "content": "I agree.
\nbut will it provide any kind of confidence or estimate that how our model is going to perform on private leaderboard ?
The competition metric is of course accuracy. So, one night be tempted to rely solely on validation accuracy. However, it's a very noisy low information metric (just 0 or 1), where small changes can just be by chance. On the other hand, I believe there's supposed to be a phenomenon where you overfit just a little bit and it can make accuracy better, but not binary cross-entropy. Some other loss functions might help there (e.g. label smoothed cross entropy, focal loss), while also helping with noisy labels.
", "votes": null, "replies": [ { "id": 1167403, "author_name": "prashantarorat", "author_url": "", "post_date": "01/24/2021 08:29:57", "content": "Did you tried it with focal loss ?
\nI don't think it would be beneficial, although its just my point of view , how does it performs is still based on experiment.
I've not seen a real difference between focal loss and label smoothed cross entropy once I had tweaked the parameters of both, but for me both seemed to lead to better CV accuracy and slightly more closely correlated CV loss/CV accuracy than binary cross entropy. Not a huge effect, but I'm inclined to think it's a real effect, because it makes sense on theoretical grounds.
", "votes": null, "replies": [] }, { "id": 1167790, "author_name": "prashantarorat", "author_url": "", "post_date": "01/24/2021 13:34:21", "content": "Nicely experimented.
\nThank You for sharing this.
Also keep in mind that our dataset is imbalanced and accuracy is a very bad metric for imbalanced datasets.
", "votes": null, "replies": [] }, { "id": 1168835, "author_name": "prashantarorat", "author_url": "", "post_date": "01/25/2021 07:40:41", "content": "Yeah that correct. so instead of val_acc should we give more focus to any other metric also ? like f1_score …etc.
", "votes": null, "replies": [] }, { "id": 1168915, "author_name": "bjoernholzhauer", "author_url": "", "post_date": "01/25/2021 08:49:44", "content": "While it's a bad metric in a sense for an imbalanced dataset, it is the competition metric. Some of the ways in which it is a bad metric really don't matter for the purpose of the competition. E.g. we all know you can do things like predict the majority class when there's one class that has way more data and get a high accuracy. You can consider that misleading, but that's not a problem here (we really don't care about that) and metrics that overcome that (e.g. F1-score, log-loss) may not all correlate well with the private LB accuracy.
\nIn the end, the private LB accuracy is what counts, the question really is what validation metric you can assess will correlate with it the most reliably. So, the question is whether accuracy is too unstable/too noisy (being a crude 0 vs. 1 dichotomization that can be heavily influenced by one example having a tiny bit higher or lower predicted probability) and whether some other metrics are more stably predicting private LB accuracy despite being a different metric.
", "votes": null, "replies": [] }, { "id": 1170362, "author_name": "prashantarorat", "author_url": "", "post_date": "01/26/2021 07:04:21", "content": "what have you found more useful for this competition?
\ncan you please share , if you don't have any problem.