Better models with lower validation loss are scoring less in LB. Any ideas?
", "messages": [ { "id": 1000333, "postDate": "2020-09-06T13:22:13.863Z", "content": "Better models with lower validation loss are scoring less in LB. Any ideas?
", "rawMarkdown": "Better models with lower validation loss are scoring less in LB. Any ideas?", "votes": 7 }, { "id": 1000369, "postDate": "2020-09-06T13:41:41.303Z", "content": "Main idea - Test is completely different from train
\nCompute a bunch of metrics and find the best correlated one
Yah I have made some trial and error and computed some relevant metrics. But with those metrics, I'm stuck with the score and unable to improve.
\nTried differently models with relevant metrics but the scores in LB is decreasing very much. Whereas in training the validation scores are good for those models.
", "rawMarkdown": "Hi @vladimirsydor \n\nYah I have made some trial and error and computed some relevant metrics. But with those metrics, I'm stuck with the score and unable to improve.\n\nTried differently models with relevant metrics but the scores in LB is decreasing very much. Whereas in training the validation scores are good for those models. " } ] }, { "id": 1000752, "postDate": "2020-09-06T19:10:07.883Z", "content": "I have the same experience and did not find out why. I will choose the highest LB submission and the highest CV submission and try my luck. I have seen others also having the same issue without any solutions.
", "rawMarkdown": "I have the same experience and did not find out why. I will choose the highest LB submission and the highest CV submission and try my luck. I have seen others also having the same issue without any solutions.", "votes": 1, "replies": [ { "id": 1001062, "postDate": "2020-09-07T03:43:35.507Z", "content": "Yah lets hope for the best👍
", "rawMarkdown": "Yah lets hope for the best👍" } ] }, { "id": 1000628, "postDate": "2020-09-06T17:03:08.603Z", "content": "Hi! @jagadish13, have you try to compare the distributions of the variables on Test vs Train sets, is there any significant difference.
", "rawMarkdown": "Hi! @jagadish13, have you try to compare the distributions of the variables on Test vs Train sets, is there any significant difference.", "votes": 1, "replies": [ { "id": 1000689, "postDate": "2020-09-06T17:49:34.270Z", "content": "Hi @cv13j0
\nThis is my first competition with audio, so I'm learning a lot from public kernels and discussions.
\nAs @vladimirsydor said the test is different from the train dataset
Thanks, @jagadish13, like @vladimirsydor, mentioned, I will identify what feature distributions differ the most and rely less on them; if the target shows the most significant difference, probably rely on my CV strategy
", "rawMarkdown": "Thanks, @jagadish13, like @vladimirsydor, mentioned, I will identify what feature distributions differ the most and rely less on them; if the target shows the most significant difference, probably rely on my CV strategy", "votes": 1 } ] }, { "id": 1001475, "postDate": "2020-09-07T11:01:23.293Z", "content": "There can sometimes be a divergence between metrics and loss. Check if your metric is improving
", "rawMarkdown": "There can sometimes be a divergence between metrics and loss. Check if your metric is improving", "votes": 2, "replies": [ { "id": 1001724, "postDate": "2020-09-07T14:16:27.320Z", "content": "With respect to my validation score, It is very low for a long time then rises and again falls
", "rawMarkdown": "With respect to my validation score, It is very low for a long time then rises and again falls", "votes": 1 }, { "id": 1001726, "postDate": "2020-09-07T14:17:35.803Z", "content": "The weird thing is models with a lower validation loss and higher validation score is performing less in LB.
", "rawMarkdown": "The weird thing is models with a lower validation loss and higher validation score is performing less in LB.", "votes": 1 }, { "id": 1001748, "postDate": "2020-09-07T14:34:02.107Z", "content": "Be careful not to overfit your training set, you may risk doing bad on test set
", "rawMarkdown": "Be careful not to overfit your training set, you may risk doing bad on test set" }, { "id": 1001976, "postDate": "2020-09-07T17:58:43.617Z", "content": "But when using a metric you are preventing to chose the overfitted model. I have exactly the same scenario. A high CV results in a low LB score, short trained models (e.g. 25 epochs) result in higher public LB scores. I'm definetly sure, that there are no errors in evaluation nor in training but I cant figure out where this unusual difference comes from.
", "rawMarkdown": "But when using a metric you are preventing to chose the overfitted model. I have exactly the same scenario. A high CV results in a low LB score, short trained models (e.g. 25 epochs) result in higher public LB scores. I'm definetly sure, that there are no errors in evaluation nor in training but I cant figure out where this unusual difference comes from." }, { "id": 1002091, "postDate": "2020-09-07T20:09:59.407Z", "content": "Yes, you have overfit your training set, that's why short trained models result in better LB score. A high CV just means you can do well on your training set, unfortunately it doesn't guarantee a good LB score in this competition
", "rawMarkdown": "Yes, you have overfit your training set, that's why short trained models result in better LB score. A high CV just means you can do well on your training set, unfortunately it doesn't guarantee a good LB score in this competition", "votes": 2 }, { "id": 1002096, "postDate": "2020-09-07T20:13:36.893Z", "content": "By my investigations a shorter trained model just delievers more nocall predictions and thus it comes very close to the all nocall submission score of 0.544 thats why it gets better when its shorter trained. A longer trained model has an answer for everything but these answers are seemingly completly wrong. I'm using validation to ensure to pick the best model during training. I dont believe that its overfitted.
How do you compute your validation score?
", "rawMarkdown": "How do you compute your validation score?", "replies": [ { "id": 1000681, "postDate": "2020-09-06T17:40:49.877Z", "content": "Hi @cpmpml
\nFor the val_score i'm using something similar to the class Fitter from the kernel:
\nhttps://www.kaggle.com/chanhu/training-bird-simple-baseline/data
With few modifications according to my model.
", "rawMarkdown": "Hi @cpmpml \nFor the val_score i'm using something similar to the `class Fitter` from the kernel:\nhttps://www.kaggle.com/chanhu/training-bird-simple-baseline/data\n\nWith few modifications according to my model." }, { "id": 1000685, "postDate": "2020-09-06T17:44:01.667Z", "content": "Did you try any ensemble? @cpmpml
\nI have tried ensembling models with different params but stuck with the same score. It's scoring less actually.
I will not spend time analyzing that code to get n answer to my question… Good luck then on finding why CV isn't reliable.
\nTo answer your question I have not ensembled so far. My LB score comes from a single model.
", "rawMarkdown": "I will not spend time analyzing that code to get n answer to my question... Good luck then on finding why CV isn't reliable.\n\nTo answer your question I have not ensembled so far. My LB score comes from a single model.\n\n", "votes": 2 } ] }, { "id": 1000645, "postDate": "2020-09-06T17:11:38.230Z", "content": "more datasets and overfitting. please have a look into it.
", "rawMarkdown": "more datasets and overfitting. please have a look into it. ", "votes": -3 }, { "id": 1000465, "postDate": "2020-09-06T14:44:59.157Z", "content": "Hey,
\nMaybe you can take look at the overfitting and validation set of the data.
Just to exclude a thought, without sounding paranoid, it can not be about leakage, in the data or how one splits the folds? How is the experience with Stratified Kfold, is it waterproof? Have a vague memory that there may be problems there.
", "rawMarkdown": "Just to exclude a thought, without sounding paranoid, it can not be about leakage, in the data or how one splits the folds? How is the experience with Stratified Kfold, is it waterproof? Have a vague memory that there may be problems there.", "replies": [ { "id": 1001979, "postDate": "2020-09-07T18:00:07.060Z", "content": "How did you solve your problem? I remember you opening an equal discussion about this issue. Regarding your competition ranking, you should have solved it or?
", "rawMarkdown": "How did you solve your problem? I remember you opening an equal discussion about this issue. Regarding your competition ranking, you should have solved it or?" }, { "id": 1002040, "postDate": "2020-09-07T19:08:16.197Z", "content": "Not really with you, have I written about kfold before, when, what and where? There is some writing about Stratified Kfold on Kaggle regarding leakage, hence the question of whether it can apply here, or maybe I have go it all wrong.
\n\"Regarding your competition ranking, you should have solved it or?\" After 10 month with Kaggle I'm more a learner and questioner than a knower, every day new questions in this area, making it all fun to continue.
You mentioned the topic problem on: https://www.kaggle.com/c/birdsong-recognition/discussion/178753
\n\"With the same training setup, picking the best val checkpoint gets worse Public LB.
\nMore training epochs with higher val loss beats a lower val loss with less epochs.
\nEven with same epochs and higher val loss can get a better Public LBs score.
\nHave also noticed that SWA from best ckps doesn't work well.\"
Thats exactly the same problem that I still have. I'm not talking about KFolds, I mean the topic question. Do you have any results to that issue?
", "rawMarkdown": "You mentioned the topic problem on: https://www.kaggle.com/c/birdsong-recognition/discussion/178753\n\n\"With the same training setup, picking the best val checkpoint gets worse Public LB.\nMore training epochs with higher val loss beats a lower val loss with less epochs.\nEven with same epochs and higher val loss can get a better Public LBs score.\nHave also noticed that SWA from best ckps doesn't work well.\"\n\nThats exactly the same problem that I still have. I'm not talking about KFolds, I mean the topic question. Do you have any results to that issue?" }, { "id": 1002069, "postDate": "2020-09-07T19:38:10.430Z", "content": "Same situation as then, but have now implemented some more metrics, see if that can help. Seems many of us have the same problem, hence my comment, with the hope of a reflection about StKFold which is related to the topic.
", "rawMarkdown": "Same situation as then, but have now implemented some more metrics, see if that can help. Seems many of us have the same problem, hence my comment, with the hope of a reflection about StKFold which is related to the topic." }, { "id": 1002073, "postDate": "2020-09-07T19:41:46.063Z", "content": "Alright, but you have a good LB rank I tought you might not have the problem anymore.
\nI'm using StratifiedKFold without shuffeling and the problem still exists for me.
", "rawMarkdown": "Alright, but you have a good LB rank I tought you might not have the problem anymore.\n\nI'm using StratifiedKFold without shuffeling and the problem still exists for me." } ] }, { "id": 1001148, "postDate": "2020-09-07T05:15:35.357Z", "content": "What is your local val-acc score?
", "rawMarkdown": "What is your local val-acc score?" }, { "id": 1000347, "postDate": "2020-09-06T13:29:52.180Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 1000362, "postDate": "2020-09-06T13:37:08.170Z", "content": "Hi @samkrishna
\nI haven't faced such an issue.
@samkrishna , you should restart the notebook and run the whole notebook codes again. Have good internet connection. if issue unresolved then there may be some issues in codes or datasets, correct it. I have faced such issues. I love to hear if issues resolved
", "rawMarkdown": "@samkrishna , you should restart the notebook and run the whole notebook codes again. Have good internet connection. if issue unresolved then there may be some issues in codes or datasets, correct it. I have faced such issues. I love to hear if issues resolved", "votes": -1 }, { "id": 1001090, "postDate": "2020-09-07T04:15:17.300Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 1000369, "author_name": "Volodymyr", "author_url": "", "post_date": "2020-09-06T13:41:41.303000", "content": "Main idea - Test is completely different from train
\nCompute a bunch of metrics and find the best correlated one
Yah I have made some trial and error and computed some relevant metrics. But with those metrics, I'm stuck with the score and unable to improve.
\nTried differently models with relevant metrics but the scores in LB is decreasing very much. Whereas in training the validation scores are good for those models.
", "votes": 0, "replies": [] } ] }, { "id": 1000752, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-09-06T19:10:07.883000", "content": "I have the same experience and did not find out why. I will choose the highest LB submission and the highest CV submission and try my luck. I have seen others also having the same issue without any solutions.
", "votes": 1, "replies": [ { "id": 1001062, "author_name": "Jagadish Sivakumar", "author_url": "", "post_date": "2020-09-07T03:43:35.507000", "content": "Yah lets hope for the best👍
", "votes": 0, "replies": [] } ] }, { "id": 1000628, "author_name": "C4rl05/V", "author_url": "", "post_date": "2020-09-06T17:03:08.603000", "content": "Hi! @jagadish13, have you try to compare the distributions of the variables on Test vs Train sets, is there any significant difference.
", "votes": 1, "replies": [ { "id": 1000689, "author_name": "Jagadish Sivakumar", "author_url": "", "post_date": "2020-09-06T17:49:34.270000", "content": "Hi @cv13j0
\nThis is my first competition with audio, so I'm learning a lot from public kernels and discussions.
\nAs @vladimirsydor said the test is different from the train dataset
Thanks, @jagadish13, like @vladimirsydor, mentioned, I will identify what feature distributions differ the most and rely less on them; if the target shows the most significant difference, probably rely on my CV strategy
", "votes": 1, "replies": [] } ] }, { "id": 1001475, "author_name": "Alan Choon", "author_url": "", "post_date": "2020-09-07T11:01:23.293000", "content": "There can sometimes be a divergence between metrics and loss. Check if your metric is improving
", "votes": 2, "replies": [ { "id": 1001724, "author_name": "Jagadish Sivakumar", "author_url": "", "post_date": "2020-09-07T14:16:27.320000", "content": "With respect to my validation score, It is very low for a long time then rises and again falls
", "votes": 1, "replies": [] }, { "id": 1001726, "author_name": "Jagadish Sivakumar", "author_url": "", "post_date": "2020-09-07T14:17:35.803000", "content": "The weird thing is models with a lower validation loss and higher validation score is performing less in LB.
", "votes": 1, "replies": [] }, { "id": 1001748, "author_name": "Jie Lu", "author_url": "", "post_date": "2020-09-07T14:34:02.107000", "content": "Be careful not to overfit your training set, you may risk doing bad on test set
", "votes": 0, "replies": [] }, { "id": 1001976, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-09-07T17:58:43.617000", "content": "But when using a metric you are preventing to chose the overfitted model. I have exactly the same scenario. A high CV results in a low LB score, short trained models (e.g. 25 epochs) result in higher public LB scores. I'm definetly sure, that there are no errors in evaluation nor in training but I cant figure out where this unusual difference comes from.
", "votes": 0, "replies": [] }, { "id": 1002091, "author_name": "Jie Lu", "author_url": "", "post_date": "2020-09-07T20:09:59.407000", "content": "Yes, you have overfit your training set, that's why short trained models result in better LB score. A high CV just means you can do well on your training set, unfortunately it doesn't guarantee a good LB score in this competition
", "votes": 2, "replies": [] }, { "id": 1002096, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-09-07T20:13:36.893000", "content": "By my investigations a shorter trained model just delievers more nocall predictions and thus it comes very close to the all nocall submission score of 0.544 thats why it gets better when its shorter trained. A longer trained model has an answer for everything but these answers are seemingly completly wrong. I'm using validation to ensure to pick the best model during training. I dont believe that its overfitted.
How do you compute your validation score?
", "votes": 0, "replies": [ { "id": 1000681, "author_name": "Jagadish Sivakumar", "author_url": "", "post_date": "2020-09-06T17:40:49.877000", "content": "Hi @cpmpml
\nFor the val_score i'm using something similar to the class Fitter from the kernel:
\nhttps://www.kaggle.com/chanhu/training-bird-simple-baseline/data
With few modifications according to my model.
", "votes": 0, "replies": [] }, { "id": 1000685, "author_name": "Jagadish Sivakumar", "author_url": "", "post_date": "2020-09-06T17:44:01.667000", "content": "Did you try any ensemble? @cpmpml
\nI have tried ensembling models with different params but stuck with the same score. It's scoring less actually.
I will not spend time analyzing that code to get n answer to my question… Good luck then on finding why CV isn't reliable.
\nTo answer your question I have not ensembled so far. My LB score comes from a single model.
", "votes": 2, "replies": [] } ] }, { "id": 1000645, "author_name": "Ankit Gupta", "author_url": "", "post_date": "2020-09-06T17:11:38.230000", "content": "more datasets and overfitting. please have a look into it.
", "votes": -3, "replies": [] }, { "id": 1000465, "author_name": "Akshay Chavan", "author_url": "", "post_date": "2020-09-06T14:44:59.157000", "content": "Hey,
\nMaybe you can take look at the overfitting and validation set of the data.
Just to exclude a thought, without sounding paranoid, it can not be about leakage, in the data or how one splits the folds? How is the experience with Stratified Kfold, is it waterproof? Have a vague memory that there may be problems there.
", "votes": 0, "replies": [ { "id": 1001979, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-09-07T18:00:07.060000", "content": "How did you solve your problem? I remember you opening an equal discussion about this issue. Regarding your competition ranking, you should have solved it or?
", "votes": 0, "replies": [] }, { "id": 1002040, "author_name": "Kirderf", "author_url": "", "post_date": "2020-09-07T19:08:16.197000", "content": "Not really with you, have I written about kfold before, when, what and where? There is some writing about Stratified Kfold on Kaggle regarding leakage, hence the question of whether it can apply here, or maybe I have go it all wrong.
\n\"Regarding your competition ranking, you should have solved it or?\" After 10 month with Kaggle I'm more a learner and questioner than a knower, every day new questions in this area, making it all fun to continue.
You mentioned the topic problem on: https://www.kaggle.com/c/birdsong-recognition/discussion/178753
\n\"With the same training setup, picking the best val checkpoint gets worse Public LB.
\nMore training epochs with higher val loss beats a lower val loss with less epochs.
\nEven with same epochs and higher val loss can get a better Public LBs score.
\nHave also noticed that SWA from best ckps doesn't work well.\"
Thats exactly the same problem that I still have. I'm not talking about KFolds, I mean the topic question. Do you have any results to that issue?
", "votes": 0, "replies": [] }, { "id": 1002069, "author_name": "Kirderf", "author_url": "", "post_date": "2020-09-07T19:38:10.430000", "content": "Same situation as then, but have now implemented some more metrics, see if that can help. Seems many of us have the same problem, hence my comment, with the hope of a reflection about StKFold which is related to the topic.
", "votes": 0, "replies": [] }, { "id": 1002073, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-09-07T19:41:46.063000", "content": "Alright, but you have a good LB rank I tought you might not have the problem anymore.
\nI'm using StratifiedKFold without shuffeling and the problem still exists for me.
", "votes": 0, "replies": [] } ] }, { "id": 1001148, "author_name": "LeoF", "author_url": "", "post_date": "2020-09-07T05:15:35.357000", "content": "What is your local val-acc score?
", "votes": 0, "replies": [] }, { "id": 1000347, "author_name": "", "author_url": "", "post_date": "2020-09-06T13:29:52.180000", "content": "", "votes": 1, "replies": [ { "id": 1000362, "author_name": "Jagadish Sivakumar", "author_url": "", "post_date": "2020-09-06T13:37:08.170000", "content": "Hi @samkrishna
\nI haven't faced such an issue.
@samkrishna , you should restart the notebook and run the whole notebook codes again. Have good internet connection. if issue unresolved then there may be some issues in codes or datasets, correct it. I have faced such issues. I love to hear if issues resolved
", "votes": -1, "replies": [] }, { "id": 1001090, "author_name": "", "author_url": "", "post_date": "2020-09-07T04:15:17.300000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1000333": "Better models with lower validation loss are scoring less in LB. Any ideas?", "1000369": "Main idea - Test is completely different from train\nCompute a bunch of metrics and find the best correlated one ", "1000752": "I have the same experience and did not find out why. I will choose the highest LB submission and the highest CV submission and try my luck. I have seen others also having the same issue without any solutions.", "1000628": "Hi! @jagadish13, have you try to compare the distributions of the variables on Test vs Train sets, is there any significant difference.", "1001475": "There can sometimes be a divergence between metrics and loss. Check if your metric is improving", "1000633": "How do you compute your validation score?", "1000645": "more datasets and overfitting. please have a look into it. ", "1000465": "Hey,\nMaybe you can take look at the overfitting and validation set of the data. ", "1001872": "Just to exclude a thought, without sounding paranoid, it can not be about leakage, in the data or how one splits the folds? How is the experience with Stratified Kfold, is it waterproof? Have a vague memory that there may be problems there.", "1001148": "What is your local val-acc score?", "1000347": "" } }