CV improved but LB dropped.
\n1 CV:0.02685/LB:0.0235
\n2 CV:0.02743/LB:0.0230
Why does this happen?
\nDid I let the CV fit too much?
CV improved but LB dropped.
\n1 CV:0.02685/LB:0.0235
\n2 CV:0.02743/LB:0.0230
Why does this happen?
\nDid I let the CV fit too much?
It happend also in my case. Even though I controlled every thing I could imagine that may leak anything about future info. Some steps that may help in this case would be:
\n1: Check every thing that could leak future info.
\n2: Using multi folds to validate, which means not only using last week as validation week, you can also use second last and third last week to validate etc.
\n3: If you still have this problem. Just live with it and keep optimizing local CV. And when the local CV improves bigger enough, the problem may disappear.
P.S. the gap between CV and LB is actually small in your case. As a comparison, when my local CV score is 0.0437, LB score is about 0.0337.
", "rawMarkdown": "It happend also in my case. Even though I controlled every thing I could imagine that may leak anything about future info. Some steps that may help in this case would be: \n\n1: Check every thing that could leak future info.\n2: Using multi folds to validate, which means not only using last week as validation week, you can also use second last and third last week to validate etc.\n3: If you still have this problem. Just live with it and keep optimizing local CV. And when the local CV improves bigger enough, the problem may disappear. \n\nP.S. the gap between CV and LB is actually small in your case. As a comparison, when my local CV score is 0.0437, LB score is about 0.0337.", "votes": 3 }, { "id": 1771639, "postDate": "2022-04-29T12:36:28.723Z", "content": "It's hard to say, perhaps nobody knows the answer. But just like the old saying says, \"Always trust your CV than LB\".๐
", "rawMarkdown": "It's hard to say, perhaps nobody knows the answer. But just like the old saying says, \"Always trust your CV than LB\".๐", "votes": 4 }, { "id": 1774827, "postDate": "2022-05-02T13:29:36.323Z", "content": "It also might be a slightly different data distribution. Especially, if the lb test data is small. Usually, it's only 10% of all test dataโฆ
", "rawMarkdown": "It also might be a slightly different data distribution. Especially, if the lb test data is small. Usually, it's only 10% of all test data...", "votes": 1 }, { "id": 1767270, "postDate": "2022-04-25T07:54:53.083Z", "content": "@lihaorocky
\nThanks for the reply.
\nAll potential future information leaks will be checked.
It's happening to a lot of people. As of now, nobody knows the answer. I would recommend trying different hyperparameters and seeing if that helps.
", "rawMarkdown": "It's happening to a lot of people. As of now, nobody knows the answer. I would recommend trying different hyperparameters and seeing if that helps.", "votes": 2 }, { "id": 1768356, "postDate": "2022-04-26T07:45:24.407Z", "content": "If your CV isn't leaky, trust your CV
", "rawMarkdown": "If your CV isn't leaky, trust your CV" }, { "id": 1768292, "postDate": "2022-04-26T06:28:41.543Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1767241, "author_name": "HAO", "author_url": "", "post_date": "2022-04-25T07:34:19.960000", "content": "It happend also in my case. Even though I controlled every thing I could imagine that may leak anything about future info. Some steps that may help in this case would be:
\n1: Check every thing that could leak future info.
\n2: Using multi folds to validate, which means not only using last week as validation week, you can also use second last and third last week to validate etc.
\n3: If you still have this problem. Just live with it and keep optimizing local CV. And when the local CV improves bigger enough, the problem may disappear.
P.S. the gap between CV and LB is actually small in your case. As a comparison, when my local CV score is 0.0437, LB score is about 0.0337.
", "votes": 3, "replies": [] }, { "id": 1771639, "author_name": "", "author_url": "", "post_date": "2022-04-29T12:36:28.723000", "content": "It's hard to say, perhaps nobody knows the answer. But just like the old saying says, \"Always trust your CV than LB\".๐
", "votes": 4, "replies": [] }, { "id": 1774827, "author_name": "Doctor", "author_url": "", "post_date": "2022-05-02T13:29:36.323000", "content": "It also might be a slightly different data distribution. Especially, if the lb test data is small. Usually, it's only 10% of all test dataโฆ
", "votes": 1, "replies": [] }, { "id": 1767270, "author_name": "tatsun", "author_url": "", "post_date": "2022-04-25T07:54:53.083000", "content": "@lihaorocky
\nThanks for the reply.
\nAll potential future information leaks will be checked.
It's happening to a lot of people. As of now, nobody knows the answer. I would recommend trying different hyperparameters and seeing if that helps.
", "votes": 2, "replies": [] }, { "id": 1768356, "author_name": "Marco Gorelli", "author_url": "", "post_date": "2022-04-26T07:45:24.407000", "content": "If your CV isn't leaky, trust your CV
", "votes": 0, "replies": [] }, { "id": 1768292, "author_name": "", "author_url": "", "post_date": "2022-04-26T06:28:41.543000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1767205": "CV improved but LB dropped.\n1 CV:0.02685/LB:0.0235\n2 CV:0.02743/LB:0.0230\n\nWhy does this happen?\nDid I let the CV fit too much?", "1767241": "It happend also in my case. Even though I controlled every thing I could imagine that may leak anything about future info. Some steps that may help in this case would be: \n\n1: Check every thing that could leak future info.\n2: Using multi folds to validate, which means not only using last week as validation week, you can also use second last and third last week to validate etc.\n3: If you still have this problem. Just live with it and keep optimizing local CV. And when the local CV improves bigger enough, the problem may disappear. \n\nP.S. the gap between CV and LB is actually small in your case. As a comparison, when my local CV score is 0.0437, LB score is about 0.0337.", "1771639": "It's hard to say, perhaps nobody knows the answer. But just like the old saying says, \"Always trust your CV than LB\".๐", "1774827": "It also might be a slightly different data distribution. Especially, if the lb test data is small. Usually, it's only 10% of all test data...", "1767270": "@lihaorocky \nThanks for the reply.\nAll potential future information leaks will be checked.", "1767960": "It's happening to a lot of people. As of now, nobody knows the answer. I would recommend trying different hyperparameters and seeing if that helps.", "1768356": "If your CV isn't leaky, trust your CV", "1768292": "" } }