I added new features to my base model, got my train cv score increased. But after I made an submission, the leaderboard score dropped. I am curious what can the reasons be?
", "messages": [ { "id": "2208697", "postDate": "04/04/2023 09:05:23", "content": "I added new features to my base model, got my train cv score increased. But after I made an submission, the leaderboard score dropped. I am curious what can the reasons be?
", "rawMarkdown": "I added new features to my base model, got my train cv score increased. But after I made an submission, the leaderboard score dropped. I am curious what can the reasons be?", "votes": null }, { "id": "2208927", "postDate": "04/04/2023 11:22:44", "content": "Hii @huangsixuan05,
\nThis depends on many things including what your cv strategy is and if you are introducing any leaks through the new features that you added.
\nJust make sure that the cv is representative of the lb and there are no leaks.
Cv leakage could be one of the reasons but I believe it's more than that. I think according to some other discussion leakage is most likely to happen if you do feature selection/elimination, but what op talks about is adding new features. You can easily avoid leakage by adding new features to all models, but in my experience it can still lead to misalignments between cv and lb.
", "rawMarkdown": "Cv leakage could be one of the reasons but I believe it's more than that. I think according to some other discussion leakage is most likely to happen if you do feature selection/elimination, but what op talks about is adding new features. You can easily avoid leakage by adding new features to all models, but in my experience it can still lead to misalignments between cv and lb.", "votes": null }, { "id": "2210578", "postDate": "04/05/2023 13:33:32", "content": "Is the change important? There is always some noise in score. Here, I think that a LB score change of 0.001 is noise. Maybe 0.002 is also noise.
\nSame is true for CV scores. I only keep a new feature if the improvement in CV score is at least 0.003. But i am known to be quite conservative. I fear overfititng like hell, esp. on small data competitons like here.
", "rawMarkdown": "Is the change important? There is always some noise in score. Here, I think that a LB score change of 0.001 is noise. Maybe 0.002 is also noise.\n\nSame is true for CV scores. I only keep a new feature if the improvement in CV score is at least 0.003. But i am known to be quite conservative. I fear overfititng like hell, esp. on small data competitons like here.", "votes": null }, { "id": "2210713", "postDate": "04/05/2023 15:15:45", "content": "Observing the same , some cv show good correlation but some as CPMP mentioned have around 0.001 -0.002 of noise . I guess at the end we have 3 submission can go with best LB , BEST CV and best correlated ones at the end [personal opinion].
", "rawMarkdown": "Observing the same , some cv show good correlation but some as CPMP mentioned have around 0.001 -0.002 of noise . I guess at the end we have 3 submission can go with best LB , BEST CV and best correlated ones at the end [personal opinion].", "votes": null }, { "id": "2211399", "postDate": "04/06/2023 02:37:01", "content": "I didn't see this noise in my last competition. This is very annoying when trying new features and judging whether new features are effective. I think your suggestion on the 3 submissions are reasonable.
", "rawMarkdown": "I didn't see this noise in my last competition. This is very annoying when trying new features and judging whether new features are effective. I think your suggestion on the 3 submissions are reasonable.", "votes": null }, { "id": "2211400", "postDate": "04/06/2023 02:43:04", "content": "A noise of 0.001-0.002 is quite normal based on my results. Sometimes I even see a drop of 0.003 after adding new features 😂. I guess I might also try to only keep the new features which improve the cv score by 0.003. But it is also very difficult to improve cv score when it has achieved the level of 0.697.
", "rawMarkdown": "A noise of 0.001-0.002 is quite normal based on my results. Sometimes I even see a drop of 0.003 after adding new features 😂. I guess I might also try to only keep the new features which improve the cv score by 0.003. But it is also very difficult to improve cv score when it has achieved the level of 0.697.", "votes": null }, { "id": "2211404", "postDate": "04/06/2023 02:48:11", "content": "Agree. I cannot really think of any leakage in my model. I will try some other cv strategies and see how it goes.
", "rawMarkdown": "Agree. I cannot really think of any leakage in my model. I will try some other cv strategies and see how it goes.", "votes": null } ], "comments": [ { "id": 2208927, "author_name": "shashwatraman", "author_url": "", "post_date": "04/04/2023 11:22:44", "content": "Hii @huangsixuan05,
\nThis depends on many things including what your cv strategy is and if you are introducing any leaks through the new features that you added.
\nJust make sure that the cv is representative of the lb and there are no leaks.
Cv leakage could be one of the reasons but I believe it's more than that. I think according to some other discussion leakage is most likely to happen if you do feature selection/elimination, but what op talks about is adding new features. You can easily avoid leakage by adding new features to all models, but in my experience it can still lead to misalignments between cv and lb.
", "votes": null, "replies": [ { "id": 2211404, "author_name": "", "author_url": "", "post_date": "04/06/2023 02:48:11", "content": "Agree. I cannot really think of any leakage in my model. I will try some other cv strategies and see how it goes.
", "votes": null, "replies": [] } ] } ] }, { "id": 2210578, "author_name": "cpmpml", "author_url": "", "post_date": "04/05/2023 13:33:32", "content": "Is the change important? There is always some noise in score. Here, I think that a LB score change of 0.001 is noise. Maybe 0.002 is also noise.
\nSame is true for CV scores. I only keep a new feature if the improvement in CV score is at least 0.003. But i am known to be quite conservative. I fear overfititng like hell, esp. on small data competitons like here.
", "votes": null, "replies": [ { "id": 2211400, "author_name": "", "author_url": "", "post_date": "04/06/2023 02:43:04", "content": "A noise of 0.001-0.002 is quite normal based on my results. Sometimes I even see a drop of 0.003 after adding new features 😂. I guess I might also try to only keep the new features which improve the cv score by 0.003. But it is also very difficult to improve cv score when it has achieved the level of 0.697.
", "votes": null, "replies": [] } ] }, { "id": 2210713, "author_name": "gauravbrills", "author_url": "", "post_date": "04/05/2023 15:15:45", "content": "Observing the same , some cv show good correlation but some as CPMP mentioned have around 0.001 -0.002 of noise . I guess at the end we have 3 submission can go with best LB , BEST CV and best correlated ones at the end [personal opinion].
", "votes": null, "replies": [ { "id": 2211399, "author_name": "", "author_url": "", "post_date": "04/06/2023 02:37:01", "content": "I didn't see this noise in my last competition. This is very annoying when trying new features and judging whether new features are effective. I think your suggestion on the 3 submissions are reasonable.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2208697": "I added new features to my base model, got my train cv score increased. But after I made an submission, the leaderboard score dropped. I am curious what can the reasons be?", "2208927": "Hii @huangsixuan05,\nThis depends on many things including what your cv strategy is and if you are introducing any leaks through the new features that you added.\nJust make sure that the cv is representative of the lb and there are no leaks.", "2210160": "Cv leakage could be one of the reasons but I believe it's more than that. I think according to some other discussion leakage is most likely to happen if you do feature selection/elimination, but what op talks about is adding new features. You can easily avoid leakage by adding new features to all models, but in my experience it can still lead to misalignments between cv and lb.", "2210578": "Is the change important? There is always some noise in score. Here, I think that a LB score change of 0.001 is noise. Maybe 0.002 is also noise.\n\nSame is true for CV scores. I only keep a new feature if the improvement in CV score is at least 0.003. But i am known to be quite conservative. I fear overfititng like hell, esp. on small data competitons like here.", "2210713": "Observing the same , some cv show good correlation but some as CPMP mentioned have around 0.001 -0.002 of noise . I guess at the end we have 3 submission can go with best LB , BEST CV and best correlated ones at the end [personal opinion].", "2211399": "I didn't see this noise in my last competition. This is very annoying when trying new features and judging whether new features are effective. I think your suggestion on the 3 submissions are reasonable.", "2211400": "A noise of 0.001-0.002 is quite normal based on my results. Sometimes I even see a drop of 0.003 after adding new features 😂. I guess I might also try to only keep the new features which improve the cv score by 0.003. But it is also very difficult to improve cv score when it has achieved the level of 0.697.", "2211404": "Agree. I cannot really think of any leakage in my model. I will try some other cv strategies and see how it goes." }, "source": "meta" }