Has anyone found a reliable CV strategy that correlates well with the public LB?
\nThe portion of the hidden test set used to calculate the public LB is quite large in this competition (49%), so I don't think we can fully trust our CV. High LB scores should also be an indicator of how good you will do in private LB. I may be wrong, but if you're getting high CV scores but low public LB scores, you likely won't perform well on the private LB.
\nI started the competition with a somewhat leaky CV strategy that used KFold without shuffling. This obviously gave overly optimistic CV scores, so over the past few days, I've been experimenting with various CV strategies to find one that correlates well with the public LB. Unfortunately, I haven’t succeeded yet. Using TimeSeriesSplit, both with and without shuffling, produces somewhat realistic CV scores, but still doesn't correlate well with the public LB. The same goes for doing no cross-validation, but a standard train_test_split.
Feel free to share your experience and what strategies you've found to be effective.
", "messages": [ { "id": 3216863, "postDate": "2025-06-04T07:33:56.813Z", "content": "Has anyone found a reliable CV strategy that correlates well with the public LB?
\nThe portion of the hidden test set used to calculate the public LB is quite large in this competition (49%), so I don't think we can fully trust our CV. High LB scores should also be an indicator of how good you will do in private LB. I may be wrong, but if you're getting high CV scores but low public LB scores, you likely won't perform well on the private LB.
\nI started the competition with a somewhat leaky CV strategy that used KFold without shuffling. This obviously gave overly optimistic CV scores, so over the past few days, I've been experimenting with various CV strategies to find one that correlates well with the public LB. Unfortunately, I haven’t succeeded yet. Using TimeSeriesSplit, both with and without shuffling, produces somewhat realistic CV scores, but still doesn't correlate well with the public LB. The same goes for doing no cross-validation, but a standard train_test_split.
Feel free to share your experience and what strategies you've found to be effective.
", "rawMarkdown": "Has anyone found a reliable CV strategy that correlates well with the public LB?\n\nThe portion of the hidden test set used to calculate the public LB is quite large in this competition (49%), so I don't think we can fully trust our CV. High LB scores should also be an indicator of how good you will do in private LB. I may be wrong, but if you're getting high CV scores but low public LB scores, you likely won't perform well on the private LB.\n\nI started the competition with a somewhat leaky CV strategy that used KFold without shuffling. This obviously gave overly optimistic CV scores, so over the past few days, I've been experimenting with various CV strategies to find one that correlates well with the public LB. Unfortunately, I haven’t succeeded yet. Using `TimeSeriesSplit`, both with and without shuffling, produces somewhat realistic CV scores, but still doesn't correlate well with the public LB. The same goes for doing no cross-validation, but a standard `train_test_split`.\n\nFeel free to share your experience and what strategies you've found to be effective.", "votes": 13 }, { "id": 3224484, "postDate": "2025-06-14T23:44:46.447Z", "content": "Same for me, all validation methods i have tried so far were unreliable. Currently I am questioning if it even makes sense to continue the competition since I am not sure that in the end my submissions will be different then just gambling. (The pearsonr between time series cv splits varies greatly and we have no reason to assume that this will be any different between public and private LB)
\nMethods i have tried so far:
\nKFold
\nShuffle KFold
\nTime Series Split
\nTime Series Split with Gaps
\nShuffle time series split
\nOnly Train/Test on Time Periods in which the public/private lb likely take place (Public is likely between 02-29 and 09-02, private likely between 09-02 and 03-08)
\nTrain test split
it explains why jane street can rent 6 floors of office in HK recently with 2.6m usd rent per month, while DRW is falling
", "rawMarkdown": "it explains why jane street can rent 6 floors of office in HK recently with 2.6m usd rent per month, while DRW is falling" }, { "id": 3225030, "postDate": "2025-06-15T20:42:29.977Z", "content": "It's unfortunate, but good to know I'm not the only one struggling to find a good CV–LB correlation. I've come to the same conclusion as you and have decided not to spend any more time on this competition.
", "rawMarkdown": "It's unfortunate, but good to know I'm not the only one struggling to find a good CV–LB correlation. I've come to the same conclusion as you and have decided not to spend any more time on this competition.", "votes": 1 } ] }, { "id": 3219345, "postDate": "2025-06-07T14:47:54.120Z", "content": "@ravaghi this problem is because the timestamps in the test data are masked and there is no particular order of time in the test data
", "rawMarkdown": "@ravaghi this problem is because the timestamps in the test data are masked and there is no particular order of time in the test data", "votes": 3, "replies": [ { "id": 3219352, "postDate": "2025-06-07T15:01:36.960Z", "content": "I'm aware of that. I'm not using the timestamps and have even tried shuffling the data and using fewer folds to mimic the test set, but still no success. I've increased my CV by a lot since I started, but that hasn't translated well to public LB.
", "rawMarkdown": "I'm aware of that. I'm not using the timestamps and have even tried shuffling the data and using fewer folds to mimic the test set, but still no success. I've increased my CV by a lot since I started, but that hasn't translated well to public LB. ", "votes": 2 } ] }, { "id": 3217035, "postDate": "2025-06-04T12:54:25.320Z", "content": "Have noticed the same thing as you have mentioned. CV scores doesn't correlate well with the public LB.
", "rawMarkdown": "Have noticed the same thing as you have mentioned. CV scores doesn't correlate well with the public LB. ", "votes": 3 }, { "id": 3216908, "postDate": "2025-06-04T09:19:34.677Z", "content": "Maximize oof score with TimeSeriesSplit(n_splits=5) or put a gap = 24x60x7 (7, 14 or 30 days)
\nNote: oof score will not cover all train data but the average score can fluctuate a lot, you should prefer oof score instead.
I've been using oof score, but the correlation is still bad. Are you seeing good correlation with TimeSeriesSplit?
", "rawMarkdown": "I've been using oof score, but the correlation is still bad. Are you seeing good correlation with TimeSeriesSplit?", "votes": 1, "replies": [ { "id": 3216917, "postDate": "2025-06-04T09:30:30.690Z", "content": "It is not always compatible, but the reason for this may be somewhere else. For example, the selected features. However, the oof score should still be trusted. The public score is misleading.
", "rawMarkdown": "It is not always compatible, but the reason for this may be somewhere else. For example, the selected features. However, the oof score should still be trusted. The public score is misleading.", "votes": 1 }, { "id": 3216920, "postDate": "2025-06-04T09:34:53.210Z", "content": "However, the oof score should still be trusted. The public score is misleading.\nThis is what I do for most competitions, but usually because only ~20% of the hidden test set is used for public LB. In this competition though, half the test set is used for public LB, so I think it should give some indication of how well you will do in private LB.
", "rawMarkdown": "```text\nHowever, the oof score should still be trusted. The public score is misleading.\n```\nThis is what I do for most competitions, but usually because only ~20% of the hidden test set is used for public LB. In this competition though, half the test set is used for public LB, so I think it should give some indication of how well you will do in private LB.", "votes": 4 }, { "id": 3217017, "postDate": "2025-06-04T12:22:25.903Z", "content": "Does the test data start the day after the train's last date or is there a gap?
", "rawMarkdown": "Does the test data start the day after the train's last date or is there a gap?", "votes": 1 }, { "id": 3217037, "postDate": "2025-06-04T13:00:05.853Z", "content": "As far as I know, we don't have that information.
", "rawMarkdown": "As far as I know, we don't have that information.", "votes": 2 }, { "id": 3217050, "postDate": "2025-06-04T13:20:33.540Z", "content": "The LB surely is still a valuable indicator for the overall performance, but roughly glancing at the historical bit-coin prices after the cut-off shows two very different stories in the first and second half.
\nDidn't look further into it, but I'm skeptical both of my CV- and LB-score.🤷♂️
bit-coin shows different trend in the first and second halves of the year, but we only got 1 year data and lb is half year data. That may be why cv and lb is relatively irrelevant.
", "rawMarkdown": "bit-coin shows different trend in the first and second halves of the year, but we only got 1 year data and lb is half year data. That may be why cv and lb is relatively irrelevant.", "votes": 2 }, { "id": 3243747, "postDate": "2025-07-07T13:44:19.163Z", "content": "Bro, I have all of them. I feel the same way as you. I think the key to improvement in this competition is feature selection.
", "rawMarkdown": "Bro, I have all of them. I feel the same way as you. I think the key to improvement in this competition is feature selection." }, { "id": 3226962, "postDate": "2025-06-18T10:04:22.973Z", "content": "The differences between my CV score and the LB score are so large that I suspect either:
\nWouldn’t shuffling the data before doing a TimeSeriesSplit ruin the whole purpose of using TimeSeriesSplit?
", "rawMarkdown": "Wouldn’t shuffling the data before doing a TimeSeriesSplit ruin the whole purpose of using TimeSeriesSplit?", "replies": [ { "id": 3224481, "postDate": "2025-06-14T23:36:55.993Z", "content": "He does not shuffle before the split, he shuffles after splitting, as it is done in the actual test set
", "rawMarkdown": "He does not shuffle before the split, he shuffles after splitting, as it is done in the actual test set", "votes": 1, "replies": [ { "id": 3225023, "postDate": "2025-06-15T20:30:36.127Z", "content": "That is correct.
", "rawMarkdown": "That is correct." } ] } ] }, { "id": 3218568, "postDate": "2025-06-06T11:28:39.223Z", "rawMarkdown": "", "votes": -3, "isDeleted": true } ], "comments": [ { "id": 3224484, "author_name": "MaxUhl98", "author_url": "", "post_date": "2025-06-14T23:44:46.447000", "content": "Same for me, all validation methods i have tried so far were unreliable. Currently I am questioning if it even makes sense to continue the competition since I am not sure that in the end my submissions will be different then just gambling. (The pearsonr between time series cv splits varies greatly and we have no reason to assume that this will be any different between public and private LB)
\nMethods i have tried so far:
\nKFold
\nShuffle KFold
\nTime Series Split
\nTime Series Split with Gaps
\nShuffle time series split
\nOnly Train/Test on Time Periods in which the public/private lb likely take place (Public is likely between 02-29 and 09-02, private likely between 09-02 and 03-08)
\nTrain test split
it explains why jane street can rent 6 floors of office in HK recently with 2.6m usd rent per month, while DRW is falling
", "votes": 0, "replies": [] }, { "id": 3225030, "author_name": "Mahdi Ravaghi", "author_url": "", "post_date": "2025-06-15T20:42:29.977000", "content": "It's unfortunate, but good to know I'm not the only one struggling to find a good CV–LB correlation. I've come to the same conclusion as you and have decided not to spend any more time on this competition.
", "votes": 1, "replies": [] } ] }, { "id": 3219345, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2025-06-07T14:47:54.120000", "content": "@ravaghi this problem is because the timestamps in the test data are masked and there is no particular order of time in the test data
", "votes": 3, "replies": [ { "id": 3219352, "author_name": "Mahdi Ravaghi", "author_url": "", "post_date": "2025-06-07T15:01:36.960000", "content": "I'm aware of that. I'm not using the timestamps and have even tried shuffling the data and using fewer folds to mimic the test set, but still no success. I've increased my CV by a lot since I started, but that hasn't translated well to public LB.
", "votes": 2, "replies": [] } ] }, { "id": 3217035, "author_name": "byunjins", "author_url": "", "post_date": "2025-06-04T12:54:25.320000", "content": "Have noticed the same thing as you have mentioned. CV scores doesn't correlate well with the public LB.
", "votes": 3, "replies": [] }, { "id": 3216908, "author_name": "United States", "author_url": "", "post_date": "2025-06-04T09:19:34.677000", "content": "Maximize oof score with TimeSeriesSplit(n_splits=5) or put a gap = 24x60x7 (7, 14 or 30 days)
\nNote: oof score will not cover all train data but the average score can fluctuate a lot, you should prefer oof score instead.
I've been using oof score, but the correlation is still bad. Are you seeing good correlation with TimeSeriesSplit?
", "votes": 1, "replies": [ { "id": 3216917, "author_name": "United States", "author_url": "", "post_date": "2025-06-04T09:30:30.690000", "content": "It is not always compatible, but the reason for this may be somewhere else. For example, the selected features. However, the oof score should still be trusted. The public score is misleading.
", "votes": 1, "replies": [] }, { "id": 3216920, "author_name": "Mahdi Ravaghi", "author_url": "", "post_date": "2025-06-04T09:34:53.210000", "content": "However, the oof score should still be trusted. The public score is misleading.\nThis is what I do for most competitions, but usually because only ~20% of the hidden test set is used for public LB. In this competition though, half the test set is used for public LB, so I think it should give some indication of how well you will do in private LB.
", "votes": 4, "replies": [] }, { "id": 3217017, "author_name": "United States", "author_url": "", "post_date": "2025-06-04T12:22:25.903000", "content": "Does the test data start the day after the train's last date or is there a gap?
", "votes": 1, "replies": [] }, { "id": 3217037, "author_name": "Mahdi Ravaghi", "author_url": "", "post_date": "2025-06-04T13:00:05.853000", "content": "As far as I know, we don't have that information.
", "votes": 2, "replies": [] }, { "id": 3217050, "author_name": "Lennart Haupts", "author_url": "", "post_date": "2025-06-04T13:20:33.540000", "content": "The LB surely is still a valuable indicator for the overall performance, but roughly glancing at the historical bit-coin prices after the cut-off shows two very different stories in the first and second half.
\nDidn't look further into it, but I'm skeptical both of my CV- and LB-score.🤷♂️
bit-coin shows different trend in the first and second halves of the year, but we only got 1 year data and lb is half year data. That may be why cv and lb is relatively irrelevant.
", "votes": 2, "replies": [] }, { "id": 3243747, "author_name": "Idrissa M Dicko", "author_url": "", "post_date": "2025-07-07T13:44:19.163000", "content": "Bro, I have all of them. I feel the same way as you. I think the key to improvement in this competition is feature selection.
", "votes": 0, "replies": [] }, { "id": 3226962, "author_name": "Joseph ", "author_url": "", "post_date": "2025-06-18T10:04:22.973000", "content": "The differences between my CV score and the LB score are so large that I suspect either:
\nWouldn’t shuffling the data before doing a TimeSeriesSplit ruin the whole purpose of using TimeSeriesSplit?
", "votes": 0, "replies": [ { "id": 3224481, "author_name": "MaxUhl98", "author_url": "", "post_date": "2025-06-14T23:36:55.993000", "content": "He does not shuffle before the split, he shuffles after splitting, as it is done in the actual test set
", "votes": 1, "replies": [ { "id": 3225023, "author_name": "Mahdi Ravaghi", "author_url": "", "post_date": "2025-06-15T20:30:36.127000", "content": "That is correct.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3218568, "author_name": "", "author_url": "", "post_date": "2025-06-06T11:28:39.223000", "content": "", "votes": -3, "replies": [] } ], "raw_markdown_by_id": { "3216863": "Has anyone found a reliable CV strategy that correlates well with the public LB?\n\nThe portion of the hidden test set used to calculate the public LB is quite large in this competition (49%), so I don't think we can fully trust our CV. High LB scores should also be an indicator of how good you will do in private LB. I may be wrong, but if you're getting high CV scores but low public LB scores, you likely won't perform well on the private LB.\n\nI started the competition with a somewhat leaky CV strategy that used KFold without shuffling. This obviously gave overly optimistic CV scores, so over the past few days, I've been experimenting with various CV strategies to find one that correlates well with the public LB. Unfortunately, I haven’t succeeded yet. Using `TimeSeriesSplit`, both with and without shuffling, produces somewhat realistic CV scores, but still doesn't correlate well with the public LB. The same goes for doing no cross-validation, but a standard `train_test_split`.\n\nFeel free to share your experience and what strategies you've found to be effective.", "3224484": "Same for me, all validation methods i have tried so far were unreliable. Currently I am questioning if it even makes sense to continue the competition since I am not sure that in the end my submissions will be different then just gambling. (The pearsonr between time series cv splits varies greatly and we have no reason to assume that this will be any different between public and private LB)\nMethods i have tried so far: \nKFold\nShuffle KFold\nTime Series Split\nTime Series Split with Gaps\nShuffle time series split \nOnly Train/Test on Time Periods in which the public/private lb likely take place (Public is likely between 02-29 and 09-02, private likely between 09-02 and 03-08)\nTrain test split", "3219345": "@ravaghi this problem is because the timestamps in the test data are masked and there is no particular order of time in the test data", "3217035": "Have noticed the same thing as you have mentioned. CV scores doesn't correlate well with the public LB. ", "3216908": "Maximize oof score with TimeSeriesSplit(n_splits=5) or put a gap = 24x60x7 (7, 14 or 30 days)\nNote: oof score will not cover all train data but the average score can fluctuate a lot, you should prefer oof score instead.\n", "3219367": "bit-coin shows different trend in the first and second halves of the year, but we only got 1 year data and lb is half year data. That may be why cv and lb is relatively irrelevant.", "3243747": "Bro, I have all of them. I feel the same way as you. I think the key to improvement in this competition is feature selection.", "3226962": "The differences between my CV score and the LB score are so large that I suspect either:\n1. the training and test data are sampled from completely different distributions, or\n2. their LB score is erroneous", "3224438": "Wouldn’t shuffling the data before doing a TimeSeriesSplit ruin the whole purpose of using TimeSeriesSplit?", "3218568": "" } }