{ "id": 589052, "title": "[Important] Leaderboard Reboot with Updated Competition Data", "url": "/competitions/drw-crypto-market-prediction/discussion/589052", "author_name": "DRW Trading", "post_date": "2025-07-09T23:27:44.667000", "votes": 12, "comment_count": 69, "views": 0, "content": "

Hi Kagglers,

\n

First, thank you all for your enthusiasm, effort, and valuable contributions to the competition so far. We understand it may be frustrating to see a data update mid-competition, but this decision was made after careful consideration to ensure fairness and integrity.

\n

The competition data has been updated with a newly shuffled test set, and as a result, the leaderboard will be rebooted. The previous leaderboard no longer reflects the current data (which now has different true test timestamps), so please re-download the dataset, rerun your code, and submit updated predictions.

\n

We are in contact with Kaggle regarding a leaderboard reboot. Please don’t be concerned if the updated scores are not yet visible.

\n

As part of this update:

\n\n

While the proprietary and market features have been revised, they remain accurate and consistent in meaning. We've taken great care to preserve feature semantics and ensure that prior analyses remain valid and minimally impacted by these changes.

\n

Thank you again for your understanding and continued participation—we're excited to see how your models perform on the refreshed dataset!

\n

Best regards,
\nThe DRW & Cumberland Team

\n
\n

Update on Jul 11 (UTC): All submissions made before the dataset update have been removed. A very small number may appear rescored if the same participant submitted again around the time of the dataset update, due to technical reasons—but this has no impact on the leaderboard.

", "messages": [ { "id": 3247577, "postDate": "2025-07-13T03:14:29.867Z", "content": "

@drwtrading , can you help us understand what the code review will entail? I still don't fully understand how notebooks will be approved/disproved.

\n

If you look at the results of running simple linear regression based feature analysis on the train dataset and the test dataset (using pseudo labels through unshuffling the data and using time series to estimate the prediction values), you can extract insights on the behavior of the test dataset. The below is based on a linear regression analysis (for simplicity) but the impact is the same. Anyone who is able to create psuedo labels based on the unshuffling and time-series prediction of the test data is then able to chose features that are most compatible with the train and test dataset - significantly improving scores and avoiding instability, noise, and drift.

\n

\"\"

\n

I am not suggesting anyone do this. Nor will I submit a notebook using these forward looking insights. However, I want to emphasize again, that in a submission notebook it would be impossible to tell if someone find a good feature or feature interaction through time series based psuedo label analysis (against the rules) or whether they found the feature through some other analysis that is based off train only.

", "rawMarkdown": "@drwtrading , can you help us understand what the code review will entail? I still don't fully understand how notebooks will be approved/disproved.\n\nIf you look at the results of running simple linear regression based feature analysis on the train dataset and the test dataset (using pseudo labels through unshuffling the data and using time series to estimate the prediction values), you can extract insights on the behavior of the test dataset. The below is based on a linear regression analysis (for simplicity) but the impact is the same. Anyone who is able to create psuedo labels based on the unshuffling and time-series prediction of the test data is then able to chose features that are most compatible with the train and test dataset - significantly improving scores and avoiding instability, noise, and drift.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10414474%2Fb5be68800f0317246b0846671f70b379%2F__results___1_1.png?generation=1752376298824875&alt=media)\n\nI am not suggesting anyone do this. Nor will I submit a notebook using these forward looking insights. However, I want to emphasize again, that in a submission notebook it would be impossible to tell if someone find a good feature or feature interaction through time series based psuedo label analysis (against the rules) or whether they found the feature through some other analysis that is based off train only.", "votes": 16 }, { "id": 3245833, "postDate": "2025-07-09T23:27:44.667Z", "content": "

Hi Kagglers,

\n

First, thank you all for your enthusiasm, effort, and valuable contributions to the competition so far. We understand it may be frustrating to see a data update mid-competition, but this decision was made after careful consideration to ensure fairness and integrity.

\n

The competition data has been updated with a newly shuffled test set, and as a result, the leaderboard will be rebooted. The previous leaderboard no longer reflects the current data (which now has different true test timestamps), so please re-download the dataset, rerun your code, and submit updated predictions.

\n

We are in contact with Kaggle regarding a leaderboard reboot. Please don’t be concerned if the updated scores are not yet visible.

\n

As part of this update:

\n\n

While the proprietary and market features have been revised, they remain accurate and consistent in meaning. We've taken great care to preserve feature semantics and ensure that prior analyses remain valid and minimally impacted by these changes.

\n

Thank you again for your understanding and continued participation—we're excited to see how your models perform on the refreshed dataset!

\n

Best regards,
\nThe DRW & Cumberland Team

\n
\n

Update on Jul 11 (UTC): All submissions made before the dataset update have been removed. A very small number may appear rescored if the same participant submitted again around the time of the dataset update, due to technical reasons—but this has no impact on the leaderboard.

", "rawMarkdown": "Hi Kagglers,\n\nFirst, thank you all for your enthusiasm, effort, and valuable contributions to the competition so far. We understand it may be frustrating to see a data update mid-competition, but this decision was made after careful consideration to ensure fairness and integrity.\n\nThe competition data has been updated with a newly shuffled test set, and as a result, the leaderboard will be rebooted. The previous leaderboard no longer reflects the current data (which now has different true test timestamps), so please **re-download the dataset**, rerun your code, and submit updated predictions.\n\nWe are in contact with Kaggle regarding a leaderboard reboot. Please don’t be concerned if the updated scores are not yet visible.\n\nAs part of this update:\n\n- Test data have been newly shuffled. The rule against future peaking still applies.\n- Columns with excessive abnormal values have been removed to streamline data preprocessing. You will now see **X1–X780 instead of X1–X890**.\n- The remaining features have been normalized.\n\nWhile the proprietary and market features have been revised, they remain accurate and consistent in meaning. We've taken great care to preserve feature semantics and ensure that prior analyses remain valid and minimally impacted by these changes.\n\nThank you again for your understanding and continued participation—we're excited to see how your models perform on the refreshed dataset!\n\nBest regards,\nThe DRW & Cumberland Team\n\n------------\n\n**Update on Jul 11 (UTC): All submissions made before the dataset update have been removed. A very small number may appear rescored if the same participant submitted again around the time of the dataset update, due to technical reasons—but this has no impact on the leaderboard.**", "votes": 11 }, { "id": 3250135, "postDate": "2025-07-17T18:43:04.133Z", "content": "

Hi @drwtrading,

\n

Could you specify the format of final submission, given lots of abnormal public scores? Is everyone required to submit a Kaggle Notebook, or you will reach out to potential winners for their code solutions

", "rawMarkdown": "Hi @drwtrading,\n\nCould you specify the format of final submission, given lots of abnormal public scores? Is everyone required to submit a Kaggle Notebook, or you will reach out to potential winners for their code solutions", "votes": 5, "replies": [ { "id": 3250170, "postDate": "2025-07-17T20:08:59.463Z", "content": "

+1 Same question as him.

", "rawMarkdown": "+1 Same question as him.", "votes": 1 } ] }, { "id": 3245903, "postDate": "2025-07-10T02:39:46.153Z", "content": "

Hi DRW,

\n

What do you mean by the remaining features have been normalized, does that mean you applied a z-score standardizer on the train dataset and transformed the test dataset? Also did all the columns get shuffled?

\n

Thanks

", "rawMarkdown": "Hi DRW,\n\nWhat do you mean by the remaining features have been normalized, does that mean you applied a z-score standardizer on the train dataset and transformed the test dataset? Also did all the columns get shuffled?\n\nThanks", "votes": 5, "replies": [ { "id": 3245915, "postDate": "2025-07-10T03:24:26.383Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3246196, "postDate": "2025-07-10T14:30:48.687Z", "content": "

Did DRW respond to you and then delete the comment? I've seen a few deleted comments, but I'm not sure if they are from DRW, although I think at least one of them was.

\n

People who saw this comment before it was deleted might have a competitive advantage.

", "rawMarkdown": "Did DRW respond to you and then delete the comment? I've seen a few deleted comments, but I'm not sure if they are from DRW, although I think at least one of them was.\n\nPeople who saw this comment before it was deleted might have a competitive advantage.", "replies": [ { "id": 3246217, "postDate": "2025-07-10T14:58:27.490Z", "content": "

I don’t know. I hope DRW responds because if the columns are shuffled it will affect our models.

", "rawMarkdown": "I don’t know. I hope DRW responds because if the columns are shuffled it will affect our models." }, { "id": 3246220, "postDate": "2025-07-10T15:05:45.917Z", "content": "

I mean, we probably have to redo our feature engineering and selection from scratch anyway right? Some of the features have been wiped and I assume all the features have been significantly transformed to the point of being essentially new features (otherwise competitors could just apply the same transformation to the unscrambled test data and have twice as much data to train on)

", "rawMarkdown": "I mean, we probably have to redo our feature engineering and selection from scratch anyway right? Some of the features have been wiped and I assume all the features have been significantly transformed to the point of being essentially new features (otherwise competitors could just apply the same transformation to the unscrambled test data and have twice as much data to train on)" }, { "id": 3246247, "postDate": "2025-07-10T15:22:51.523Z", "content": "

My initial research suggest that while the features are not the same, there are patterns that can be found between the old data and the new data. Thus people who tuned their models on the unmasked data can run sub-models to figure out relationships between new features and old features in order to incorporate ill-derived learnings from the unmasked data into the new data.

\n

An oversimplified way of deconstructing this would be to:

\n
    \n
  1. Review the most important old features, especially the old features that were tuned based on the unmasked data.
    2. \n
  2. Train a model that predicts these old features using the new proprietary features. If you can train a model that accurately predict important old features (again especially old features that were tuned based on the unmasked data) with certain new features, then you can effectively generate proxies for the old features.
    3. \n
\n

For example, using the traditional XGBoost configuration that got scores of around .13 on the LB, I was able to do the following:

\n

1.) Use the original features in the XGBoost model.
\n2.) Find important features in the unmasked dataset, with consideration for drift, stability, etc.
\n3.) Incorporate those important features and/or derivatives thereof into the XGBoost model.
\n4.) X856_times_X598 was a feature that showed preliminary promise in the unmasked dataset.
\n5.) With the new dataset, I just need to figure out what new features can accurately predict the old features of X856_times_X598, then select those new features themselves or use a sub-model to recreate a proxy variable that would estimate the old X856_times_X598.
\n6.) Similar relationships and patterns between the old and new data can likely be found, even if the new features are truly shuffled and transformed, the power of models allows us to find relationships.

", "rawMarkdown": "My initial research suggest that while the features are not the same, there are patterns that can be found between the old data and the new data. Thus people who tuned their models on the unmasked data can run sub-models to figure out relationships between new features and old features in order to incorporate ill-derived learnings from the unmasked data into the new data.\n\nAn oversimplified way of deconstructing this would be to:\n\n1. Review the most important old features, especially the old features that were tuned based on the unmasked data.\n2. Train a model that predicts these old features using the new proprietary features. If you can train a model that accurately predict important old features (again especially old features that were tuned based on the unmasked data) with certain new features, then you can effectively generate proxies for the old features.\n\nFor example, using the traditional XGBoost configuration that got scores of around .13 on the LB, I was able to do the following:\n\n1.) Use the original features in the XGBoost model.\n2.) Find important features in the unmasked dataset, with consideration for drift, stability, etc.\n3.) Incorporate those important features and/or derivatives thereof into the XGBoost model.\n4.) X856_times_X598 was a feature that showed preliminary promise in the unmasked dataset.\n5.) With the new dataset, I just need to figure out what new features can accurately predict the old features of X856_times_X598, then select those new features themselves or use a sub-model to recreate a proxy variable that would estimate the old X856_times_X598.\n6.) Similar relationships and patterns between the old and new data can likely be found, even if the new features are truly shuffled and transformed, the power of models allows us to find relationships." }, { "id": 3246292, "postDate": "2025-07-10T16:31:10.057Z", "content": "

Thanks @longissachairfay for sharing your valuable insight, but how do you make sure that patterns found before will generalize well in the new private test set

", "rawMarkdown": "Thanks @longissachairfay for sharing your valuable insight, but how do you make sure that patterns found before will generalize well in the new private test set" }, { "id": 3246298, "postDate": "2025-07-10T16:43:53.800Z", "content": "

Tracking train vs CV scores and stability will help with that.

\n

I'd say that building a model that generalizes well has been the primary challenge in this competition, or any competition for that matter, and this challenge existed even before the dataset was reset.

\n

Using the previously unmasked test data can help guide feature engineering, model architectures, and other decisions if done correctly, regardless of the dataset being shuffled and transformed again. Even though transformations was made with the new dataset, I believe the following assumptions true:

\n

1.) The old dataset and new dataset are both crypto datasets and may follow similar patterns.
\n2.) Transformations and adjustments in the new dataset can occur, but it is unlikely these transformations are so extreme and obfuscated as to eliminate all possible patterns and relationships between the original and new dataset. If the data was completely obfuscated it would provide provide less value to DRW in real world trading exercises. The new data could be mixed up, adjusted in scale, and transformed using various patterns and functions, but ML models may be able to pickup and estimate these patterns, at least in part.

", "rawMarkdown": "Tracking train vs CV scores and stability will help with that.\n\nI'd say that building a model that generalizes well has been the primary challenge in this competition, or any competition for that matter, and this challenge existed even before the dataset was reset.\n\nUsing the previously unmasked test data can help guide feature engineering, model architectures, and other decisions if done correctly, regardless of the dataset being shuffled and transformed again. Even though transformations was made with the new dataset, I believe the following assumptions true:\n\n1.) The old dataset and new dataset are both crypto datasets and may follow similar patterns.\n2.) Transformations and adjustments in the new dataset can occur, but it is unlikely these transformations are so extreme and obfuscated as to eliminate all possible patterns and relationships between the original and new dataset. If the data was completely obfuscated it would provide provide less value to DRW in real world trading exercises. The new data could be mixed up, adjusted in scale, and transformed using various patterns and functions, but ML models may be able to pickup and estimate these patterns, at least in part.\n\n" }, { "id": 3246358, "postDate": "2025-07-10T18:52:37.967Z", "content": "

@drwtrading please tell us how you normalized the data and if the column names were shuffled, the distribution seems to be different.

", "rawMarkdown": "@drwtrading please tell us how you normalized the data and if the column names were shuffled, the distribution seems to be different." } ] } ] }, { "id": 3246454, "postDate": "2025-07-11T01:34:48.243Z", "content": "

Dear @drwtrading , I know you are working hard to manage the current situation, but I would suggest that you provide an update on what is happening with scoring and leaderboard changes, preferably before changes are made.

\n

It is very confusing seeing the leaderboard change, some notebooks removed, other notebooks staying - there needs to be an official announcement on what is happening with more detail.

", "rawMarkdown": "Dear @drwtrading , I know you are working hard to manage the current situation, but I would suggest that you provide an update on what is happening with scoring and leaderboard changes, preferably before changes are made.\n\nIt is very confusing seeing the leaderboard change, some notebooks removed, other notebooks staying - there needs to be an official announcement on what is happening with more detail.", "votes": 6, "replies": [ { "id": 3246571, "postDate": "2025-07-11T06:54:35.583Z", "content": "

Dear Kaggler, thank you for your feedback. All submissions made before the dataset update have now been removed or rescored.

", "rawMarkdown": "Dear Kaggler, thank you for your feedback. All submissions made before the dataset update have now been removed or rescored.", "votes": -1, "replies": [ { "id": 3246572, "postDate": "2025-07-11T07:01:55.730Z", "content": "

Have you noticed the data has leak again? Someone has figured it out.

", "rawMarkdown": "Have you noticed the data has leak again? Someone has figured it out.", "votes": 2 }, { "id": 3246739, "postDate": "2025-07-11T12:40:21.223Z", "content": "

How about the \"Code\" section? There are still notebooks showing scores of 0.5+.

", "rawMarkdown": "How about the \"Code\" section? There are still notebooks showing scores of 0.5+." }, { "id": 3246745, "postDate": "2025-07-11T12:47:15.777Z", "content": "

DRW, are you going to take any action to make this competition fair again? As myself and others have discovered, in a very short amount of time, it is entirely possible to use insights from the old dataset and its unshuffled test component to get a competitive edge in the rebooted competition.

", "rawMarkdown": "DRW, are you going to take any action to make this competition fair again? As myself and others have discovered, in a very short amount of time, it is entirely possible to use insights from the old dataset and its unshuffled test component to get a competitive edge in the rebooted competition.", "votes": 4 } ] } ] }, { "id": 3245858, "postDate": "2025-07-10T00:45:28.960Z", "content": "

I'm honestly at a loss here. All of this is very disappointing to be honest, which from your message I think you already understand.

\n

However, reshuffling the data does not solve the problem because it does not eliminate the insights that people already gathered when analyzing the unshuffled dataset. These insights can be used to intelligently create features for training and test which will give significant advantages to some teams, without showing any visible signs of forward peaking in the notebook.

\n

Also, can you help me understand what the statement below means:

\n

\"While the proprietary and market features have been revised, they remain accurate and consistent in meaning.\"

\n

If they have been revised, how are they consistent in meaning? I hope this wasn't just a scaling adjustment, but that is the only way I could see \"consistent in meaning\" to be accurate.

", "rawMarkdown": "I'm honestly at a loss here. All of this is very disappointing to be honest, which from your message I think you already understand.\n\nHowever, reshuffling the data does not solve the problem because it does not eliminate the insights that people already gathered when analyzing the unshuffled dataset. These insights can be used to intelligently create features for training and test which will give significant advantages to some teams, without showing any visible signs of forward peaking in the notebook.\n\nAlso, can you help me understand what the statement below means:\n\n\"While the proprietary and market features have been revised, they remain accurate and consistent in meaning.\"\n\nIf they have been revised, how are they consistent in meaning? I hope this wasn't just a scaling adjustment, but that is the only way I could see \"consistent in meaning\" to be accurate.\n\n\n\n", "votes": 4, "replies": [ { "id": 3245860, "postDate": "2025-07-10T00:49:11.340Z", "content": "

We understand your concerns and appreciate you sharing them openly. You're absolutely right that once certain insights are gained, they can't be “unlearned.” However, we’ve employed several changes—not just reshuffling—to ensure that the current test set remains valid, while also rendering previously hacked labels ineffective. These measures aim to level the playing field as much as possible going forward.

\n

We’re committed to maintaining fairness and will continue monitoring for potential leakage during the review phase.

", "rawMarkdown": "We understand your concerns and appreciate you sharing them openly. You're absolutely right that once certain insights are gained, they can't be “unlearned.” However, we’ve employed several changes—not just reshuffling—to ensure that the current test set remains valid, while also rendering previously hacked labels ineffective. These measures aim to level the playing field as much as possible going forward.\n\nWe’re committed to maintaining fairness and will continue monitoring for potential leakage during the review phase.", "replies": [ { "id": 3245864, "postDate": "2025-07-10T00:53:19.003Z", "content": "

Thank you for your efforts on this, I do appreciate this.

\n

I do have my doubts about the ability to render insights gained through unshuffling as ineffective. But I suppose there are ways to highly obfuscate the data in a way that makes it highly unlikely those insights would be useful.

\n

Also, I do appreciate you cleaning the proprietary dataset. There were some unnecessary columns in the original dataset in my opinion.

\n

I'm still in the game, just a bit of a speed bump.

", "rawMarkdown": "Thank you for your efforts on this, I do appreciate this.\n\nI do have my doubts about the ability to render insights gained through unshuffling as ineffective. But I suppose there are ways to highly obfuscate the data in a way that makes it highly unlikely those insights would be useful.\n\nAlso, I do appreciate you cleaning the proprietary dataset. There were some unnecessary columns in the original dataset in my opinion.\n\nI'm still in the game, just a bit of a speed bump." } ] } ] }, { "id": 3246872, "postDate": "2025-07-11T17:09:41.047Z", "content": "

I found yet another way to reverse the time for the test set. At this point I am not sure if the organizers will be able to keep the competition fair.

\n

We will only know after code review, I guess.
\nThen, again, how would one control for subjective calls about features (some of which can be pure lookahead)

", "rawMarkdown": "I found yet another way to reverse the time for the test set. At this point I am not sure if the organizers will be able to keep the competition fair. \n\nWe will only know after code review, I guess.\nThen, again, how would one control for subjective calls about features (some of which can be pure lookahead)", "votes": 3, "replies": [ { "id": 3246875, "postDate": "2025-07-11T17:17:54.520Z", "content": "

In my opinion, it is impossible to keep it fair without a complete reset.

\n

Once you reverse engineer the order of the test labels, then you use these labels to intelligently augment your training model by selecting features that better align with the test labels.

\n

There is no way to catch this in code review because there is no way to prove that someone engineered a feature based on the reverse engineered test labels or if they are just good at feature engineering.

", "rawMarkdown": "In my opinion, it is impossible to keep it fair without a complete reset.\n\nOnce you reverse engineer the order of the test labels, then you use these labels to intelligently augment your training model by selecting features that better align with the test labels.\n\nThere is no way to catch this in code review because there is no way to prove that someone engineered a feature based on the reverse engineered test labels or if they are just good at feature engineering.", "votes": 6, "replies": [ { "id": 3247058, "postDate": "2025-07-12T03:05:45.063Z", "content": "

Completely agree. It is impossible to quantify to which extent this is an issue. Subjective calls in this case can be completely lookahead. Right now, there are several ways to re-construct the test order and then choose the best features based on the test set.

\n

The only way out is completely restarting the competitions, with entirely different features and NO test data available for download, with a model-based submission (like most other trading competitions these days).

\n

This can be summarized by the title of the piece of code posted by quantju: \"ALL U NEED IS GIVE UP\".

", "rawMarkdown": "Completely agree. It is impossible to quantify to which extent this is an issue. Subjective calls in this case can be completely lookahead. Right now, there are several ways to re-construct the test order and then choose the best features based on the test set.\n\nThe only way out is completely restarting the competitions, with entirely different features and NO test data available for download, with a model-based submission (like most other trading competitions these days).\n\nThis can be summarized by the title of the piece of code posted by quantju: \"ALL U NEED IS GIVE UP\".\n", "votes": 2 }, { "id": 3247066, "postDate": "2025-07-12T03:24:47.427Z", "content": "

All true, I am still learning new things when playing around with the data though, so its all good. I'm not really here hoping I would get a prize.

", "rawMarkdown": "All true, I am still learning new things when playing around with the data though, so its all good. I'm not really here hoping I would get a prize." }, { "id": 3247145, "postDate": "2025-07-12T07:28:49.710Z", "content": "

I think DRW should have a new test set, with half of existing data from the first test set that is for the public leaderboard, and half of new data for the private leaderboard. Future-peeking the public leaderboard has happened in a lot of past trading competitions, what we want is for it not to happen in the private one. So if they give a new test set, one where only the half of the data that is for the public leaderboard has already been seen then the people that do future-peeking would just hurt themselves. I think this is the best way forward for DRW. The public leaderboard will be a mess but again this has always happened in trading competitions.

", "rawMarkdown": "I think DRW should have a new test set, with half of existing data from the first test set that is for the public leaderboard, and half of new data for the private leaderboard. Future-peeking the public leaderboard has happened in a lot of past trading competitions, what we want is for it not to happen in the private one. So if they give a new test set, one where only the half of the data that is for the public leaderboard has already been seen then the people that do future-peeking would just hurt themselves. I think this is the best way forward for DRW. The public leaderboard will be a mess but again this has always happened in trading competitions.", "votes": 2 } ] }, { "id": 3246946, "postDate": "2025-07-11T20:26:06.793Z", "content": "

@drwtrading you might want to recheck the data now with the high LB score

", "rawMarkdown": "@drwtrading you might want to recheck the data now with the high LB score ", "votes": 1 } ] }, { "id": 3246003, "postDate": "2025-07-10T06:22:08.177Z", "content": "

Are column names this same?

", "rawMarkdown": "Are column names this same?", "votes": 1, "replies": [ { "id": 3246186, "postDate": "2025-07-10T14:16:50.887Z", "content": "

I would also like to know this! Would save time checking things like constant columns, etc.

", "rawMarkdown": "I would also like to know this! Would save time checking things like constant columns, etc." }, { "id": 3246191, "postDate": "2025-07-10T14:22:25.480Z", "content": "

OK just checked and all the constant columns have been removed, i.e. the column names are not the same. I'm assuming some of the very highly-correlated columns have also been removed?

", "rawMarkdown": "OK just checked and all the constant columns have been removed, i.e. the column names are _not_ the same. I'm assuming some of the very highly-correlated columns have also been removed?", "replies": [ { "id": 3246193, "postDate": "2025-07-10T14:25:37.737Z", "content": "

That is what I noticed too

", "rawMarkdown": "That is what I noticed too" } ] } ] }, { "id": 3250183, "postDate": "2025-07-17T20:42:37.293Z", "content": "

it doesn't make sense to go through all the CSV files and manually check every code solution. There should be a new test set along with a notebook to run everything efficiently!!!

", "rawMarkdown": "it doesn't make sense to go through all the CSV files and manually check every code solution. There should be a new test set along with a notebook to run everything efficiently!!!", "votes": 2 }, { "id": 3249373, "postDate": "2025-07-16T10:31:46.583Z", "content": "

Hi, thank you for your effort. For the final submission, do we need to load and process the test data using our model, or is submitting a CSV file with the predictions based on the current test data sufficient?

", "rawMarkdown": "Hi, thank you for your effort. For the final submission, do we need to load and process the test data using our model, or is submitting a CSV file with the predictions based on the current test data sufficient?", "replies": [ { "id": 3249445, "postDate": "2025-07-16T13:38:19.843Z", "content": "

same question here

", "rawMarkdown": "same question here", "votes": 1 } ] }, { "id": 3246170, "postDate": "2025-07-10T13:22:36.910Z", "content": "

Dear DRW Hosts (or anyone who can help clarify),

\n

I'm puzzled by how this solves the issue. Users will already have downloaded and unshuffled test data locally. At the very least this allows them to train their models on the test data period biasing their models to the correct regime? As stated this is still against the rules, but who is going to catch the 79th place entry who boosted their score by a 0.05 by training on the forbidden data?

\n

I'm a total beginner so possibly I'm missing something obvious but as it stands I'm releuctant to spend the time to adapt to the new data and chasing an extra 0.01 if it'll be impossible to beat someone who just uses a baseline model and cheats for just enough gain to look believable.

\n

Thanks

", "rawMarkdown": "Dear DRW Hosts (or anyone who can help clarify),\n\nI'm puzzled by how this solves the issue. Users will already have downloaded and unshuffled test data locally. At the very least this allows them to train their models on the test data period biasing their models to the correct regime? As stated this is still against the rules, but who is going to catch the 79th place entry who boosted their score by a 0.05 by training on the forbidden data?\n\nI'm a total beginner so possibly I'm missing something obvious but as it stands I'm releuctant to spend the time to adapt to the new data and chasing an extra 0.01 if it'll be impossible to beat someone who just uses a baseline model and cheats for just enough gain to look believable.\n\nThanks ", "votes": 2, "replies": [ { "id": 3246175, "postDate": "2025-07-10T13:36:04.110Z", "content": "

The updated test data includes different true timestamps, making it difficult to map or reuse previous labels. We anticipated this and applied additional manipulations to ensure that prior future-peaking insights are no longer effective.

\n

We will continuously monitor for suspicious submissions with unusually high scores and conduct thorough code reviews to uphold fairness throughout the competition.

", "rawMarkdown": "The updated test data includes different true timestamps, making it difficult to map or reuse previous labels. We anticipated this and applied additional manipulations to ensure that prior future-peaking insights are no longer effective.\n\nWe will continuously monitor for suspicious submissions with unusually high scores and conduct thorough code reviews to uphold fairness throughout the competition.", "votes": 4, "replies": [ { "id": 3246180, "postDate": "2025-07-10T13:58:10.120Z", "content": "

ah ok, the new test data is a completely different time period? Thanks for clarifying.
\nActually just one more question for clarity if you don't mind; at the very least those with the old, unshuffled test data have twice as much data to train on, can we assume the new data is sufficiently transformed so that the old test data can't simply be transformed, clipped of the now-redundant features and trained on? Even if its a different period now, having twice as much data to train on is a big advantage.

", "rawMarkdown": "ah ok, the new test data is a completely different time period? Thanks for clarifying.\nActually just one more question for clarity if you don't mind; at the very least those with the old, unshuffled test data have twice as much data to train on, can we assume the new data is sufficiently transformed so that the old test data can't simply be transformed, clipped of the now-redundant features and trained on? Even if its a different period now, having twice as much data to train on is a big advantage." }, { "id": 3249017, "postDate": "2025-07-15T14:20:30.490Z", "content": "

@drwtrading what about a mid-range score that uses the leak? Perhaps one could use the leak, drop useless features and then train a model.
\nWhat is the intended response in this case?

", "rawMarkdown": "@drwtrading what about a mid-range score that uses the leak? Perhaps one could use the leak, drop useless features and then train a model. \nWhat is the intended response in this case?", "votes": 1 }, { "id": 3249848, "postDate": "2025-07-17T08:13:50.683Z", "content": "

it seems a waste of time here. not a good setup.

", "rawMarkdown": "it seems a waste of time here. not a good setup.", "votes": 3 } ] } ] }, { "id": 3245896, "postDate": "2025-07-10T02:25:23.673Z", "content": "

Hi DRW & Cumberland Team,

\n

Thanks for your efforts and fast response to keep this match fair. Do you intend to extend the deadline given new dataset used, or the deadline will stay the same?

", "rawMarkdown": "Hi DRW & Cumberland Team,\n\nThanks for your efforts and fast response to keep this match fair. Do you intend to extend the deadline given new dataset used, or the deadline will stay the same? ", "replies": [ { "id": 3245897, "postDate": "2025-07-10T02:26:06.553Z", "content": "

We're still discussing the possibility of an extension and will share an update soon.

", "rawMarkdown": "We're still discussing the possibility of an extension and will share an update soon.\n", "replies": [ { "id": 3245902, "postDate": "2025-07-10T02:36:43.983Z", "content": "

Its already been two months though

", "rawMarkdown": "Its already been two months though", "votes": 1 }, { "id": 3245918, "postDate": "2025-07-10T03:29:07.983Z", "content": "

True
\nThis should have been planned better
\nHope kaggle takes necessary measures to prevent effort and time waste of participants @paperxd

", "rawMarkdown": "True \nThis should have been planned better \nHope kaggle takes necessary measures to prevent effort and time waste of participants @paperxd ", "votes": 2 }, { "id": 3246104, "postDate": "2025-07-10T10:24:57.750Z", "content": "

since the feature has been updated, 2-4 weeks extension may be fine. If there is a new leaderboard without hacking, that would be better. Thanks for your organizing

", "rawMarkdown": "since the feature has been updated, 2-4 weeks extension may be fine. If there is a new leaderboard without hacking, that would be better. Thanks for your organizing", "votes": 1 }, { "id": 3246200, "postDate": "2025-07-10T14:38:39.993Z", "content": "

I don't think an extension will cut it. I've looked at the new dataset along with the old train data and unmasked old test data. In my opinion, there are insights gathered through the unmasking of the old test data that can be applied to the new data to give a competitive advantage.

", "rawMarkdown": "I don't think an extension will cut it. I've looked at the new dataset along with the old train data and unmasked old test data. In my opinion, there are insights gathered through the unmasking of the old test data that can be applied to the new data to give a competitive advantage." }, { "id": 3247002, "postDate": "2025-07-12T00:20:57.843Z", "content": "

We’re looking forward to the final outcome of your discussions. We truly hope that, if an extension is granted, it could be for at least two more weeks — that would be greatly appreciated!

", "rawMarkdown": "We’re looking forward to the final outcome of your discussions. We truly hope that, if an extension is granted, it could be for at least two more weeks — that would be greatly appreciated!" }, { "id": 3247005, "postDate": "2025-07-12T00:26:29.740Z", "content": "

We think there is no point in an extension, if we have an extension more people will just find ways to hack the leaderboard.

", "rawMarkdown": "We think there is no point in an extension, if we have an extension more people will just find ways to hack the leaderboard." }, { "id": 3247008, "postDate": "2025-07-12T00:31:19.457Z", "content": "

They hacked the leaderboard already

", "rawMarkdown": "They hacked the leaderboard already" } ] } ] }, { "id": 3253401, "postDate": "2025-07-24T15:51:46.837Z", "content": "

@drwtrading I was wondering if it is possible to use different test set for the final submission.

", "rawMarkdown": "@drwtrading I was wondering if it is possible to use different test set for the final submission. ", "replies": [ { "id": 3253403, "postDate": "2025-07-24T16:01:14.267Z", "content": "

I believe a straightforward way to detect overfitted notebook submissions is by reshuffling the test set. This would cause models that rely heavily on timestamp order or lag-based features to perform poorly.

", "rawMarkdown": "I believe a straightforward way to detect overfitted notebook submissions is by reshuffling the test set. This would cause models that rely heavily on timestamp order or lag-based features to perform poorly.\n", "replies": [ { "id": 3253404, "postDate": "2025-07-24T16:02:55.533Z", "content": "

I agree with you. I think another option is using different set of test data.

", "rawMarkdown": "I agree with you. I think another option is using different set of test data." }, { "id": 3253411, "postDate": "2025-07-24T16:15:50.780Z", "content": "

Using a completely different test set would make the competition winners largely a matter of random luck. To be honest, no model can generalize perfectly to any arbitrary dataset — it's like trying to use one key to open every door, which is impossible. A better way to eliminate notebooks that rely on future-peeking is by reshuffling the test set during the evaluation phase.

", "rawMarkdown": "Using a completely different test set would make the competition winners largely a matter of random luck. To be honest, no model can generalize perfectly to any arbitrary dataset — it's like trying to use one key to open every door, which is impossible. A better way to eliminate notebooks that rely on future-peeking is by reshuffling the test set during the evaluation phase." } ] } ] }, { "id": 3245865, "postDate": "2025-07-10T00:55:20.103Z", "content": "

Would it be possible to clarify if all prior submissions will be removed? Otherwise, the leaderboard is misleading. Additionally, will the competition time be extended?

", "rawMarkdown": "Would it be possible to clarify if all prior submissions will be removed? Otherwise, the leaderboard is misleading. Additionally, will the competition time be extended?", "replies": [ { "id": 3245870, "postDate": "2025-07-10T00:58:21.600Z", "content": "

All prior submissions will be considered invalid since the test data has changed. The leaderboard scores will be updated accordingly to reflect submissions made on the new dataset.

\n

As for the competition timeline, we’re still discussing whether an extension will be granted and will share an update soon.

", "rawMarkdown": "All prior submissions will be considered invalid since the test data has changed. The leaderboard scores will be updated accordingly to reflect submissions made on the new dataset.\n\nAs for the competition timeline, we’re still discussing whether an extension will be granted and will share an update soon.", "votes": 1, "replies": [ { "id": 3246197, "postDate": "2025-07-10T14:32:12.220Z", "content": "

In my opinion, the current competition should be shutdown, an apology issued. Then a reboot competition started, and you may want to increase the number of award slots to quell angst and discontent. Just my 2 cents.

\n

There is no way to ensure that any of the future peeking doesn't taint the results with unfair advantages.

", "rawMarkdown": "In my opinion, the current competition should be shutdown, an apology issued. Then a reboot competition started, and you may want to increase the number of award slots to quell angst and discontent. Just my 2 cents.\n\nThere is no way to ensure that any of the future peeking doesn't taint the results with unfair advantages.", "votes": 3 }, { "id": 3246460, "postDate": "2025-07-11T01:46:04.553Z", "content": "

This seems like the only way forward. Even with adjusted features, teams/individuals can use insights gained from the previous unshuffled test dataset to get an unfair advantage.

", "rawMarkdown": "This seems like the only way forward. Even with adjusted features, teams/individuals can use insights gained from the previous unshuffled test dataset to get an unfair advantage. " }, { "id": 3246762, "postDate": "2025-07-11T13:46:18.893Z", "content": "

As expected, someone has likely already reversed engineered the new dataset, possibly using the old dataset as well for guidance.
\n\"\"

", "rawMarkdown": "As expected, someone has likely already reversed engineered the new dataset, possibly using the old dataset as well for guidance.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10414474%2Fe527868ac7281b3a70f7858f8e302d51%2F2025-07-11_09-45.png?generation=1752241573177090&alt=media)\n", "votes": 2 }, { "id": 3246853, "postDate": "2025-07-11T16:35:16.570Z", "content": "

This might have been a glitch that was using the old LB

", "rawMarkdown": "This might have been a glitch that was using the old LB" }, { "id": 3246868, "postDate": "2025-07-11T16:53:56.990Z", "content": "

Unfortunately, it is not. The new order within the test dataset has been decoded. The obfuscation and reordering done by DRW was insufficient. I was able to score 0.65 just now.

", "rawMarkdown": "Unfortunately, it is not. The new order within the test dataset has been decoded. The obfuscation and reordering done by DRW was insufficient. I was able to score 0.65 just now.", "votes": 2 }, { "id": 3246877, "postDate": "2025-07-11T17:27:39.963Z", "content": "

Hi @taylorsamarel , are you sure that you reorder the entire test set right or only the public score part?

", "rawMarkdown": "Hi @taylorsamarel , are you sure that you reorder the entire test set right or only the public score part?", "votes": 1 }, { "id": 3246884, "postDate": "2025-07-11T17:35:58.857Z", "content": "

It may not be perfect, but it is pretty close. It wasn't all of my work, I was able to build off of other notebooks.

\n

It seems that the administrators of this competition are discounting notebooks that score greater than ~0.6, but I didn't see this in an announcement anywhere. I understand forward looking is against the competition rules, but I think there is learning to be had when experimenting with notebooks, as long as participants - like myself - understand that these 'peaking' notebooks won't be considered for a prize.

\n

I think the fundamental issue with this competition is that somewhat arbitrary and non-transparent actions are being taken in an attempt to 'level the playing field'. However, the data continues to provide 'leaked' insights for those that can decompose the new dataset.

\n

The issue comes from the fact that this 'decomposed insights' can be used to intelligently tune a model, that, at its face, does not involve rule breaking forward peaking.

", "rawMarkdown": "It may not be perfect, but it is pretty close. It wasn't all of my work, I was able to build off of other notebooks.\n\nIt seems that the administrators of this competition are discounting notebooks that score greater than ~0.6, but I didn't see this in an announcement anywhere. I understand forward looking is against the competition rules, but I think there is learning to be had when experimenting with notebooks, as long as participants - like myself - understand that these 'peaking' notebooks won't be considered for a prize.\n\nI think the fundamental issue with this competition is that somewhat arbitrary and non-transparent actions are being taken in an attempt to 'level the playing field'. However, the data continues to provide 'leaked' insights for those that can decompose the new dataset.\n\nThe issue comes from the fact that this 'decomposed insights' can be used to intelligently tune a model, that, at its face, does not involve rule breaking forward peaking.", "votes": 1 }, { "id": 3246915, "postDate": "2025-07-11T19:11:54.910Z", "content": "

DRW will conduct code reviews though

", "rawMarkdown": "DRW will conduct code reviews though" }, { "id": 3246923, "postDate": "2025-07-11T19:30:24.260Z", "content": "

Have you deleted your submission yourself ? Or has it been moderated

", "rawMarkdown": "Have you deleted your submission yourself ? Or has it been moderated" }, { "id": 3246929, "postDate": "2025-07-11T19:49:22.433Z", "content": "

Seems like it was administratively removed, not exactly sure what that means. Anything above 0.6 didn't stay on the board, but I still see the .599 score on the code section if you sort by score (not on the leaderboard though).

", "rawMarkdown": "Seems like it was administratively removed, not exactly sure what that means. Anything above 0.6 didn't stay on the board, but I still see the .599 score on the code section if you sort by score (not on the leaderboard though)." }, { "id": 3246930, "postDate": "2025-07-11T19:50:19.257Z", "content": "

There is no way to determine if an engineered feature was a good guess, or if it was derived through future peaking. Someone could derive the perfect features to engineer using the shuffled label data and then just hard code those features into a separate notebook.

", "rawMarkdown": "There is no way to determine if an engineered feature was a good guess, or if it was derived through future peaking. Someone could derive the perfect features to engineer using the shuffled label data and then just hard code those features into a separate notebook." } ] } ] }, { "id": 3246069, "postDate": "2025-07-10T08:37:53.613Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3246011, "postDate": "2025-07-10T06:47:00.427Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3245877, "postDate": "2025-07-10T01:25:21.040Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 3245879, "postDate": "2025-07-10T01:43:28.370Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 3245838, "postDate": "2025-07-09T23:41:56.770Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 3247577, "author_name": "Taylor S. Amarel", "author_url": "", "post_date": "2025-07-13T03:14:29.867000", "content": "

@drwtrading , can you help us understand what the code review will entail? I still don't fully understand how notebooks will be approved/disproved.

\n

If you look at the results of running simple linear regression based feature analysis on the train dataset and the test dataset (using pseudo labels through unshuffling the data and using time series to estimate the prediction values), you can extract insights on the behavior of the test dataset. The below is based on a linear regression analysis (for simplicity) but the impact is the same. Anyone who is able to create psuedo labels based on the unshuffling and time-series prediction of the test data is then able to chose features that are most compatible with the train and test dataset - significantly improving scores and avoiding instability, noise, and drift.

\n

\"\"

\n

I am not suggesting anyone do this. Nor will I submit a notebook using these forward looking insights. However, I want to emphasize again, that in a submission notebook it would be impossible to tell if someone find a good feature or feature interaction through time series based psuedo label analysis (against the rules) or whether they found the feature through some other analysis that is based off train only.

", "votes": 16, "replies": [] }, { "id": 3250135, "author_name": "Alex Zhongs", "author_url": "", "post_date": "2025-07-17T18:43:04.133000", "content": "

Hi @drwtrading,

\n

Could you specify the format of final submission, given lots of abnormal public scores? Is everyone required to submit a Kaggle Notebook, or you will reach out to potential winners for their code solutions

", "votes": 5, "replies": [ { "id": 3250170, "author_name": "paperxd", "author_url": "", "post_date": "2025-07-17T20:08:59.463000", "content": "

+1 Same question as him.

", "votes": 1, "replies": [] } ] }, { "id": 3245903, "author_name": "paperxd", "author_url": "", "post_date": "2025-07-10T02:39:46.153000", "content": "

Hi DRW,

\n

What do you mean by the remaining features have been normalized, does that mean you applied a z-score standardizer on the train dataset and transformed the test dataset? Also did all the columns get shuffled?

\n

Thanks

", "votes": 5, "replies": [ { "id": 3245915, "author_name": "", "author_url": "", "post_date": "2025-07-10T03:24:26.383000", "content": "", "votes": 0, "replies": [] }, { "id": 3246196, "author_name": "Taylor S. Amarel", "author_url": "", "post_date": "2025-07-10T14:30:48.687000", "content": "

Did DRW respond to you and then delete the comment? I've seen a few deleted comments, but I'm not sure if they are from DRW, although I think at least one of them was.

\n

People who saw this comment before it was deleted might have a competitive advantage.

", "votes": 0, "replies": [ { "id": 3246217, "author_name": "paperxd", "author_url": "", "post_date": "2025-07-10T14:58:27.490000", "content": "

I don’t know. I hope DRW responds because if the columns are shuffled it will affect our models.

", "votes": 0, "replies": [] }, { "id": 3246220, "author_name": "GreatestCutie", "author_url": "", "post_date": "2025-07-10T15:05:45.917000", "content": "

I mean, we probably have to redo our feature engineering and selection from scratch anyway right? Some of the features have been wiped and I assume all the features have been significantly transformed to the point of being essentially new features (otherwise competitors could just apply the same transformation to the unscrambled test data and have twice as much data to train on)

", "votes": 0, "replies": [] }, { "id": 3246247, "author_name": "Long Issac Hair Fay", "author_url": "", "post_date": "2025-07-10T15:22:51.523000", "content": "

My initial research suggest that while the features are not the same, there are patterns that can be found between the old data and the new data. Thus people who tuned their models on the unmasked data can run sub-models to figure out relationships between new features and old features in order to incorporate ill-derived learnings from the unmasked data into the new data.

\n

An oversimplified way of deconstructing this would be to:

\n
    \n
  1. Review the most important old features, especially the old features that were tuned based on the unmasked data.
    2. \n
  2. Train a model that predicts these old features using the new proprietary features. If you can train a model that accurately predict important old features (again especially old features that were tuned based on the unmasked data) with certain new features, then you can effectively generate proxies for the old features.
    3. \n
\n

For example, using the traditional XGBoost configuration that got scores of around .13 on the LB, I was able to do the following:

\n

1.) Use the original features in the XGBoost model.
\n2.) Find important features in the unmasked dataset, with consideration for drift, stability, etc.
\n3.) Incorporate those important features and/or derivatives thereof into the XGBoost model.
\n4.) X856_times_X598 was a feature that showed preliminary promise in the unmasked dataset.
\n5.) With the new dataset, I just need to figure out what new features can accurately predict the old features of X856_times_X598, then select those new features themselves or use a sub-model to recreate a proxy variable that would estimate the old X856_times_X598.
\n6.) Similar relationships and patterns between the old and new data can likely be found, even if the new features are truly shuffled and transformed, the power of models allows us to find relationships.

", "votes": 0, "replies": [] }, { "id": 3246292, "author_name": "Alex Zhongs", "author_url": "", "post_date": "2025-07-10T16:31:10.057000", "content": "

Thanks @longissachairfay for sharing your valuable insight, but how do you make sure that patterns found before will generalize well in the new private test set

", "votes": 0, "replies": [] }, { "id": 3246298, "author_name": "Long Issac Hair Fay", "author_url": "", "post_date": "2025-07-10T16:43:53.800000", "content": "

Tracking train vs CV scores and stability will help with that.

\n

I'd say that building a model that generalizes well has been the primary challenge in this competition, or any competition for that matter, and this challenge existed even before the dataset was reset.

\n

Using the previously unmasked test data can help guide feature engineering, model architectures, and other decisions if done correctly, regardless of the dataset being shuffled and transformed again. Even though transformations was made with the new dataset, I believe the following assumptions true:

\n

1.) The old dataset and new dataset are both crypto datasets and may follow similar patterns.
\n2.) Transformations and adjustments in the new dataset can occur, but it is unlikely these transformations are so extreme and obfuscated as to eliminate all possible patterns and relationships between the original and new dataset. If the data was completely obfuscated it would provide provide less value to DRW in real world trading exercises. The new data could be mixed up, adjusted in scale, and transformed using various patterns and functions, but ML models may be able to pickup and estimate these patterns, at least in part.

", "votes": 0, "replies": [] }, { "id": 3246358, "author_name": "paperxd", "author_url": "", "post_date": "2025-07-10T18:52:37.967000", "content": "

@drwtrading please tell us how you normalized the data and if the column names were shuffled, the distribution seems to be different.

", "votes": 0, "replies": [] } ] } ] }, { "id": 3246454, "author_name": "Taylor S. Amarel", "author_url": "", "post_date": "2025-07-11T01:34:48.243000", "content": "

Dear @drwtrading , I know you are working hard to manage the current situation, but I would suggest that you provide an update on what is happening with scoring and leaderboard changes, preferably before changes are made.

\n

It is very confusing seeing the leaderboard change, some notebooks removed, other notebooks staying - there needs to be an official announcement on what is happening with more detail.

", "votes": 6, "replies": [ { "id": 3246571, "author_name": "DRW Trading", "author_url": "", "post_date": "2025-07-11T06:54:35.583000", "content": "

Dear Kaggler, thank you for your feedback. All submissions made before the dataset update have now been removed or rescored.

", "votes": -1, "replies": [ { "id": 3246572, "author_name": "wangliang9274", "author_url": "", "post_date": "2025-07-11T07:01:55.730000", "content": "

Have you noticed the data has leak again? Someone has figured it out.

", "votes": 2, "replies": [] }, { "id": 3246739, "author_name": "Taylor S. Amarel", "author_url": "", "post_date": "2025-07-11T12:40:21.223000", "content": "

How about the \"Code\" section? There are still notebooks showing scores of 0.5+.

", "votes": 0, "replies": [] }, { "id": 3246745, "author_name": "Long Issac Hair Fay", "author_url": "", "post_date": "2025-07-11T12:47:15.777000", "content": "

DRW, are you going to take any action to make this competition fair again? As myself and others have discovered, in a very short amount of time, it is entirely possible to use insights from the old dataset and its unshuffled test component to get a competitive edge in the rebooted competition.

", "votes": 4, "replies": [] } ] } ] }, { "id": 3245858, "author_name": "Taylor S. Amarel", "author_url": "", "post_date": "2025-07-10T00:45:28.960000", "content": "

I'm honestly at a loss here. All of this is very disappointing to be honest, which from your message I think you already understand.

\n

However, reshuffling the data does not solve the problem because it does not eliminate the insights that people already gathered when analyzing the unshuffled dataset. These insights can be used to intelligently create features for training and test which will give significant advantages to some teams, without showing any visible signs of forward peaking in the notebook.

\n

Also, can you help me understand what the statement below means:

\n

\"While the proprietary and market features have been revised, they remain accurate and consistent in meaning.\"

\n

If they have been revised, how are they consistent in meaning? I hope this wasn't just a scaling adjustment, but that is the only way I could see \"consistent in meaning\" to be accurate.

", "votes": 4, "replies": [ { "id": 3245860, "author_name": "DRW Trading", "author_url": "", "post_date": "2025-07-10T00:49:11.340000", "content": "

We understand your concerns and appreciate you sharing them openly. You're absolutely right that once certain insights are gained, they can't be “unlearned.” However, we’ve employed several changes—not just reshuffling—to ensure that the current test set remains valid, while also rendering previously hacked labels ineffective. These measures aim to level the playing field as much as possible going forward.

\n

We’re committed to maintaining fairness and will continue monitoring for potential leakage during the review phase.

", "votes": 0, "replies": [ { "id": 3245864, "author_name": "Taylor S. Amarel", "author_url": "", "post_date": "2025-07-10T00:53:19.003000", "content": "

Thank you for your efforts on this, I do appreciate this.

\n

I do have my doubts about the ability to render insights gained through unshuffling as ineffective. But I suppose there are ways to highly obfuscate the data in a way that makes it highly unlikely those insights would be useful.

\n

Also, I do appreciate you cleaning the proprietary dataset. There were some unnecessary columns in the original dataset in my opinion.

\n

I'm still in the game, just a bit of a speed bump.

", "votes": 0, "replies": [] } ] } ] }, { "id": 3246872, "author_name": "Vladimir Khismatullin", "author_url": "", "post_date": "2025-07-11T17:09:41.047000", "content": "

I found yet another way to reverse the time for the test set. At this point I am not sure if the organizers will be able to keep the competition fair.

\n

We will only know after code review, I guess.
\nThen, again, how would one control for subjective calls about features (some of which can be pure lookahead)

", "votes": 3, "replies": [ { "id": 3246875, "author_name": "Taylor S. Amarel", "author_url": "", "post_date": "2025-07-11T17:17:54.520000", "content": "

In my opinion, it is impossible to keep it fair without a complete reset.

\n

Once you reverse engineer the order of the test labels, then you use these labels to intelligently augment your training model by selecting features that better align with the test labels.

\n

There is no way to catch this in code review because there is no way to prove that someone engineered a feature based on the reverse engineered test labels or if they are just good at feature engineering.

", "votes": 6, "replies": [ { "id": 3247058, "author_name": "Vladimir Khismatullin", "author_url": "", "post_date": "2025-07-12T03:05:45.063000", "content": "

Completely agree. It is impossible to quantify to which extent this is an issue. Subjective calls in this case can be completely lookahead. Right now, there are several ways to re-construct the test order and then choose the best features based on the test set.

\n

The only way out is completely restarting the competitions, with entirely different features and NO test data available for download, with a model-based submission (like most other trading competitions these days).

\n

This can be summarized by the title of the piece of code posted by quantju: \"ALL U NEED IS GIVE UP\".

", "votes": 2, "replies": [] }, { "id": 3247066, "author_name": "Taylor S. Amarel", "author_url": "", "post_date": "2025-07-12T03:24:47.427000", "content": "

All true, I am still learning new things when playing around with the data though, so its all good. I'm not really here hoping I would get a prize.

", "votes": 0, "replies": [] }, { "id": 3247145, "author_name": "NN", "author_url": "", "post_date": "2025-07-12T07:28:49.710000", "content": "

I think DRW should have a new test set, with half of existing data from the first test set that is for the public leaderboard, and half of new data for the private leaderboard. Future-peeking the public leaderboard has happened in a lot of past trading competitions, what we want is for it not to happen in the private one. So if they give a new test set, one where only the half of the data that is for the public leaderboard has already been seen then the people that do future-peeking would just hurt themselves. I think this is the best way forward for DRW. The public leaderboard will be a mess but again this has always happened in trading competitions.

", "votes": 2, "replies": [] } ] }, { "id": 3246946, "author_name": "Anh Quang Phan", "author_url": "", "post_date": "2025-07-11T20:26:06.793000", "content": "

@drwtrading you might want to recheck the data now with the high LB score

", "votes": 1, "replies": [] } ] }, { "id": 3246003, "author_name": "Rafał Pawłowski", "author_url": "", "post_date": "2025-07-10T06:22:08.177000", "content": "

Are column names this same?

", "votes": 1, "replies": [ { "id": 3246186, "author_name": "Sarah Jeffreson", "author_url": "", "post_date": "2025-07-10T14:16:50.887000", "content": "

I would also like to know this! Would save time checking things like constant columns, etc.

", "votes": 0, "replies": [] }, { "id": 3246191, "author_name": "Sarah Jeffreson", "author_url": "", "post_date": "2025-07-10T14:22:25.480000", "content": "

OK just checked and all the constant columns have been removed, i.e. the column names are not the same. I'm assuming some of the very highly-correlated columns have also been removed?

", "votes": 0, "replies": [ { "id": 3246193, "author_name": "byunjins", "author_url": "", "post_date": "2025-07-10T14:25:37.737000", "content": "

That is what I noticed too

", "votes": 0, "replies": [] } ] } ] }, { "id": 3250183, "author_name": "Cyrus", "author_url": "", "post_date": "2025-07-17T20:42:37.293000", "content": "

it doesn't make sense to go through all the CSV files and manually check every code solution. There should be a new test set along with a notebook to run everything efficiently!!!

", "votes": 2, "replies": [] }, { "id": 3249373, "author_name": "Cyrus", "author_url": "", "post_date": "2025-07-16T10:31:46.583000", "content": "

Hi, thank you for your effort. For the final submission, do we need to load and process the test data using our model, or is submitting a CSV file with the predictions based on the current test data sufficient?

", "votes": 0, "replies": [ { "id": 3249445, "author_name": "Alex Zhongs", "author_url": "", "post_date": "2025-07-16T13:38:19.843000", "content": "

same question here

", "votes": 1, "replies": [] } ] }, { "id": 3246170, "author_name": "GreatestCutie", "author_url": "", "post_date": "2025-07-10T13:22:36.910000", "content": "

Dear DRW Hosts (or anyone who can help clarify),

\n

I'm puzzled by how this solves the issue. Users will already have downloaded and unshuffled test data locally. At the very least this allows them to train their models on the test data period biasing their models to the correct regime? As stated this is still against the rules, but who is going to catch the 79th place entry who boosted their score by a 0.05 by training on the forbidden data?

\n

I'm a total beginner so possibly I'm missing something obvious but as it stands I'm releuctant to spend the time to adapt to the new data and chasing an extra 0.01 if it'll be impossible to beat someone who just uses a baseline model and cheats for just enough gain to look believable.

\n

Thanks

", "votes": 2, "replies": [ { "id": 3246175, "author_name": "DRW Trading", "author_url": "", "post_date": "2025-07-10T13:36:04.110000", "content": "

The updated test data includes different true timestamps, making it difficult to map or reuse previous labels. We anticipated this and applied additional manipulations to ensure that prior future-peaking insights are no longer effective.

\n

We will continuously monitor for suspicious submissions with unusually high scores and conduct thorough code reviews to uphold fairness throughout the competition.

", "votes": 4, "replies": [ { "id": 3246180, "author_name": "GreatestCutie", "author_url": "", "post_date": "2025-07-10T13:58:10.120000", "content": "

ah ok, the new test data is a completely different time period? Thanks for clarifying.
\nActually just one more question for clarity if you don't mind; at the very least those with the old, unshuffled test data have twice as much data to train on, can we assume the new data is sufficiently transformed so that the old test data can't simply be transformed, clipped of the now-redundant features and trained on? Even if its a different period now, having twice as much data to train on is a big advantage.

", "votes": 0, "replies": [] }, { "id": 3249017, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2025-07-15T14:20:30.490000", "content": "

@drwtrading what about a mid-range score that uses the leak? Perhaps one could use the leak, drop useless features and then train a model.
\nWhat is the intended response in this case?

", "votes": 1, "replies": [] }, { "id": 3249848, "author_name": "Cyrus", "author_url": "", "post_date": "2025-07-17T08:13:50.683000", "content": "

it seems a waste of time here. not a good setup.

", "votes": 3, "replies": [] } ] } ] }, { "id": 3245896, "author_name": "Alex Zhongs", "author_url": "", "post_date": "2025-07-10T02:25:23.673000", "content": "

Hi DRW & Cumberland Team,

\n

Thanks for your efforts and fast response to keep this match fair. Do you intend to extend the deadline given new dataset used, or the deadline will stay the same?

", "votes": 0, "replies": [ { "id": 3245897, "author_name": "DRW Trading", "author_url": "", "post_date": "2025-07-10T02:26:06.553000", "content": "

We're still discussing the possibility of an extension and will share an update soon.

", "votes": 0, "replies": [ { "id": 3245902, "author_name": "paperxd", "author_url": "", "post_date": "2025-07-10T02:36:43.983000", "content": "

Its already been two months though

", "votes": 1, "replies": [] }, { "id": 3245918, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2025-07-10T03:29:07.983000", "content": "

True
\nThis should have been planned better
\nHope kaggle takes necessary measures to prevent effort and time waste of participants @paperxd

", "votes": 2, "replies": [] }, { "id": 3246104, "author_name": "Jackeylov3", "author_url": "", "post_date": "2025-07-10T10:24:57.750000", "content": "

since the feature has been updated, 2-4 weeks extension may be fine. If there is a new leaderboard without hacking, that would be better. Thanks for your organizing

", "votes": 1, "replies": [] }, { "id": 3246200, "author_name": "Taylor S. Amarel", "author_url": "", "post_date": "2025-07-10T14:38:39.993000", "content": "

I don't think an extension will cut it. I've looked at the new dataset along with the old train data and unmasked old test data. In my opinion, there are insights gathered through the unmasking of the old test data that can be applied to the new data to give a competitive advantage.

", "votes": 0, "replies": [] }, { "id": 3247002, "author_name": "EL Younes", "author_url": "", "post_date": "2025-07-12T00:20:57.843000", "content": "

We’re looking forward to the final outcome of your discussions. We truly hope that, if an extension is granted, it could be for at least two more weeks — that would be greatly appreciated!

", "votes": 0, "replies": [] }, { "id": 3247005, "author_name": "paperxd", "author_url": "", "post_date": "2025-07-12T00:26:29.740000", "content": "

We think there is no point in an extension, if we have an extension more people will just find ways to hack the leaderboard.

", "votes": 0, "replies": [] }, { "id": 3247008, "author_name": "EL Younes", "author_url": "", "post_date": "2025-07-12T00:31:19.457000", "content": "

They hacked the leaderboard already

", "votes": 0, "replies": [] } ] } ] }, { "id": 3253401, "author_name": "byunjins", "author_url": "", "post_date": "2025-07-24T15:51:46.837000", "content": "

@drwtrading I was wondering if it is possible to use different test set for the final submission.

", "votes": 0, "replies": [ { "id": 3253403, "author_name": "EL Younes", "author_url": "", "post_date": "2025-07-24T16:01:14.267000", "content": "

I believe a straightforward way to detect overfitted notebook submissions is by reshuffling the test set. This would cause models that rely heavily on timestamp order or lag-based features to perform poorly.

", "votes": 0, "replies": [ { "id": 3253404, "author_name": "byunjins", "author_url": "", "post_date": "2025-07-24T16:02:55.533000", "content": "

I agree with you. I think another option is using different set of test data.

", "votes": 0, "replies": [] }, { "id": 3253411, "author_name": "EL Younes", "author_url": "", "post_date": "2025-07-24T16:15:50.780000", "content": "

Using a completely different test set would make the competition winners largely a matter of random luck. To be honest, no model can generalize perfectly to any arbitrary dataset — it's like trying to use one key to open every door, which is impossible. A better way to eliminate notebooks that rely on future-peeking is by reshuffling the test set during the evaluation phase.

", "votes": 0, "replies": [] } ] } ] }, { "id": 3245865, "author_name": "KIT", "author_url": "", "post_date": "2025-07-10T00:55:20.103000", "content": "

Would it be possible to clarify if all prior submissions will be removed? Otherwise, the leaderboard is misleading. Additionally, will the competition time be extended?

", "votes": 0, "replies": [ { "id": 3245870, "author_name": "DRW Trading", "author_url": "", "post_date": "2025-07-10T00:58:21.600000", "content": "

All prior submissions will be considered invalid since the test data has changed. The leaderboard scores will be updated accordingly to reflect submissions made on the new dataset.

\n

As for the competition timeline, we’re still discussing whether an extension will be granted and will share an update soon.

", "votes": 1, "replies": [ { "id": 3246197, "author_name": "Taylor S. Amarel", "author_url": "", "post_date": "2025-07-10T14:32:12.220000", "content": "

In my opinion, the current competition should be shutdown, an apology issued. Then a reboot competition started, and you may want to increase the number of award slots to quell angst and discontent. Just my 2 cents.

\n

There is no way to ensure that any of the future peeking doesn't taint the results with unfair advantages.

", "votes": 3, "replies": [] }, { "id": 3246460, "author_name": "Long Issac Hair Fay", "author_url": "", "post_date": "2025-07-11T01:46:04.553000", "content": "

This seems like the only way forward. Even with adjusted features, teams/individuals can use insights gained from the previous unshuffled test dataset to get an unfair advantage.

", "votes": 0, "replies": [] }, { "id": 3246762, "author_name": "Taylor S. Amarel", "author_url": "", "post_date": "2025-07-11T13:46:18.893000", "content": "

As expected, someone has likely already reversed engineered the new dataset, possibly using the old dataset as well for guidance.
\n\"\"

", "votes": 2, "replies": [] }, { "id": 3246853, "author_name": "Deepak Saldanha", "author_url": "", "post_date": "2025-07-11T16:35:16.570000", "content": "

This might have been a glitch that was using the old LB

", "votes": 0, "replies": [] }, { "id": 3246868, "author_name": "Taylor S. Amarel", "author_url": "", "post_date": "2025-07-11T16:53:56.990000", "content": "

Unfortunately, it is not. The new order within the test dataset has been decoded. The obfuscation and reordering done by DRW was insufficient. I was able to score 0.65 just now.

", "votes": 2, "replies": [] }, { "id": 3246877, "author_name": "Alex Zhongs", "author_url": "", "post_date": "2025-07-11T17:27:39.963000", "content": "

Hi @taylorsamarel , are you sure that you reorder the entire test set right or only the public score part?

", "votes": 1, "replies": [] }, { "id": 3246884, "author_name": "Taylor S. Amarel", "author_url": "", "post_date": "2025-07-11T17:35:58.857000", "content": "

It may not be perfect, but it is pretty close. It wasn't all of my work, I was able to build off of other notebooks.

\n

It seems that the administrators of this competition are discounting notebooks that score greater than ~0.6, but I didn't see this in an announcement anywhere. I understand forward looking is against the competition rules, but I think there is learning to be had when experimenting with notebooks, as long as participants - like myself - understand that these 'peaking' notebooks won't be considered for a prize.

\n

I think the fundamental issue with this competition is that somewhat arbitrary and non-transparent actions are being taken in an attempt to 'level the playing field'. However, the data continues to provide 'leaked' insights for those that can decompose the new dataset.

\n

The issue comes from the fact that this 'decomposed insights' can be used to intelligently tune a model, that, at its face, does not involve rule breaking forward peaking.

", "votes": 1, "replies": [] }, { "id": 3246915, "author_name": "paperxd", "author_url": "", "post_date": "2025-07-11T19:11:54.910000", "content": "

DRW will conduct code reviews though

", "votes": 0, "replies": [] }, { "id": 3246923, "author_name": "YannFb", "author_url": "", "post_date": "2025-07-11T19:30:24.260000", "content": "

Have you deleted your submission yourself ? Or has it been moderated

", "votes": 0, "replies": [] }, { "id": 3246929, "author_name": "Taylor S. Amarel", "author_url": "", "post_date": "2025-07-11T19:49:22.433000", "content": "

Seems like it was administratively removed, not exactly sure what that means. Anything above 0.6 didn't stay on the board, but I still see the .599 score on the code section if you sort by score (not on the leaderboard though).

", "votes": 0, "replies": [] }, { "id": 3246930, "author_name": "Taylor S. Amarel", "author_url": "", "post_date": "2025-07-11T19:50:19.257000", "content": "

There is no way to determine if an engineered feature was a good guess, or if it was derived through future peaking. Someone could derive the perfect features to engineer using the shuffled label data and then just hard code those features into a separate notebook.

", "votes": 0, "replies": [] } ] } ] }, { "id": 3246069, "author_name": "", "author_url": "", "post_date": "2025-07-10T08:37:53.613000", "content": "", "votes": 0, "replies": [] }, { "id": 3246011, "author_name": "", "author_url": "", "post_date": "2025-07-10T06:47:00.427000", "content": "", "votes": 0, "replies": [] }, { "id": 3245877, "author_name": "", "author_url": "", "post_date": "2025-07-10T01:25:21.040000", "content": "", "votes": 0, "replies": [ { "id": 3245879, "author_name": "", "author_url": "", "post_date": "2025-07-10T01:43:28.370000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 3245838, "author_name": "", "author_url": "", "post_date": "2025-07-09T23:41:56.770000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3247577": "@drwtrading , can you help us understand what the code review will entail? I still don't fully understand how notebooks will be approved/disproved.\n\nIf you look at the results of running simple linear regression based feature analysis on the train dataset and the test dataset (using pseudo labels through unshuffling the data and using time series to estimate the prediction values), you can extract insights on the behavior of the test dataset. The below is based on a linear regression analysis (for simplicity) but the impact is the same. Anyone who is able to create psuedo labels based on the unshuffling and time-series prediction of the test data is then able to chose features that are most compatible with the train and test dataset - significantly improving scores and avoiding instability, noise, and drift.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10414474%2Fb5be68800f0317246b0846671f70b379%2F__results___1_1.png?generation=1752376298824875&alt=media)\n\nI am not suggesting anyone do this. Nor will I submit a notebook using these forward looking insights. However, I want to emphasize again, that in a submission notebook it would be impossible to tell if someone find a good feature or feature interaction through time series based psuedo label analysis (against the rules) or whether they found the feature through some other analysis that is based off train only.", "3245833": "Hi Kagglers,\n\nFirst, thank you all for your enthusiasm, effort, and valuable contributions to the competition so far. We understand it may be frustrating to see a data update mid-competition, but this decision was made after careful consideration to ensure fairness and integrity.\n\nThe competition data has been updated with a newly shuffled test set, and as a result, the leaderboard will be rebooted. The previous leaderboard no longer reflects the current data (which now has different true test timestamps), so please **re-download the dataset**, rerun your code, and submit updated predictions.\n\nWe are in contact with Kaggle regarding a leaderboard reboot. Please don’t be concerned if the updated scores are not yet visible.\n\nAs part of this update:\n\n- Test data have been newly shuffled. The rule against future peaking still applies.\n- Columns with excessive abnormal values have been removed to streamline data preprocessing. You will now see **X1–X780 instead of X1–X890**.\n- The remaining features have been normalized.\n\nWhile the proprietary and market features have been revised, they remain accurate and consistent in meaning. We've taken great care to preserve feature semantics and ensure that prior analyses remain valid and minimally impacted by these changes.\n\nThank you again for your understanding and continued participation—we're excited to see how your models perform on the refreshed dataset!\n\nBest regards,\nThe DRW & Cumberland Team\n\n------------\n\n**Update on Jul 11 (UTC): All submissions made before the dataset update have been removed. A very small number may appear rescored if the same participant submitted again around the time of the dataset update, due to technical reasons—but this has no impact on the leaderboard.**", "3250135": "Hi @drwtrading,\n\nCould you specify the format of final submission, given lots of abnormal public scores? Is everyone required to submit a Kaggle Notebook, or you will reach out to potential winners for their code solutions", "3245903": "Hi DRW,\n\nWhat do you mean by the remaining features have been normalized, does that mean you applied a z-score standardizer on the train dataset and transformed the test dataset? Also did all the columns get shuffled?\n\nThanks", "3246454": "Dear @drwtrading , I know you are working hard to manage the current situation, but I would suggest that you provide an update on what is happening with scoring and leaderboard changes, preferably before changes are made.\n\nIt is very confusing seeing the leaderboard change, some notebooks removed, other notebooks staying - there needs to be an official announcement on what is happening with more detail.", "3245858": "I'm honestly at a loss here. All of this is very disappointing to be honest, which from your message I think you already understand.\n\nHowever, reshuffling the data does not solve the problem because it does not eliminate the insights that people already gathered when analyzing the unshuffled dataset. These insights can be used to intelligently create features for training and test which will give significant advantages to some teams, without showing any visible signs of forward peaking in the notebook.\n\nAlso, can you help me understand what the statement below means:\n\n\"While the proprietary and market features have been revised, they remain accurate and consistent in meaning.\"\n\nIf they have been revised, how are they consistent in meaning? I hope this wasn't just a scaling adjustment, but that is the only way I could see \"consistent in meaning\" to be accurate.\n\n\n\n", "3246872": "I found yet another way to reverse the time for the test set. At this point I am not sure if the organizers will be able to keep the competition fair. \n\nWe will only know after code review, I guess.\nThen, again, how would one control for subjective calls about features (some of which can be pure lookahead)", "3246003": "Are column names this same?", "3250183": "it doesn't make sense to go through all the CSV files and manually check every code solution. There should be a new test set along with a notebook to run everything efficiently!!!", "3249373": "Hi, thank you for your effort. For the final submission, do we need to load and process the test data using our model, or is submitting a CSV file with the predictions based on the current test data sufficient?", "3246170": "Dear DRW Hosts (or anyone who can help clarify),\n\nI'm puzzled by how this solves the issue. Users will already have downloaded and unshuffled test data locally. At the very least this allows them to train their models on the test data period biasing their models to the correct regime? As stated this is still against the rules, but who is going to catch the 79th place entry who boosted their score by a 0.05 by training on the forbidden data?\n\nI'm a total beginner so possibly I'm missing something obvious but as it stands I'm releuctant to spend the time to adapt to the new data and chasing an extra 0.01 if it'll be impossible to beat someone who just uses a baseline model and cheats for just enough gain to look believable.\n\nThanks ", "3245896": "Hi DRW & Cumberland Team,\n\nThanks for your efforts and fast response to keep this match fair. Do you intend to extend the deadline given new dataset used, or the deadline will stay the same? ", "3253401": "@drwtrading I was wondering if it is possible to use different test set for the final submission. ", "3245865": "Would it be possible to clarify if all prior submissions will be removed? Otherwise, the leaderboard is misleading. Additionally, will the competition time be extended?", "3246069": "", "3246011": "", "3245877": "", "3245838": "" } }