![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16949276%2F92d3db9ddb6fbd059651a3d707332ddd%2FScreenshot_select-area_20250711182638.png?generation=1752222435764286&alt=media\")
reverse-engineer the test order. This creates a path to overfitting the test set by using it as a validation set in disguise — a known leaderboard gaming technique.First and foremost, I want to clarify my intent with this post:
\n❗I am not sharing this to encourage any reverse-engineering. I am sharing this because I believe there is a potential vulnerability in the current competition setup that could allow participants to intentionally exploit the test set — and achieve artificially high scores with/without using any future-looking features.
Despite the test set being reshuffled (which I sincerely appreciate — thank you DRW!), I’ve found that it is still possible to reverse-engineer the test order. This creates a path to overfitting the test set by using it as a validation set in disguise — a known leaderboard gaming technique.
This means that a model can be fine-tuned directly on the test set, potentially yielding high leaderboard scores that don't reflect true forecasting performance. While this does not involve explicit use of future-looking features, it creates an unfair advantage for those who are able to reconstruct the order.
\nI’ve learned from my previous post, where sharing certain code unintentionally caused confusion and controversy.
\nSo, let me be very clear:
\n🔒 I will not share the code or specific methods used to restore the order this time — doing so risks further disruption and defeats the purpose of raising this issue responsibly.
Once again, I appreciate the effort DRW has put into organizing and adjusting this competition. However, I believe this issue still compromises the fairness and integrity of the leaderboard.
\nI’ve shared two potential solutions in my earlier post:
\n🔗 Can We Still Fix This Competition?
I sincerely ask the organizers to consider these or alternative mechanisms to ensure a level playing field.
\nI want to stress again that I am fully against any strategy that restores the test time order or uses future-looking features to boost performance. Such approaches are unrealistic in real-world trading, and unfair to participants who are following the rules in good faith.
\nThanks again to DRW and the community for the opportunity to participate and learn — I hope this discussion contributes positively toward strengthening the competition's design.
", "messages": [ { "id": 3246614, "postDate": "2025-07-11T08:27:27.300Z", "content": "First and foremost, I want to clarify my intent with this post:
\n❗I am not sharing this to encourage any reverse-engineering. I am sharing this because I believe there is a potential vulnerability in the current competition setup that could allow participants to intentionally exploit the test set — and achieve artificially high scores with/without using any future-looking features.
Despite the test set being reshuffled (which I sincerely appreciate — thank you DRW!), I’ve found that it is still possible to reverse-engineer the test order. This creates a path to overfitting the test set by using it as a validation set in disguise — a known leaderboard gaming technique.
This means that a model can be fine-tuned directly on the test set, potentially yielding high leaderboard scores that don't reflect true forecasting performance. While this does not involve explicit use of future-looking features, it creates an unfair advantage for those who are able to reconstruct the order.
\nI’ve learned from my previous post, where sharing certain code unintentionally caused confusion and controversy.
\nSo, let me be very clear:
\n🔒 I will not share the code or specific methods used to restore the order this time — doing so risks further disruption and defeats the purpose of raising this issue responsibly.
Once again, I appreciate the effort DRW has put into organizing and adjusting this competition. However, I believe this issue still compromises the fairness and integrity of the leaderboard.
\nI’ve shared two potential solutions in my earlier post:
\n🔗 Can We Still Fix This Competition?
I sincerely ask the organizers to consider these or alternative mechanisms to ensure a level playing field.
\nI want to stress again that I am fully against any strategy that restores the test time order or uses future-looking features to boost performance. Such approaches are unrealistic in real-world trading, and unfair to participants who are following the rules in good faith.
\nThanks again to DRW and the community for the opportunity to participate and learn — I hope this discussion contributes positively toward strengthening the competition's design.
", "rawMarkdown": "# A Follow-Up on the Test Set Reshuffling\n\nFirst and foremost, I want to clarify my **intent** with this post: \n❗**I am not sharing this to encourage any reverse-engineering.** I am sharing this because I believe there is a **potential vulnerability** in the current competition setup that could allow participants to intentionally exploit the test set — and achieve artificially high scores **with/without using any future-looking features**.\n\n---\n\n## The Core Issue\n\nDespite the test set being reshuffled (which I sincerely appreciate — thank you DRW!), I’ve found that it is still possible to **reverse-engineer the test order**. This creates a path to overfitting the test set by using it as a validation set in disguise — a known leaderboard gaming technique.\n\nThis means that a model can be fine-tuned directly on the test set, potentially yielding high leaderboard scores that don't reflect true forecasting performance. While this does not involve explicit use of future-looking features, it creates **an unfair advantage** for those who are able to reconstruct the order.\n\n---\n\n## A Lesson Learned\n\nI’ve **learned from my previous post**, where sharing certain code unintentionally caused confusion and controversy. \nSo, let me be very clear: \n🔒 **I will not share the code** or specific methods used to restore the order this time — doing so risks further disruption and defeats the purpose of raising this issue responsibly.\n\n---\n\n## A Sincere Request to DRW\n\nOnce again, I appreciate the effort DRW has put into organizing and adjusting this competition. However, I believe this issue still compromises the fairness and integrity of the leaderboard.\n\nI’ve shared two potential solutions in my earlier post: \n🔗 [**Can We Still Fix This Competition?**](https://www.kaggle.com/competitions/drw-crypto-market-prediction/discussion/588896)\n\nI sincerely ask the organizers to consider these or alternative mechanisms to ensure a level playing field.\n\n---\n\n## Final Note\n\nI want to stress again that I am **fully against** any strategy that restores the test time order or uses future-looking features to boost performance. Such approaches are **unrealistic in real-world trading**, and **unfair to participants** who are following the rules in good faith.\n\nThanks again to DRW and the community for the opportunity to participate and learn — I hope this discussion contributes positively toward strengthening the competition's design.\n", "votes": 5 }, { "id": 3246803, "postDate": "2025-07-11T15:22:17.373Z", "content": "After everything has happened so far, I have no doubt that only when someone discloses the hacking strategy and everyone begins to blow up the leaderboard will the host show up and communicate
", "rawMarkdown": "After everything has happened so far, I have no doubt that only when someone discloses the hacking strategy and **everyone begins to blow up the leaderboard** will the host show up and communicate", "votes": 1 }, { "id": 3246779, "postDate": "2025-07-11T14:16:43.107Z", "content": "Ya they should probably just give all the data and then score on a new set. No amount of shuffling can can fix the issue
", "rawMarkdown": "Ya they should probably just give all the data and then score on a new set. No amount of shuffling can can fix the issue" }, { "id": 3246703, "postDate": "2025-07-11T11:26:52.117Z", "content": "Hi @mirko45 , I honestly don’t know the person who upvoted my post — but I appreciate the support regardless.
\n@oraclesleeping I’m also confused by your comment. If you believe it’s difficult to reach the top of the leaderboard after fully uncovering the structure of the test set, I’d have to disagree. Once the labels are essentially known, it's not hard to tune a model to appear impressive on the LB — but that doesn't reflect real-world forecasting ability.
\nMy English isn’t great, so I may have misunderstood your message. If possible, please try to use simpler language — I want to make sure we can have a fair and constructive discussion.
", "rawMarkdown": "Hi @mirko45 , I honestly don’t know the person who upvoted my post — but I appreciate the support regardless.\n\n@oraclesleeping I’m also confused by your comment. If you believe it’s difficult to reach the top of the leaderboard after fully uncovering the structure of the test set, I’d have to disagree. Once the labels are essentially known, it's not hard to tune a model to appear impressive on the LB — but that doesn't reflect real-world forecasting ability.\n\nMy English isn’t great, so I may have misunderstood your message. If possible, please try to use simpler language — I want to make sure we can have a fair and constructive discussion.", "replies": [ { "id": 3246723, "postDate": "2025-07-11T11:59:19.830Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3246732, "postDate": "2025-07-11T12:18:05.557Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3246781, "postDate": "2025-07-11T14:20:28.607Z", "content": "The guy is just unreasonably angry you're bringing attention to this exploit
", "rawMarkdown": "The guy is just unreasonably angry you're bringing attention to this exploit" }, { "id": 3246789, "postDate": "2025-07-11T14:39:57.457Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 3246638, "postDate": "2025-07-11T09:08:25.013Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 3246803, "author_name": "A_A", "author_url": "", "post_date": "2025-07-11T15:22:17.373000", "content": "After everything has happened so far, I have no doubt that only when someone discloses the hacking strategy and everyone begins to blow up the leaderboard will the host show up and communicate
", "votes": 1, "replies": [] }, { "id": 3246779, "author_name": "Jackson Fenner", "author_url": "", "post_date": "2025-07-11T14:16:43.107000", "content": "Ya they should probably just give all the data and then score on a new set. No amount of shuffling can can fix the issue
", "votes": 0, "replies": [] }, { "id": 3246703, "author_name": "ShinC", "author_url": "", "post_date": "2025-07-11T11:26:52.117000", "content": "Hi @mirko45 , I honestly don’t know the person who upvoted my post — but I appreciate the support regardless.
\n@oraclesleeping I’m also confused by your comment. If you believe it’s difficult to reach the top of the leaderboard after fully uncovering the structure of the test set, I’d have to disagree. Once the labels are essentially known, it's not hard to tune a model to appear impressive on the LB — but that doesn't reflect real-world forecasting ability.
\nMy English isn’t great, so I may have misunderstood your message. If possible, please try to use simpler language — I want to make sure we can have a fair and constructive discussion.
", "votes": 0, "replies": [ { "id": 3246723, "author_name": "", "author_url": "", "post_date": "2025-07-11T11:59:19.830000", "content": "", "votes": 0, "replies": [] }, { "id": 3246732, "author_name": "", "author_url": "", "post_date": "2025-07-11T12:18:05.557000", "content": "", "votes": 0, "replies": [] }, { "id": 3246781, "author_name": "Jackson Fenner", "author_url": "", "post_date": "2025-07-11T14:20:28.607000", "content": "The guy is just unreasonably angry you're bringing attention to this exploit
", "votes": 0, "replies": [] }, { "id": 3246789, "author_name": "", "author_url": "", "post_date": "2025-07-11T14:39:57.457000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 3246638, "author_name": "", "author_url": "", "post_date": "2025-07-11T09:08:25.013000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3246614": "# A Follow-Up on the Test Set Reshuffling\n\nFirst and foremost, I want to clarify my **intent** with this post: \n❗**I am not sharing this to encourage any reverse-engineering.** I am sharing this because I believe there is a **potential vulnerability** in the current competition setup that could allow participants to intentionally exploit the test set — and achieve artificially high scores **with/without using any future-looking features**.\n\n---\n\n## The Core Issue\n\nDespite the test set being reshuffled (which I sincerely appreciate — thank you DRW!), I’ve found that it is still possible to **reverse-engineer the test order**. This creates a path to overfitting the test set by using it as a validation set in disguise — a known leaderboard gaming technique.\n\nThis means that a model can be fine-tuned directly on the test set, potentially yielding high leaderboard scores that don't reflect true forecasting performance. While this does not involve explicit use of future-looking features, it creates **an unfair advantage** for those who are able to reconstruct the order.\n\n---\n\n## A Lesson Learned\n\nI’ve **learned from my previous post**, where sharing certain code unintentionally caused confusion and controversy. \nSo, let me be very clear: \n🔒 **I will not share the code** or specific methods used to restore the order this time — doing so risks further disruption and defeats the purpose of raising this issue responsibly.\n\n---\n\n## A Sincere Request to DRW\n\nOnce again, I appreciate the effort DRW has put into organizing and adjusting this competition. However, I believe this issue still compromises the fairness and integrity of the leaderboard.\n\nI’ve shared two potential solutions in my earlier post: \n🔗 [**Can We Still Fix This Competition?**](https://www.kaggle.com/competitions/drw-crypto-market-prediction/discussion/588896)\n\nI sincerely ask the organizers to consider these or alternative mechanisms to ensure a level playing field.\n\n---\n\n## Final Note\n\nI want to stress again that I am **fully against** any strategy that restores the test time order or uses future-looking features to boost performance. Such approaches are **unrealistic in real-world trading**, and **unfair to participants** who are following the rules in good faith.\n\nThanks again to DRW and the community for the opportunity to participate and learn — I hope this discussion contributes positively toward strengthening the competition's design.\n", "3246803": "After everything has happened so far, I have no doubt that only when someone discloses the hacking strategy and **everyone begins to blow up the leaderboard** will the host show up and communicate", "3246779": "Ya they should probably just give all the data and then score on a new set. No amount of shuffling can can fix the issue", "3246703": "Hi @mirko45 , I honestly don’t know the person who upvoted my post — but I appreciate the support regardless.\n\n@oraclesleeping I’m also confused by your comment. If you believe it’s difficult to reach the top of the leaderboard after fully uncovering the structure of the test set, I’d have to disagree. Once the labels are essentially known, it's not hard to tune a model to appear impressive on the LB — but that doesn't reflect real-world forecasting ability.\n\nMy English isn’t great, so I may have misunderstood your message. If possible, please try to use simpler language — I want to make sure we can have a fair and constructive discussion.", "3246638": "" } }