After the leaderboard was hacked for the second time, I believe the organizers must take a serious decision — not just regarding the public leaderboard, but also the private one. Both should be disregarded. Instead, the winning solution should be selected based on the best strategy or methodology that can generalize and work well in a real-world environment.
\nThis is a time series regression problem, and I fully understand why the organizers masked the timestamps before shuffling the test set — to prevent cheating by using future information to predict the present. However, the downside is that masking and shuffling add artificial noise that doesn't exist in real scenarios. As a result, only naive models tend to perform well under these conditions, but they are unlikely to be useful in real-life applications.
\nSo I strongly urge the organizers to take meaningful action: the winner of this competition should be selected based on a useful and realistic methodology, not just leaderboard performance. And in future competitions, I suggest using API-based submissions only, without masking or shuffling the time series, to ensure integrity and real-world relevance.
", "messages": [ { "id": "3248994", "postDate": "07/15/2025 13:58:52", "content": "After the leaderboard was hacked for the second time, I believe the organizers must take a serious decision — not just regarding the public leaderboard, but also the private one. Both should be disregarded. Instead, the winning solution should be selected based on the best strategy or methodology that can generalize and work well in a real-world environment.
\nThis is a time series regression problem, and I fully understand why the organizers masked the timestamps before shuffling the test set — to prevent cheating by using future information to predict the present. However, the downside is that masking and shuffling add artificial noise that doesn't exist in real scenarios. As a result, only naive models tend to perform well under these conditions, but they are unlikely to be useful in real-life applications.
\nSo I strongly urge the organizers to take meaningful action: the winner of this competition should be selected based on a useful and realistic methodology, not just leaderboard performance. And in future competitions, I suggest using API-based submissions only, without masking or shuffling the time series, to ensure integrity and real-world relevance.
", "rawMarkdown": "After the leaderboard was hacked for the second time, I believe the organizers must take a serious decision — not just regarding the public leaderboard, but also the private one. Both should be disregarded. Instead, the winning solution should be selected based on the best strategy or methodology that can generalize and work well in a real-world environment.\n\nThis is a time series regression problem, and I fully understand why the organizers masked the timestamps before shuffling the test set — to prevent cheating by using future information to predict the present. However, the downside is that masking and shuffling add artificial noise that doesn't exist in real scenarios. As a result, only naive models tend to perform well under these conditions, but they are unlikely to be useful in real-life applications.\n\nSo I strongly urge the organizers to take meaningful action: the winner of this competition should be selected based on a useful and realistic methodology, not just leaderboard performance. And in future competitions, I suggest using API-based submissions only, without masking or shuffling the time series, to ensure integrity and real-world relevance.", "votes": null }, { "id": "3249034", "postDate": "07/15/2025 14:41:18", "content": "Cannot agree more.
\nMultiple testing the test set is always a temptation hard to resist when doing research on ones' own or inside quant firms. Asking winners for their thorough in-sample research processes is of the best ways to prevent this problem.
", "rawMarkdown": "Cannot agree more. \n\nMultiple testing the test set is always a temptation hard to resist when doing research on ones' own or inside quant firms. Asking winners for their thorough in-sample research processes is of the best ways to prevent this problem.", "votes": null }, { "id": "3249150", "postDate": "07/15/2025 20:05:39", "content": "I think we should just finish the competition because I don’t think it’s salvageable. It’s a bit demotivating but I guess it is what it is.
", "rawMarkdown": "I think we should just finish the competition because I don’t think it’s salvageable. It’s a bit demotivating but I guess it is what it is.", "votes": null }, { "id": "3249154", "postDate": "07/15/2025 20:13:55", "content": "I think its unfair and unreasonable to suddenly change the judging criteria with a week left. People have tailored their model to the competition data, I would imagine almost no one has gone out of their way to maximise real-world utility at the cost of compeition performance. This would then become a competition between the few people that just decided to do that for some reason.
", "rawMarkdown": "I think its unfair and unreasonable to suddenly change the judging criteria with a week left. People have tailored their model to the competition data, I would imagine almost no one has gone out of their way to maximise real-world utility at the cost of compeition performance. This would then become a competition between the few people that just decided to do that for some reason.", "votes": null }, { "id": "3249155", "postDate": "07/15/2025 20:15:04", "content": "Agreed, we have already spent a lot of time on it.
", "rawMarkdown": "Agreed, we have already spent a lot of time on it.", "votes": null }, { "id": "3249157", "postDate": "07/15/2025 20:20:43", "content": "True, lots time spent lol
", "rawMarkdown": "True, lots time spent lol", "votes": null }, { "id": "3249208", "postDate": "07/16/2025 03:06:11", "content": "There is nothing wrong to hack and do data reverse engineering and submit.
\nWhat we are lacking are some brave guys to tell the truth at the beginning of the competition.
\nSome people know the trick/hack early and think they can profit from it. In the end it is loss for everyone.
\nTo be honest , I don’t think it’s salvageable. It’s a bit demotivating but I guess it is what it is. Maybe we can finish this competition only like one assignment from college lol.
", "rawMarkdown": "To be honest , I don’t think it’s salvageable. It’s a bit demotivating but I guess it is what it is. Maybe we can finish this competition only like one assignment from college lol.", "votes": null }, { "id": "3249327", "postDate": "07/16/2025 08:49:47", "content": "I think it there are several realistic possibilities.
\nAt inference you are only allowed to use the current test row + the much older training data (as is currently the case in this competition) and then that will of course hurt the model performance, since there is only so much you can do with that information.
At inference you are allowed to use the current test row, the previous test rows, and the training data --> then the results will likely become much better since you have access to additional newer information (but still older than the current test row).
In this competition, it seems they are mainly interested in scenario 1, and then I don't believe its realistic to expect much higher correlation than around 0.1. Given that, one could still take precautions to ensure that no data newer than the current test row is used at any point during inference.
", "rawMarkdown": "I think it there are several realistic possibilities.\n\n1. At inference you are only allowed to use the current test row + the much older training data (as is currently the case in this competition) and then that will of course hurt the model performance, since there is only so much you can do with that information.\n\n2. At inference you are allowed to use the current test row, the previous test rows, and the training data --> then the results will likely become much better since you have access to additional newer information (but still older than the current test row). \n\nIn this competition, it seems they are mainly interested in scenario 1, and then I don't believe its realistic to expect much higher correlation than around 0.1. Given that, one could still take precautions to ensure that no data newer than the current test row is used at any point during inference.", "votes": null }, { "id": "3249750", "postDate": "07/17/2025 03:50:55", "content": "Yes you are right
", "rawMarkdown": "Yes you are right", "votes": null }, { "id": "3249915", "postDate": "07/17/2025 10:42:01", "content": "The Kaggle community is full of smart people who are really good at finding structure and signals in data. Ideally, a competition would be set up to reward those skills, rather than equate them with cheating. Given the size of the generous prizes the hosts are offering, it should have been possible to set this up as a featured competition using Kaggle's time series submission API (or something similar). This would have aligned data science skills with success rather than with misconduct, and avoided arguments over what fraction of the juice within the data comes from forbidden fruit.
\nAs to the OP's original suggestion, I don't believe it's reasonable to pivot to a completely different victory condition at this stage of a competition.
", "rawMarkdown": "The Kaggle community is full of smart people who are really good at finding structure and signals in data. Ideally, a competition would be set up to reward those skills, rather than equate them with cheating. Given the size of the generous prizes the hosts are offering, it should have been possible to set this up as a featured competition using Kaggle's time series submission API (or something similar). This would have aligned data science skills with success rather than with misconduct, and avoided arguments over what fraction of the juice within the data comes from forbidden fruit.\n\nAs to the OP's original suggestion, I don't believe it's reasonable to pivot to a completely different victory condition at this stage of a competition.", "votes": null }, { "id": "3250301", "postDate": "07/18/2025 05:07:37", "content": "And let's don't forget the biggest ostrich behind the scene.
", "rawMarkdown": "And let's don't forget the biggest ostrich behind the scene.", "votes": null } ], "comments": [ { "id": 3249034, "author_name": "alexzhongs", "author_url": "", "post_date": "07/15/2025 14:41:18", "content": "Cannot agree more.
\nMultiple testing the test set is always a temptation hard to resist when doing research on ones' own or inside quant firms. Asking winners for their thorough in-sample research processes is of the best ways to prevent this problem.
", "votes": null, "replies": [] }, { "id": 3249150, "author_name": "paperxd", "author_url": "", "post_date": "07/15/2025 20:05:39", "content": "I think we should just finish the competition because I don’t think it’s salvageable. It’s a bit demotivating but I guess it is what it is.
", "votes": null, "replies": [] }, { "id": 3249154, "author_name": "greatestcutie", "author_url": "", "post_date": "07/15/2025 20:13:55", "content": "I think its unfair and unreasonable to suddenly change the judging criteria with a week left. People have tailored their model to the competition data, I would imagine almost no one has gone out of their way to maximise real-world utility at the cost of compeition performance. This would then become a competition between the few people that just decided to do that for some reason.
", "votes": null, "replies": [ { "id": 3249155, "author_name": "paperxd", "author_url": "", "post_date": "07/15/2025 20:15:04", "content": "Agreed, we have already spent a lot of time on it.
", "votes": null, "replies": [ { "id": 3249157, "author_name": "alexzhongs", "author_url": "", "post_date": "07/15/2025 20:20:43", "content": "True, lots time spent lol
", "votes": null, "replies": [] } ] } ] }, { "id": 3249208, "author_name": "hengck23", "author_url": "", "post_date": "07/16/2025 03:06:11", "content": "There is nothing wrong to hack and do data reverse engineering and submit.
\nWhat we are lacking are some brave guys to tell the truth at the beginning of the competition.
\nSome people know the trick/hack early and think they can profit from it. In the end it is loss for everyone.
\nThe Kaggle community is full of smart people who are really good at finding structure and signals in data. Ideally, a competition would be set up to reward those skills, rather than equate them with cheating. Given the size of the generous prizes the hosts are offering, it should have been possible to set this up as a featured competition using Kaggle's time series submission API (or something similar). This would have aligned data science skills with success rather than with misconduct, and avoided arguments over what fraction of the juice within the data comes from forbidden fruit.
\nAs to the OP's original suggestion, I don't believe it's reasonable to pivot to a completely different victory condition at this stage of a competition.
", "votes": null, "replies": [] }, { "id": 3250301, "author_name": "tony271ynot", "author_url": "", "post_date": "07/18/2025 05:07:37", "content": "And let's don't forget the biggest ostrich behind the scene.
", "votes": null, "replies": [] } ] }, { "id": 3249291, "author_name": "meijieyue2025", "author_url": "", "post_date": "07/16/2025 07:20:30", "content": "To be honest , I don’t think it’s salvageable. It’s a bit demotivating but I guess it is what it is. Maybe we can finish this competition only like one assignment from college lol.
", "votes": null, "replies": [] }, { "id": 3249327, "author_name": "ern711", "author_url": "", "post_date": "07/16/2025 08:49:47", "content": "I think it there are several realistic possibilities.
\nAt inference you are only allowed to use the current test row + the much older training data (as is currently the case in this competition) and then that will of course hurt the model performance, since there is only so much you can do with that information.
At inference you are allowed to use the current test row, the previous test rows, and the training data --> then the results will likely become much better since you have access to additional newer information (but still older than the current test row).
In this competition, it seems they are mainly interested in scenario 1, and then I don't believe its realistic to expect much higher correlation than around 0.1. Given that, one could still take precautions to ensure that no data newer than the current test row is used at any point during inference.
", "votes": null, "replies": [] }, { "id": 3249750, "author_name": "zunairaamanshad", "author_url": "", "post_date": "07/17/2025 03:50:55", "content": "Yes you are right
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3248994": "After the leaderboard was hacked for the second time, I believe the organizers must take a serious decision — not just regarding the public leaderboard, but also the private one. Both should be disregarded. Instead, the winning solution should be selected based on the best strategy or methodology that can generalize and work well in a real-world environment.\n\nThis is a time series regression problem, and I fully understand why the organizers masked the timestamps before shuffling the test set — to prevent cheating by using future information to predict the present. However, the downside is that masking and shuffling add artificial noise that doesn't exist in real scenarios. As a result, only naive models tend to perform well under these conditions, but they are unlikely to be useful in real-life applications.\n\nSo I strongly urge the organizers to take meaningful action: the winner of this competition should be selected based on a useful and realistic methodology, not just leaderboard performance. And in future competitions, I suggest using API-based submissions only, without masking or shuffling the time series, to ensure integrity and real-world relevance.", "3249034": "Cannot agree more. \n\nMultiple testing the test set is always a temptation hard to resist when doing research on ones' own or inside quant firms. Asking winners for their thorough in-sample research processes is of the best ways to prevent this problem.", "3249150": "I think we should just finish the competition because I don’t think it’s salvageable. It’s a bit demotivating but I guess it is what it is.", "3249154": "I think its unfair and unreasonable to suddenly change the judging criteria with a week left. People have tailored their model to the competition data, I would imagine almost no one has gone out of their way to maximise real-world utility at the cost of compeition performance. This would then become a competition between the few people that just decided to do that for some reason.", "3249155": "Agreed, we have already spent a lot of time on it.", "3249157": "True, lots time spent lol", "3249208": "There is nothing wrong to hack and do data reverse engineering and submit.\n\nWhat we are lacking are some brave guys to tell the truth at the beginning of the competition.\n\nSome people know the trick/hack early and think they can profit from it. In the end it is loss for everyone.\n\n###Conclusion: community competition needs community responsibility", "3249291": "To be honest , I don’t think it’s salvageable. It’s a bit demotivating but I guess it is what it is. Maybe we can finish this competition only like one assignment from college lol.", "3249327": "I think it there are several realistic possibilities.\n\n1. At inference you are only allowed to use the current test row + the much older training data (as is currently the case in this competition) and then that will of course hurt the model performance, since there is only so much you can do with that information.\n\n2. At inference you are allowed to use the current test row, the previous test rows, and the training data --> then the results will likely become much better since you have access to additional newer information (but still older than the current test row). \n\nIn this competition, it seems they are mainly interested in scenario 1, and then I don't believe its realistic to expect much higher correlation than around 0.1. Given that, one could still take precautions to ensure that no data newer than the current test row is used at any point during inference.", "3249750": "Yes you are right", "3249915": "The Kaggle community is full of smart people who are really good at finding structure and signals in data. Ideally, a competition would be set up to reward those skills, rather than equate them with cheating. Given the size of the generous prizes the hosts are offering, it should have been possible to set this up as a featured competition using Kaggle's time series submission API (or something similar). This would have aligned data science skills with success rather than with misconduct, and avoided arguments over what fraction of the juice within the data comes from forbidden fruit.\n\nAs to the OP's original suggestion, I don't believe it's reasonable to pivot to a completely different victory condition at this stage of a competition.", "3250301": "And let's don't forget the biggest ostrich behind the scene." }, "source": "meta" }