![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F715257%2F9c4996f8a1c8e4b5705479554f392598%2Fslow_or_fast_leak.png?generation=1700273145933699&alt=media\")
We can know using the prediction results of random as the default results can greatly improve both lb and pb from this discussion.
\nhttps://www.kaggle.com/competitions/predict-ai-model-runtime/discussion/456083
My team also can achieve a very high score with this leakage
After discussed with other teams, we think the possible reason is from test data sampling. The original test lables(random and default both) are ranked first,then use the same seed for sampling,so they become same lable for this metrics(Kendal Tau Correlation ).
\nSome teams maybe used this leakage, some didn't, it's not fair to all teams, we hope can re-calculate the learderboard with same leakage or without leakage.
\n@mangpophothilimthana @ashleychow
", "messages": [ { "id": "2529160", "postDate": "11/18/2023 02:07:23", "content": "We can know using the prediction results of random as the default results can greatly improve both lb and pb from this discussion.
\nhttps://www.kaggle.com/competitions/predict-ai-model-runtime/discussion/456083
My team also can achieve a very high score with this leakage
After discussed with other teams, we think the possible reason is from test data sampling. The original test lables(random and default both) are ranked first,then use the same seed for sampling,so they become same lable for this metrics(Kendal Tau Correlation ).
\nSome teams maybe used this leakage, some didn't, it's not fair to all teams, we hope can re-calculate the learderboard with same leakage or without leakage.
\n@mangpophothilimthana @ashleychow
", "rawMarkdown": "We can know using the prediction results of random as the default results can greatly improve both lb and pb from this discussion.\nhttps://www.kaggle.com/competitions/predict-ai-model-runtime/discussion/456083\n\nMy team also can achieve a very high score with this leakage\n\nAfter discussed with other teams, we think the possible reason is from test data sampling. The original test lables(random and default both) are ranked first,then use the same seed for sampling,so they become same lable for this metrics(Kendal Tau Correlation ).\n\nSome teams maybe used this leakage, some didn't, it's not fair to all teams, we hope can re-calculate the learderboard with same leakage or without leakage. \n\n@mangpophothilimthana @ashleychow", "votes": null }, { "id": "2532344", "postDate": "11/21/2023 01:24:59", "content": "Thanks @senkin13! The leaderboard has been finalized. Thanks!
", "rawMarkdown": "Thanks @senkin13! The leaderboard has been finalized. Thanks!", "votes": null }, { "id": "2533077", "postDate": "11/21/2023 15:16:14", "content": "We did some analysis. We see that none of the top 6 winning teams have cheated. Specifically, because all of their scores on \"{nlp, xla}:default\" collection are much worse than their scores on \"{nlp, xla}:random\" collection.
\nHowever, in the ranks of >6 (7 through 30), we find there are two teams with very similar \"default\" and \"random\" scores, which might've used the trick in discussion/456083
", "rawMarkdown": "We did some analysis. We see that none of the top 6 winning teams have cheated. Specifically, because all of their scores on \"{nlp, xla}:default\" collection are much worse than their scores on \"{nlp, xla}:random\" collection.\n\nHowever, in the ranks of >6 (7 through 30), we find there are two teams with very similar \"default\" and \"random\" scores, which might've used the trick in [discussion/456083](https://www.kaggle.com/competitions/predict-ai-model-runtime/discussion/456083)", "votes": null } ], "comments": [ { "id": 2532344, "author_name": "ashleychow", "author_url": "", "post_date": "11/21/2023 01:24:59", "content": "Thanks @senkin13! The leaderboard has been finalized. Thanks!
", "votes": null, "replies": [] }, { "id": 2533077, "author_name": "samihaija", "author_url": "", "post_date": "11/21/2023 15:16:14", "content": "We did some analysis. We see that none of the top 6 winning teams have cheated. Specifically, because all of their scores on \"{nlp, xla}:default\" collection are much worse than their scores on \"{nlp, xla}:random\" collection.
\nHowever, in the ranks of >6 (7 through 30), we find there are two teams with very similar \"default\" and \"random\" scores, which might've used the trick in discussion/456083
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2529160": "We can know using the prediction results of random as the default results can greatly improve both lb and pb from this discussion.\nhttps://www.kaggle.com/competitions/predict-ai-model-runtime/discussion/456083\n\nMy team also can achieve a very high score with this leakage\n\nAfter discussed with other teams, we think the possible reason is from test data sampling. The original test lables(random and default both) are ranked first,then use the same seed for sampling,so they become same lable for this metrics(Kendal Tau Correlation ).\n\nSome teams maybe used this leakage, some didn't, it's not fair to all teams, we hope can re-calculate the learderboard with same leakage or without leakage. \n\n@mangpophothilimthana @ashleychow", "2532344": "Thanks @senkin13! The leaderboard has been finalized. Thanks!", "2533077": "We did some analysis. We see that none of the top 6 winning teams have cheated. Specifically, because all of their scores on \"{nlp, xla}:default\" collection are much worse than their scores on \"{nlp, xla}:random\" collection.\n\nHowever, in the ranks of >6 (7 through 30), we find there are two teams with very similar \"default\" and \"random\" scores, which might've used the trick in [discussion/456083](https://www.kaggle.com/competitions/predict-ai-model-runtime/discussion/456083)" }, "source": "meta" }