A serious bug in Kaggle API let people see their private scores unintentionally. The following topic has been discussed in several comments but I'd like to discuss it separately.
\nEven though a majority of people think the effect on Riiid is smaller compared to other competitions because CV/LB are correlated closely, I still think it gave a slight advantage. For example, checking scores of models focused on old users/new users. At least, if you check your private scores, it gave a mental relief.
\nIf the host changes the private test set, then some of the advantages might be gone but still there could be some. So, I suggest Kaggle reveal private scores and its rank before the bug fixed.
\nI want this competition go back to a fair competition where winners win because of their good models and everybody else can think so.
\nWhat do you think? Is there anything bad by revealing private scores so far?
\n( Personally, I didn't see my scores but it's just because it was fixed already. I might have tried if not fixed.)
EDIT: (12-30-2020) After give it another thought, I realize that revealing private score before the bug doesn't resolve all problems. It found out that the bug has been there since November 18, 2019. So, people could have experienced a lot of things.
\nSo, how can we make this competition a fair competition again?
\nRevealing private scores is one thing but we have no longer an advantage of being able to experience with private data.
Revealing private data set and replace it with new data. This might cause a discontinuity of users' records.
Revealing private data, and introduce new train and private data sets…
A serious bug in Kaggle API let people see their private scores unintentionally. The following topic has been discussed in several comments but I'd like to discuss it separately.
\nEven though a majority of people think the effect on Riiid is smaller compared to other competitions because CV/LB are correlated closely, I still think it gave a slight advantage. For example, checking scores of models focused on old users/new users. At least, if you check your private scores, it gave a mental relief.
\nIf the host changes the private test set, then some of the advantages might be gone but still there could be some. So, I suggest Kaggle reveal private scores and its rank before the bug fixed.
\nI want this competition go back to a fair competition where winners win because of their good models and everybody else can think so.
\nWhat do you think? Is there anything bad by revealing private scores so far?
\n( Personally, I didn't see my scores but it's just because it was fixed already. I might have tried if not fixed.)
EDIT: (12-30-2020) After give it another thought, I realize that revealing private score before the bug doesn't resolve all problems. It found out that the bug has been there since November 18, 2019. So, people could have experienced a lot of things.
\nSo, how can we make this competition a fair competition again?
\nRevealing private scores is one thing but we have no longer an advantage of being able to experience with private data.
Revealing private data set and replace it with new data. This might cause a discontinuity of users' records.
Revealing private data, and introduce new train and private data sets…
I hesitated to check (although it had been fixed when I saw the discussion about the bug) - but back to the time, I would rather not to check the private score - at least, I will enjoy the surprise that I might ends in a higher position in the private LB (the probability is not 0.0, so let me dream about it …)
\nWith the action of checking my private LB, the joy of winning disappears … for me
", "rawMarkdown": "I hesitated to check (although it had been fixed when I saw the discussion about the bug) - but back to the time, I would rather not to check the private score - at least, I will enjoy the surprise that I might ends in a higher position in the private LB (the probability is not 0.0, so let me dream about it ...)\n\nWith the action of checking my private LB, the joy of winning disappears ... for me", "votes": null }, { "id": "1133110", "postDate": "12/31/2020 00:21:15", "content": "Yes, I understand and I agree the excitement of the surprise. What's problem is that some people who checked their private score might take advantage of it and deprive you of the joy of winning.
", "rawMarkdown": "Yes, I understand and I agree the excitement of the surprise. What's problem is that some people who checked their private score might take advantage of it and deprive you of the joy of winning.", "votes": null }, { "id": "1133168", "postDate": "12/31/2020 02:04:16", "content": "This seems like throwing out the baby and the bathwater to me.
\nI think with two subs and good cv/lb correlation, this competition isn't so much at risk as some other historic competitions. Then again, you might see me eating my words in seven days…
", "rawMarkdown": "This seems like throwing out the baby and the bathwater to me.\n\nI think with two subs and good cv/lb correlation, this competition isn't so much at risk as some other historic competitions. Then again, you might see me eating my words in seven days...", "votes": null }, { "id": "1133185", "postDate": "12/31/2020 02:31:22", "content": "I agree with the idea of revealing private scores before the bug fixed. Introducing new private test set is recommended, but it will possibly make the competition meaningless if kaggle admins and the hosts cannot prepare a proper private test set. Specifically, if they have already used all of the history of the users in the training set (i.e. \"all-in\"), it will be hard to prepare the proper private test set. If so, they probably need to introduce new training set, which costs the participants a lot of money…
", "rawMarkdown": "I agree with the idea of revealing private scores before the bug fixed. Introducing new private test set is recommended, but it will possibly make the competition meaningless if kaggle admins and the hosts cannot prepare a proper private test set. Specifically, if they have already used all of the history of the users in the training set (i.e. \"all-in\"), it will be hard to prepare the proper private test set. If so, they probably need to introduce new training set, which costs the participants a lot of money...", "votes": null }, { "id": "1133187", "postDate": "12/31/2020 02:33:32", "content": "In this thread https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/191106, @sohier said \"The train/test data is complete, in the sense that there are no missing interactions in the union of train and test data.\", which suggests sadly they did \"all-in\"…
", "rawMarkdown": "In this thread https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/191106, @sohier said \"The train/test data is complete, in the sense that there are no missing interactions in the union of train and test data.\", which suggests sadly they did \"all-in\"...", "votes": null }, { "id": "1133219", "postDate": "12/31/2020 03:10:21", "content": "If we all know previous private LB score, the competition would become fair in some degree.
\nAt the same time, it makes the competition completely deviated from the standard practice of ML/DS or that of Kaggle.
\nThat's why I did not see my score, though I can not prove myself about it unfortunately.
\nIf anyone say the competition has already deviated from the standard, I have to admit it😓.
Anyway, I think only thing we can do currently is being the first on both public and private LB. Nobody can have silly suspicion of cheating.
", "rawMarkdown": "If we all know previous private LB score, the competition would become fair in some degree. \nAt the same time, it makes the competition completely deviated from the standard practice of ML/DS or that of Kaggle.\nThat's why I did not see my score, though I can not prove myself about it unfortunately.\nIf anyone say the competition has already deviated from the standard, I have to admit it😓.\n\nAnyway, I think only thing we can do currently is being the first on both public and private LB. Nobody can have silly suspicion of cheating.", "votes": null } ], "comments": [ { "id": 1132750, "author_name": "yihdarshieh", "author_url": "", "post_date": "12/30/2020 16:53:24", "content": "I hesitated to check (although it had been fixed when I saw the discussion about the bug) - but back to the time, I would rather not to check the private score - at least, I will enjoy the surprise that I might ends in a higher position in the private LB (the probability is not 0.0, so let me dream about it …)
\nWith the action of checking my private LB, the joy of winning disappears … for me
", "votes": null, "replies": [ { "id": 1133110, "author_name": "yutsumura", "author_url": "", "post_date": "12/31/2020 00:21:15", "content": "Yes, I understand and I agree the excitement of the surprise. What's problem is that some people who checked their private score might take advantage of it and deprive you of the joy of winning.
", "votes": null, "replies": [] } ] }, { "id": 1133168, "author_name": "authman", "author_url": "", "post_date": "12/31/2020 02:04:16", "content": "This seems like throwing out the baby and the bathwater to me.
\nI think with two subs and good cv/lb correlation, this competition isn't so much at risk as some other historic competitions. Then again, you might see me eating my words in seven days…
", "votes": null, "replies": [] }, { "id": 1133185, "author_name": "mamasinkgs", "author_url": "", "post_date": "12/31/2020 02:31:22", "content": "I agree with the idea of revealing private scores before the bug fixed. Introducing new private test set is recommended, but it will possibly make the competition meaningless if kaggle admins and the hosts cannot prepare a proper private test set. Specifically, if they have already used all of the history of the users in the training set (i.e. \"all-in\"), it will be hard to prepare the proper private test set. If so, they probably need to introduce new training set, which costs the participants a lot of money…
", "votes": null, "replies": [ { "id": 1133187, "author_name": "mamasinkgs", "author_url": "", "post_date": "12/31/2020 02:33:32", "content": "In this thread https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/191106, @sohier said \"The train/test data is complete, in the sense that there are no missing interactions in the union of train and test data.\", which suggests sadly they did \"all-in\"…
", "votes": null, "replies": [] } ] }, { "id": 1133219, "author_name": "tomooinubushi", "author_url": "", "post_date": "12/31/2020 03:10:21", "content": "If we all know previous private LB score, the competition would become fair in some degree.
\nAt the same time, it makes the competition completely deviated from the standard practice of ML/DS or that of Kaggle.
\nThat's why I did not see my score, though I can not prove myself about it unfortunately.
\nIf anyone say the competition has already deviated from the standard, I have to admit it😓.
Anyway, I think only thing we can do currently is being the first on both public and private LB. Nobody can have silly suspicion of cheating.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1132747": "A [serious bug](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/207390) in Kaggle API let people see their private scores unintentionally. The following topic has been discussed in several comments but I'd like to discuss it separately.\n\nEven though a majority of people think the effect on Riiid is smaller compared to other competitions because CV/LB are correlated closely, I still think it gave a slight advantage. For example, checking scores of models focused on old users/new users. At least, if you check your private scores, it gave a mental relief.\n\nIf the host changes the private test set, then some of the advantages might be gone but still there could be some. So, I suggest Kaggle reveal private scores and its rank before the bug fixed.\n\nI want this competition go back to a fair competition where winners win because of their good models and everybody else can think so.\n\nWhat do you think? Is there anything bad by revealing private scores so far?\n( Personally, I didn't see my scores but it's just because it was fixed already. I might have tried if not fixed.)\n\n**EDIT:** (12-30-2020) After give it another thought, I realize that revealing private score before the bug doesn't resolve all problems. [It found out that the bug has been there since November 18, 2019](https://www.kaggle.com/private-leaderboard-bug-dec-2020). So, people could have experienced a lot of things. \n\nSo, how can we make this competition a fair competition again? \n\n1. Revealing private scores is one thing but we have no longer an advantage of being able to experience with private data.\n\n2. Revealing private data set and replace it with new data. This might cause a discontinuity of users' records.\n\n3. Revealing private data, and introduce new train and private data sets...", "1132750": "I hesitated to check (although it had been fixed when I saw the discussion about the bug) - but back to the time, I would rather not to check the private score - at least, I will enjoy the surprise that I might ends in a higher position in the private LB (the probability is not 0.0, so let me dream about it ...)\n\nWith the action of checking my private LB, the joy of winning disappears ... for me", "1133110": "Yes, I understand and I agree the excitement of the surprise. What's problem is that some people who checked their private score might take advantage of it and deprive you of the joy of winning.", "1133168": "This seems like throwing out the baby and the bathwater to me.\n\nI think with two subs and good cv/lb correlation, this competition isn't so much at risk as some other historic competitions. Then again, you might see me eating my words in seven days...", "1133185": "I agree with the idea of revealing private scores before the bug fixed. Introducing new private test set is recommended, but it will possibly make the competition meaningless if kaggle admins and the hosts cannot prepare a proper private test set. Specifically, if they have already used all of the history of the users in the training set (i.e. \"all-in\"), it will be hard to prepare the proper private test set. If so, they probably need to introduce new training set, which costs the participants a lot of money...", "1133187": "In this thread https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/191106, @sohier said \"The train/test data is complete, in the sense that there are no missing interactions in the union of train and test data.\", which suggests sadly they did \"all-in\"...", "1133219": "If we all know previous private LB score, the competition would become fair in some degree. \nAt the same time, it makes the competition completely deviated from the standard practice of ML/DS or that of Kaggle.\nThat's why I did not see my score, though I can not prove myself about it unfortunately.\nIf anyone say the competition has already deviated from the standard, I have to admit it😓.\n\nAnyway, I think only thing we can do currently is being the first on both public and private LB. Nobody can have silly suspicion of cheating." }, "source": "meta" }