![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F590240%2F530e6c7a233d299906f2d445f7db3e5d%2F2020-12-29%2023.34.02.png?generation=1609252473508532&alt=media\")
According to this discussion, you can see your 'private score'. I can see one in this competition too.
\n
(P.S.)
\nThis apparent bug has been resolved already (12/30/2020). I am sure that the Kaggle team will find a solution to this situation.
According to this discussion, you can see your 'private score'. I can see one in this competition too.
\n
(P.S.)
\nThis apparent bug has been resolved already (12/30/2020). I am sure that the Kaggle team will find a solution to this situation.
Dear all,
\nThis is Tim, organizer of this year's AIEd Challenge. Thank you for bringing this issue to our attention. We truly appreciate the collective integrity of Kagglers that helps this community stay vibrant and keep thriving.
\nJin and I, as well as our organizing team at Riiid, are also aware of the gravity of this situation and are currently working together with Kaggle to find the optimal solution ASAP. We will do our best to make sure that our competition lives up to its original vision, which is to encourage research in AI Education through fair and healthy competition.
\nThank you for your participation in our AIEd Challenge and we wish you all a Happy New Year!
\nSincerely,
\nTim
@sohier This is very very big problem, all kaggle admins should discuss this problem!!!
", "rawMarkdown": "@sohier This is very very big problem, all kaggle admins should discuss this problem!!!", "votes": 13 }, { "id": 1131275, "postDate": "2020-12-29T16:19:47.010Z", "content": "Hey all,
\nPlease know we're aware of this, looking into it, and working on it! It may take a little bit of time for us to release a full statement as we evaluate the full breadth and depth of the issue, including which competitions may have been impacted and the full extent of the impact on leaderboard results.
\nThanks for your patience! We'll let you know when we know more.
", "rawMarkdown": "Hey all,\n\nPlease know we're aware of this, looking into it, and working on it! It may take a little bit of time for us to release a full statement as we evaluate the full breadth and depth of the issue, including which competitions may have been impacted and the full extent of the impact on leaderboard results.\n\nThanks for your patience! We'll let you know when we know more.", "votes": 14, "replies": [ { "id": 1132173, "postDate": "2020-12-30T08:15:38.243Z", "content": "@addisonhoward will competition deadline be extended.
\nPlease confirm
Hi all - please see this statement from our team regarding the leaderboard bug.
", "rawMarkdown": "Hi all - please see [this statement](https://www.kaggle.com/private-leaderboard-bug-dec-2020) from our team regarding the leaderboard bug.", "votes": 12 }, { "id": 1132051, "postDate": "2020-12-30T06:20:42.407Z", "content": "In my opinion, in Riiid competition, the problem is not so big as in other competitions, because LB-CV is consistent (at least for me) and the image also suggests that the public-private LB is basically consistent and big shakeup is not expected, which means it's not easy to exploit this bug.
\nBut I think kaggle admins should do best efforts to make the competitions as fair as possible, like adding or making a new private test set, or maybe revealing the private score of people's current best submissions so that everyone can access the same information.
Can't imagine if anyone used this to win previous competitions
", "rawMarkdown": "Can't imagine if anyone used this to win previous competitions", "votes": 10, "replies": [ { "id": 1131144, "postDate": "2020-12-29T15:05:52.547Z", "content": "Yes, now we cannot believe big shakeup teams, because they may have cheated.
", "rawMarkdown": "Yes, now we cannot believe big shakeup teams, because they may have cheated.", "votes": 1 }, { "id": 1131820, "postDate": "2020-12-30T02:16:57.170Z", "content": "That is a liitle bit too harsh. I choose to believe them, unless proved otherwise.
", "rawMarkdown": "That is a liitle bit too harsh. I choose to believe them, unless proved otherwise.", "votes": 2 }, { "id": 1131867, "postDate": "2020-12-30T03:33:00.113Z", "content": "Yes I would like to believe them too, but I'm really sad I cannot genuinely believe them :(
", "rawMarkdown": "Yes I would like to believe them too, but I'm really sad I cannot genuinely believe them :(", "votes": -1 }, { "id": 1133754, "postDate": "2020-12-31T13:53:41.680Z", "content": "@mamasinkgs Who is Big Shakeup Team ?
", "rawMarkdown": "@mamasinkgs Who is Big Shakeup Team ?", "votes": 1 }, { "id": 1133760, "postDate": "2020-12-31T14:02:36.770Z", "content": "sorry for the confusion, I mean \"teams who experienced shakeup in past competitions\".
", "rawMarkdown": "sorry for the confusion, I mean \"teams who experienced shakeup in past competitions\".", "votes": 2 } ] }, { "id": 1131375, "postDate": "2020-12-29T17:33:47.697Z", "content": "I think the host should change the private test set in order to maintain a competition with fair results
", "rawMarkdown": "I think the host should change the private test set in order to maintain a competition with fair results", "votes": 5 }, { "id": 1131182, "postDate": "2020-12-29T15:26:05.010Z", "content": "I think kaggle team should release the access log of this page before 2020/12/28, we need to know the teams who cheated in the past competitions if possible !
", "rawMarkdown": "I think kaggle team should release the access log of this page before 2020/12/28, we need to know the teams who cheated in the past competitions if possible !", "votes": 5, "replies": [ { "id": 1131248, "postDate": "2020-12-29T16:04:36.823Z", "content": "It's not a separate web page as such. Anyone who has looked at their submission summary (and possibly other pages as well) will have accessed this information, so, pretty much everyone. 😕
\nIf this really was the actual scores, hopefully it was just a very recent thing that very few knew of. At least it appears to be fixed now.
", "rawMarkdown": "It's not a separate web page as such. Anyone who has looked at their submission summary (and possibly other pages as well) will have accessed this information, so, pretty much everyone. 😕 \n\nIf this really was the actual scores, hopefully it was just a very recent thing that very few knew of. At least it appears to be fixed now.", "votes": 1 }, { "id": 1131262, "postDate": "2020-12-29T16:13:19.213Z", "content": "Oh………
", "rawMarkdown": "Oh........." }, { "id": 1131358, "postDate": "2020-12-29T17:18:31.413Z", "content": "OMG……..
", "rawMarkdown": "OMG........" }, { "id": 1131364, "postDate": "2020-12-29T17:26:47.920Z", "content": "Can someone elaborate on the consequences of such a thing?
", "rawMarkdown": "Can someone elaborate on the consequences of such a thing?" } ] }, { "id": 1133997, "postDate": "2020-12-31T18:19:11.107Z", "content": "Hey all,
\nWe've made a post here that addresses the latest update on the bug, and what we're providing to all users.
", "rawMarkdown": "Hey all,\n\nWe've made a post [here](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/207875) that addresses the latest update on the bug, and what we're providing to all users.", "votes": 6 }, { "id": 1132721, "postDate": "2020-12-30T16:27:18.707Z", "content": "Perhaps a solution would be for Kaggle to reveal to each team their private scores to date yesterday.
", "rawMarkdown": "Perhaps a solution would be for Kaggle to reveal to each team their private scores to date yesterday.", "votes": 3 }, { "id": 1131354, "postDate": "2020-12-29T17:13:15.947Z", "content": "Have been fixed. currently, the private related fields are null. I think the host needs to consider changing the private test set. Otherwise, these teams who have checked the private scores could choose the one which has the best private score.
Revealing all of the private score (until 2020/12/29) may be a possible option. Of course it is also somewhat unfair because people who have made many submissions can get more information, though.
", "rawMarkdown": "Revealing all of the private score (until 2020/12/29) may be a possible option. Of course it is also somewhat unfair because people who have made many submissions can get more information, though.", "votes": 2 }, { "id": 1131363, "postDate": "2020-12-29T17:25:25.090Z", "content": "Yup, maybe these teams were utilizing this bug to overfit the private test set……
", "rawMarkdown": "Yup, maybe these teams were utilizing this bug to overfit the private test set......" }, { "id": 1131386, "postDate": "2020-12-29T17:42:43.237Z", "content": "From the screenshot above, maybe you could \"only\" have seen the private score of your top public model, not the submission id of the top private one … ?
", "rawMarkdown": "From the screenshot above, maybe you could \"only\" have seen the private score of your top public model, not the submission id of the top private one ... ?" }, { "id": 1131393, "postDate": "2020-12-29T17:46:43.527Z", "content": "But I could have checked the private score every time I made a submission. So I would have the private scores for all my submissions.
", "rawMarkdown": "But I could have checked the private score every time I made a submission. So I would have the private scores for all my submissions." }, { "id": 1131416, "postDate": "2020-12-29T18:00:16.107Z", "content": "You are right, thanks for pointing it out!
", "rawMarkdown": "You are right, thanks for pointing it out!" }, { "id": 1131691, "postDate": "2020-12-29T22:24:56.363Z", "content": "Your point is also fair, to check the private score of the current submission I need to make sure the current submission has the best score on Private LB.
", "rawMarkdown": "Your point is also fair, to check the private score of the current submission I need to make sure the current submission has the best score on Private LB." }, { "id": 1132177, "postDate": "2020-12-30T08:18:29.883Z", "content": "I think merging public data d private data is useful ,and then building up new private set . Extending the deadline by 20 days . This is happened in past with airbus ship
", "rawMarkdown": "I think merging public data d private data is useful ,and then building up new private set . Extending the deadline by 20 days . This is happened in past with airbus ship " } ] }, { "id": 1131146, "postDate": "2020-12-29T15:08:56.293Z", "content": "I am happy that the CV is stable enough for me ; No plans to exploit this information.
", "rawMarkdown": "I am happy that the CV is stable enough for me ; No plans to exploit this information.", "votes": 4, "replies": [ { "id": 1131222, "postDate": "2020-12-29T15:49:53.180Z", "content": "me too，i can trust my cv score。but still unbelievable。。。。feel sad for my first competition
", "rawMarkdown": "me too，i can trust my cv score。but still unbelievable。。。。feel sad for my first competition", "votes": 1 } ] }, { "id": 1132276, "postDate": "2020-12-30T09:29:22.810Z", "content": "Imagine this information in MoA…
", "rawMarkdown": "Imagine this information in MoA...", "votes": 1, "replies": [ { "id": 1132336, "postDate": "2020-12-30T10:19:05.820Z", "content": "list can be even longer.. Pandas,Melanoma, OSIC.. etc etc
", "rawMarkdown": "list can be even longer.. Pandas,Melanoma, OSIC.. etc etc" }, { "id": 1135825, "postDate": "2021-01-02T14:37:13.690Z", "content": "Maybe it can avoid the scared shake up.😄
", "rawMarkdown": "Maybe it can avoid the scared shake up.😄" } ] }, { "id": 1131253, "postDate": "2020-12-29T16:09:22.707Z", "content": "I just wonder how the private score is calculated before we select two kernels for final submission.
", "rawMarkdown": "I just wonder how the private score is calculated before we select two kernels for final submission.", "votes": 1, "replies": [ { "id": 1131357, "postDate": "2020-12-29T17:17:08.893Z", "content": "The private score will be calculated every time you successfully make a submission. It's designed by Kaggle to avoid failure during the last days. Think about this case, you selected two kernels at the end, then you come across Submission Error:)
", "rawMarkdown": "The private score will be calculated every time you successfully make a submission. It's designed by Kaggle to avoid failure during the last days. Think about this case, you selected two kernels at the end, then you come across Submission Error:)" }, { "id": 1131756, "postDate": "2020-12-29T23:59:36.167Z", "content": "Thank you. I understand why.
\nIt is very serious problem. If someone get shakeup in this or previous competition, he/she would receive unnecessary suspicion.
\nNobody can deny it. It is sad.
Am I a fool if I refuse to look? Maybe…
", "rawMarkdown": "Am I a fool if I refuse to look? Maybe...", "votes": 1 }, { "id": 1131174, "postDate": "2020-12-29T15:24:01.640Z", "content": "The funniest thing is that We are using publically available data 😂😂😂 to get our ranks. As time passes by, more and more people will exploit it in the currently going competitions too :( . And you thought 2020 was over
", "rawMarkdown": "The funniest thing is that **We are using publically available data** 😂😂😂 to get our ranks. As time passes by, more and more people will exploit it in the currently going competitions too :( . And you thought 2020 was over", "votes": 1 }, { "id": 1131127, "postDate": "2020-12-29T14:58:06.783Z", "content": "I can confirm, the glitch is still there just by pressing F12 in Chrome to view the network requests.
\nThe information seems to be too detailed to be fake.
\nEDIT: for those of you who are curious, go to the leaderboard page, press F12 (or Fn+F12) to invoke the Chrome console->Network->Search privateLeaderboardRank in the search window->You will see your userID information in the response window.
This can be very frightening for competition with a high shakeup rate…
", "rawMarkdown": "This can be very frightening for competition with a high shakeup rate...", "votes": 2 }, { "id": 1131190, "postDate": "2020-12-29T15:28:53.423Z", "content": "How is it possible? I can't believe it… since how long ago?
", "rawMarkdown": "How is it possible? I can't believe it... since how long ago?", "votes": 2 }, { "id": 1131763, "postDate": "2020-12-30T00:11:56.997Z", "content": "Please change the content of private leaderboard test set or add more data e.g. from 2.5 million to 7.5 million. Some of the participants had already accessed that private leaderboard information. We need a sense of urgency here as competition deadline is approaching. thanks
", "rawMarkdown": "Please change the content of private leaderboard test set or add more data e.g. from 2.5 million to 7.5 million. Some of the participants had already accessed that private leaderboard information. We need a sense of urgency here as competition deadline is approaching. thanks" }, { "id": 1131456, "postDate": "2020-12-29T18:23:54.983Z", "content": "Houston, we have a problem… This is really insane…
", "rawMarkdown": "Houston, we have a problem... This is really insane..." }, { "id": 1131343, "postDate": "2020-12-29T17:07:28.657Z", "content": "Interesting. I see null in privateLeaderboardScore
", "rawMarkdown": "Interesting. I see null in privateLeaderboardScore", "replies": [ { "id": 1131355, "postDate": "2020-12-29T17:15:17.740Z", "content": "same here, maybe they fixed the bug?
", "rawMarkdown": "same here, maybe they fixed the bug?" }, { "id": 1131361, "postDate": "2020-12-29T17:22:29.737Z", "content": "I also think they fixed the bug
", "rawMarkdown": "I also think they fixed the bug" } ] }, { "id": 1131296, "postDate": "2020-12-29T16:37:45.460Z", "content": "I think they have fixed it. The API for getUser is not working.
", "rawMarkdown": "I think they have fixed it. The API for getUser is not working." }, { "id": 1131257, "postDate": "2020-12-29T16:11:18.783Z", "rawMarkdown": "", "votes": -1, "isDeleted": true, "replies": [ { "id": 1131270, "postDate": "2020-12-29T16:17:11.803Z", "content": "You can select a different kernel than the top public scoring one for final submission (maybe more based on CV than public LB)
", "rawMarkdown": "You can select a different kernel than the top public scoring one for final submission (maybe more based on CV than public LB)", "votes": 1 } ] }, { "id": 1131252, "postDate": "2020-12-29T16:09:09.083Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1132347, "author_name": "Hoon Pyo (Tim) Jeon", "author_url": "", "post_date": "2020-12-30T10:29:21.410000", "content": "Dear all,
\nThis is Tim, organizer of this year's AIEd Challenge. Thank you for bringing this issue to our attention. We truly appreciate the collective integrity of Kagglers that helps this community stay vibrant and keep thriving.
\nJin and I, as well as our organizing team at Riiid, are also aware of the gravity of this situation and are currently working together with Kaggle to find the optimal solution ASAP. We will do our best to make sure that our competition lives up to its original vision, which is to encourage research in AI Education through fair and healthy competition.
\nThank you for your participation in our AIEd Challenge and we wish you all a Happy New Year!
\nSincerely,
\nTim
@sohier This is very very big problem, all kaggle admins should discuss this problem!!!
", "votes": 13, "replies": [] }, { "id": 1131275, "author_name": "Addison Howard", "author_url": "", "post_date": "2020-12-29T16:19:47.010000", "content": "Hey all,
\nPlease know we're aware of this, looking into it, and working on it! It may take a little bit of time for us to release a full statement as we evaluate the full breadth and depth of the issue, including which competitions may have been impacted and the full extent of the impact on leaderboard results.
\nThanks for your patience! We'll let you know when we know more.
", "votes": 14, "replies": [ { "id": 1132173, "author_name": "Jaideep", "author_url": "", "post_date": "2020-12-30T08:15:38.243000", "content": "@addisonhoward will competition deadline be extended.
\nPlease confirm
Hi all - please see this statement from our team regarding the leaderboard bug.
", "votes": 12, "replies": [] }, { "id": 1132051, "author_name": "mamas", "author_url": "", "post_date": "2020-12-30T06:20:42.407000", "content": "In my opinion, in Riiid competition, the problem is not so big as in other competitions, because LB-CV is consistent (at least for me) and the image also suggests that the public-private LB is basically consistent and big shakeup is not expected, which means it's not easy to exploit this bug.
\nBut I think kaggle admins should do best efforts to make the competitions as fair as possible, like adding or making a new private test set, or maybe revealing the private score of people's current best submissions so that everyone can access the same information.
Can't imagine if anyone used this to win previous competitions
", "votes": 10, "replies": [ { "id": 1131144, "author_name": "mamas", "author_url": "", "post_date": "2020-12-29T15:05:52.547000", "content": "Yes, now we cannot believe big shakeup teams, because they may have cheated.
", "votes": 1, "replies": [] }, { "id": 1131820, "author_name": "Frank Pan", "author_url": "", "post_date": "2020-12-30T02:16:57.170000", "content": "That is a liitle bit too harsh. I choose to believe them, unless proved otherwise.
", "votes": 2, "replies": [] }, { "id": 1131867, "author_name": "mamas", "author_url": "", "post_date": "2020-12-30T03:33:00.113000", "content": "Yes I would like to believe them too, but I'm really sad I cannot genuinely believe them :(
", "votes": -1, "replies": [] }, { "id": 1133754, "author_name": "Tensor Girl", "author_url": "", "post_date": "2020-12-31T13:53:41.680000", "content": "@mamasinkgs Who is Big Shakeup Team ?
", "votes": 1, "replies": [] }, { "id": 1133760, "author_name": "mamas", "author_url": "", "post_date": "2020-12-31T14:02:36.770000", "content": "sorry for the confusion, I mean \"teams who experienced shakeup in past competitions\".
", "votes": 2, "replies": [] } ] }, { "id": 1131375, "author_name": "Maher el Ouahabi", "author_url": "", "post_date": "2020-12-29T17:33:47.697000", "content": "I think the host should change the private test set in order to maintain a competition with fair results
", "votes": 5, "replies": [] }, { "id": 1131182, "author_name": "mamas", "author_url": "", "post_date": "2020-12-29T15:26:05.010000", "content": "I think kaggle team should release the access log of this page before 2020/12/28, we need to know the teams who cheated in the past competitions if possible !
", "votes": 5, "replies": [ { "id": 1131248, "author_name": "Christoffer Karlsson", "author_url": "", "post_date": "2020-12-29T16:04:36.823000", "content": "It's not a separate web page as such. Anyone who has looked at their submission summary (and possibly other pages as well) will have accessed this information, so, pretty much everyone. 😕
\nIf this really was the actual scores, hopefully it was just a very recent thing that very few knew of. At least it appears to be fixed now.
", "votes": 1, "replies": [] }, { "id": 1131262, "author_name": "mamas", "author_url": "", "post_date": "2020-12-29T16:13:19.213000", "content": "Oh………
", "votes": 0, "replies": [] }, { "id": 1131358, "author_name": "william.wu", "author_url": "", "post_date": "2020-12-29T17:18:31.413000", "content": "OMG……..
", "votes": 0, "replies": [] }, { "id": 1131364, "author_name": "Abdessalem Boukil", "author_url": "", "post_date": "2020-12-29T17:26:47.920000", "content": "Can someone elaborate on the consequences of such a thing?
", "votes": 0, "replies": [] } ] }, { "id": 1133997, "author_name": "Addison Howard", "author_url": "", "post_date": "2020-12-31T18:19:11.107000", "content": "Hey all,
\nWe've made a post here that addresses the latest update on the bug, and what we're providing to all users.
", "votes": 6, "replies": [] }, { "id": 1132721, "author_name": "Manuel Campos", "author_url": "", "post_date": "2020-12-30T16:27:18.707000", "content": "Perhaps a solution would be for Kaggle to reveal to each team their private scores to date yesterday.
", "votes": 3, "replies": [] }, { "id": 1131354, "author_name": "william.wu", "author_url": "", "post_date": "2020-12-29T17:13:15.947000", "content": "Have been fixed. currently, the private related fields are null. I think the host needs to consider changing the private test set. Otherwise, these teams who have checked the private scores could choose the one which has the best private score.
Revealing all of the private score (until 2020/12/29) may be a possible option. Of course it is also somewhat unfair because people who have made many submissions can get more information, though.
", "votes": 2, "replies": [] }, { "id": 1131363, "author_name": "william.wu", "author_url": "", "post_date": "2020-12-29T17:25:25.090000", "content": "Yup, maybe these teams were utilizing this bug to overfit the private test set……
", "votes": 0, "replies": [] }, { "id": 1131386, "author_name": "Stefan Mandl", "author_url": "", "post_date": "2020-12-29T17:42:43.237000", "content": "From the screenshot above, maybe you could \"only\" have seen the private score of your top public model, not the submission id of the top private one … ?
", "votes": 0, "replies": [] }, { "id": 1131393, "author_name": "william.wu", "author_url": "", "post_date": "2020-12-29T17:46:43.527000", "content": "But I could have checked the private score every time I made a submission. So I would have the private scores for all my submissions.
", "votes": 0, "replies": [] }, { "id": 1131416, "author_name": "Stefan Mandl", "author_url": "", "post_date": "2020-12-29T18:00:16.107000", "content": "You are right, thanks for pointing it out!
", "votes": 0, "replies": [] }, { "id": 1131691, "author_name": "william.wu", "author_url": "", "post_date": "2020-12-29T22:24:56.363000", "content": "Your point is also fair, to check the private score of the current submission I need to make sure the current submission has the best score on Private LB.
", "votes": 0, "replies": [] }, { "id": 1132177, "author_name": "Jaideep", "author_url": "", "post_date": "2020-12-30T08:18:29.883000", "content": "I think merging public data d private data is useful ,and then building up new private set . Extending the deadline by 20 days . This is happened in past with airbus ship
", "votes": 0, "replies": [] } ] }, { "id": 1131146, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-12-29T15:08:56.293000", "content": "I am happy that the CV is stable enough for me ; No plans to exploit this information.
", "votes": 4, "replies": [ { "id": 1131222, "author_name": "qiaqia", "author_url": "", "post_date": "2020-12-29T15:49:53.180000", "content": "me too，i can trust my cv score。but still unbelievable。。。。feel sad for my first competition
", "votes": 1, "replies": [] } ] }, { "id": 1132276, "author_name": "Claudio Verdú Ruiz", "author_url": "", "post_date": "2020-12-30T09:29:22.810000", "content": "Imagine this information in MoA…
", "votes": 1, "replies": [ { "id": 1132336, "author_name": "Jaideep", "author_url": "", "post_date": "2020-12-30T10:19:05.820000", "content": "list can be even longer.. Pandas,Melanoma, OSIC.. etc etc
", "votes": 0, "replies": [] }, { "id": 1135825, "author_name": "HenryHZY", "author_url": "", "post_date": "2021-01-02T14:37:13.690000", "content": "Maybe it can avoid the scared shake up.😄
", "votes": 0, "replies": [] } ] }, { "id": 1131253, "author_name": "tomoo inubushi", "author_url": "", "post_date": "2020-12-29T16:09:22.707000", "content": "I just wonder how the private score is calculated before we select two kernels for final submission.
", "votes": 1, "replies": [ { "id": 1131357, "author_name": "william.wu", "author_url": "", "post_date": "2020-12-29T17:17:08.893000", "content": "The private score will be calculated every time you successfully make a submission. It's designed by Kaggle to avoid failure during the last days. Think about this case, you selected two kernels at the end, then you come across Submission Error:)
", "votes": 0, "replies": [] }, { "id": 1131756, "author_name": "tomoo inubushi", "author_url": "", "post_date": "2020-12-29T23:59:36.167000", "content": "Thank you. I understand why.
\nIt is very serious problem. If someone get shakeup in this or previous competition, he/she would receive unnecessary suspicion.
\nNobody can deny it. It is sad.
Am I a fool if I refuse to look? Maybe…
", "votes": 1, "replies": [] }, { "id": 1131174, "author_name": "NikhilMishra", "author_url": "", "post_date": "2020-12-29T15:24:01.640000", "content": "The funniest thing is that We are using publically available data 😂😂😂 to get our ranks. As time passes by, more and more people will exploit it in the currently going competitions too :( . And you thought 2020 was over
", "votes": 1, "replies": [] }, { "id": 1131127, "author_name": "Shuhao Cao", "author_url": "", "post_date": "2020-12-29T14:58:06.783000", "content": "I can confirm, the glitch is still there just by pressing F12 in Chrome to view the network requests.
\nThe information seems to be too detailed to be fake.
\nEDIT: for those of you who are curious, go to the leaderboard page, press F12 (or Fn+F12) to invoke the Chrome console->Network->Search privateLeaderboardRank in the search window->You will see your userID information in the response window.
This can be very frightening for competition with a high shakeup rate…
", "votes": 2, "replies": [] }, { "id": 1131190, "author_name": "Manuel Campos", "author_url": "", "post_date": "2020-12-29T15:28:53.423000", "content": "How is it possible? I can't believe it… since how long ago?
", "votes": 2, "replies": [] }, { "id": 1131763, "author_name": "FGPC", "author_url": "", "post_date": "2020-12-30T00:11:56.997000", "content": "Please change the content of private leaderboard test set or add more data e.g. from 2.5 million to 7.5 million. Some of the participants had already accessed that private leaderboard information. We need a sense of urgency here as competition deadline is approaching. thanks
", "votes": 0, "replies": [] }, { "id": 1131456, "author_name": "Chris X", "author_url": "", "post_date": "2020-12-29T18:23:54.983000", "content": "Houston, we have a problem… This is really insane…
", "votes": 0, "replies": [] }, { "id": 1131343, "author_name": "Alyona Pasevieva", "author_url": "", "post_date": "2020-12-29T17:07:28.657000", "content": "Interesting. I see null in privateLeaderboardScore
", "votes": 0, "replies": [ { "id": 1131355, "author_name": "bluetrain", "author_url": "", "post_date": "2020-12-29T17:15:17.740000", "content": "same here, maybe they fixed the bug?
", "votes": 0, "replies": [] }, { "id": 1131361, "author_name": "Alyona Pasevieva", "author_url": "", "post_date": "2020-12-29T17:22:29.737000", "content": "I also think they fixed the bug
", "votes": 0, "replies": [] } ] }, { "id": 1131296, "author_name": "Manikanth Reddy", "author_url": "", "post_date": "2020-12-29T16:37:45.460000", "content": "I think they have fixed it. The API for getUser is not working.
", "votes": 0, "replies": [] }, { "id": 1131257, "author_name": "", "author_url": "", "post_date": "2020-12-29T16:11:18.783000", "content": "", "votes": -1, "replies": [ { "id": 1131270, "author_name": "Stefan Mandl", "author_url": "", "post_date": "2020-12-29T16:17:11.803000", "content": "You can select a different kernel than the top public scoring one for final submission (maybe more based on CV than public LB)
", "votes": 1, "replies": [] } ] }, { "id": 1131252, "author_name": "", "author_url": "", "post_date": "2020-12-29T16:09:09.083000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1131080": "According to [this discussion](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/207357), you can see your 'private score'. I can see one in this competition too.\n\n\n\n(P.S.)\nThis apparent bug has been resolved already (12/30/2020). I am sure that the Kaggle team will find a solution to this situation.", "1132347": "Dear all,\n\nThis is Tim, organizer of this year's AIEd Challenge. Thank you for bringing this issue to our attention. We truly appreciate the collective integrity of Kagglers that helps this community stay vibrant and keep thriving.\n\nJin and I, as well as our organizing team at Riiid, are also aware of the gravity of this situation and are **currently working together with Kaggle to find the optimal solution ASAP.** We will do our best to make sure that our competition lives up to its original vision, which is to encourage research in AI Education through fair and healthy competition.\n\nThank you for your participation in our AIEd Challenge and we wish you all a Happy New Year!\n\nSincerely,\nTim", "1131124": "@sohier This is very very big problem, all kaggle admins should discuss this problem!!!", "1131275": "Hey all,\n\nPlease know we're aware of this, looking into it, and working on it! It may take a little bit of time for us to release a full statement as we evaluate the full breadth and depth of the issue, including which competitions may have been impacted and the full extent of the impact on leaderboard results.\n\nThanks for your patience! We'll let you know when we know more.", "1132929": "Hi all - please see [this statement](https://www.kaggle.com/private-leaderboard-bug-dec-2020) from our team regarding the leaderboard bug.", "1132051": "In my opinion, in Riiid competition, the problem is not so big as in other competitions, because LB-CV is consistent (at least for me) and the image also suggests that the public-private LB is basically consistent and big shakeup is not expected, which means it's not easy to exploit this bug.\nBut I think kaggle admins should do best efforts to make the competitions as fair as possible, like adding or making a new private test set, or maybe revealing the private score of people's current best submissions so that everyone can access the same information.", "1131135": "Can't imagine if anyone used this to win previous competitions", "1131375": "I think the host should change the private test set in order to maintain a competition with fair results", "1131182": "I think kaggle team should release the access log of this page before 2020/12/28, we need to know the teams who cheated in the past competitions if possible !", "1133997": "Hey all,\n\nWe've made a post [here](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/207875) that addresses the latest update on the bug, and what we're providing to all users.", "1132721": "Perhaps a solution would be for Kaggle to reveal to each team their private scores to date yesterday.", "1131354": "Have been fixed. currently, the `private` related fields are `null`. I think the host needs to consider changing the private test set. Otherwise, these teams who have checked the private scores could choose the one which has the best private score.", "1131146": "I am happy that the CV is stable enough for me ; No plans to exploit this information.", "1132276": "Imagine this information in MoA...", "1131253": "I just wonder how the private score is calculated before we select two kernels for final submission.", "1131247": "Am I a fool if I refuse to look? Maybe...", "1131174": "The funniest thing is that **We are using publically available data** 😂😂😂 to get our ranks. As time passes by, more and more people will exploit it in the currently going competitions too :( . And you thought 2020 was over", "1131127": "I can confirm, the glitch is still there just by pressing F12 in Chrome to view the network requests.\n\nThe information seems to be too detailed to be fake.\n\nEDIT: for those of you who are curious, go to the leaderboard page, press F12 (or Fn+F12) to invoke the Chrome console->Network->Search `privateLeaderboardRank` in the search window->You will see your `userID` information in the response window.", "1131244": "This can be very frightening for competition with a high shakeup rate...", "1131190": "How is it possible? I can't believe it... since how long ago?", "1131763": "Please change the content of private leaderboard test set or add more data e.g. from 2.5 million to 7.5 million. Some of the participants had already accessed that private leaderboard information. We need a sense of urgency here as competition deadline is approaching. thanks", "1131456": "Houston, we have a problem... This is really insane...", "1131343": "Interesting. I see null in privateLeaderboardScore", "1131296": "I think they have fixed it. The API for getUser is not working.", "1131257": "", "1131252": "" } }