![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fcbba131c09ce15ceacdb7d90496bd54a%2FSelection_246.png?generation=1609257605183898&alt=media\")
I am not sure if it is the right leaderboard, but I am able to see my private leaderboard rank (see the attachment). I was able to get it by looking at the network requests and responses on the leaderboard page.
\nEdit 1: I had sent a message to kaggle support in https://www.kaggle.com/contact in the ' I found an issue with the website' section at around 23:50 UTC, Sunday detailing how the private leaderboard could be accessible. This was for the challenge https://www.kaggle.com/c/cassava-leaf-disease-classification. Unfortunately, kaggle did not send me a copy of the message, however I replied to (what I think is automatically generated mail) with a screenshot similar to what I posted earlier but for the other competition. I did not receive (to the best of my knowledge) any response. Attached the screenshot of the mail thread (note that in India it is +05:30 UTC).
Edit 2: It seems (as acknowledged by others) that the bug is fixed. Good.
", "messages": [ { "id": 1130871, "postDate": "2020-12-29T11:08:44.100Z", "content": "I am not sure if it is the right leaderboard, but I am able to see my private leaderboard rank (see the attachment). I was able to get it by looking at the network requests and responses on the leaderboard page.
\nEdit 1: I had sent a message to kaggle support in https://www.kaggle.com/contact in the ' I found an issue with the website' section at around 23:50 UTC, Sunday detailing how the private leaderboard could be accessible. This was for the challenge https://www.kaggle.com/c/cassava-leaf-disease-classification. Unfortunately, kaggle did not send me a copy of the message, however I replied to (what I think is automatically generated mail) with a screenshot similar to what I posted earlier but for the other competition. I did not receive (to the best of my knowledge) any response. Attached the screenshot of the mail thread (note that in India it is +05:30 UTC).
Edit 2: It seems (as acknowledged by others) that the bug is fixed. Good.
", "rawMarkdown": "I am not sure if it is the right leaderboard, but I am able to see my private leaderboard rank (see the attachment). I was able to get it by looking at the network requests and responses on the leaderboard page.\n\nEdit 1: I had sent a message to kaggle support in `https://www.kaggle.com/contact` in the ' I found an issue with the website' section at around 23:50 UTC, Sunday detailing how the private leaderboard could be accessible. This was for the challenge `https://www.kaggle.com/c/cassava-leaf-disease-classification`. Unfortunately, kaggle did not send me a copy of the message, however I replied to (what I think is automatically generated mail) with a screenshot similar to what I posted earlier but for the other competition. I did not receive (to the best of my knowledge) any response. Attached the screenshot of the mail thread (note that in India it is +05:30 UTC).\n\nEdit 2: It seems (as acknowledged by others) that the bug is fixed. Good.", "votes": 240 }, { "id": 1131271, "postDate": "2020-12-29T16:17:38.927Z", "content": "Hey all,
\nPlease know we're aware of this, looking into it, and working on it! It may take a little bit of time for us to release a full statement as we evaluate the full breadth and depth of the issue, including which competitions may have been impacted and the full extent of the impact on leaderboard results.
\nThanks for your patience! We'll let you know when we know more.
", "rawMarkdown": "Hey all,\n\nPlease know we're aware of this, looking into it, and working on it! It may take a little bit of time for us to release a full statement as we evaluate the full breadth and depth of the issue, including which competitions may have been impacted and the full extent of the impact on leaderboard results.\n\nThanks for your patience! We'll let you know when we know more.", "votes": 53, "replies": [ { "id": 1132139, "postDate": "2020-12-30T07:34:04.087Z", "content": "So many people looked their private lb, how to keep fair, change private test set temporary？ @addisonhoward
", "rawMarkdown": "So many people looked their private lb, how to keep fair, change private test set temporary？ @addisonhoward" } ] }, { "id": 1131099, "postDate": "2020-12-29T14:44:29.147Z", "content": "The biggest leak in Kaggle history? 😃
", "rawMarkdown": "The biggest leak in Kaggle history? 😃", "votes": 45 }, { "id": 1131110, "postDate": "2020-12-29T14:51:21.717Z", "content": "I think Unfair can only be seen by those who know that.
\nI just write how to see.(Google Chrome)
Note: this is an important bug, please fix that
", "rawMarkdown": "I think Unfair can only be seen by those who know that.\nI just write how to see.(Google Chrome)\n\n1. Go to Leaderboard\n2. open Devtool -> Network -> Reload\n3. XHR only->select getUser request\n4. result->team, maybe you can see your public/private score and ranking\n\nNote: this is an important bug, please fix that", "votes": 43, "replies": [ { "id": 1131125, "postDate": "2020-12-29T14:57:19.793Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1131139, "postDate": "2020-12-29T15:04:19.837Z", "content": "I don't agree with that.
\nHe didn't say how to, but he said there is information in the network.
\nAlmost kaggler can use network information using browser and dev tools.
\nThat thread Implicitly said method.
If when hiding, someone who din't use devtool cannot see private score. I think that is very unfair.
\nNOTE: In my opinion, he report to admin for silent(don't use competitions thread).
", "rawMarkdown": "I don't agree with that. \nHe didn't say how to, but he said there is information in the network.\nAlmost kaggler can use network information using browser and dev tools.\nThat thread Implicitly said method.\n\nIf when hiding, someone who din't use devtool cannot see private score. I think that is very unfair.\n\nNOTE: In my opinion, he report to admin for silent(don't use competitions thread).", "votes": 5 }, { "id": 1131143, "postDate": "2020-12-29T15:05:22.630Z", "content": "it happens in all compeitions
", "rawMarkdown": "it happens in all compeitions", "votes": 8 }, { "id": 1131178, "postDate": "2020-12-29T15:24:52.957Z", "content": "When I select submission files for final scores, I can see the change the value of privateLeaderboardRank, privateLeaderboardScore, and privateLeaderboardSubmissionId. Therefore, it seems that we can check all private scores of our own submission files.
Where is getUser ?
", "rawMarkdown": "Where is getUser ?" }, { "id": 1131193, "postDate": "2020-12-29T15:30:59.447Z", "content": "You can see it when you access the leaderboard page. You might need to reload the page after opening the developer tool.
", "rawMarkdown": "You can see it when you access the leaderboard page. You might need to reload the page after opening the developer tool." }, { "id": 1131206, "postDate": "2020-12-29T15:35:48.617Z", "content": "I found it… It's so ..crazy…
", "rawMarkdown": "I found it... It's so ..crazy...", "votes": 1 }, { "id": 1132057, "postDate": "2020-12-30T06:24:35.090Z", "content": "@li325040229 i dont see it for my competition atleaset as shown in screen shot. does it affects all competition ?
", "rawMarkdown": "@li325040229 i dont see it for my competition atleaset as shown in screen shot. does it affects all competition ?" }, { "id": 1132062, "postDate": "2020-12-30T06:30:40.480Z", "content": "@sishihara i can see what value to put
", "rawMarkdown": "@sishihara i can see what value to put" } ] }, { "id": 1131043, "postDate": "2020-12-29T14:14:35.610Z", "content": "give this man a medal! well done
", "rawMarkdown": "give this man a medal! well done", "votes": 34 }, { "id": 1131246, "postDate": "2020-12-29T16:04:05.387Z", "content": "When this ends, I think we should also get to know how long ago this bug appeared - how many competitions could have been affected by it.
", "rawMarkdown": "When this ends, I think we should also get to know how long ago this bug appeared - how many competitions could have been affected by it.\n", "votes": 31, "replies": [ { "id": 1131255, "postDate": "2020-12-29T16:09:26.897Z", "content": "Yes, we need a full statement from Kaggle on this and how they can guarantee integrity of results.
", "rawMarkdown": "Yes, we need a full statement from Kaggle on this and how they can guarantee integrity of results.", "votes": 24 } ] }, { "id": 1131239, "postDate": "2020-12-29T16:00:07.143Z", "content": "
Now, all those fake accounts are starting to make sense 😄
\nI just hope the leak wasn't active for too long. Every winner of past shakeup competitions would be questioned. Hello magic scaling factors.
", "rawMarkdown": "Now, all those fake accounts are starting to make sense 😄\n\nI just hope the leak wasn't active for too long. Every winner of past shakeup competitions would be questioned. Hello magic scaling factors. ", "votes": 20 }, { "id": 1132928, "postDate": "2020-12-30T19:44:32.273Z", "content": "Hi all - please see this statement from our team regarding the leaderboard bug.
", "rawMarkdown": "Hi all - please see [this statement](https://www.kaggle.com/private-leaderboard-bug-dec-2020) from our team regarding the leaderboard bug.", "votes": 19, "replies": [ { "id": 1132934, "postDate": "2020-12-30T19:47:49.013Z", "content": "Thanks @addisonhoward. Do you already have a plan about ongoing competitions, specifically those ending very soon?
", "rawMarkdown": "Thanks @addisonhoward. Do you already have a plan about ongoing competitions, specifically those ending very soon?", "votes": 5 }, { "id": 1132937, "postDate": "2020-12-30T19:52:39.613Z", "content": "We are in contact with the hosts of ongoing competitions to determine if and what any immediate actions will be.
", "rawMarkdown": "We are in contact with the hosts of ongoing competitions to determine if and what any immediate actions will be.", "votes": 7 }, { "id": 1132977, "postDate": "2020-12-30T20:49:35.820Z", "content": "thanks for the statement @addisonhoward
\nhopefully, it's not just: \"one individual used it heavily enough to be detected\"
@Thanks for the statement @addisonhoward
\nHopefully fairer actions will be taken about ongoing competitions.
", "rawMarkdown": "@Thanks for the statement @addisonhoward \n\nHopefully fairer actions will be taken about ongoing competitions. ", "votes": 1 } ] }, { "id": 1131202, "postDate": "2020-12-29T15:34:34.107Z", "content": "And we thought 2020 was over 😂
", "rawMarkdown": "And we thought 2020 was over 😂", "votes": 12, "replies": [ { "id": 1132683, "postDate": "2020-12-30T15:57:54.303Z", "content": "ulti😂😂😂😂
", "rawMarkdown": "ulti😂😂😂😂" } ] }, { "id": 1132178, "postDate": "2020-12-30T08:18:31.463Z", "content": "Someone who has already seen private scores yesterday for multiple submissions could be benefited by comparing and finding the correlation between public, private and cv rank/scores. Wouldn't it also affect the ongoing competitions??
", "rawMarkdown": "Someone who has already seen private scores yesterday for multiple submissions could be benefited by comparing and finding the correlation between public, private and cv rank/scores. Wouldn't it also affect the ongoing competitions??", "votes": 8, "replies": [ { "id": 1132360, "postDate": "2020-12-30T10:44:45.700Z", "content": "@nischaydnk
\nI think so.
\nBut there probably won't be a tool to distinguish between these two groups.
\nBesides, I think there is still a long time before the competition is over, so the benefits at this point may disappear.
\nSo I will try to make a good one without worrying about it.
Agreed, but there are still some competitions which are about to end in next weeks.
", "rawMarkdown": "Agreed, but there are still some competitions which are about to end in next weeks. ", "votes": 1 } ] }, { "id": 1131701, "postDate": "2020-12-29T22:36:04.573Z", "content": "At this point Kaggle should make a public statement explaining for how long that has been exposed and if that has been exploited by someone. When such a thing happens in a company, we call it a \"data breach\", and this is truly a data breach for the importance of the critical information that leaked out without anyone being aware of it.
", "rawMarkdown": "At this point Kaggle should make a public statement explaining for how long that has been exposed and if that has been exploited by someone. When such a thing happens in a company, we call it a \"data breach\", and this is truly a data breach for the importance of the critical information that leaked out without anyone being aware of it.", "votes": 8 }, { "id": 1131349, "postDate": "2020-12-29T17:10:39.870Z", "content": "This is a very serious and unfortunate issue.
\nWinning or getting at top of kaggle competitions require huge workload and dedication. Knowing that people could easily probe the private results to adjust their model and submissions can be discouraging.
\nI can't wait to read the statement about the full breadth and depth of the issue,
", "rawMarkdown": "This is a very serious and unfortunate issue.\n\nWinning or getting at top of kaggle competitions require huge workload and dedication. Knowing that people could easily probe the private results to adjust their model and submissions can be discouraging.\n\nI can't wait to read the statement about the full breadth and depth of the issue,\n", "votes": 8 }, { "id": 1131154, "postDate": "2020-12-29T15:11:40.507Z", "content": "I am now wondering if some people already knew about the trick and used it to tweak hyper-parameters (thresholds, blending coefficients, …) using the private leaderboard.
", "rawMarkdown": "I am now wondering if some people already knew about the trick and used it to tweak hyper-parameters (thresholds, blending coefficients, ...) using the private leaderboard. ", "votes": 6, "replies": [ { "id": 1131161, "postDate": "2020-12-29T15:13:44.157Z", "content": "There indeed is such a possibility, and could have been used to probe the private LB
", "rawMarkdown": "There indeed is such a possibility, and could have been used to probe the private LB", "votes": 2 }, { "id": 1131163, "postDate": "2020-12-29T15:15:56.403Z", "content": "I wonder indeed for how long this already has existed… jesus this is too problematic
", "rawMarkdown": "I wonder indeed for how long this already has existed... jesus this is too problematic", "votes": 6 }, { "id": 1131177, "postDate": "2020-12-29T15:24:39.983Z", "content": "Kagglers who never shake down should be investigated:) I think only the current best public submission's private score is leaked, right? If all is leaked, it gets even worse.
", "rawMarkdown": "Kagglers who never shake down should be investigated:) I think only the current best public submission's private score is leaked, right? If all is leaked, it gets even worse.", "votes": 9 }, { "id": 1131180, "postDate": "2020-12-29T15:25:27.570Z", "content": "The question is if it would show the private score based on what you selected manually?
\nEdit: confirmed below
I can confirm this, when I selected 2 different submissions, the private score just changed
", "rawMarkdown": "I can confirm this, when I selected 2 different submissions, the private score just changed", "votes": 1 }, { "id": 1131185, "postDate": "2020-12-29T15:27:40.473Z", "content": "this is very serious then, it means you can see the private LB of any submission. just imagine someone creating 100 fake accounts, and using them to get correct label, it could be disastrous
", "rawMarkdown": "this is very serious then, it means you can see the private LB of any submission. just imagine someone creating 100 fake accounts, and using them to get correct label, it could be disastrous", "votes": 1 }, { "id": 1131225, "postDate": "2020-12-29T15:50:36.803Z", "content": "Only we can trust the competition which had broken private LB such as OpenVaccine.
\nFYI, I was 2nd place on OpenVaccine😉
After that many years still cannot beat the curse of 2nd - maybe this will help to remove the curse ?:D
", "rawMarkdown": "After that many years still cannot beat the curse of 2nd - maybe this will help to remove the curse ?:D", "votes": 3 }, { "id": 1131306, "postDate": "2020-12-29T16:41:41.507Z", "content": "\n\nOnly we can trust the competition which had broken private LB such as OpenVaccine.
\n
Or competitions where the private LB was based on future data- like last years NFL Big Data Bowl and Two Sigma.
", "rawMarkdown": "> Only we can trust the competition which had broken private LB such as OpenVaccine.\n\nOr competitions where the private LB was based on future data- like last years NFL Big Data Bowl and Two Sigma.", "votes": 2 }, { "id": 1131319, "postDate": "2020-12-29T16:51:16.100Z", "content": "\n\nKagglers who never shake down should be investigated:)
\n
And often shake down in those competitions which have broken private LB?
", "rawMarkdown": "> Kagglers who never shake down should be investigated:) \n\nAnd often shake down in those competitions which have broken private LB?" }, { "id": 1132324, "postDate": "2020-12-30T10:06:06.093Z", "content": "Probably it is not possibile to prove anyone has used that leakage because the data leaked for all the submissions and if you took advantage of it or not is not something that could be recorded anywhere. Also questioning about the optimal behaviour of a team in shake down competitions is not a definitive proof of having used the leak. Most likely they will not be able to point out anyone who took advantage of it. We expect at least to know how many competitions this leakage may have affected the results.
", "rawMarkdown": "Probably it is not possibile to prove anyone has used that leakage because the data leaked for all the submissions and if you took advantage of it or not is not something that could be recorded anywhere. Also questioning about the optimal behaviour of a team in shake down competitions is not a definitive proof of having used the leak. Most likely they will not be able to point out anyone who took advantage of it. We expect at least to know how many competitions this leakage may have affected the results.", "votes": 1 } ] }, { "id": 1131447, "postDate": "2020-12-29T18:20:22.360Z", "content": "Ain't even mad. This is more hilarious than anything. Everyone makes mistakes and trusting the front end to hide logic never intended for the user rather than filtering on the back end is one of the classical ones. Whenever Kaggle gets around to publishing the write up on this, I'd just request it's also shared with the community how long this issue has been present / how many competitions this has affected.
\nEdit: oops just read through some of the other posts and looks like a few people also shared pretty much verbatim sentiments.
\nSo that my post isn't completely useless, I'll just drop this little nugget here:
\nhttps://twitter.com/twimlai/status/1343976072577073152
\nNot just from an ethics standpoint, but also from a software engineering standpoint. And in some cases such as this, the results of one directly affect the other.
", "rawMarkdown": "Ain't even mad. This is more hilarious than anything. Everyone makes mistakes and trusting the front end to hide logic never intended for the user rather than filtering on the back end is one of the classical ones. Whenever Kaggle gets around to publishing the write up on this, I'd just request it's also shared with the community how long this issue has been present / how many competitions this has affected.\n\nEdit: oops just read through some of the other posts and looks like a few people also shared pretty much verbatim sentiments. \n\nSo that my post isn't completely useless, I'll just drop this little nugget here:\n\nhttps://twitter.com/twimlai/status/1343976072577073152\n\nNot just from an ethics standpoint, but also from a software engineering standpoint. And in some cases such as this, the results of one directly affect the other.", "votes": 6 }, { "id": 1131580, "postDate": "2020-12-29T20:24:19.143Z", "content": "I wonder for how long that has been available to a few ones
", "rawMarkdown": "I wonder for how long that has been available to a few ones", "votes": 3 }, { "id": 1131481, "postDate": "2020-12-29T18:43:33.447Z", "content": "wow this is bad..
\nThank you for sharing rather than taking advantage of it!
", "rawMarkdown": "wow this is bad..\n\n\nThank you for sharing rather than taking advantage of it! ", "votes": 3 }, { "id": 1131424, "postDate": "2020-12-29T18:07:31.437Z", "content": "This is a very intriguing find so thanks for sharing. If I may ask, how did you find out about it? Just by randomly checking HTTP responses or did you had an intuition that something was off?
\nAs pointed by others, it would be interesting for how long this have been going on and why it was there on the first place (was the code reviewed and approved?).
\nA postmortem by Kaggle would be very valuable.
", "rawMarkdown": "This is a very intriguing find so thanks for sharing. If I may ask, how did you find out about it? Just by randomly checking HTTP responses or did you had an intuition that something was off?\n\nAs pointed by others, it would be interesting for how long this have been going on and why it was there on the first place (was the code reviewed and approved?). \n\nA postmortem by Kaggle would be very valuable. ", "votes": 3 }, { "id": 1131291, "postDate": "2020-12-29T16:33:51.543Z", "content": "What's most important is \"when\" it has started. Please include timeline in the full statement, unless it will be understood \"always\"…
\nHope it's only this 2~3 months or less, for the honor of past competition winners.
@kaggle should go down, and fix this, even if its for a few hours or one or two days. So that people stop exploiting
", "rawMarkdown": "@kaggle should go down, and fix this, even if its for a few hours or one or two days. So that people stop exploiting ", "votes": 3, "replies": [ { "id": 1131245, "postDate": "2020-12-29T16:03:49.320Z", "content": "Kaggle staff is on holidays, this makes it even worse :)
", "rawMarkdown": "Kaggle staff is on holidays, this makes it even worse :)", "votes": 2 }, { "id": 1131264, "postDate": "2020-12-29T16:15:16.657Z", "content": "fixed now, thank god
", "rawMarkdown": "fixed now, thank god\n", "votes": 2 }, { "id": 1131464, "postDate": "2020-12-29T18:30:05.730Z", "content": "API returns null now for me on those fields, looks like they fixed it that way temporarily.
The funniest thing is that We are using publically available data 😂😂😂 to get our ranks. As time passes by, more and more people will exploit it in the currently going competitions too :(
", "rawMarkdown": "The funniest thing is that **We are using publically available data** 😂😂😂 to get our ranks. As time passes by, more and more people will exploit it in the currently going competitions too :(", "votes": 3 }, { "id": 1131028, "postDate": "2020-12-29T14:04:49.533Z", "content": "I can confirm this!
", "rawMarkdown": "I can confirm this!\n\n", "votes": 3 }, { "id": 1132129, "postDate": "2020-12-30T07:17:23.657Z", "content": "It looks like the bug has been fixed.
\nBut some may have peeked into the private testset, and I suggest that the sponsor replace the private testset.
2020 still surprises
", "rawMarkdown": "2020 still surprises", "votes": 4 }, { "id": 1131003, "postDate": "2020-12-29T13:44:21.307Z", "content": "@maggiemd please check this issue ASAP.
", "rawMarkdown": "@maggiemd please check this issue ASAP.", "votes": 3 }, { "id": 1132087, "postDate": "2020-12-30T06:55:44.753Z", "content": "Thanks for reporting the issue.
\nHas the issue fixed?
\nI cannot see my private score at this time (Google Chrome).
I tried @tereka method.
\n@sakuraandblackcat It was fixed
", "rawMarkdown": "@sakuraandblackcat It was fixed", "votes": 1 }, { "id": 1132354, "postDate": "2020-12-30T10:36:37.277Z", "content": "@yovinyahathugoda Thanks!
", "rawMarkdown": "@yovinyahathugoda Thanks!" } ] }, { "id": 1131215, "postDate": "2020-12-29T15:43:43.957Z", "content": "Ouch. That's a awkward bug. A way of overfitting the (private) leaderboard \"successfully\". And it's obviously tough to decide what can be done about ongoing and completed competitions where some may have used this. There may not even be any real way to tell who looked in this fashion, even if it were obvious how to treat this situation (e.g. if one wanted to disqualify any team that looked)?!
", "rawMarkdown": "Ouch. That's a awkward bug. A way of overfitting the (private) leaderboard \"successfully\". And it's obviously tough to decide what can be done about ongoing and completed competitions where some may have used this. There may not even be any real way to tell who looked in this fashion, even if it were obvious how to treat this situation (e.g. if one wanted to disqualify any team that looked)?!", "votes": 1 }, { "id": 1131194, "postDate": "2020-12-29T15:31:25.310Z", "content": "Having this information is like being immortal …
", "rawMarkdown": "Having this information is like being immortal ...", "votes": 1 }, { "id": 1131604, "postDate": "2020-12-29T20:46:22.860Z", "content": "Well, there is no way a random user found such serious bug, this might have been always there, never reported though!
", "rawMarkdown": "Well, there is no way a random user found such serious bug, this might have been always there, never reported though!", "votes": -1 }, { "id": 1137536, "postDate": "2021-01-04T03:01:24.107Z", "content": "Awesome find..
", "rawMarkdown": "Awesome find.." }, { "id": 1132906, "postDate": "2020-12-30T19:18:58.733Z", "content": "Thank you for sharing and correcting this. Does anyone know whether or not this will affect the final test set, and if we will have to run our data on a knew model? It seems this is not the first correction in this competition (https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/203581). Keep up the great work, and here's an upvote.
", "rawMarkdown": "Thank you for sharing and correcting this. Does anyone know whether or not this will affect the final test set, and if we will have to run our data on a knew model? It seems this is not the first correction in this competition (https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/203581). Keep up the great work, and here's an upvote." }, { "id": 1131217, "postDate": "2020-12-29T15:47:21.343Z", "content": "The Best CV… I just checked it!
", "rawMarkdown": "The Best CV... I just checked it!" }, { "id": 1131214, "postDate": "2020-12-29T15:43:38.187Z", "content": "The dark side of strength :(
", "rawMarkdown": "\nThe dark side of strength :(" }, { "id": 1131211, "postDate": "2020-12-29T15:40:29.880Z", "content": "In past simulation competition, we can get almost same json, like \"privateLeaderboardScore\".
\nI thought that column available after competition completed.
\nhttps://www.kaggle.com/robga/google-football-episode-scraper
Thanks for reporting this!
", "rawMarkdown": "Thanks for reporting this!" } ], "comments": [ { "id": 1131271, "author_name": "Addison Howard", "author_url": "", "post_date": "2020-12-29T16:17:38.927000", "content": "Hey all,
\nPlease know we're aware of this, looking into it, and working on it! It may take a little bit of time for us to release a full statement as we evaluate the full breadth and depth of the issue, including which competitions may have been impacted and the full extent of the impact on leaderboard results.
\nThanks for your patience! We'll let you know when we know more.
", "votes": 53, "replies": [ { "id": 1132139, "author_name": "cswwp", "author_url": "", "post_date": "2020-12-30T07:34:04.087000", "content": "So many people looked their private lb, how to keep fair, change private test set temporary？ @addisonhoward
", "votes": 0, "replies": [] } ] }, { "id": 1131099, "author_name": "Andrey Lukyanenko", "author_url": "", "post_date": "2020-12-29T14:44:29.147000", "content": "The biggest leak in Kaggle history? 😃
", "votes": 45, "replies": [] }, { "id": 1131110, "author_name": "tereka", "author_url": "", "post_date": "2020-12-29T14:51:21.717000", "content": "I think Unfair can only be seen by those who know that.
\nI just write how to see.(Google Chrome)
Note: this is an important bug, please fix that
", "votes": 43, "replies": [ { "id": 1131125, "author_name": "", "author_url": "", "post_date": "2020-12-29T14:57:19.793000", "content": "", "votes": 0, "replies": [] }, { "id": 1131139, "author_name": "tereka", "author_url": "", "post_date": "2020-12-29T15:04:19.837000", "content": "I don't agree with that.
\nHe didn't say how to, but he said there is information in the network.
\nAlmost kaggler can use network information using browser and dev tools.
\nThat thread Implicitly said method.
If when hiding, someone who din't use devtool cannot see private score. I think that is very unfair.
\nNOTE: In my opinion, he report to admin for silent(don't use competitions thread).
", "votes": 5, "replies": [] }, { "id": 1131143, "author_name": "hengck23", "author_url": "", "post_date": "2020-12-29T15:05:22.630000", "content": "it happens in all compeitions
", "votes": 8, "replies": [] }, { "id": 1131178, "author_name": "u++", "author_url": "", "post_date": "2020-12-29T15:24:52.957000", "content": "When I select submission files for final scores, I can see the change the value of privateLeaderboardRank, privateLeaderboardScore, and privateLeaderboardSubmissionId. Therefore, it seems that we can check all private scores of our own submission files.
Where is getUser ?
", "votes": 0, "replies": [] }, { "id": 1131193, "author_name": "u++", "author_url": "", "post_date": "2020-12-29T15:30:59.447000", "content": "You can see it when you access the leaderboard page. You might need to reload the page after opening the developer tool.
", "votes": 0, "replies": [] }, { "id": 1131206, "author_name": "Dapeng", "author_url": "", "post_date": "2020-12-29T15:35:48.617000", "content": "I found it… It's so ..crazy…
", "votes": 1, "replies": [] }, { "id": 1132057, "author_name": "Jaideep", "author_url": "", "post_date": "2020-12-30T06:24:35.090000", "content": "@li325040229 i dont see it for my competition atleaset as shown in screen shot. does it affects all competition ?
", "votes": 0, "replies": [] }, { "id": 1132062, "author_name": "Jaideep", "author_url": "", "post_date": "2020-12-30T06:30:40.480000", "content": "@sishihara i can see what value to put
", "votes": 0, "replies": [] } ] }, { "id": 1131043, "author_name": "raddar", "author_url": "", "post_date": "2020-12-29T14:14:35.610000", "content": "give this man a medal! well done
", "votes": 34, "replies": [] }, { "id": 1131246, "author_name": "Andrey Lukyanenko", "author_url": "", "post_date": "2020-12-29T16:04:05.387000", "content": "When this ends, I think we should also get to know how long ago this bug appeared - how many competitions could have been affected by it.
", "votes": 31, "replies": [ { "id": 1131255, "author_name": "Psi", "author_url": "", "post_date": "2020-12-29T16:09:26.897000", "content": "Yes, we need a full statement from Kaggle on this and how they can guarantee integrity of results.
", "votes": 24, "replies": [] } ] }, { "id": 1131239, "author_name": "hengck23", "author_url": "", "post_date": "2020-12-29T16:00:07.143000", "content": "
Now, all those fake accounts are starting to make sense 😄
\nI just hope the leak wasn't active for too long. Every winner of past shakeup competitions would be questioned. Hello magic scaling factors.
", "votes": 20, "replies": [] }, { "id": 1132928, "author_name": "Addison Howard", "author_url": "", "post_date": "2020-12-30T19:44:32.273000", "content": "Hi all - please see this statement from our team regarding the leaderboard bug.
", "votes": 19, "replies": [ { "id": 1132934, "author_name": "Psi", "author_url": "", "post_date": "2020-12-30T19:47:49.013000", "content": "Thanks @addisonhoward. Do you already have a plan about ongoing competitions, specifically those ending very soon?
", "votes": 5, "replies": [] }, { "id": 1132937, "author_name": "Addison Howard", "author_url": "", "post_date": "2020-12-30T19:52:39.613000", "content": "We are in contact with the hosts of ongoing competitions to determine if and what any immediate actions will be.
", "votes": 7, "replies": [] }, { "id": 1132977, "author_name": "Pascal Pfeiffer", "author_url": "", "post_date": "2020-12-30T20:49:35.820000", "content": "thanks for the statement @addisonhoward
\nhopefully, it's not just: \"one individual used it heavily enough to be detected\"
@Thanks for the statement @addisonhoward
\nHopefully fairer actions will be taken about ongoing competitions.
", "votes": 1, "replies": [] } ] }, { "id": 1131202, "author_name": "NikhilMishra", "author_url": "", "post_date": "2020-12-29T15:34:34.107000", "content": "And we thought 2020 was over 😂
", "votes": 12, "replies": [ { "id": 1132683, "author_name": "TEnsorSAge", "author_url": "", "post_date": "2020-12-30T15:57:54.303000", "content": "ulti😂😂😂😂
", "votes": 0, "replies": [] } ] }, { "id": 1132178, "author_name": "Nischay Dhankhar", "author_url": "", "post_date": "2020-12-30T08:18:31.463000", "content": "Someone who has already seen private scores yesterday for multiple submissions could be benefited by comparing and finding the correlation between public, private and cv rank/scores. Wouldn't it also affect the ongoing competitions??
", "votes": 8, "replies": [ { "id": 1132360, "author_name": "ANZ", "author_url": "", "post_date": "2020-12-30T10:44:45.700000", "content": "@nischaydnk
\nI think so.
\nBut there probably won't be a tool to distinguish between these two groups.
\nBesides, I think there is still a long time before the competition is over, so the benefits at this point may disappear.
\nSo I will try to make a good one without worrying about it.
Agreed, but there are still some competitions which are about to end in next weeks.
", "votes": 1, "replies": [] } ] }, { "id": 1131701, "author_name": "Luca Massaron", "author_url": "", "post_date": "2020-12-29T22:36:04.573000", "content": "At this point Kaggle should make a public statement explaining for how long that has been exposed and if that has been exploited by someone. When such a thing happens in a company, we call it a \"data breach\", and this is truly a data breach for the importance of the critical information that leaked out without anyone being aware of it.
", "votes": 8, "replies": [] }, { "id": 1131349, "author_name": "Serigne ", "author_url": "", "post_date": "2020-12-29T17:10:39.870000", "content": "This is a very serious and unfortunate issue.
\nWinning or getting at top of kaggle competitions require huge workload and dedication. Knowing that people could easily probe the private results to adjust their model and submissions can be discouraging.
\nI can't wait to read the statement about the full breadth and depth of the issue,
", "votes": 8, "replies": [] }, { "id": 1131154, "author_name": "Theo Viel", "author_url": "", "post_date": "2020-12-29T15:11:40.507000", "content": "I am now wondering if some people already knew about the trick and used it to tweak hyper-parameters (thresholds, blending coefficients, …) using the private leaderboard.
", "votes": 6, "replies": [ { "id": 1131161, "author_name": "raddar", "author_url": "", "post_date": "2020-12-29T15:13:44.157000", "content": "There indeed is such a possibility, and could have been used to probe the private LB
", "votes": 2, "replies": [] }, { "id": 1131163, "author_name": "Psi", "author_url": "", "post_date": "2020-12-29T15:15:56.403000", "content": "I wonder indeed for how long this already has existed… jesus this is too problematic
", "votes": 6, "replies": [] }, { "id": 1131177, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2020-12-29T15:24:39.983000", "content": "Kagglers who never shake down should be investigated:) I think only the current best public submission's private score is leaked, right? If all is leaked, it gets even worse.
", "votes": 9, "replies": [] }, { "id": 1131180, "author_name": "Psi", "author_url": "", "post_date": "2020-12-29T15:25:27.570000", "content": "The question is if it would show the private score based on what you selected manually?
\nEdit: confirmed below
I can confirm this, when I selected 2 different submissions, the private score just changed
", "votes": 1, "replies": [] }, { "id": 1131185, "author_name": "NikhilMishra", "author_url": "", "post_date": "2020-12-29T15:27:40.473000", "content": "this is very serious then, it means you can see the private LB of any submission. just imagine someone creating 100 fake accounts, and using them to get correct label, it could be disastrous
", "votes": 1, "replies": [] }, { "id": 1131225, "author_name": "ONODERA", "author_url": "", "post_date": "2020-12-29T15:50:36.803000", "content": "Only we can trust the competition which had broken private LB such as OpenVaccine.
\nFYI, I was 2nd place on OpenVaccine😉
After that many years still cannot beat the curse of 2nd - maybe this will help to remove the curse ?:D
", "votes": 3, "replies": [] }, { "id": 1131306, "author_name": "Rob Mulla", "author_url": "", "post_date": "2020-12-29T16:41:41.507000", "content": "\n\nOnly we can trust the competition which had broken private LB such as OpenVaccine.
\n
Or competitions where the private LB was based on future data- like last years NFL Big Data Bowl and Two Sigma.
", "votes": 2, "replies": [] }, { "id": 1131319, "author_name": "Dapeng", "author_url": "", "post_date": "2020-12-29T16:51:16.100000", "content": "\n\nKagglers who never shake down should be investigated:)
\n
And often shake down in those competitions which have broken private LB?
", "votes": 0, "replies": [] }, { "id": 1132324, "author_name": "Luca Massaron", "author_url": "", "post_date": "2020-12-30T10:06:06.093000", "content": "Probably it is not possibile to prove anyone has used that leakage because the data leaked for all the submissions and if you took advantage of it or not is not something that could be recorded anywhere. Also questioning about the optimal behaviour of a team in shake down competitions is not a definitive proof of having used the leak. Most likely they will not be able to point out anyone who took advantage of it. We expect at least to know how many competitions this leakage may have affected the results.
", "votes": 1, "replies": [] } ] }, { "id": 1131447, "author_name": "عثمان", "author_url": "", "post_date": "2020-12-29T18:20:22.360000", "content": "Ain't even mad. This is more hilarious than anything. Everyone makes mistakes and trusting the front end to hide logic never intended for the user rather than filtering on the back end is one of the classical ones. Whenever Kaggle gets around to publishing the write up on this, I'd just request it's also shared with the community how long this issue has been present / how many competitions this has affected.
\nEdit: oops just read through some of the other posts and looks like a few people also shared pretty much verbatim sentiments.
\nSo that my post isn't completely useless, I'll just drop this little nugget here:
\nhttps://twitter.com/twimlai/status/1343976072577073152
\nNot just from an ethics standpoint, but also from a software engineering standpoint. And in some cases such as this, the results of one directly affect the other.
", "votes": 6, "replies": [] }, { "id": 1131580, "author_name": "Luca Massaron", "author_url": "", "post_date": "2020-12-29T20:24:19.143000", "content": "I wonder for how long that has been available to a few ones
", "votes": 3, "replies": [] }, { "id": 1131481, "author_name": "", "author_url": "", "post_date": "2020-12-29T18:43:33.447000", "content": "wow this is bad..
\nThank you for sharing rather than taking advantage of it!
", "votes": 3, "replies": [] }, { "id": 1131424, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2020-12-29T18:07:31.437000", "content": "This is a very intriguing find so thanks for sharing. If I may ask, how did you find out about it? Just by randomly checking HTTP responses or did you had an intuition that something was off?
\nAs pointed by others, it would be interesting for how long this have been going on and why it was there on the first place (was the code reviewed and approved?).
\nA postmortem by Kaggle would be very valuable.
", "votes": 3, "replies": [] }, { "id": 1131291, "author_name": "Camaro", "author_url": "", "post_date": "2020-12-29T16:33:51.543000", "content": "What's most important is \"when\" it has started. Please include timeline in the full statement, unless it will be understood \"always\"…
\nHope it's only this 2~3 months or less, for the honor of past competition winners.
@kaggle should go down, and fix this, even if its for a few hours or one or two days. So that people stop exploiting
", "votes": 3, "replies": [ { "id": 1131245, "author_name": "raddar", "author_url": "", "post_date": "2020-12-29T16:03:49.320000", "content": "Kaggle staff is on holidays, this makes it even worse :)
", "votes": 2, "replies": [] }, { "id": 1131264, "author_name": "NikhilMishra", "author_url": "", "post_date": "2020-12-29T16:15:16.657000", "content": "fixed now, thank god
", "votes": 2, "replies": [] }, { "id": 1131464, "author_name": "Bouwe Ceunen", "author_url": "", "post_date": "2020-12-29T18:30:05.730000", "content": "API returns null now for me on those fields, looks like they fixed it that way temporarily.
The funniest thing is that We are using publically available data 😂😂😂 to get our ranks. As time passes by, more and more people will exploit it in the currently going competitions too :(
", "votes": 3, "replies": [] }, { "id": 1131028, "author_name": "Abhishek Thakur", "author_url": "", "post_date": "2020-12-29T14:04:49.533000", "content": "I can confirm this!
", "votes": 3, "replies": [] }, { "id": 1132129, "author_name": "Tian", "author_url": "", "post_date": "2020-12-30T07:17:23.657000", "content": "It looks like the bug has been fixed.
\nBut some may have peeked into the private testset, and I suggest that the sponsor replace the private testset.
2020 still surprises
", "votes": 4, "replies": [] }, { "id": 1131003, "author_name": "Y.Nakama", "author_url": "", "post_date": "2020-12-29T13:44:21.307000", "content": "@maggiemd please check this issue ASAP.
", "votes": 3, "replies": [] }, { "id": 1132087, "author_name": "ANZ", "author_url": "", "post_date": "2020-12-30T06:55:44.753000", "content": "Thanks for reporting the issue.
\nHas the issue fixed?
\nI cannot see my private score at this time (Google Chrome).
I tried @tereka method.
\n@sakuraandblackcat It was fixed
", "votes": 1, "replies": [] }, { "id": 1132354, "author_name": "ANZ", "author_url": "", "post_date": "2020-12-30T10:36:37.277000", "content": "@yovinyahathugoda Thanks!
", "votes": 0, "replies": [] } ] }, { "id": 1131215, "author_name": "Björn", "author_url": "", "post_date": "2020-12-29T15:43:43.957000", "content": "Ouch. That's a awkward bug. A way of overfitting the (private) leaderboard \"successfully\". And it's obviously tough to decide what can be done about ongoing and completed competitions where some may have used this. There may not even be any real way to tell who looked in this fashion, even if it were obvious how to treat this situation (e.g. if one wanted to disqualify any team that looked)?!
", "votes": 1, "replies": [] }, { "id": 1131194, "author_name": "Manuel Campos", "author_url": "", "post_date": "2020-12-29T15:31:25.310000", "content": "Having this information is like being immortal …
", "votes": 1, "replies": [] }, { "id": 1131604, "author_name": "Harveen Singh Chadha", "author_url": "", "post_date": "2020-12-29T20:46:22.860000", "content": "Well, there is no way a random user found such serious bug, this might have been always there, never reported though!
", "votes": -1, "replies": [] }, { "id": 1137536, "author_name": "Vishy", "author_url": "", "post_date": "2021-01-04T03:01:24.107000", "content": "Awesome find..
", "votes": 0, "replies": [] }, { "id": 1132906, "author_name": "", "author_url": "", "post_date": "2020-12-30T19:18:58.733000", "content": "", "votes": 0, "replies": [] }, { "id": 1131217, "author_name": "", "author_url": "", "post_date": "2020-12-29T15:47:21.343000", "content": "", "votes": 0, "replies": [] }, { "id": 1131214, "author_name": "", "author_url": "", "post_date": "2020-12-29T15:43:38.187000", "content": "", "votes": 0, "replies": [] }, { "id": 1131211, "author_name": "", "author_url": "", "post_date": "2020-12-29T15:40:29.880000", "content": "", "votes": 0, "replies": [] }, { "id": 1132007, "author_name": "", "author_url": "", "post_date": "2020-12-30T05:39:16.433000", "content": "", "votes": -4, "replies": [ { "id": 1132080, "author_name": "", "author_url": "", "post_date": "2020-12-30T06:45:40.387000", "content": "", "votes": -1, "replies": [] } ] }, { "id": 1131249, "author_name": "", "author_url": "", "post_date": "2020-12-29T16:07:21.720000", "content": "", "votes": 5, "replies": [ { "id": 1131250, "author_name": "", "author_url": "", "post_date": "2020-12-29T16:08:48.843000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 1131591, "author_name": "", "author_url": "", "post_date": "2020-12-29T20:30:24.967000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1130871": "I am not sure if it is the right leaderboard, but I am able to see my private leaderboard rank (see the attachment). I was able to get it by looking at the network requests and responses on the leaderboard page.\n\nEdit 1: I had sent a message to kaggle support in `https://www.kaggle.com/contact` in the ' I found an issue with the website' section at around 23:50 UTC, Sunday detailing how the private leaderboard could be accessible. This was for the challenge `https://www.kaggle.com/c/cassava-leaf-disease-classification`. Unfortunately, kaggle did not send me a copy of the message, however I replied to (what I think is automatically generated mail) with a screenshot similar to what I posted earlier but for the other competition. I did not receive (to the best of my knowledge) any response. Attached the screenshot of the mail thread (note that in India it is +05:30 UTC).\n\nEdit 2: It seems (as acknowledged by others) that the bug is fixed. Good.", "1131271": "Hey all,\n\nPlease know we're aware of this, looking into it, and working on it! It may take a little bit of time for us to release a full statement as we evaluate the full breadth and depth of the issue, including which competitions may have been impacted and the full extent of the impact on leaderboard results.\n\nThanks for your patience! We'll let you know when we know more.", "1131099": "The biggest leak in Kaggle history? 😃", "1131110": "I think Unfair can only be seen by those who know that.\nI just write how to see.(Google Chrome)\n\n1. Go to Leaderboard\n2. open Devtool -> Network -> Reload\n3. XHR only->select getUser request\n4. result->team, maybe you can see your public/private score and ranking\n\nNote: this is an important bug, please fix that", "1131043": "give this man a medal! well done", "1131246": "When this ends, I think we should also get to know how long ago this bug appeared - how many competitions could have been affected by it.\n", "1131239": "", "1131471": "Now, all those fake accounts are starting to make sense 😄\n\nI just hope the leak wasn't active for too long. Every winner of past shakeup competitions would be questioned. Hello magic scaling factors. ", "1132928": "Hi all - please see [this statement](https://www.kaggle.com/private-leaderboard-bug-dec-2020) from our team regarding the leaderboard bug.", "1131202": "And we thought 2020 was over 😂", "1132178": "Someone who has already seen private scores yesterday for multiple submissions could be benefited by comparing and finding the correlation between public, private and cv rank/scores. Wouldn't it also affect the ongoing competitions??", "1131701": "At this point Kaggle should make a public statement explaining for how long that has been exposed and if that has been exploited by someone. When such a thing happens in a company, we call it a \"data breach\", and this is truly a data breach for the importance of the critical information that leaked out without anyone being aware of it.", "1131349": "This is a very serious and unfortunate issue.\n\nWinning or getting at top of kaggle competitions require huge workload and dedication. Knowing that people could easily probe the private results to adjust their model and submissions can be discouraging.\n\nI can't wait to read the statement about the full breadth and depth of the issue,\n", "1131154": "I am now wondering if some people already knew about the trick and used it to tweak hyper-parameters (thresholds, blending coefficients, ...) using the private leaderboard. ", "1131447": "Ain't even mad. This is more hilarious than anything. Everyone makes mistakes and trusting the front end to hide logic never intended for the user rather than filtering on the back end is one of the classical ones. Whenever Kaggle gets around to publishing the write up on this, I'd just request it's also shared with the community how long this issue has been present / how many competitions this has affected.\n\nEdit: oops just read through some of the other posts and looks like a few people also shared pretty much verbatim sentiments. \n\nSo that my post isn't completely useless, I'll just drop this little nugget here:\n\nhttps://twitter.com/twimlai/status/1343976072577073152\n\nNot just from an ethics standpoint, but also from a software engineering standpoint. And in some cases such as this, the results of one directly affect the other.", "1131580": "I wonder for how long that has been available to a few ones", "1131481": "wow this is bad..\n\n\nThank you for sharing rather than taking advantage of it! ", "1131424": "This is a very intriguing find so thanks for sharing. If I may ask, how did you find out about it? Just by randomly checking HTTP responses or did you had an intuition that something was off?\n\nAs pointed by others, it would be interesting for how long this have been going on and why it was there on the first place (was the code reviewed and approved?). \n\nA postmortem by Kaggle would be very valuable. ", "1131291": "What's most important is \"when\" it has started. Please include timeline in the full statement, unless it will be understood \"always\"...\nHope it's only this 2~3 months or less, for the honor of past competition winners.", "1131234": "@kaggle should go down, and fix this, even if its for a few hours or one or two days. So that people stop exploiting ", "1131179": "The funniest thing is that **We are using publically available data** 😂😂😂 to get our ranks. As time passes by, more and more people will exploit it in the currently going competitions too :(", "1131028": "I can confirm this!\n\n", "1132129": "It looks like the bug has been fixed.\nBut some may have peeked into the private testset, and I suggest that the sponsor replace the private testset.", "1131205": "2020 still surprises", "1131003": "@maggiemd please check this issue ASAP.", "1132087": "Thanks for reporting the issue.\n\nHas the issue fixed?\nI cannot see my private score at this time (Google Chrome).\n\nI tried @tereka method.\n1. Go to Leaderboard\n2. open Devtool -> Network -> Reload\n3. XHR only->select getUser request\n4. result->team, maybe you can see your public/private score and ranking", "1131215": "Ouch. That's a awkward bug. A way of overfitting the (private) leaderboard \"successfully\". And it's obviously tough to decide what can be done about ongoing and completed competitions where some may have used this. There may not even be any real way to tell who looked in this fashion, even if it were obvious how to treat this situation (e.g. if one wanted to disqualify any team that looked)?!", "1131194": "Having this information is like being immortal ...", "1131604": "Well, there is no way a random user found such serious bug, this might have been always there, never reported though!", "1137536": "Awesome find..", "1132906": "Thank you for sharing and correcting this. Does anyone know whether or not this will affect the final test set, and if we will have to run our data on a knew model? It seems this is not the first correction in this competition (https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/203581). Keep up the great work, and here's an upvote.", "1131217": "The Best CV... I just checked it!", "1131214": "\nThe dark side of strength :(", "1131211": "In past simulation competition, we can get almost same json, like \"privateLeaderboardScore\".\nI thought that column available after competition completed.\nhttps://www.kaggle.com/robga/google-football-episode-scraper", "1132007": "", "1131249": "", "1131591": "Thanks for reporting this!" } }