{ "id": 207875, "title": "Private Score Release", "url": "/competitions/riiid-test-answer-prediction/discussion/207875", "author_name": "Addison Howard", "post_date": "2020-12-31T18:10:50.765000", "votes": 69, "comment_count": 35, "views": 0, "content": "

Hello Kagglers,

\n

As you may be aware, a recently disclosed vulnerability allowed teams to view their own private leaderboard score prior to the deadline. We have released a statement describing the bug here.

\n

Since the vulnerability was publicly disclosed for approximately five hours, a small number of teams accessed their team’s private score. Because this competition is so close to its deadline and we want teams to have a fair footing, Kaggle and the hosts have agreed to make available the identity and score of your team’s best private submission as of the time the bug was fixed.

\n

To retrieve your team’s score as of 2020-12-29 16:00:48 UTC, please visit this link.

\n

We believe this approach is a fair and minimally disruptive way to finish this competition. In coming to this decision, we considered a number of alternatives, including creating a new test set. The disadvantages introduced by a new test set and model reruns are not warranted, given the limited impact resulting from our investigation. Similarly, we do not believe this incident merits a deadline extension. In contrast to other competition interventions, such as the introduction of new data or discovery of critical data flaws, the underlying task and data remains the same, just with an additional cross-validation input to consider.

\n

We apologize again for the inconvenience this bug caused and thank you for your participation and understanding.

\n

Kaggle Team

", "messages": [ { "id": 1133987, "postDate": "2020-12-31T18:10:50.767Z", "content": "

Hello Kagglers,

\n

As you may be aware, a recently disclosed vulnerability allowed teams to view their own private leaderboard score prior to the deadline. We have released a statement describing the bug here.

\n

Since the vulnerability was publicly disclosed for approximately five hours, a small number of teams accessed their team’s private score. Because this competition is so close to its deadline and we want teams to have a fair footing, Kaggle and the hosts have agreed to make available the identity and score of your team’s best private submission as of the time the bug was fixed.

\n

To retrieve your team’s score as of 2020-12-29 16:00:48 UTC, please visit this link.

\n

We believe this approach is a fair and minimally disruptive way to finish this competition. In coming to this decision, we considered a number of alternatives, including creating a new test set. The disadvantages introduced by a new test set and model reruns are not warranted, given the limited impact resulting from our investigation. Similarly, we do not believe this incident merits a deadline extension. In contrast to other competition interventions, such as the introduction of new data or discovery of critical data flaws, the underlying task and data remains the same, just with an additional cross-validation input to consider.

\n

We apologize again for the inconvenience this bug caused and thank you for your participation and understanding.

\n

Kaggle Team

", "rawMarkdown": "Hello Kagglers,\n\nAs you may be aware, a recently disclosed vulnerability allowed teams to view their own private leaderboard score prior to the deadline. We have released a statement describing the bug [here](https://www.kaggle.com/private-leaderboard-bug-dec-2020).\n\nSince the vulnerability was publicly disclosed for approximately five hours, a small number of teams accessed their team’s private score. Because this competition is so close to its deadline and we want teams to have a fair footing, Kaggle and the hosts have agreed to make available the identity and score of your team’s best private submission as of the time the bug was fixed. \n\nTo retrieve your team’s score as of 2020-12-29 16:00:48 UTC, please visit [this link](https://www.kaggle.com/c/riiid-test-answer-prediction/best-private-submission-dec-2020).\n\nWe believe this approach is a fair and minimally disruptive way to finish this competition. In coming to this decision, we considered a number of alternatives, including creating a new test set. The disadvantages introduced by a new test set and model reruns are not warranted, given the limited impact resulting from our investigation. Similarly, we do not believe this incident merits a deadline extension. In contrast to other competition interventions, such as the introduction of new data or discovery of critical data flaws, the underlying task and data remains the same, just with an additional cross-validation input to consider.\n\nWe apologize again for the inconvenience this bug caused and thank you for your participation and understanding.\n\nKaggle Team\n", "votes": 68 }, { "id": 1147882, "postDate": "2021-01-10T19:23:40.227Z", "content": "

This kind of bug should have been disclosed after the competition is over or bug is fixed.
\nI think the impact of this case would have been much smaller if it had been released after this bug is fixed.

\n

However, as a matter of reality, if the issue is released after the competition, it will not attract much attention, whereas if it is released during the competition period, it will attract attention and be treated as a hero.
\nThis means that in the future as well, this kind of issues will be disclosed during the competition period in the discussion.

\n

So, I propose to give some kind of incentive to those who find this kind of bug and report it to the administration.

\n

As a start, I think it would be a good idea to give this prize to @nyanpn.
\nHe found another way to know private score for this competition, reported it to the administration, and waited until the end of the competition to announce it.

\n

@addisonhoward, @sohier,
\nI would appreciate it if you would consider it.

", "rawMarkdown": "This kind of bug should have been disclosed after the competition is over or bug is fixed.\nI think the impact of this case would have been much smaller if it had been released ~~after the competition~~ after this bug is fixed.\n\nHowever, as a matter of reality, if the issue is released after the competition, it will not attract much attention, whereas if it is released during the competition period, it will attract attention and be treated as a hero.\n**This means that in the future as well, this kind of issues will be disclosed during the competition period in the discussion.**\n\n**So, I propose to give some kind of incentive to those who find this kind of bug and report it to the administration.**\n\nAs a start, I think it would be a good idea to give this prize to @nyanpn.\nHe found [another way to know private score for this competition](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/210354), reported it to the administration, and waited until the end of the competition to announce it.\n\n@addisonhoward, @sohier, \nI would appreciate it if you would consider it.", "votes": 11, "replies": [ { "id": 1148107, "postDate": "2021-01-10T22:41:43.683Z", "content": "

😶 there were multiple concurrent running competitions that this affected though. That said, I agree with you that the accepted white-hat etiquette for 0-day disclosure is first to notify the company/devs in charge and see how they respond to and handle the situation before sharing with the planet =]. IIRC it took Kaggle less than 3 hours to make the patch.

", "rawMarkdown": "😶 there were multiple concurrent running competitions that this affected though. That said, I agree with you that the *accepted* white-hat etiquette for 0-day disclosure is first to notify the company/devs in charge and see how they respond to and handle the situation before sharing with the planet =]. IIRC it took Kaggle less than 3 hours to make the patch.", "votes": 2 }, { "id": 1148118, "postDate": "2021-01-10T23:01:45.310Z", "content": "

This one has been affected most because the correct target is being updated and available to the users, after the prediction is made, which is the bug @its7171 mentioned in his post.

", "rawMarkdown": "This one has been affected most because the correct target is being updated and available to the users, after the prediction is made, which is the bug @its7171 mentioned in his post.", "votes": 1 }, { "id": 1148255, "postDate": "2021-01-11T03:00:53.917Z", "content": "

Thanks for your comment.

\n

I should have said, I think the impact of this case would have been much smaller if it had been released after this bug is fixed.

\n

Announcement should be made when the bug no longer affects.
\nI think this kind of bugs should be published after the bug has been fixed, and the kind of issues should be published after the competition has ended.

\n

Anyway, I think the best way to deal with this kind of bug is to report it to the administration and follow their instructions before publishing it.

\n

And my point is to give the motivation to do so.

", "rawMarkdown": "Thanks for your comment.\n\nI should have said, I think the impact of this case would have been much smaller if it had been released *after this bug is fixed*.\n\nAnnouncement should be made when the bug no longer affects.\nI think [this kind of bugs](https://www.kaggle.com/private-leaderboard-bug-dec-2020) should be published after the bug has been fixed, and [the kind of issues](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/210354) should be published after the competition has ended.\n\nAnyway, I think the best way to deal with this kind of bug is to report it to the administration and follow their instructions before publishing it.\n\nAnd my point is to give the motivation to do so." }, { "id": 1149093, "postDate": "2021-01-11T15:29:55.983Z", "content": "

Hey all,

\n

We're thankful the Kaggle community comes to us with bug reports throughout competitions. We almost always reward users for doing so. In the event you find a security issue on Kaggle, you can report it via https://kaggle.com/contact or https://goo.gl/vulnz.

\n

The user did report it via our Support channels, however with the holidays and weekend hours, we had not yet seen the report by the time it was raised in the Kaggle forums.

\n

I hope that provides some additional context!

\n

Addison

", "rawMarkdown": "Hey all,\n\nWe're thankful the Kaggle community comes to us with bug reports throughout competitions. We almost always reward users for doing so. In the event you find a security issue on Kaggle, you can report it via https://kaggle.com/contact or https://goo.gl/vulnz.\n\nThe user did report it via our Support channels, however with the holidays and weekend hours, we had not yet seen the report by the time it was raised in the Kaggle forums. \n\nI hope that provides some additional context!\n\nAddison", "votes": 2 }, { "id": 1149658, "postDate": "2021-01-12T03:42:05.723Z", "content": "

Thanks for the information!
\nI did not know about the reward.

\n

So you have a system for reporting issues and rewarding it.
\nBut there might be some room for improvement.

\n

I think it might be a good idea to publish this information.

\n\n

This would help to motivate to report issues before disclose it.

\n

Also, I am interested in how you deal with this private score probing issue.
\nI think this is a very big issue because we can know the private score with enough accuracy by binary search.

", "rawMarkdown": "Thanks for the information!\nI did not know about the reward.\n\nSo you have a system for reporting issues and rewarding it.\nBut there might be some room for improvement.\n\nI think it might be a good idea to publish this information.\n* Who reported it\n* What the issue is about\n* When it was fixed or a policy for handling the issue\n* The reward he got\n* etc\n\nThis would help to motivate to report issues before disclose it.\n\nAlso, I am interested in how you deal with [this private score probing issue](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/210354).\nI think this is a very big issue because **we can know the private score with enough accuracy** by binary search.", "votes": 3 }, { "id": 1149677, "postDate": "2021-01-12T04:08:09.743Z", "content": "

I agree with tito, private score probing issue will be a very big problem in future competitions that use Time-Series API. @addisonhoward please consider adding some dummy data/target when using Time-Series API to prevent private score probing.

", "rawMarkdown": "I agree with tito, [private score probing](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/210354) issue will be a very big problem in future competitions that use Time-Series API. @addisonhoward please consider adding some dummy data/target when using Time-Series API to prevent private score probing.", "votes": 2 }, { "id": 1151866, "postDate": "2021-01-13T16:12:53.090Z", "content": "

Regarding disclosures, I believe there's two types:

\n
    \n

  1. Security vulnerabilities or exploits, which if you actually use/exploit it would be considered bad and would get banned/etc. So you have little to no incentive to not disclose.

    2. \n

  2. Data leaks. As fas as I know there's no actual policy wrt to rewards for data leaks and there's been instances in which participants have disclosed leaks and as a consequence competition restarted (e.g https://www.kaggle.com/c/airbus-ship-detection/discussion/64355 in which we got rewarded a t-shirt). Other people may find a t-shirt is not good enough and would use the privately found leak (the line between leak, magic and correlation is blurry) e.g. https://www.kaggle.com/c/liverpool-ion-switching/discussion/153940 and win.

    3. \n
", "rawMarkdown": "Regarding disclosures, I believe there's two types:\n\n1. Security vulnerabilities or exploits, which if you actually use/exploit it would be considered bad and would get banned/etc. So you have little to no incentive to not disclose.\n\n2. Data leaks. As fas as I know there's no actual policy wrt to rewards for data leaks and there's been instances in which participants have disclosed leaks and as a consequence competition restarted (e.g https://www.kaggle.com/c/airbus-ship-detection/discussion/64355 in which we got rewarded a t-shirt). Other people may find a t-shirt is not good enough and would use the privately found leak (the line between leak, magic and correlation is blurry) e.g. https://www.kaggle.com/c/liverpool-ion-switching/discussion/153940 and win.", "votes": 2 } ] }, { "id": 1135908, "postDate": "2021-01-02T15:35:35.200Z", "content": "

", "rawMarkdown": "~~I would be thankful if someone who has used the public 0.783/0.781 notebook(s) could post the private score for the same. It helps to understand the minimum baseline people have to beat to get a medal. ~~", "votes": 4, "replies": [ { "id": 1136802, "postDate": "2021-01-03T12:57:24.177Z", "content": "

I think this could be against the spirit of the competition. I take it back.

", "rawMarkdown": "I think this could be [against the spirit of the competition](#1134028). I take it back. ", "votes": 4 } ] }, { "id": 1134192, "postDate": "2021-01-01T01:12:45.053Z", "content": "

Thank you Addison, This is a very simple, and realistic solution to this situation, as LB-CV is very stable and the effect of the bug is not so big at least in this competition! :)

", "rawMarkdown": "Thank you Addison, This is a very simple, and realistic solution to this situation, as LB-CV is very stable and the effect of the bug is not so big at least in this competition! :)", "votes": 4 }, { "id": 1134013, "postDate": "2020-12-31T18:40:48.720Z", "content": "

Thank you for finding an alternative and updating us @addisonhoward. It would be helpful if you also clarify that if teams are allowed to share their private score or mention it anywhere in discussions??

", "rawMarkdown": "Thank you for finding an alternative and updating us @addisonhoward. It would be helpful if you also clarify that if teams are allowed to share their private score or mention it anywhere in discussions??", "votes": 4, "replies": [ { "id": 1134028, "postDate": "2020-12-31T18:54:30.470Z", "content": "

We don’t explicitly prohibit the sharing of cross-validation scores at your chosen discretion, as long as it still fits in accordance with our rules that restrict private data sharing outside of teams. However, sharing this private score could be viewed by your peers as against the spirit of the competition, not unlike sharing a high-scoring notebook toward the end of the competition.

", "rawMarkdown": "We don’t explicitly prohibit the sharing of cross-validation scores at your chosen discretion, as long as it still fits in accordance with our rules that restrict private data sharing outside of teams. However, sharing this private score could be viewed by your peers as against the spirit of the competition, not unlike sharing a high-scoring notebook toward the end of the competition.", "votes": 3 } ] }, { "id": 1134227, "postDate": "2021-01-01T02:59:08.577Z", "content": "

@addisonhoward i think this is very fair solution to reduce the impact of bug however it would been even more fair to the participants who dint see scores by letting all know private score for all sub till that date .
\nAs people who have viewed score while big was on would have seen all model scored they used till that time

", "rawMarkdown": "@addisonhoward i think this is very fair solution to reduce the impact of bug however it would been even more fair to the participants who dint see scores by letting all know private score for all sub till that date .\nAs people who have viewed score while big was on would have seen all model scored they used till that time ", "votes": 2, "replies": [ { "id": 1134260, "postDate": "2021-01-01T04:31:50.463Z", "content": "

I agree with your idea. However, I also agree with @addisonhoward's idea,

\n
releasing all submission scores would be detrimental from an overfitting standpoint, unfairly benefit teams with large numbers of submissions, and would be out-of-proportion to the information we believe was viewed while the bug was live.\n
\n

It's difficult to say which is better :(

", "rawMarkdown": "I agree with your idea. However, I also agree with @addisonhoward's idea, \n```\nreleasing all submission scores would be detrimental from an overfitting standpoint, unfairly benefit teams with large numbers of submissions, and would be out-of-proportion to the information we believe was viewed while the bug was live.\n```\nIt's difficult to say which is better :(", "votes": 1 }, { "id": 1134266, "postDate": "2021-01-01T04:50:47.783Z", "content": "

I suggest to release at least top 5 submissions private leaderboard score because some participants already had accessed on private leaderboard scores before Kaggle investigation on this issue

", "rawMarkdown": "I suggest to release at least top 5 submissions private leaderboard score because some participants already had accessed on private leaderboard scores before Kaggle investigation on this issue", "votes": 5 }, { "id": 1135284, "postDate": "2021-01-02T05:28:44.033Z", "content": "

@addisonhoward this is very fair. Top 5 private score till date .
\nPlease consider this. I am currently confused between the model whose private score i seen, and model whose private score i dint see and is 3rd best. But there is atleast person in similar state must have seen all the scores and has by now decided which one to keep and which one to discard. Saving time ,resources

", "rawMarkdown": "@addisonhoward this is very fair. Top 5 private score till date . \nPlease consider this. I am currently confused between the model whose private score i seen, and model whose private score i dint see and is 3rd best. But there is atleast person in similar state must have seen all the scores and has by now decided which one to keep and which one to discard. Saving time ,resources \n", "votes": 2 } ] }, { "id": 1134104, "postDate": "2020-12-31T20:37:30.007Z", "content": "

@addisonhoward I believe people could see their private rank. Could you also provide the rank as well?

", "rawMarkdown": "@addisonhoward I believe people could see [their private rank] (https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/207390). Could you also provide the rank as well?", "votes": 2, "replies": [ { "id": 1134117, "postDate": "2020-12-31T21:10:37.900Z", "content": "

While it is true that this was visible as part of the bug, private rank is conditional on a number of factors outside of your own performance--including submission selections of other teams--and is not directly relevant to model cross validation. We will not be providing it here.

", "rawMarkdown": "While it is true that this was visible as part of the bug, private rank is conditional on a number of factors outside of your own performance--including submission selections of other teams--and is not directly relevant to model cross validation. We will not be providing it here.\n", "votes": 2 } ] }, { "id": 1134054, "postDate": "2020-12-31T19:37:21.773Z", "content": "

Liked what you saw guys? 😁

", "rawMarkdown": "Liked what you saw guys? 😁", "votes": 2, "replies": [ { "id": 1134061, "postDate": "2020-12-31T19:49:57.417Z", "content": "

I have cv0805 but public0799 private0800 single tfm model. Must be a bug during inference or I miss something. I thought my private is equivalent to cv. So thanks to this I know there must be something wrong that needs to be fixed.

", "rawMarkdown": "I have cv0805 but public0799 private0800 single tfm model. Must be a bug during inference or I miss something. I thought my private is equivalent to cv. So thanks to this I know there must be something wrong that needs to be fixed. ", "votes": 1 }, { "id": 1134069, "postDate": "2020-12-31T19:58:42.900Z", "content": "
\n

\"publicScore\":\"0.763\",
\n \"privateScore\":\"0.762\",

\n
\n

🙃 Perfect correlation. On a serious note, I think the approach the Kaggle took here strikes a good balance.

", "rawMarkdown": "> \"publicScore\":\"0.763\",\n> \"privateScore\":\"0.762\",\n\n🙃 Perfect correlation. On a serious note, I think the approach the Kaggle took here strikes a good balance.", "votes": 2 }, { "id": 1134111, "postDate": "2020-12-31T20:58:33.643Z", "content": "
\"publicScore\":\"0.789\",\n\"privateScore\":\"0.789\"\n
\n

Perfect correlation for me too. I don't think our competition was affected as much as others because of this correlation between the public and private LBs.

", "rawMarkdown": "```\n\"publicScore\":\"0.789\",\n\"privateScore\":\"0.789\"\n```\nPerfect correlation for me too. I don't think our competition was affected as much as others because of this correlation between the public and private LBs.", "votes": 2 }, { "id": 1134112, "postDate": "2020-12-31T20:59:02.033Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1134121, "postDate": "2020-12-31T21:27:15.847Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1134194, "postDate": "2021-01-01T01:24:26.640Z", "content": "

\"publicScore\":\"0.786\",\"privateScore\":\"0.789\"
\nShake is big 🤔

", "rawMarkdown": "\"publicScore\":\"0.786\",\"privateScore\":\"0.789\"\nShake is big 🤔", "votes": 1 }, { "id": 1134229, "postDate": "2021-01-01T03:01:43.210Z", "content": "

I think every one is getting .001 to.004 increase in scores ..

", "rawMarkdown": "I think every one is getting .001 to.004 increase in scores ..", "votes": 1 } ] }, { "id": 1134032, "postDate": "2020-12-31T19:03:25.103Z", "content": "

I think the fairest approach is to release all private scores for the submissions up to just before the bug fix.

", "rawMarkdown": "I think the fairest approach is to release all private scores for the submissions up to just before the bug fix.", "replies": [ { "id": 1134040, "postDate": "2020-12-31T19:19:13.563Z", "content": "

Our analysis suggests that releasing all submission scores would be detrimental from an overfitting standpoint, unfairly benefit teams with large numbers of submissions, and would be out-of-proportion to the information we believe was viewed while the bug was live.

", "rawMarkdown": "Our analysis suggests that releasing all submission scores would be detrimental from an overfitting standpoint, unfairly benefit teams with large numbers of submissions, and would be out-of-proportion to the information we believe was viewed while the bug was live.", "votes": 15 }, { "id": 1134334, "postDate": "2021-01-01T07:03:16.227Z", "content": "

at least top 5 submission scores which is equivalent of maximum number of submissions per day.

", "rawMarkdown": "at least top 5 submission scores which is equivalent of maximum number of submissions per day." } ] }, { "id": 1135038, "postDate": "2021-01-01T20:14:05.580Z", "content": "

@addisonhoward - The report said my 44 version have public score - 0.794:
\n{\"name\":\"Baseline-v8\",\"notebookVersion\":\"(version 44/60)\",\"details\":\"From Notebook [Baseline-v8]\",\"publicScore\":\"0.794\",\"privateScore\":\"0.795\",\"submittedBy\":\"chistyakov\",\"dateSubmittedUtc\":\"2020-12-12T10:08:21.697Z\"}

\n

But in the run history version 44 of Baseline-v8 have score 0.792 which is significant lower.
\n\"\n\"

", "rawMarkdown": "@addisonhoward - The report said my 44 version have public score - `0.794`:\n`{\"name\":\"Baseline-v8\",\"notebookVersion\":\"(version 44/60)\",\"details\":\"From Notebook [Baseline-v8]\",\"publicScore\":\"0.794\",\"privateScore\":\"0.795\",\"submittedBy\":\"chistyakov\",\"dateSubmittedUtc\":\"2020-12-12T10:08:21.697Z\"}`\n\nBut in the run history version 44 of Baseline-v8 have score `0.792` which is significant lower.\n![\n](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F453677%2Fdfc1c88bdcb24f8acc0f454f43c69098%2FScreenshot%202021-01-01%20at%2023.12.17.png?generation=1609531967158528&alt=media)\n\n", "replies": [ { "id": 1135119, "postDate": "2021-01-01T22:45:31.330Z", "content": "

I think there is a gap between the version shown in the report and in the notebook. I guess that the version in the report ignores the versions that failed because of some errors. So, in your case, 44th successfully saved version is your best one shown in the report.

", "rawMarkdown": "I think there is a gap between the version shown in the report and in the notebook. I guess that the version in the report ignores the versions that failed because of some errors. So, in your case, 44th successfully saved version is your best one shown in the report. ", "votes": 1 }, { "id": 1135132, "postDate": "2021-01-01T23:18:03.827Z", "content": "

Hey Stanislav, Yu is correct that this is a mismatch between Kernel Version number logic. We've recently added special logic to the My Submissions page to reconcile those mismatches, and unfortunately we neglected to copy that special logic over to this new tool, so that bug has resurfaced. Indeed the mismatch is due to two failed Kernel Versions prior to your best version, and so while the tool reports 44/60, the true Kernel Version number for that submission is 46/62.

\n

Since it's risky to make changes during the holidays and the tool will only be around a few more days, we won't fix the tool, and hope that impacted users can work around the issue using the above guidance. We apologize for the inconvenience here.

", "rawMarkdown": "Hey Stanislav, Yu is correct that this is a mismatch between Kernel Version number logic. We've recently added special logic to the My Submissions page to reconcile those mismatches, and unfortunately we neglected to copy that special logic over to this new tool, so that bug has resurfaced. Indeed the mismatch is due to two failed Kernel Versions prior to your best version, and so while the tool reports 44/60, the true Kernel Version number for that submission is 46/62.\n\nSince it's risky to make changes during the holidays and the tool will only be around a few more days, we won't fix the tool, and hope that impacted users can work around the issue using the above guidance. We apologize for the inconvenience here.", "votes": 3 } ] }, { "id": 1134550, "postDate": "2021-01-01T11:32:16.237Z", "content": "

I think this is a reasonable decision and action to take.
\nThank you, @addisonhoward.

", "rawMarkdown": "I think this is a reasonable decision and action to take.\nThank you, @addisonhoward." }, { "id": 1134161, "postDate": "2020-12-31T22:59:29.613Z", "content": "
\n

\"publicScore\":\"0.771\", \"privateScore\":\"0.779\", \"submittedBy\":\"scaomath\",\"dateSubmittedUtc\":\"2020-12-28T19:57:28.72Z\"

\n
\n

I haven't submit any of fully trained models until yesterday. To be honest it gives people like myself a slight edge because I know which of my prototypes is working best.

", "rawMarkdown": ">\"publicScore\":\"0.771\", \"privateScore\":\"0.779\", \"submittedBy\":\"scaomath\",\"dateSubmittedUtc\":\"2020-12-28T19:57:28.72Z\"\n\nI haven't submit any of fully trained models until yesterday. To be honest it gives people like myself a slight edge because I know which of my prototypes is working best. " } ], "comments": [ { "id": 1147882, "author_name": "tito", "author_url": "", "post_date": "2021-01-10T19:23:40.227000", "content": "

This kind of bug should have been disclosed after the competition is over or bug is fixed.
\nI think the impact of this case would have been much smaller if it had been released after this bug is fixed.

\n

However, as a matter of reality, if the issue is released after the competition, it will not attract much attention, whereas if it is released during the competition period, it will attract attention and be treated as a hero.
\nThis means that in the future as well, this kind of issues will be disclosed during the competition period in the discussion.

\n

So, I propose to give some kind of incentive to those who find this kind of bug and report it to the administration.

\n

As a start, I think it would be a good idea to give this prize to @nyanpn.
\nHe found another way to know private score for this competition, reported it to the administration, and waited until the end of the competition to announce it.

\n

@addisonhoward, @sohier,
\nI would appreciate it if you would consider it.

", "votes": 11, "replies": [ { "id": 1148107, "author_name": "عثمان", "author_url": "", "post_date": "2021-01-10T22:41:43.683000", "content": "

😶 there were multiple concurrent running competitions that this affected though. That said, I agree with you that the accepted white-hat etiquette for 0-day disclosure is first to notify the company/devs in charge and see how they respond to and handle the situation before sharing with the planet =]. IIRC it took Kaggle less than 3 hours to make the patch.

", "votes": 2, "replies": [] }, { "id": 1148118, "author_name": "Shuhao Cao", "author_url": "", "post_date": "2021-01-10T23:01:45.310000", "content": "

This one has been affected most because the correct target is being updated and available to the users, after the prediction is made, which is the bug @its7171 mentioned in his post.

", "votes": 1, "replies": [] }, { "id": 1148255, "author_name": "tito", "author_url": "", "post_date": "2021-01-11T03:00:53.917000", "content": "

Thanks for your comment.

\n

I should have said, I think the impact of this case would have been much smaller if it had been released after this bug is fixed.

\n

Announcement should be made when the bug no longer affects.
\nI think this kind of bugs should be published after the bug has been fixed, and the kind of issues should be published after the competition has ended.

\n

Anyway, I think the best way to deal with this kind of bug is to report it to the administration and follow their instructions before publishing it.

\n

And my point is to give the motivation to do so.

", "votes": 0, "replies": [] }, { "id": 1149093, "author_name": "Addison Howard", "author_url": "", "post_date": "2021-01-11T15:29:55.983000", "content": "

Hey all,

\n

We're thankful the Kaggle community comes to us with bug reports throughout competitions. We almost always reward users for doing so. In the event you find a security issue on Kaggle, you can report it via https://kaggle.com/contact or https://goo.gl/vulnz.

\n

The user did report it via our Support channels, however with the holidays and weekend hours, we had not yet seen the report by the time it was raised in the Kaggle forums.

\n

I hope that provides some additional context!

\n

Addison

", "votes": 2, "replies": [] }, { "id": 1149658, "author_name": "tito", "author_url": "", "post_date": "2021-01-12T03:42:05.723000", "content": "

Thanks for the information!
\nI did not know about the reward.

\n

So you have a system for reporting issues and rewarding it.
\nBut there might be some room for improvement.

\n

I think it might be a good idea to publish this information.

\n\n

This would help to motivate to report issues before disclose it.

\n

Also, I am interested in how you deal with this private score probing issue.
\nI think this is a very big issue because we can know the private score with enough accuracy by binary search.

", "votes": 3, "replies": [] }, { "id": 1149677, "author_name": "mamas", "author_url": "", "post_date": "2021-01-12T04:08:09.743000", "content": "

I agree with tito, private score probing issue will be a very big problem in future competitions that use Time-Series API. @addisonhoward please consider adding some dummy data/target when using Time-Series API to prevent private score probing.

", "votes": 2, "replies": [] }, { "id": 1151866, "author_name": "Andrés Miguel Torrubia Sáez", "author_url": "", "post_date": "2021-01-13T16:12:53.090000", "content": "

Regarding disclosures, I believe there's two types:

\n
    \n

  1. Security vulnerabilities or exploits, which if you actually use/exploit it would be considered bad and would get banned/etc. So you have little to no incentive to not disclose.

    2. \n

  2. Data leaks. As fas as I know there's no actual policy wrt to rewards for data leaks and there's been instances in which participants have disclosed leaks and as a consequence competition restarted (e.g https://www.kaggle.com/c/airbus-ship-detection/discussion/64355 in which we got rewarded a t-shirt). Other people may find a t-shirt is not good enough and would use the privately found leak (the line between leak, magic and correlation is blurry) e.g. https://www.kaggle.com/c/liverpool-ion-switching/discussion/153940 and win.

    3. \n
", "votes": 2, "replies": [] } ] }, { "id": 1135908, "author_name": "Manikanth Reddy", "author_url": "", "post_date": "2021-01-02T15:35:35.200000", "content": "

", "votes": 4, "replies": [ { "id": 1136802, "author_name": "Manikanth Reddy", "author_url": "", "post_date": "2021-01-03T12:57:24.177000", "content": "

I think this could be against the spirit of the competition. I take it back.

", "votes": 4, "replies": [] } ] }, { "id": 1134192, "author_name": "mamas", "author_url": "", "post_date": "2021-01-01T01:12:45.053000", "content": "

Thank you Addison, This is a very simple, and realistic solution to this situation, as LB-CV is very stable and the effect of the bug is not so big at least in this competition! :)

", "votes": 4, "replies": [] }, { "id": 1134013, "author_name": "Nischay Dhankhar", "author_url": "", "post_date": "2020-12-31T18:40:48.720000", "content": "

Thank you for finding an alternative and updating us @addisonhoward. It would be helpful if you also clarify that if teams are allowed to share their private score or mention it anywhere in discussions??

", "votes": 4, "replies": [ { "id": 1134028, "author_name": "Addison Howard", "author_url": "", "post_date": "2020-12-31T18:54:30.470000", "content": "

We don’t explicitly prohibit the sharing of cross-validation scores at your chosen discretion, as long as it still fits in accordance with our rules that restrict private data sharing outside of teams. However, sharing this private score could be viewed by your peers as against the spirit of the competition, not unlike sharing a high-scoring notebook toward the end of the competition.

", "votes": 3, "replies": [] } ] }, { "id": 1134227, "author_name": "Jaideep", "author_url": "", "post_date": "2021-01-01T02:59:08.577000", "content": "

@addisonhoward i think this is very fair solution to reduce the impact of bug however it would been even more fair to the participants who dint see scores by letting all know private score for all sub till that date .
\nAs people who have viewed score while big was on would have seen all model scored they used till that time

", "votes": 2, "replies": [ { "id": 1134260, "author_name": "mamas", "author_url": "", "post_date": "2021-01-01T04:31:50.463000", "content": "

I agree with your idea. However, I also agree with @addisonhoward's idea,

\n
releasing all submission scores would be detrimental from an overfitting standpoint, unfairly benefit teams with large numbers of submissions, and would be out-of-proportion to the information we believe was viewed while the bug was live.\n
\n

It's difficult to say which is better :(

", "votes": 1, "replies": [] }, { "id": 1134266, "author_name": "FGPC", "author_url": "", "post_date": "2021-01-01T04:50:47.783000", "content": "

I suggest to release at least top 5 submissions private leaderboard score because some participants already had accessed on private leaderboard scores before Kaggle investigation on this issue

", "votes": 5, "replies": [] }, { "id": 1135284, "author_name": "Jaideep", "author_url": "", "post_date": "2021-01-02T05:28:44.033000", "content": "

@addisonhoward this is very fair. Top 5 private score till date .
\nPlease consider this. I am currently confused between the model whose private score i seen, and model whose private score i dint see and is 3rd best. But there is atleast person in similar state must have seen all the scores and has by now decided which one to keep and which one to discard. Saving time ,resources

", "votes": 2, "replies": [] } ] }, { "id": 1134104, "author_name": "Yu Tsumura", "author_url": "", "post_date": "2020-12-31T20:37:30.007000", "content": "

@addisonhoward I believe people could see their private rank. Could you also provide the rank as well?

", "votes": 2, "replies": [ { "id": 1134117, "author_name": "Addison Howard", "author_url": "", "post_date": "2020-12-31T21:10:37.900000", "content": "

While it is true that this was visible as part of the bug, private rank is conditional on a number of factors outside of your own performance--including submission selections of other teams--and is not directly relevant to model cross validation. We will not be providing it here.

", "votes": 2, "replies": [] } ] }, { "id": 1134054, "author_name": "Nikola Bacic", "author_url": "", "post_date": "2020-12-31T19:37:21.773000", "content": "

Liked what you saw guys? 😁

", "votes": 2, "replies": [ { "id": 1134061, "author_name": "Kha Vo", "author_url": "", "post_date": "2020-12-31T19:49:57.417000", "content": "

I have cv0805 but public0799 private0800 single tfm model. Must be a bug during inference or I miss something. I thought my private is equivalent to cv. So thanks to this I know there must be something wrong that needs to be fixed.

", "votes": 1, "replies": [] }, { "id": 1134069, "author_name": "عثمان", "author_url": "", "post_date": "2020-12-31T19:58:42.900000", "content": "
\n

\"publicScore\":\"0.763\",
\n \"privateScore\":\"0.762\",

\n
\n

🙃 Perfect correlation. On a serious note, I think the approach the Kaggle took here strikes a good balance.

", "votes": 2, "replies": [] }, { "id": 1134111, "author_name": "Moeen Bagheri", "author_url": "", "post_date": "2020-12-31T20:58:33.643000", "content": "
\"publicScore\":\"0.789\",\n\"privateScore\":\"0.789\"\n
\n

Perfect correlation for me too. I don't think our competition was affected as much as others because of this correlation between the public and private LBs.

", "votes": 2, "replies": [] }, { "id": 1134112, "author_name": "", "author_url": "", "post_date": "2020-12-31T20:59:02.033000", "content": "", "votes": 1, "replies": [] }, { "id": 1134121, "author_name": "", "author_url": "", "post_date": "2020-12-31T21:27:15.847000", "content": "", "votes": 0, "replies": [] }, { "id": 1134194, "author_name": "pin-rui", "author_url": "", "post_date": "2021-01-01T01:24:26.640000", "content": "

\"publicScore\":\"0.786\",\"privateScore\":\"0.789\"
\nShake is big 🤔

", "votes": 1, "replies": [] }, { "id": 1134229, "author_name": "Jaideep", "author_url": "", "post_date": "2021-01-01T03:01:43.210000", "content": "

I think every one is getting .001 to.004 increase in scores ..

", "votes": 1, "replies": [] } ] }, { "id": 1134032, "author_name": "Sugawarya", "author_url": "", "post_date": "2020-12-31T19:03:25.103000", "content": "

I think the fairest approach is to release all private scores for the submissions up to just before the bug fix.

", "votes": 0, "replies": [ { "id": 1134040, "author_name": "Addison Howard", "author_url": "", "post_date": "2020-12-31T19:19:13.563000", "content": "

Our analysis suggests that releasing all submission scores would be detrimental from an overfitting standpoint, unfairly benefit teams with large numbers of submissions, and would be out-of-proportion to the information we believe was viewed while the bug was live.

", "votes": 15, "replies": [] }, { "id": 1134334, "author_name": "FGPC", "author_url": "", "post_date": "2021-01-01T07:03:16.227000", "content": "

at least top 5 submission scores which is equivalent of maximum number of submissions per day.

", "votes": 0, "replies": [] } ] }, { "id": 1135038, "author_name": "Stanislav Chistyakov", "author_url": "", "post_date": "2021-01-01T20:14:05.580000", "content": "

@addisonhoward - The report said my 44 version have public score - 0.794:
\n{\"name\":\"Baseline-v8\",\"notebookVersion\":\"(version 44/60)\",\"details\":\"From Notebook [Baseline-v8]\",\"publicScore\":\"0.794\",\"privateScore\":\"0.795\",\"submittedBy\":\"chistyakov\",\"dateSubmittedUtc\":\"2020-12-12T10:08:21.697Z\"}

\n

But in the run history version 44 of Baseline-v8 have score 0.792 which is significant lower.
\n\"\n\"

", "votes": 0, "replies": [ { "id": 1135119, "author_name": "Yu Tsumura", "author_url": "", "post_date": "2021-01-01T22:45:31.330000", "content": "

I think there is a gap between the version shown in the report and in the notebook. I guess that the version in the report ignores the versions that failed because of some errors. So, in your case, 44th successfully saved version is your best one shown in the report.

", "votes": 1, "replies": [] }, { "id": 1135132, "author_name": "DJ Sterling", "author_url": "", "post_date": "2021-01-01T23:18:03.827000", "content": "

Hey Stanislav, Yu is correct that this is a mismatch between Kernel Version number logic. We've recently added special logic to the My Submissions page to reconcile those mismatches, and unfortunately we neglected to copy that special logic over to this new tool, so that bug has resurfaced. Indeed the mismatch is due to two failed Kernel Versions prior to your best version, and so while the tool reports 44/60, the true Kernel Version number for that submission is 46/62.

\n

Since it's risky to make changes during the holidays and the tool will only be around a few more days, we won't fix the tool, and hope that impacted users can work around the issue using the above guidance. We apologize for the inconvenience here.

", "votes": 3, "replies": [] } ] }, { "id": 1134550, "author_name": "ANZ", "author_url": "", "post_date": "2021-01-01T11:32:16.237000", "content": "

I think this is a reasonable decision and action to take.
\nThank you, @addisonhoward.

", "votes": 0, "replies": [] }, { "id": 1134161, "author_name": "Shuhao Cao", "author_url": "", "post_date": "2020-12-31T22:59:29.613000", "content": "
\n

\"publicScore\":\"0.771\", \"privateScore\":\"0.779\", \"submittedBy\":\"scaomath\",\"dateSubmittedUtc\":\"2020-12-28T19:57:28.72Z\"

\n
\n

I haven't submit any of fully trained models until yesterday. To be honest it gives people like myself a slight edge because I know which of my prototypes is working best.

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1133987": "Hello Kagglers,\n\nAs you may be aware, a recently disclosed vulnerability allowed teams to view their own private leaderboard score prior to the deadline. We have released a statement describing the bug [here](https://www.kaggle.com/private-leaderboard-bug-dec-2020).\n\nSince the vulnerability was publicly disclosed for approximately five hours, a small number of teams accessed their team’s private score. Because this competition is so close to its deadline and we want teams to have a fair footing, Kaggle and the hosts have agreed to make available the identity and score of your team’s best private submission as of the time the bug was fixed. \n\nTo retrieve your team’s score as of 2020-12-29 16:00:48 UTC, please visit [this link](https://www.kaggle.com/c/riiid-test-answer-prediction/best-private-submission-dec-2020).\n\nWe believe this approach is a fair and minimally disruptive way to finish this competition. In coming to this decision, we considered a number of alternatives, including creating a new test set. The disadvantages introduced by a new test set and model reruns are not warranted, given the limited impact resulting from our investigation. Similarly, we do not believe this incident merits a deadline extension. In contrast to other competition interventions, such as the introduction of new data or discovery of critical data flaws, the underlying task and data remains the same, just with an additional cross-validation input to consider.\n\nWe apologize again for the inconvenience this bug caused and thank you for your participation and understanding.\n\nKaggle Team\n", "1147882": "This kind of bug should have been disclosed after the competition is over or bug is fixed.\nI think the impact of this case would have been much smaller if it had been released ~~after the competition~~ after this bug is fixed.\n\nHowever, as a matter of reality, if the issue is released after the competition, it will not attract much attention, whereas if it is released during the competition period, it will attract attention and be treated as a hero.\n**This means that in the future as well, this kind of issues will be disclosed during the competition period in the discussion.**\n\n**So, I propose to give some kind of incentive to those who find this kind of bug and report it to the administration.**\n\nAs a start, I think it would be a good idea to give this prize to @nyanpn.\nHe found [another way to know private score for this competition](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/210354), reported it to the administration, and waited until the end of the competition to announce it.\n\n@addisonhoward, @sohier, \nI would appreciate it if you would consider it.", "1135908": "~~I would be thankful if someone who has used the public 0.783/0.781 notebook(s) could post the private score for the same. It helps to understand the minimum baseline people have to beat to get a medal. ~~", "1134192": "Thank you Addison, This is a very simple, and realistic solution to this situation, as LB-CV is very stable and the effect of the bug is not so big at least in this competition! :)", "1134013": "Thank you for finding an alternative and updating us @addisonhoward. It would be helpful if you also clarify that if teams are allowed to share their private score or mention it anywhere in discussions??", "1134227": "@addisonhoward i think this is very fair solution to reduce the impact of bug however it would been even more fair to the participants who dint see scores by letting all know private score for all sub till that date .\nAs people who have viewed score while big was on would have seen all model scored they used till that time ", "1134104": "@addisonhoward I believe people could see [their private rank] (https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/207390). Could you also provide the rank as well?", "1134054": "Liked what you saw guys? 😁", "1134032": "I think the fairest approach is to release all private scores for the submissions up to just before the bug fix.", "1135038": "@addisonhoward - The report said my 44 version have public score - `0.794`:\n`{\"name\":\"Baseline-v8\",\"notebookVersion\":\"(version 44/60)\",\"details\":\"From Notebook [Baseline-v8]\",\"publicScore\":\"0.794\",\"privateScore\":\"0.795\",\"submittedBy\":\"chistyakov\",\"dateSubmittedUtc\":\"2020-12-12T10:08:21.697Z\"}`\n\nBut in the run history version 44 of Baseline-v8 have score `0.792` which is significant lower.\n![\n](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F453677%2Fdfc1c88bdcb24f8acc0f454f43c69098%2FScreenshot%202021-01-01%20at%2023.12.17.png?generation=1609531967158528&alt=media)\n\n", "1134550": "I think this is a reasonable decision and action to take.\nThank you, @addisonhoward.", "1134161": ">\"publicScore\":\"0.771\", \"privateScore\":\"0.779\", \"submittedBy\":\"scaomath\",\"dateSubmittedUtc\":\"2020-12-28T19:57:28.72Z\"\n\nI haven't submit any of fully trained models until yesterday. To be honest it gives people like myself a slight edge because I know which of my prototypes is working best. " } }