{ "id": 275173, "title": "Such a high scoring kernel with 3 days left : Potential Cheating Strategy", "url": "/competitions/landmark-retrieval-2021/discussion/275173", "author_name": "", "post_date": "2021-09-29T06:38:18.637643800Z", "votes": 17, "comment_count": 16, "views": 0, "content": "

I am frustrated with this high scoring kernel. I feel like all my efforts to get a decent score are vain. That's really a joke 😨.

\n

About cheating (update): I was about to submit a new kernel in this competition and i noticed a high scoring public kernel with LB=0.304 (Hopefully i improved my models). At that time my best LB was 0.303. But what was strange is that :

\n\n

So @addisonhoward @juliaelliott can you check it ?

", "messages": [ { "id": "1527850", "postDate": "09/29/2021 06:38:18", "content": "

I am frustrated with this high scoring kernel. I feel like all my efforts to get a decent score are vain. That's really a joke 😨.

\n

About cheating (update): I was about to submit a new kernel in this competition and i noticed a high scoring public kernel with LB=0.304 (Hopefully i improved my models). At that time my best LB was 0.303. But what was strange is that :

\n\n

So @addisonhoward @juliaelliott can you check it ?

", "rawMarkdown": "I am frustrated with this high scoring [kernel](https://www.kaggle.com/raproderick/glret21-efficientnetb0-baseline-inference-v2/notebook). I feel like all my efforts to get a decent score are vain. That's really a joke 😨. \n\nAbout cheating (**update**): I was about to submit a new kernel in this competition and i noticed a high scoring public kernel with LB=0.304 (Hopefully i improved my models). At that time my best LB was 0.303. But what was strange is that :\n* (1) this notebook was based on this outstanding @hidehisaarai1213 [kernel](https://www.kaggle.com/hidehisaarai1213/glret21-efficientnetb0-baseline-inference) using EfficientnetB7 as backbone; \n* (2) it was submitted by a member of the team **landmarker** with one team mate having a deleted account, \n* (3) the weights of the model were a public dataset of another team (unfortunately i didn't save the name account of the dataset host). One thing is sure this team (dataset host) were ranked between 170 and 180, once i looked after they were no more in this ranking zone.\n* (4) Once i reported it in this discussion: the kernel was removed and the dataset afterwards. But i wisely downloaded the dataset and made it public [here](https://www.kaggle.com/ulrich07/glret21-effb7) to keep an evidence and allow anybody to score at least 0.304.\n* (5) the first ones to fork that kernel had previously not submitted in this competition: look at @hiramatsuryosuke and @tomoyaonishi. Notice that both of them joined 6 months ago at the same time\n\nSo @addisonhoward @juliaelliott can you check it ?", "votes": null }, { "id": "1527889", "postDate": "09/29/2021 07:13:04", "content": "

It has been taken down now

", "rawMarkdown": "It has been taken down now", "votes": null }, { "id": "1527902", "postDate": "09/29/2021 07:23:48", "content": "

Thanks for you quick response. I am relieved

", "rawMarkdown": "Thanks for you quick response. I am relieved", "votes": null }, { "id": "1527983", "postDate": "09/29/2021 08:50:02", "content": "

i wanna know if i use this kernel for the LB, but now it is taken down, will it break the competition rule?

", "rawMarkdown": "i wanna know if i use this kernel for the LB, but now it is taken down, will it break the competition rule?", "votes": null }, { "id": "1527996", "postDate": "09/29/2021 09:11:53", "content": "

Now everybody is getting 0.304, some forked it before its removal. Sorry for people aroudn 0.30x and down

", "rawMarkdown": "Now everybody is getting 0.304, some forked it before its removal. Sorry for people aroudn 0.30x and down", "votes": null }, { "id": "1528080", "postDate": "09/29/2021 10:39:09", "content": "

I wonder if this goes under private sharing, a notebook that only a few have seen? My best was 0.304 before this last-minute publishing notebook, too bad for me that someone couldn’t stay to the rules, but hope I can use the last 3 days to increase the score again.

", "rawMarkdown": "I wonder if this goes under private sharing, a notebook that only a few have seen? My best was 0.304 before this last-minute publishing notebook, too bad for me that someone couldn’t stay to the rules, but hope I can use the last 3 days to increase the score again.", "votes": null }, { "id": "1528103", "postDate": "09/29/2021 11:03:25", "content": "

Quite annoying that this is still possible after such a long time with the problem.
\nI remember a suggestion that I wrote a long time ago.

\n

https://www.kaggle.com/product-feedback/190541

\n
\n

Problem – 1
\n Posting high score kernels within the last week of a competition.
\n Already a rule? Yes
\n Solution – 1
\n Close the window for posting new kernels the final week or even better the final two weeks of the competitions. One has had many weeks and month digging into the public codes. And it’s a 100 % effective solution.

\n
\n

Hope that there is a solution in the near future for this problem with last-minute posting of top notebooks.

\n

@addisonhoward

", "rawMarkdown": "Quite annoying that this is still possible after such a long time with the problem.\nI remember a suggestion that I wrote a long time ago.\n\nhttps://www.kaggle.com/product-feedback/190541\n\n> Problem – 1\nPosting high score kernels within the last week of a competition.\nAlready a rule? Yes\nSolution – 1\nClose the window for posting new kernels the final week or even better the final two weeks of the competitions. One has had many weeks and month digging into the public codes. And it’s a 100 % effective solution.\n\nHope that there is a solution in the near future for this problem with last-minute posting of top notebooks.\n\n\n\n @addisonhoward", "votes": null }, { "id": "1528125", "postDate": "09/29/2021 11:45:21", "content": "

Yup, this should be implemented

", "rawMarkdown": "Yup, this should be implemented", "votes": null }, { "id": "1528298", "postDate": "09/29/2021 14:41:34", "content": "

I agree to your points…making kernels public in last one week of competitions should be blocked…that can solve the problem to a large extent.

", "rawMarkdown": "I agree to your points...making kernels public in last one week of competitions should be blocked...that can solve the problem to a large extent.", "votes": null }, { "id": "1528301", "postDate": "09/29/2021 14:44:32", "content": "

That explains why a few entries are there with just one sub , score of 0.304 all submitted in last few hours…this seems to be happening in every other competition for the past few years.

", "rawMarkdown": "That explains why a few entries are there with just one sub , score of 0.304 all submitted in last few hours...this seems to be happening in every other competition for the past few years.", "votes": null }, { "id": "1528316", "postDate": "09/29/2021 14:50:12", "content": "

Probably not, since you forked it from a public kernel (at that point of time) and that would not qualify as private sharing. I maybe wrong here…best answered by the organizers. @maggiemd would it be possible for you to confirm this?

", "rawMarkdown": "Probably not, since you forked it from a public kernel (at that point of time) and that would not qualify as private sharing. I maybe wrong here...best answered by the organizers. @maggiemd would it be possible for you to confirm this?", "votes": null }, { "id": "1528328", "postDate": "09/29/2021 14:58:22", "content": "

Yes its a 3 second implementation without additional cost, resulting in a happy Kaggle community that can lay these complaints minutes to modeling instead and with knowledge that this cant happen again, win-win. At least do it for some month and a few competitions, review the result, easy to undo, but I doubt it will have any negative effect more the opposite taking account the past years discussions.

", "rawMarkdown": "Yes its a 3 second implementation without additional cost, resulting in a happy Kaggle community that can lay these complaints minutes to modeling instead and with knowledge that this cant happen again, win-win. At least do it for some month and a few competitions, review the result, easy to undo, but I doubt it will have any negative effect more the opposite taking account the past years discussions.", "votes": null }, { "id": "1528391", "postDate": "09/29/2021 15:48:30", "content": "

If that kernel is not open to everyone, it is a violation of Kaggle rules even if it was revealed once and deleted.

", "rawMarkdown": "If that kernel is not open to everyone, it is a violation of Kaggle rules even if it was revealed once and deleted.", "votes": null }, { "id": "1528981", "postDate": "09/30/2021 04:20:29", "content": "

I can see your frustration, it must be difficult when you put in a lot of work to get a high scoring model then someone above you shares there code with days to go and suddenly people that did no work at come in above you and you drop down the list at the last moment.

\n

I guess that in some ways part of kaggle is about finding solutions to problems that might benefit everyone for example detecting cancers earlier, so working together and sharing code so that together we come up with the best solutions to finding solutions for health, climate change, science problems etc. But part of kaggle is the idea of competition which needs to be fair and reward the right people. So its difficult.

\n

I think its already been suggested but introducing a rule that code should not be shared in the final 7-14 days of a competition might help with the problem by encouraging sharing ideas in the early part of competition but then limiting it for the final stages. Once the competition is closed, code could then be shared again and if for example further submissions were allowed for another few weeks aften the closing date it would allow people that were interest in building on the top scoring public submissions could do so. Allowing better solutions that are in the public interest to be made without spoiling the competition.

\n

I really hope that this does put you off further competions. You should at the very least be proud of your score in the knowledge that it was your own work not simply a fork of someone elses.

", "rawMarkdown": "I can see your frustration, it must be difficult when you put in a lot of work to get a high scoring model then someone above you shares there code with days to go and suddenly people that did no work at come in above you and you drop down the list at the last moment.\n\nI guess that in some ways part of kaggle is about finding solutions to problems that might benefit everyone for example detecting cancers earlier, so working together and sharing code so that together we come up with the best solutions to finding solutions for health, climate change, science problems etc. But part of kaggle is the idea of competition which needs to be fair and reward the right people. So its difficult.\n\nI think its already been suggested but introducing a rule that code should not be shared in the final 7-14 days of a competition might help with the problem by encouraging sharing ideas in the early part of competition but then limiting it for the final stages. Once the competition is closed, code could then be shared again and if for example further submissions were allowed for another few weeks aften the closing date it would allow people that were interest in building on the top scoring public submissions could do so. Allowing better solutions that are in the public interest to be made without spoiling the competition.\n\nI really hope that this does put you off further competions. You should at the very least be proud of your score in the knowledge that it was your own work not simply a fork of someone elses.", "votes": null }, { "id": "1530043", "postDate": "09/30/2021 22:01:04", "content": "

@ulrich07 it happened in other comps, due to a bug some high score nbs were made public https://www.kaggle.com/c/g2net-gravitational-wave-detection/discussion/275316

\n

that kernel was made public and then private again? The user does not exist now lol

\n

how do we know that it was not done intentionally to share the solution with another team real quick and then make it private again? I have to say that I think it is not the case here, I don't know the nb nor the team, but just saying it could happen in other circumstances (ie new kind of private sharing using public notebooks?).

\n

I would not expect a solution from kaggle, feels bad 😄

", "rawMarkdown": "ulrich07 it happened in other comps, due to a bug some high score nbs were made public https://www.kaggle.com/c/g2net-gravitational-wave-detection/discussion/275316\n\nthat kernel was made public and then private again? The user does not exist now lol\n\nhow do we know that it was not done intentionally to share the solution with another team real quick and then make it private again? I have to say that I think it is not the case here, I don't know the nb nor the team, but just saying it could happen in other circumstances (ie new kind of private sharing using public notebooks?).\n\nI would not expect a solution from kaggle, feels bad 😄", "votes": null }, { "id": "1530050", "postDate": "09/30/2021 22:10:17", "content": "

The bug occurred at a different time I believe. I saw some notebooks by a top team public but all the weights were private datasets so it wasn't a problem. They became private after some time.
\nBut this notebook was published way before (maybe they were new and weren't aware of the rule)

", "rawMarkdown": "The bug occurred at a different time I believe. I saw some notebooks by a top team public but all the weights were private datasets so it wasn't a problem. They became private after some time.\nBut this notebook was published way before (maybe they were new and weren't aware of the rule)", "votes": null }, { "id": "1530461", "postDate": "10/01/2021 07:08:35", "content": "

I think so @jesucristo

", "rawMarkdown": "I think so @jesucristo", "votes": null } ], "comments": [ { "id": 1527889, "author_name": "debarshichanda", "author_url": "", "post_date": "09/29/2021 07:13:04", "content": "

It has been taken down now

", "votes": null, "replies": [ { "id": 1527902, "author_name": "ulrich07", "author_url": "", "post_date": "09/29/2021 07:23:48", "content": "

Thanks for you quick response. I am relieved

", "votes": null, "replies": [] } ] }, { "id": 1527983, "author_name": "guohey", "author_url": "", "post_date": "09/29/2021 08:50:02", "content": "

i wanna know if i use this kernel for the LB, but now it is taken down, will it break the competition rule?

", "votes": null, "replies": [ { "id": 1527996, "author_name": "louise2001", "author_url": "", "post_date": "09/29/2021 09:11:53", "content": "

Now everybody is getting 0.304, some forked it before its removal. Sorry for people aroudn 0.30x and down

", "votes": null, "replies": [] }, { "id": 1528316, "author_name": "sandy1112", "author_url": "", "post_date": "09/29/2021 14:50:12", "content": "

Probably not, since you forked it from a public kernel (at that point of time) and that would not qualify as private sharing. I maybe wrong here…best answered by the organizers. @maggiemd would it be possible for you to confirm this?

", "votes": null, "replies": [] } ] }, { "id": 1528080, "author_name": "kirderf", "author_url": "", "post_date": "09/29/2021 10:39:09", "content": "

I wonder if this goes under private sharing, a notebook that only a few have seen? My best was 0.304 before this last-minute publishing notebook, too bad for me that someone couldn’t stay to the rules, but hope I can use the last 3 days to increase the score again.

", "votes": null, "replies": [ { "id": 1528103, "author_name": "kirderf", "author_url": "", "post_date": "09/29/2021 11:03:25", "content": "

Quite annoying that this is still possible after such a long time with the problem.
\nI remember a suggestion that I wrote a long time ago.

\n

https://www.kaggle.com/product-feedback/190541

\n
\n

Problem – 1
\n Posting high score kernels within the last week of a competition.
\n Already a rule? Yes
\n Solution – 1
\n Close the window for posting new kernels the final week or even better the final two weeks of the competitions. One has had many weeks and month digging into the public codes. And it’s a 100 % effective solution.

\n
\n

Hope that there is a solution in the near future for this problem with last-minute posting of top notebooks.

\n

@addisonhoward

", "votes": null, "replies": [] }, { "id": 1528125, "author_name": "debarshichanda", "author_url": "", "post_date": "09/29/2021 11:45:21", "content": "

Yup, this should be implemented

", "votes": null, "replies": [] }, { "id": 1528298, "author_name": "sandy1112", "author_url": "", "post_date": "09/29/2021 14:41:34", "content": "

I agree to your points…making kernels public in last one week of competitions should be blocked…that can solve the problem to a large extent.

", "votes": null, "replies": [] }, { "id": 1528328, "author_name": "kirderf", "author_url": "", "post_date": "09/29/2021 14:58:22", "content": "

Yes its a 3 second implementation without additional cost, resulting in a happy Kaggle community that can lay these complaints minutes to modeling instead and with knowledge that this cant happen again, win-win. At least do it for some month and a few competitions, review the result, easy to undo, but I doubt it will have any negative effect more the opposite taking account the past years discussions.

", "votes": null, "replies": [] } ] }, { "id": 1528301, "author_name": "sandy1112", "author_url": "", "post_date": "09/29/2021 14:44:32", "content": "

That explains why a few entries are there with just one sub , score of 0.304 all submitted in last few hours…this seems to be happening in every other competition for the past few years.

", "votes": null, "replies": [] }, { "id": 1528391, "author_name": "deepkim", "author_url": "", "post_date": "09/29/2021 15:48:30", "content": "

If that kernel is not open to everyone, it is a violation of Kaggle rules even if it was revealed once and deleted.

", "votes": null, "replies": [] }, { "id": 1528981, "author_name": "davidcoxon", "author_url": "", "post_date": "09/30/2021 04:20:29", "content": "

I can see your frustration, it must be difficult when you put in a lot of work to get a high scoring model then someone above you shares there code with days to go and suddenly people that did no work at come in above you and you drop down the list at the last moment.

\n

I guess that in some ways part of kaggle is about finding solutions to problems that might benefit everyone for example detecting cancers earlier, so working together and sharing code so that together we come up with the best solutions to finding solutions for health, climate change, science problems etc. But part of kaggle is the idea of competition which needs to be fair and reward the right people. So its difficult.

\n

I think its already been suggested but introducing a rule that code should not be shared in the final 7-14 days of a competition might help with the problem by encouraging sharing ideas in the early part of competition but then limiting it for the final stages. Once the competition is closed, code could then be shared again and if for example further submissions were allowed for another few weeks aften the closing date it would allow people that were interest in building on the top scoring public submissions could do so. Allowing better solutions that are in the public interest to be made without spoiling the competition.

\n

I really hope that this does put you off further competions. You should at the very least be proud of your score in the knowledge that it was your own work not simply a fork of someone elses.

", "votes": null, "replies": [] }, { "id": 1530043, "author_name": "jesucristo", "author_url": "", "post_date": "09/30/2021 22:01:04", "content": "

@ulrich07 it happened in other comps, due to a bug some high score nbs were made public https://www.kaggle.com/c/g2net-gravitational-wave-detection/discussion/275316

\n

that kernel was made public and then private again? The user does not exist now lol

\n

how do we know that it was not done intentionally to share the solution with another team real quick and then make it private again? I have to say that I think it is not the case here, I don't know the nb nor the team, but just saying it could happen in other circumstances (ie new kind of private sharing using public notebooks?).

\n

I would not expect a solution from kaggle, feels bad 😄

", "votes": null, "replies": [ { "id": 1530050, "author_name": "debarshichanda", "author_url": "", "post_date": "09/30/2021 22:10:17", "content": "

The bug occurred at a different time I believe. I saw some notebooks by a top team public but all the weights were private datasets so it wasn't a problem. They became private after some time.
\nBut this notebook was published way before (maybe they were new and weren't aware of the rule)

", "votes": null, "replies": [] }, { "id": 1530461, "author_name": "ulrich07", "author_url": "", "post_date": "10/01/2021 07:08:35", "content": "

I think so @jesucristo

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1527850": "I am frustrated with this high scoring [kernel](https://www.kaggle.com/raproderick/glret21-efficientnetb0-baseline-inference-v2/notebook). I feel like all my efforts to get a decent score are vain. That's really a joke 😨. \n\nAbout cheating (**update**): I was about to submit a new kernel in this competition and i noticed a high scoring public kernel with LB=0.304 (Hopefully i improved my models). At that time my best LB was 0.303. But what was strange is that :\n* (1) this notebook was based on this outstanding @hidehisaarai1213 [kernel](https://www.kaggle.com/hidehisaarai1213/glret21-efficientnetb0-baseline-inference) using EfficientnetB7 as backbone; \n* (2) it was submitted by a member of the team **landmarker** with one team mate having a deleted account, \n* (3) the weights of the model were a public dataset of another team (unfortunately i didn't save the name account of the dataset host). One thing is sure this team (dataset host) were ranked between 170 and 180, once i looked after they were no more in this ranking zone.\n* (4) Once i reported it in this discussion: the kernel was removed and the dataset afterwards. But i wisely downloaded the dataset and made it public [here](https://www.kaggle.com/ulrich07/glret21-effb7) to keep an evidence and allow anybody to score at least 0.304.\n* (5) the first ones to fork that kernel had previously not submitted in this competition: look at @hiramatsuryosuke and @tomoyaonishi. Notice that both of them joined 6 months ago at the same time\n\nSo @addisonhoward @juliaelliott can you check it ?", "1527889": "It has been taken down now", "1527902": "Thanks for you quick response. I am relieved", "1527983": "i wanna know if i use this kernel for the LB, but now it is taken down, will it break the competition rule?", "1527996": "Now everybody is getting 0.304, some forked it before its removal. Sorry for people aroudn 0.30x and down", "1528080": "I wonder if this goes under private sharing, a notebook that only a few have seen? My best was 0.304 before this last-minute publishing notebook, too bad for me that someone couldn’t stay to the rules, but hope I can use the last 3 days to increase the score again.", "1528103": "Quite annoying that this is still possible after such a long time with the problem.\nI remember a suggestion that I wrote a long time ago.\n\nhttps://www.kaggle.com/product-feedback/190541\n\n> Problem – 1\nPosting high score kernels within the last week of a competition.\nAlready a rule? Yes\nSolution – 1\nClose the window for posting new kernels the final week or even better the final two weeks of the competitions. One has had many weeks and month digging into the public codes. And it’s a 100 % effective solution.\n\nHope that there is a solution in the near future for this problem with last-minute posting of top notebooks.\n\n\n\n @addisonhoward", "1528125": "Yup, this should be implemented", "1528298": "I agree to your points...making kernels public in last one week of competitions should be blocked...that can solve the problem to a large extent.", "1528301": "That explains why a few entries are there with just one sub , score of 0.304 all submitted in last few hours...this seems to be happening in every other competition for the past few years.", "1528316": "Probably not, since you forked it from a public kernel (at that point of time) and that would not qualify as private sharing. I maybe wrong here...best answered by the organizers. @maggiemd would it be possible for you to confirm this?", "1528328": "Yes its a 3 second implementation without additional cost, resulting in a happy Kaggle community that can lay these complaints minutes to modeling instead and with knowledge that this cant happen again, win-win. At least do it for some month and a few competitions, review the result, easy to undo, but I doubt it will have any negative effect more the opposite taking account the past years discussions.", "1528391": "If that kernel is not open to everyone, it is a violation of Kaggle rules even if it was revealed once and deleted.", "1528981": "I can see your frustration, it must be difficult when you put in a lot of work to get a high scoring model then someone above you shares there code with days to go and suddenly people that did no work at come in above you and you drop down the list at the last moment.\n\nI guess that in some ways part of kaggle is about finding solutions to problems that might benefit everyone for example detecting cancers earlier, so working together and sharing code so that together we come up with the best solutions to finding solutions for health, climate change, science problems etc. But part of kaggle is the idea of competition which needs to be fair and reward the right people. So its difficult.\n\nI think its already been suggested but introducing a rule that code should not be shared in the final 7-14 days of a competition might help with the problem by encouraging sharing ideas in the early part of competition but then limiting it for the final stages. Once the competition is closed, code could then be shared again and if for example further submissions were allowed for another few weeks aften the closing date it would allow people that were interest in building on the top scoring public submissions could do so. Allowing better solutions that are in the public interest to be made without spoiling the competition.\n\nI really hope that this does put you off further competions. You should at the very least be proud of your score in the knowledge that it was your own work not simply a fork of someone elses.", "1530043": "ulrich07 it happened in other comps, due to a bug some high score nbs were made public https://www.kaggle.com/c/g2net-gravitational-wave-detection/discussion/275316\n\nthat kernel was made public and then private again? The user does not exist now lol\n\nhow do we know that it was not done intentionally to share the solution with another team real quick and then make it private again? I have to say that I think it is not the case here, I don't know the nb nor the team, but just saying it could happen in other circumstances (ie new kind of private sharing using public notebooks?).\n\nI would not expect a solution from kaggle, feels bad 😄", "1530050": "The bug occurred at a different time I believe. I saw some notebooks by a top team public but all the weights were private datasets so it wasn't a problem. They became private after some time.\nBut this notebook was published way before (maybe they were new and weren't aware of the rule)", "1530461": "I think so @jesucristo" }, "source": "meta" }