\nPlease include how you started to approach this tough challenge!
\nIt would be the most valuable infornation for all kagglers.
\n
![\"\"](\"https://i.ibb.co/MDHvv4C/image.png\")
First of all I have to apologize to suspect that you have multiple accounts and would be disqualified, but now leaderboard was fixed and that means all participants (at least) in gold zone are certificated by kaggle host.
\nWhen I saw you jumped in 16th place with score 1.18 just 7 hours after creating kaggle account, I felt it's inpossible with considering the dataset size and the data characteristics.
\nFor me 7 hours was the time to just download, unzip and resize the data:(
Again, congrats for the incredible result, I can't wait to see your solution!
\nPlease include how you started to approach this tough challenge!
\nIt would be the most valuable infornation for all kagglers.
\n
First of all I have to apologize to suspect that you have multiple accounts and would be disqualified, but now leaderboard was fixed and that means all participants (at least) in gold zone are certificated by kaggle host.
\nWhen I saw you jumped in 16th place with score 1.18 just 7 hours after creating kaggle account, I felt it's inpossible with considering the dataset size and the data characteristics.
\nFor me 7 hours was the time to just download, unzip and resize the data:(
Again, congrats for the incredible result, I can't wait to see your solution!
\nPlease include how you started to approach this tough challenge!
\nIt would be the most valuable infornation for all kagglers.
\n
Highly suspicious. For four reasons:
\nKaggle must have overlooked the feature time elapsed since account creation / rank of the submission in their cheater detection algorithm…
In HubMap there was a lot of suspicious accounts reaching top 20 in less than 5 hours, that were not removed as well.
\nI feel like Kaggle is overlooking the verification in some competitions, that's sad because it really hurts the image of the platform.
", "rawMarkdown": "Kaggle must have overlooked the feature `time elapsed since account creation / rank of the submission` in their cheater detection algorithm...\n\nIn HubMap there was a lot of suspicious accounts reaching top 20 in less than 5 hours, that were not removed as well.\n\nI feel like Kaggle is overlooking the verification in some competitions, that's sad because it really hurts the image of the platform.", "votes": 20 }, { "id": 1335431, "postDate": "2021-06-04T07:54:20.233Z", "content": "To be generous to Kaggle, a FP in their anti cheating algorithm is arguably worse than a FN.
", "rawMarkdown": "To be generous to Kaggle, a FP in their anti cheating algorithm is arguably worse than a FN.\n\n", "votes": 15 }, { "id": 1335677, "postDate": "2021-06-04T11:12:33.310Z", "content": "We may need a competition to detect cheating by using kaggle meta data🤣
", "rawMarkdown": "We may need a competition to detect cheating by using kaggle meta data🤣", "votes": 7 }, { "id": 1335680, "postDate": "2021-06-04T11:18:56.450Z", "content": "I think a FN is worse than a FP. You can get all suspicious accounts and ask them to justify themselves. Then you can remove the ones who don't make sense. 2 Stage procedure sounds the best to me.
", "rawMarkdown": "I think a FN is worse than a FP. You can get all suspicious accounts and ask them to justify themselves. Then you can remove the ones who don't make sense. 2 Stage procedure sounds the best to me.", "votes": 3 }, { "id": 1335716, "postDate": "2021-06-04T11:50:58.497Z", "content": "\"Suspicious\" is a hard thing to pin down…
\nThe most practical solution would be for Kagglers to flag potential suspicious behavior the way this thread is doing.
\nI'm curious how you'd codify the current accusation into a heuristic - an account being setup and making a high LB sub almost immediately isn't necessarily suspicious. And if implemented, what would be considered a good \"justification\"?
\nAnti-cheating algos aren't trivial.
", "rawMarkdown": "\"Suspicious\" is a hard thing to pin down...\n\nThe most practical solution would be for Kagglers to flag potential suspicious behavior the way this thread is doing. \n\nI'm curious how you'd codify the current accusation into a heuristic - an account being setup and making a high LB sub almost immediately isn't necessarily suspicious. And if implemented, what would be considered a good \"justification\"?\n\nAnti-cheating algos aren't trivial. ", "votes": 3 }, { "id": 1335782, "postDate": "2021-06-04T12:39:52.950Z", "content": "Think there's a difference between a general anti-cheating algo (applied to all entries) versus a more detailed examination of a prize-winning competitor. This particular situation cannot be codified in a heuristic, and shouldn't be.
\nIn this case Kaggle needs to ask for the code to reproduce the first submission. Kaggle would have the data on when the competition dataset download completed. Therefore they must be able to reproduce the training + inference in order to reproduce the submission given the GPU resources that were used. If this can't be done then it's not possible that the situation is legitimate.
", "rawMarkdown": "Think there's a difference between a general anti-cheating algo (applied to all entries) versus a more detailed examination of a prize-winning competitor. This particular situation cannot be codified in a heuristic, and shouldn't be. \n\nIn this case Kaggle needs to ask for the code to reproduce the first submission. Kaggle would have the data on when the competition dataset download completed. Therefore they must be able to reproduce the training + inference in order to reproduce the submission given the GPU resources that were used. If this can't be done then it's not possible that the situation is legitimate.", "votes": 3 }, { "id": 1335829, "postDate": "2021-06-04T13:11:26.913Z", "content": "I hope a just solution is found especially given your potential standing.
\nI believe I had a gold snatched from me by private sharing in one of my first competitions so I understand the frustration.
", "rawMarkdown": "I hope a just solution is found especially given your potential standing. \n\nI believe I had a gold snatched from me by private sharing in one of my first competitions so I understand the frustration. ", "votes": 4 } ] }, { "id": 1335221, "postDate": "2021-06-04T05:06:29.510Z", "content": "Looking forward to see the solution +1 :)
", "rawMarkdown": "Looking forward to see the solution +1 :)", "votes": 10, "replies": [ { "id": 1335678, "postDate": "2021-06-04T11:13:40.397Z", "content": "I'm looking forward to see your solution too, we're scared!!!
", "rawMarkdown": "I'm looking forward to see your solution too, we're scared!!!", "votes": 1 } ] }, { "id": 1335715, "postDate": "2021-06-04T11:50:40.877Z", "content": "By the way, we've already reported this suspect to kaggle compliance team during competition.
\nThey answered that they would conduct vigorous sanity check after competition end and promissed to take this into account.
\nSo, I believe the check was completed and he passed. I'm just curious how he did it.
Provided it only took a few hours to finalize the results instead of the couple of days we're used to, they probably did not really look into the case thoroughly.
\nHowever they will have to, since additional checks are conducted for teams in the prize zone.
", "rawMarkdown": "Provided it only took a few hours to finalize the results instead of the couple of days we're used to, they probably did not really look into the case thoroughly. \n\nHowever they will have to, since additional checks are conducted for teams in the prize zone.", "votes": 4 }, { "id": 1336444, "postDate": "2021-06-04T23:04:53.237Z", "content": "@theoviel The reason the competition was verified quickly - I'm currently out on holiday. I did most of the compliance checks before I left, and then a quick review to ensure there was nothing new, and then I processed them right away at the competition close. This particular case was not on my radar, but, to your point, will get additional scrutiny as a winning team.
", "rawMarkdown": "@theoviel The reason the competition was verified quickly - I'm currently out on holiday. I did most of the compliance checks before I left, and then a quick review to ensure there was nothing new, and then I processed them right away at the competition close. This particular case was not on my radar, but, to your point, will get additional scrutiny as a winning team.", "votes": 6 }, { "id": 1336488, "postDate": "2021-06-05T01:04:20.997Z", "content": "@inversion Thanks for the comment, it's good to know to be acknowledged by the host.
\nBut to be honest your comment made me uneasy…
\n\nThis particular case was not on my radar
\n
We've already reported to kaggle compliance team and their response was below:
\n\n\nHi Kazuki,
\n
\n Thanks for reaching out. We perform our cheating removals at the end of the competition, and will certainly take this into consideration. Thanks for bringing it to our attention.
\n Kaggle Compliance
But you (Judging from your comment, the person who conduct compliance check for this competition) didn't know that. Is this system correctly working? If there could be a communication issue, we might need to raise such kind of concern in competition discussion, which seems not desirable.
\n\n\nbut, to your point, will get additional scrutiny as a winning team.
\n
I'm thinking the same, but in my understanding winner's call is not a place to do such a primitive sanity check. What if the suspects were in 4th place and I haven't raise this issue? It would be definitely missed.
\nI don't judge for this particular case, he can be valid and I have no chance to investigate by myself. Kaggle team only can know.
\nAnd also I didn't want to expose the doubt in public discussion, that's why we kept quiet and just submit to compliance team. But with considering so quick leaderboard fix, we felt it might not be checked vigorously, so we decided to publish.
As everyone says, credibility of winners is so important for all the past competition winners and platform itself. Hope fair result.
", "rawMarkdown": "@inversion Thanks for the comment, it's good to know to be acknowledged by the host.\nBut to be honest your comment made me uneasy... \n> This particular case was not on my radar\n\nWe've already reported to kaggle compliance team and their response was below:\n> Hi Kazuki,\nThanks for reaching out. We perform our cheating removals at the end of the competition, and will certainly take this into consideration. Thanks for bringing it to our attention.\nKaggle Compliance\n\nBut you (Judging from your comment, the person who conduct compliance check for this competition) didn't know that. Is this system correctly working? If there could be a communication issue, we might need to raise such kind of concern in competition discussion, which seems not desirable.\n\n> but, to your point, will get additional scrutiny as a winning team.\n\nI'm thinking the same, but in my understanding winner's call is not a place to do such a primitive sanity check. What if the suspects were in 4th place and I haven't raise this issue? It would be definitely missed.\n\nI don't judge for this particular case, he can be valid and I have no chance to investigate by myself. Kaggle team only can know.\nAnd also I didn't want to expose the doubt in public discussion, that's why we kept quiet and just submit to compliance team. But with considering so quick leaderboard fix, we felt it might not be checked vigorously, so we decided to publish.\n\nAs everyone says, credibility of winners is so important for all the past competition winners and platform itself. Hope fair result. ", "votes": 6 }, { "id": 1336498, "postDate": "2021-06-05T01:46:24.160Z", "content": "@bamps53 I will definitely look into why this did not get passed on (they generally are).
", "rawMarkdown": "@bamps53 I will definitely look into why this did not get passed on (they generally are). ", "votes": 13 }, { "id": 1336553, "postDate": "2021-06-05T03:20:36.113Z", "content": "Thanks, please post the result here so that everyone can be sure that there is nothing wrong in compliance system.
", "rawMarkdown": "Thanks, please post the result here so that everyone can be sure that there is nothing wrong in compliance system.", "votes": 4 } ] }, { "id": 1335443, "postDate": "2021-06-04T08:17:03.897Z", "content": "There must be some superpower I have not seen ever! Maybe time magic?
", "rawMarkdown": "There must be some superpower I have not seen ever! Maybe time magic?", "votes": 8, "replies": [ { "id": 1335508, "postDate": "2021-06-04T09:06:56.793Z", "content": "@steamedsheep They might have put more faith in Santa Claus than we did…
", "rawMarkdown": "@steamedsheep They might have put more faith in Santa Claus than we did...", "votes": 1 }, { "id": 1336389, "postDate": "2021-06-04T21:07:22.670Z", "content": "![\"image\"](\"https://user-images.githubusercontent.com/4602302/120865509-d7ba2900-c596-11eb-9767-56d1e285f008.png\")
\nJust find a good planet for training
Something really fishy indeed with the tight gap between subscription and the first submission
\nI second @haqishen and kaggle should check carefully prize winning solutions.
Hey all,
\nEDIT for TL;DR: We investigated and found no violation of the rules. The new account was a continuation of a previous deleted account. There were no simultaneous submissions
\nWe've investigated this on our end and will not be proceeding with any further action against the 2nd place team. We've been in contact with them, we're thankful they've been forthright throughout our investigation and have continually lead with wanting to do the right thing. We were able to investigate and validate the claims that yes, Legoshi had an old account with which they started the competition, deleted that account as the result of account access issues, and resumed on a new account. We ensured that there was no advantage gained from the multiple accounts - specifically submission limits were not exceeded on either a team basis or a daily submission basis.
\nThe initial claim about that came in through our Compliance channels was not shared with @inversion immediately, and with some overlapping holidays, it simply was overlooked. I'll take responsibility for this oversight.
\nThanks for your patience! Recap post coming soon.
", "rawMarkdown": "Hey all,\n\n*EDIT for TL;DR: We investigated and found no violation of the rules. The new account was a continuation of a previous deleted account. There were no simultaneous submissions*\n\nWe've investigated this on our end and will not be proceeding with any further action against the 2nd place team. We've been in contact with them, we're thankful they've been forthright throughout our investigation and have continually lead with wanting to do the right thing. We were able to investigate and validate the claims that yes, Legoshi had an old account with which they started the competition, deleted that account as the result of account access issues, and resumed on a new account. We ensured that there was no advantage gained from the multiple accounts - specifically submission limits were not exceeded on either a team basis or a daily submission basis.\n\nThe initial claim about that came in through our Compliance channels was not shared with @inversion immediately, and with some overlapping holidays, it simply was overlooked. I'll take responsibility for this oversight. \n\nThanks for your patience! Recap post coming soon.", "votes": 7, "replies": [ { "id": 1344392, "postDate": "2021-06-10T22:21:55.320Z", "content": "@addisonhoward Thanks for the investigation and detailed explanation. Now I understood what happened and there was no issue.
\n@legoshi sorry for raising such a frustrating topic for you. Now I can say congrats to you and your team at heart! Even though there is no more suspicion, still I can't wait to see your solution:)
", "rawMarkdown": "@addisonhoward Thanks for the investigation and detailed explanation. Now I understood what happened and there was no issue.\n\n@legoshi sorry for raising such a frustrating topic for you. Now I can say congrats to you and your team at heart! Even though there is no more suspicion, still I can't wait to see your solution:)", "votes": -2 }, { "id": 1344396, "postDate": "2021-06-10T22:30:02.727Z", "content": "@bamps53 - we are thankful the Kaggle community continues to raise issues like these. We believe our anti-cheating process is pretty thorough as it has been refined over the years, but we always want to continue to improve and be tuned in to things that we may miss. Please do not hesitate to raise the issues with us in the future as well.
", "rawMarkdown": "@bamps53 - we are thankful the Kaggle community continues to raise issues like these. We believe our anti-cheating process is pretty thorough as it has been refined over the years, but we always want to continue to improve and be tuned in to things that we may miss. Please do not hesitate to raise the issues with us in the future as well." }, { "id": 1344401, "postDate": "2021-06-10T22:40:26.703Z", "content": "As discussed in this discussion, it's better to be able to share about cheating suspicion in each competition forum.
\nNext time if I found something strange, I'll raise an issue in public discussion immediately, in addition to report to compliance team.
It's a healthy reminder not to jump to conclusions. It's become common for Kagglers to call out suspected cheaters - usually in the form of unranked competitors making big, quick jumps in the LB. But let's remember there can be perfectly innocent reasons - the ones I've seen are real-life data scientists or PhDs who've decided to try out Kaggle on a topic that they're experts in and immediately perform well.
\nOn the flip side, I'm reminded that arguably the worst cheater in Kaggle history was a highly ranked and experienced GM.
\nThere's no harm in flagging potential rules violations in a forums thread but everyone should be viewed as a competitor \"acting in good faith\" until there's clear evidence showing otherwise.
", "rawMarkdown": "It's a healthy reminder not to jump to conclusions. It's become common for Kagglers to call out suspected cheaters - usually in the form of unranked competitors making big, quick jumps in the LB. But let's remember there can be perfectly innocent reasons - the ones I've seen are real-life data scientists or PhDs who've decided to try out Kaggle on a topic that they're experts in and immediately perform well.\n\nOn the flip side, I'm reminded that arguably the worst cheater in Kaggle history was a highly ranked and experienced GM. \n\nThere's no harm in flagging potential rules violations in a forums thread but everyone should be viewed as a competitor \"acting in good faith\" until there's clear evidence showing otherwise.", "votes": 3 }, { "id": 1345914, "postDate": "2021-06-11T23:27:30.360Z", "content": "@leecming I understand your concern and that's why we haven't opened a thread like this during competition and just reported to compliance form. But at least in this case it was simply overlooked, and the more worse, there was no way to know if it was considered or not because there is no feedback. I should say current system is not working. It's not ideal, but currently we need to do witch hunt and raise an issue to get hosts' attention if we found something strange, even if it's very slight signal.
", "rawMarkdown": "@leecming I understand your concern and that's why we haven't opened a thread like this during competition and just reported to compliance form. But at least in this case it was simply overlooked, and the more worse, there was no way to know if it was considered or not because there is no feedback. I should say current system is not working. It's not ideal, but currently we need to do witch hunt and raise an issue to get hosts' attention if we found something strange, even if it's very slight signal.", "votes": 1 }, { "id": 1348813, "postDate": "2021-06-14T09:38:23.053Z", "content": "\n\nThe initial claim about that came in through our Compliance channels was not shared with @inversion immediately, and with some overlapping holidays, it simply was overlooked. I'll take responsibility for this oversight.
\n
@addisonhoward If an error like this could happen then it can happen again, and may have happened in the past. The best way to avoid repetition is to change the process that allowed for the error. Suggested in this thread, sending an acknowledgement to the case reporter that the case reported via compliance is being looked at seems like a good idea. It will lower the will to publicly share accusations.
\nMy 2 cents.
", "rawMarkdown": "> The initial claim about that came in through our Compliance channels was not shared with @inversion immediately, and with some overlapping holidays, it simply was overlooked. I'll take responsibility for this oversight. \n\n@addisonhoward If an error like this could happen then it can happen again, and may have happened in the past. The best way to avoid repetition is to change the process that allowed for the error. Suggested in this thread, sending an acknowledgement to the case reporter that the case reported via compliance is being looked at seems like a good idea. It will lower the will to publicly share accusations.\n\nMy 2 cents.", "votes": 1 } ] }, { "id": 1335553, "postDate": "2021-06-04T09:24:12.847Z", "content": "I was also taken aback when I saw that the guy created an account 7 hours ago and directly jumped to 16th place (yes I have seen this on the day itself he/she created the account). For us, it took 7 hours just to do inference and prepare a submission file (with an A100 GPU) ☹️. Kaggle must look into this.
\n@addisonhoward
Also, I wonder which wolf is this guy in zootopia
\nhttps://zootopia.fandom.com/wiki/Category:Wolves
Having seen the updated message of this topic, is there any update such that investigastion has been completed by Kaggle?
\nAs far as I read from the \"Welcome to the Molecular Translation Competition!\" discussion, they were still in the process to finalize the leaderboard yesterday.
I haven't received any yet. Hope they're working hard!
", "rawMarkdown": "I haven't received any yet. Hope they're working hard!", "votes": 1 } ] }, { "id": 1335667, "postDate": "2021-06-04T11:04:19.367Z", "content": "Hmm, maybe he was excluded by his friends for some reason. Then he went alone with their code and needed some compute resources which was solved by merging teams. Just a theory.
", "rawMarkdown": "Hmm, maybe he was excluded by his friends for some reason. Then he went alone with their code and needed some compute resources which was solved by merging teams. Just a theory.", "votes": 1 }, { "id": 1336884, "postDate": "2021-06-05T09:09:08.460Z", "content": "Hi Camaro, thanks for sharing the information. But why did you disclose this after the competition end? If you post this thread when you noticed Legoshi has multiple accounts, his teammates wouldn't have teamed up with him. I feel very sorry for his teammates, because they must have tried very hard in this competition but will probably be disqualified.
", "rawMarkdown": "Hi Camaro, thanks for sharing the information. But why did you disclose this after the competition end? If you post this thread when you noticed Legoshi has multiple accounts, his teammates wouldn't have teamed up with him. I feel very sorry for his teammates, because they must have tried very hard in this competition but will probably be disqualified.", "votes": 1, "replies": [ { "id": 1336913, "postDate": "2021-06-05T09:28:56.063Z", "content": "When I noticed that he had jumped in to LB, I had reported it to the Kaggle complience team at that point.
\nAs Camaro indicated, Kaggle compliance team responded that they would check on it.
\nSo I had assumed that he would be removed from LB after the competition if he has another accounts, but as it turned out that was not enough.
\nI agree with you that I feel very sorry for his teammates.
@bamps53 said he has already reported it. The other team members are responsible for the team merging with a suspicious account. There were many suspicious accounts in the competition. Legoshi was one of the obvious ones. They should have checked their profile, which I believe they already did but took the risk. That's why they are silent now.
\n\"When in doubt, tenuki.\"
", "rawMarkdown": "@bamps53 said he has already reported it. The other team members are responsible for the team merging with a suspicious account. There were many suspicious accounts in the competition. Legoshi was one of the obvious ones. They should have checked their profile, which I believe they already did but took the risk. That's why they are silent now.\n\n\"When in doubt, tenuki.\"", "votes": 4 }, { "id": 1336963, "postDate": "2021-06-05T10:15:30.307Z", "content": "Yes, you are completely correct. Camaro's team did nothing wrong and Legoshi's teammates are responsible for the team merging with a suspicious account. However, I think it should be much better for such a problem to be disclosed in kaggle discussion as soon as possible to prevent the \"victims\" of cheaters. In future competitions, I hope such a problem is disclosed not after the competition end, but immediately after people notice the problem.
\n\"Play Double Sente Early\"
", "rawMarkdown": "Yes, you are completely correct. Camaro's team did nothing wrong and Legoshi's teammates are responsible for the team merging with a suspicious account. However, I think it should be much better for such a problem to be disclosed in kaggle discussion as soon as possible to prevent the \"victims\" of cheaters. In future competitions, I hope such a problem is disclosed not after the competition end, but **immediately after people notice the problem**. \n\n\"Play Double Sente Early\"", "votes": 2 }, { "id": 1336973, "postDate": "2021-06-05T10:29:16.573Z", "content": "Yes, we need a cheater report thread just next to team merging thread.
", "rawMarkdown": "Yes, we need a cheater report thread just next to team merging thread.", "votes": 4 } ] }, { "id": 1336480, "postDate": "2021-06-05T00:47:45.607Z", "content": "Perhaps his/her teammates (experienced Kagglers) can shine some light on the situation?
", "rawMarkdown": "Perhaps his/her teammates (experienced Kagglers) can shine some light on the situation? " }, { "id": 1335354, "postDate": "2021-06-04T07:02:24.070Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1335205, "author_name": "sin", "author_url": "", "post_date": "2021-06-04T04:50:15.917000", "content": "Highly suspicious. For four reasons:
\nKaggle must have overlooked the feature time elapsed since account creation / rank of the submission in their cheater detection algorithm…
In HubMap there was a lot of suspicious accounts reaching top 20 in less than 5 hours, that were not removed as well.
\nI feel like Kaggle is overlooking the verification in some competitions, that's sad because it really hurts the image of the platform.
", "votes": 20, "replies": [] }, { "id": 1335431, "author_name": "Chun Ming Lee", "author_url": "", "post_date": "2021-06-04T07:54:20.233000", "content": "To be generous to Kaggle, a FP in their anti cheating algorithm is arguably worse than a FN.
", "votes": 15, "replies": [] }, { "id": 1335677, "author_name": "Camaro", "author_url": "", "post_date": "2021-06-04T11:12:33.310000", "content": "We may need a competition to detect cheating by using kaggle meta data🤣
", "votes": 7, "replies": [] }, { "id": 1335680, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2021-06-04T11:18:56.450000", "content": "I think a FN is worse than a FP. You can get all suspicious accounts and ask them to justify themselves. Then you can remove the ones who don't make sense. 2 Stage procedure sounds the best to me.
", "votes": 3, "replies": [] }, { "id": 1335716, "author_name": "Chun Ming Lee", "author_url": "", "post_date": "2021-06-04T11:50:58.497000", "content": "\"Suspicious\" is a hard thing to pin down…
\nThe most practical solution would be for Kagglers to flag potential suspicious behavior the way this thread is doing.
\nI'm curious how you'd codify the current accusation into a heuristic - an account being setup and making a high LB sub almost immediately isn't necessarily suspicious. And if implemented, what would be considered a good \"justification\"?
\nAnti-cheating algos aren't trivial.
", "votes": 3, "replies": [] }, { "id": 1335782, "author_name": "fergusoci", "author_url": "", "post_date": "2021-06-04T12:39:52.950000", "content": "Think there's a difference between a general anti-cheating algo (applied to all entries) versus a more detailed examination of a prize-winning competitor. This particular situation cannot be codified in a heuristic, and shouldn't be.
\nIn this case Kaggle needs to ask for the code to reproduce the first submission. Kaggle would have the data on when the competition dataset download completed. Therefore they must be able to reproduce the training + inference in order to reproduce the submission given the GPU resources that were used. If this can't be done then it's not possible that the situation is legitimate.
", "votes": 3, "replies": [] }, { "id": 1335829, "author_name": "Chun Ming Lee", "author_url": "", "post_date": "2021-06-04T13:11:26.913000", "content": "I hope a just solution is found especially given your potential standing.
\nI believe I had a gold snatched from me by private sharing in one of my first competitions so I understand the frustration.
", "votes": 4, "replies": [] } ] }, { "id": 1335221, "author_name": "Qishen Ha", "author_url": "", "post_date": "2021-06-04T05:06:29.510000", "content": "Looking forward to see the solution +1 :)
", "votes": 10, "replies": [ { "id": 1335678, "author_name": "Camaro", "author_url": "", "post_date": "2021-06-04T11:13:40.397000", "content": "I'm looking forward to see your solution too, we're scared!!!
", "votes": 1, "replies": [] } ] }, { "id": 1335715, "author_name": "Camaro", "author_url": "", "post_date": "2021-06-04T11:50:40.877000", "content": "By the way, we've already reported this suspect to kaggle compliance team during competition.
\nThey answered that they would conduct vigorous sanity check after competition end and promissed to take this into account.
\nSo, I believe the check was completed and he passed. I'm just curious how he did it.
Provided it only took a few hours to finalize the results instead of the couple of days we're used to, they probably did not really look into the case thoroughly.
\nHowever they will have to, since additional checks are conducted for teams in the prize zone.
", "votes": 4, "replies": [] }, { "id": 1336444, "author_name": "inversion", "author_url": "", "post_date": "2021-06-04T23:04:53.237000", "content": "@theoviel The reason the competition was verified quickly - I'm currently out on holiday. I did most of the compliance checks before I left, and then a quick review to ensure there was nothing new, and then I processed them right away at the competition close. This particular case was not on my radar, but, to your point, will get additional scrutiny as a winning team.
", "votes": 6, "replies": [] }, { "id": 1336488, "author_name": "Camaro", "author_url": "", "post_date": "2021-06-05T01:04:20.997000", "content": "@inversion Thanks for the comment, it's good to know to be acknowledged by the host.
\nBut to be honest your comment made me uneasy…
\n\nThis particular case was not on my radar
\n
We've already reported to kaggle compliance team and their response was below:
\n\n\nHi Kazuki,
\n
\n Thanks for reaching out. We perform our cheating removals at the end of the competition, and will certainly take this into consideration. Thanks for bringing it to our attention.
\n Kaggle Compliance
But you (Judging from your comment, the person who conduct compliance check for this competition) didn't know that. Is this system correctly working? If there could be a communication issue, we might need to raise such kind of concern in competition discussion, which seems not desirable.
\n\n\nbut, to your point, will get additional scrutiny as a winning team.
\n
I'm thinking the same, but in my understanding winner's call is not a place to do such a primitive sanity check. What if the suspects were in 4th place and I haven't raise this issue? It would be definitely missed.
\nI don't judge for this particular case, he can be valid and I have no chance to investigate by myself. Kaggle team only can know.
\nAnd also I didn't want to expose the doubt in public discussion, that's why we kept quiet and just submit to compliance team. But with considering so quick leaderboard fix, we felt it might not be checked vigorously, so we decided to publish.
As everyone says, credibility of winners is so important for all the past competition winners and platform itself. Hope fair result.
", "votes": 6, "replies": [] }, { "id": 1336498, "author_name": "inversion", "author_url": "", "post_date": "2021-06-05T01:46:24.160000", "content": "@bamps53 I will definitely look into why this did not get passed on (they generally are).
", "votes": 13, "replies": [] }, { "id": 1336553, "author_name": "Camaro", "author_url": "", "post_date": "2021-06-05T03:20:36.113000", "content": "Thanks, please post the result here so that everyone can be sure that there is nothing wrong in compliance system.
", "votes": 4, "replies": [] } ] }, { "id": 1335443, "author_name": "sheep", "author_url": "", "post_date": "2021-06-04T08:17:03.897000", "content": "There must be some superpower I have not seen ever! Maybe time magic?
", "votes": 8, "replies": [ { "id": 1335508, "author_name": "Gold Retriever", "author_url": "", "post_date": "2021-06-04T09:06:56.793000", "content": "@steamedsheep They might have put more faith in Santa Claus than we did…
", "votes": 1, "replies": [] }, { "id": 1336389, "author_name": "Stas Sl", "author_url": "", "post_date": "2021-06-04T21:07:22.670000", "content": "
\nJust find a good planet for training
Something really fishy indeed with the tight gap between subscription and the first submission
\nI second @haqishen and kaggle should check carefully prize winning solutions.
Hey all,
\nEDIT for TL;DR: We investigated and found no violation of the rules. The new account was a continuation of a previous deleted account. There were no simultaneous submissions
\nWe've investigated this on our end and will not be proceeding with any further action against the 2nd place team. We've been in contact with them, we're thankful they've been forthright throughout our investigation and have continually lead with wanting to do the right thing. We were able to investigate and validate the claims that yes, Legoshi had an old account with which they started the competition, deleted that account as the result of account access issues, and resumed on a new account. We ensured that there was no advantage gained from the multiple accounts - specifically submission limits were not exceeded on either a team basis or a daily submission basis.
\nThe initial claim about that came in through our Compliance channels was not shared with @inversion immediately, and with some overlapping holidays, it simply was overlooked. I'll take responsibility for this oversight.
\nThanks for your patience! Recap post coming soon.
", "votes": 7, "replies": [ { "id": 1344392, "author_name": "Camaro", "author_url": "", "post_date": "2021-06-10T22:21:55.320000", "content": "@addisonhoward Thanks for the investigation and detailed explanation. Now I understood what happened and there was no issue.
\n@legoshi sorry for raising such a frustrating topic for you. Now I can say congrats to you and your team at heart! Even though there is no more suspicion, still I can't wait to see your solution:)
", "votes": -2, "replies": [] }, { "id": 1344396, "author_name": "Addison Howard", "author_url": "", "post_date": "2021-06-10T22:30:02.727000", "content": "@bamps53 - we are thankful the Kaggle community continues to raise issues like these. We believe our anti-cheating process is pretty thorough as it has been refined over the years, but we always want to continue to improve and be tuned in to things that we may miss. Please do not hesitate to raise the issues with us in the future as well.
", "votes": 0, "replies": [] }, { "id": 1344401, "author_name": "Camaro", "author_url": "", "post_date": "2021-06-10T22:40:26.703000", "content": "As discussed in this discussion, it's better to be able to share about cheating suspicion in each competition forum.
\nNext time if I found something strange, I'll raise an issue in public discussion immediately, in addition to report to compliance team.
It's a healthy reminder not to jump to conclusions. It's become common for Kagglers to call out suspected cheaters - usually in the form of unranked competitors making big, quick jumps in the LB. But let's remember there can be perfectly innocent reasons - the ones I've seen are real-life data scientists or PhDs who've decided to try out Kaggle on a topic that they're experts in and immediately perform well.
\nOn the flip side, I'm reminded that arguably the worst cheater in Kaggle history was a highly ranked and experienced GM.
\nThere's no harm in flagging potential rules violations in a forums thread but everyone should be viewed as a competitor \"acting in good faith\" until there's clear evidence showing otherwise.
", "votes": 3, "replies": [] }, { "id": 1345914, "author_name": "Camaro", "author_url": "", "post_date": "2021-06-11T23:27:30.360000", "content": "@leecming I understand your concern and that's why we haven't opened a thread like this during competition and just reported to compliance form. But at least in this case it was simply overlooked, and the more worse, there was no way to know if it was considered or not because there is no feedback. I should say current system is not working. It's not ideal, but currently we need to do witch hunt and raise an issue to get hosts' attention if we found something strange, even if it's very slight signal.
", "votes": 1, "replies": [] }, { "id": 1348813, "author_name": "CPMP", "author_url": "", "post_date": "2021-06-14T09:38:23.053000", "content": "\n\nThe initial claim about that came in through our Compliance channels was not shared with @inversion immediately, and with some overlapping holidays, it simply was overlooked. I'll take responsibility for this oversight.
\n
@addisonhoward If an error like this could happen then it can happen again, and may have happened in the past. The best way to avoid repetition is to change the process that allowed for the error. Suggested in this thread, sending an acknowledgement to the case reporter that the case reported via compliance is being looked at seems like a good idea. It will lower the will to publicly share accusations.
\nMy 2 cents.
", "votes": 1, "replies": [] } ] }, { "id": 1335553, "author_name": "Atharva Ingle", "author_url": "", "post_date": "2021-06-04T09:24:12.847000", "content": "I was also taken aback when I saw that the guy created an account 7 hours ago and directly jumped to 16th place (yes I have seen this on the day itself he/she created the account). For us, it took 7 hours just to do inference and prepare a submission file (with an A100 GPU) ☹️. Kaggle must look into this.
\n@addisonhoward
Also, I wonder which wolf is this guy in zootopia
\nhttps://zootopia.fandom.com/wiki/Category:Wolves
Having seen the updated message of this topic, is there any update such that investigastion has been completed by Kaggle?
\nAs far as I read from the \"Welcome to the Molecular Translation Competition!\" discussion, they were still in the process to finalize the leaderboard yesterday.
I haven't received any yet. Hope they're working hard!
", "votes": 1, "replies": [] } ] }, { "id": 1335667, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2021-06-04T11:04:19.367000", "content": "Hmm, maybe he was excluded by his friends for some reason. Then he went alone with their code and needed some compute resources which was solved by merging teams. Just a theory.
", "votes": 1, "replies": [] }, { "id": 1336884, "author_name": "mamas", "author_url": "", "post_date": "2021-06-05T09:09:08.460000", "content": "Hi Camaro, thanks for sharing the information. But why did you disclose this after the competition end? If you post this thread when you noticed Legoshi has multiple accounts, his teammates wouldn't have teamed up with him. I feel very sorry for his teammates, because they must have tried very hard in this competition but will probably be disqualified.
", "votes": 1, "replies": [ { "id": 1336913, "author_name": "KF", "author_url": "", "post_date": "2021-06-05T09:28:56.063000", "content": "When I noticed that he had jumped in to LB, I had reported it to the Kaggle complience team at that point.
\nAs Camaro indicated, Kaggle compliance team responded that they would check on it.
\nSo I had assumed that he would be removed from LB after the competition if he has another accounts, but as it turned out that was not enough.
\nI agree with you that I feel very sorry for his teammates.
@bamps53 said he has already reported it. The other team members are responsible for the team merging with a suspicious account. There were many suspicious accounts in the competition. Legoshi was one of the obvious ones. They should have checked their profile, which I believe they already did but took the risk. That's why they are silent now.
\n\"When in doubt, tenuki.\"
", "votes": 4, "replies": [] }, { "id": 1336963, "author_name": "mamas", "author_url": "", "post_date": "2021-06-05T10:15:30.307000", "content": "Yes, you are completely correct. Camaro's team did nothing wrong and Legoshi's teammates are responsible for the team merging with a suspicious account. However, I think it should be much better for such a problem to be disclosed in kaggle discussion as soon as possible to prevent the \"victims\" of cheaters. In future competitions, I hope such a problem is disclosed not after the competition end, but immediately after people notice the problem.
\n\"Play Double Sente Early\"
", "votes": 2, "replies": [] }, { "id": 1336973, "author_name": "Camaro", "author_url": "", "post_date": "2021-06-05T10:29:16.573000", "content": "Yes, we need a cheater report thread just next to team merging thread.
", "votes": 4, "replies": [] } ] }, { "id": 1336480, "author_name": "Charles", "author_url": "", "post_date": "2021-06-05T00:47:45.607000", "content": "Perhaps his/her teammates (experienced Kagglers) can shine some light on the situation?
", "votes": 0, "replies": [] }, { "id": 1335354, "author_name": "", "author_url": "", "post_date": "2021-06-04T07:02:24.070000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1335185": "First of all I have to apologize to suspect that you have multiple accounts and would be disqualified, but now leaderboard was fixed and that means all participants (at least) in gold zone are certificated by kaggle host.\n\nWhen I saw you jumped in 16th place with score 1.18 just 7 hours after creating kaggle account, I felt it's inpossible with considering the dataset size and the data characteristics.\nFor me 7 hours was the time to just download, unzip and resize the data:(\n\nAgain, congrats for the incredible result, I can't wait to see your solution! \nPlease include how you started to approach this tough challenge!\nIt would be the most valuable infornation for all kagglers.\n", "1335205": "Highly suspicious. For four reasons:\n1. you cannot download the data unless you signed up on kaggle and accepted the rules. \n2. with 1 being said, this user signed up 7 hours ago and made a submission; this means the user downloaded, extracted, trained a model and inference the whole test set within 7 hours. For me, this is impossible, our team's model takes about 1 day to inference. Unless this user has 1024*A100, then yeah.\n3. the finishing score of this user(and his/her team) is very close to first place. I am not implying anything, just suspicions.\n4. the user went from a gray wolf in Texas to AI engineer in Shanghai. \n\n@addisonhoward ", "1335221": "Looking forward to see the solution +1 :)", "1335715": "By the way, we've already reported this suspect to kaggle compliance team during competition.\nThey answered that they would conduct vigorous sanity check after competition end and promissed to take this into account.\nSo, I believe the check was completed and he passed. I'm just curious how he did it.", "1335443": "There must be some superpower I have not seen ever! Maybe time magic?", "1335298": "Something really fishy indeed with the tight gap between subscription and the first submission \nI second @haqishen and kaggle should check carefully prize winning solutions. ", "1344251": "Hey all,\n\n*EDIT for TL;DR: We investigated and found no violation of the rules. The new account was a continuation of a previous deleted account. There were no simultaneous submissions*\n\nWe've investigated this on our end and will not be proceeding with any further action against the 2nd place team. We've been in contact with them, we're thankful they've been forthright throughout our investigation and have continually lead with wanting to do the right thing. We were able to investigate and validate the claims that yes, Legoshi had an old account with which they started the competition, deleted that account as the result of account access issues, and resumed on a new account. We ensured that there was no advantage gained from the multiple accounts - specifically submission limits were not exceeded on either a team basis or a daily submission basis.\n\nThe initial claim about that came in through our Compliance channels was not shared with @inversion immediately, and with some overlapping holidays, it simply was overlooked. I'll take responsibility for this oversight. \n\nThanks for your patience! Recap post coming soon.", "1335553": "I was also taken aback when I saw that the guy created an account 7 hours ago and directly jumped to 16th place (yes I have seen this on the day itself he/she created the account). For us, it took 7 hours just to do inference and prepare a submission file (with an A100 GPU) ☹️. Kaggle must look into this. \n@addisonhoward ", "1335219": "Also, I wonder which wolf is this guy in zootopia\nhttps://zootopia.fandom.com/wiki/Category:Wolves", "1342244": "Having seen the updated message of this topic, is there any update such that investigastion has been completed by Kaggle? \nAs far as I read from the \"Welcome to the Molecular Translation Competition!\" discussion, they were still in the process to finalize the leaderboard yesterday. ", "1335667": "Hmm, maybe he was excluded by his friends for some reason. Then he went alone with their code and needed some compute resources which was solved by merging teams. Just a theory.", "1336884": "Hi Camaro, thanks for sharing the information. But why did you disclose this after the competition end? If you post this thread when you noticed Legoshi has multiple accounts, his teammates wouldn't have teamed up with him. I feel very sorry for his teammates, because they must have tried very hard in this competition but will probably be disqualified.", "1336480": "Perhaps his/her teammates (experienced Kagglers) can shine some light on the situation? ", "1335354": "" } }