One week ago I asked @addisonhoward about removing the 2 teams whose accounts were recently terminated (private LB places 253 and 332) from the leaderboard. The answer was:
\n\n\nWe do not remove teams until after the competition has concluded.
\n
Apparently Kaggle team doesn't remove some properly convicted cheaters even after the competitions have concluded.
\nIn this leaderboard the teams with terminated accounts fortunately don't occupy the medal zone which can't be said for example about Feedback Prize competition (still 14 \"dead\" teams in medal zones).
\nThis way the already convicted cheaters continue \"stealing\" medals or at least competition points from fairly competing participants.
\nOn the other hand leaving lately terminated accounts of convicted cheaters in older leaderboards does not give the best picture of Kaggle anti-cheating effort as the teams were showing the typical group behavior (for which they were lately disqualified) already there.
Or @wcukierski is there any other special meaningful reason for keeping teams of convicted cheaters in leaderboards which I cannot see?
", "messages": [ { "id": "1802283", "postDate": "05/26/2022 16:05:03", "content": "One week ago I asked @addisonhoward about removing the 2 teams whose accounts were recently terminated (private LB places 253 and 332) from the leaderboard. The answer was:
\n\n\nWe do not remove teams until after the competition has concluded.
\n
Apparently Kaggle team doesn't remove some properly convicted cheaters even after the competitions have concluded.
\nIn this leaderboard the teams with terminated accounts fortunately don't occupy the medal zone which can't be said for example about Feedback Prize competition (still 14 \"dead\" teams in medal zones).
\nThis way the already convicted cheaters continue \"stealing\" medals or at least competition points from fairly competing participants.
\nOn the other hand leaving lately terminated accounts of convicted cheaters in older leaderboards does not give the best picture of Kaggle anti-cheating effort as the teams were showing the typical group behavior (for which they were lately disqualified) already there.
Or @wcukierski is there any other special meaningful reason for keeping teams of convicted cheaters in leaderboards which I cannot see?
", "rawMarkdown": "One week ago I asked @addisonhoward about removing the 2 teams whose accounts were recently terminated (private LB places 253 and 332) from the leaderboard. The answer was:\n> We do not remove teams until after the competition has concluded.\n\nApparently Kaggle team doesn't remove some properly convicted cheaters **even after the competitions** have concluded.\nIn this leaderboard the teams with terminated accounts fortunately don't occupy the medal zone which can't be said for example about Feedback Prize competition (still 14 \"dead\" teams in medal zones).\nThis way the already convicted cheaters continue \"stealing\" medals or at least competition points from fairly competing participants.\nOn the other hand leaving lately terminated accounts of convicted cheaters in older leaderboards does not give the best picture of Kaggle anti-cheating effort as the teams were showing the typical group behavior (for which they were lately disqualified) already there.\n\nOr @wcukierski is there any other special meaningful reason for keeping teams of convicted cheaters in leaderboards which I cannot see?", "votes": null }, { "id": "1802594", "postDate": "05/26/2022 23:47:00", "content": "Thanks for flagging this, we're making a bug fix as a result. Teams that contain no active users when the leaderboard is finalized should be removed and our filter was missing some relevant cases. I've removed the two teams you mentioned plus a handful of others.
\nI do want to note that if a user account is closed after a leaderboard is finalized we do not go back and remove the team. Users removed for cheating in one competition often have legitimate results in other, older, competitions. Similarly, many users choose to delete their own accounts for a variety of reasons. Removing their teams from long closed leaderboards would artificially inflate the true rankings of the other teams.
", "rawMarkdown": "Thanks for flagging this, we're making a bug fix as a result. Teams that contain no active users when the leaderboard is finalized should be removed and our filter was missing some relevant cases. I've removed the two teams you mentioned plus a handful of others.\n\nI do want to note that if a user account is closed after a leaderboard is finalized we do not go back and remove the team. Users removed for cheating in one competition often have legitimate results in other, older, competitions. Similarly, many users choose to delete their own accounts for a variety of reasons. Removing their teams from long closed leaderboards would artificially inflate the true rankings of the other teams.", "votes": null }, { "id": "1803251", "postDate": "05/27/2022 15:57:27", "content": "@sohier
\nThanks for the first part of your response, though it brings some new mysteries.
\nThe number of teams in the leaderboard is 802, in all other documents the number is 807 (minus 1 from the original). More surprisingly the numbers are different in other competitions as well (H&M 2954 or 2952?, NBME 1472 or 1471?)
\nSimply put all the leaderboards at least of this year's non-lottery type competitions are still full of surprises and miracles.
In the second part of your response you are trying to downplay all the organized cheaters' case by speaking about totally unrelated cases like accounts voluntarily deleted by users or \"occasional small cheaters\" who cheated on their own once but their accounts were not terminated.
\nI write clearly and repeatedly only about apparent members of large organized groups whose accounts were terminated by Kaggle and about their appearance in competitions closely proceeded the ones, where they were convicted of cheating.
\nMoreover their activity in these preceding competitions showed clear characteristics of the systemic common behaviour which they were lately disqualified for. The worst \"spoiled\" leaderboard happens to be the one of Feedback Prize from this March.
\nAt the end of this FP competition only 2 teams were removed from the leaderboard and dozens of others miraculously slipped through the sieve of your anti-cheating algorithm, though their highly suspicious activity could be noticed at first sight. And I wasn't the only one who noticed it, see the discussion here.
The patterns of the behaviour of members of these 2 organized groups can be observed even now by tracing the activity of some still existing accounts apparently belonging to the groups. I hate to be personal but for illustrative purposes I will mention only as an example 2 of them \"efficientalan\" and \"jesusdon\". They typically started in CommonLit competition, \"graduated\" as Experts in Feedback Prize and after reaching the tier they stopped competing, so they were not caught in subsequent competitions.
\nOn the other hand the \"new wave\" of the same group started in Feedback Prize and was mostly but not completely disqualified in 1 of the following 3 competitions (HappyWhale, NBME or H&M).
Conclusion
\nEverything would be easier and clearer if all the members of organized cheaters' groups had been caught at first try.
\nIt is completely unfair and morally unjustifiable that in these very specific cases the terminated accounts still remain in leaderboards as recent as from March.
\nIt is sad and dangerous for future that not all the members of those organized cheaters' groups have yet been identified and disqualified and they can continue their activity. I can already observe the consequences in my currently active 2 competitions - Image Matching and US Patent.
I still hope that Kaggle team will take more pro-active approach against these negative phenomena. The first step would of course be an introduction of really functional anti-cheating algorithm. Isn't it ridiculous that a platform hosting the best and smartest doesn't have any?
", "rawMarkdown": "sohier \nThanks for the first part of your response, though it brings some new mysteries.\nThe number of teams in the leaderboard is 802, in all other documents the number is 807 (minus 1 from the original). More surprisingly the numbers are different in other competitions as well (H&M 2954 or 2952?, NBME 1472 or 1471?) \nSimply put all the leaderboards at least of this year's non-lottery type competitions are still full of surprises and miracles.\n\nIn the second part of your response you are trying to downplay all the organized cheaters' case by speaking about totally unrelated cases like accounts voluntarily deleted by users or \"occasional small cheaters\" who cheated on their own once but their accounts were not terminated.\nI write clearly and repeatedly only about apparent **members of large organized groups whose accounts were terminated by Kaggle** and about their appearance in **competitions closely proceeded** the ones, where they were convicted of cheating.\nMoreover their activity in these preceding competitions showed clear characteristics of the systemic common behaviour which they were lately disqualified for. The worst \"spoiled\" leaderboard happens to be the one of Feedback Prize from this March.\nAt the end of this FP competition only 2 teams were removed from the leaderboard and dozens of others miraculously slipped through the sieve of your anti-cheating algorithm, though their highly suspicious activity could be noticed at first sight. And I wasn't the only one who noticed it, see the discussion here. \n\nThe patterns of the behaviour of members of these 2 organized groups can be observed even now by tracing the activity of some still existing accounts apparently belonging to the groups. I hate to be personal but for illustrative purposes I will mention only as an example 2 of them \"efficientalan\" and \"jesusdon\". They typically started in CommonLit competition, \"graduated\" as Experts in Feedback Prize and after reaching the tier they stopped competing, so they were not caught in subsequent competitions. \nOn the other hand the \"new wave\" of the same group started in Feedback Prize and was mostly but not completely disqualified in 1 of the following 3 competitions (HappyWhale, NBME or H&M).\n\n**Conclusion**\nEverything would be easier and clearer if all the members of organized cheaters' groups had been caught at first try.\nIt is completely unfair and morally unjustifiable that **in these very specific cases** the terminated accounts still remain in leaderboards as recent as from March.\nIt is sad and dangerous for future that not all the members of those organized cheaters' groups have yet been identified and disqualified and they can continue their activity. I can already observe the consequences in my currently active 2 competitions - Image Matching and US Patent.\n\nI still hope that Kaggle team will take more pro-active approach against these negative phenomena. The first step would of course be an introduction of really functional anti-cheating algorithm. Isn't it ridiculous that a platform hosting the best and smartest doesn't have any?", "votes": null } ], "comments": [ { "id": 1802594, "author_name": "sohier", "author_url": "", "post_date": "05/26/2022 23:47:00", "content": "Thanks for flagging this, we're making a bug fix as a result. Teams that contain no active users when the leaderboard is finalized should be removed and our filter was missing some relevant cases. I've removed the two teams you mentioned plus a handful of others.
\nI do want to note that if a user account is closed after a leaderboard is finalized we do not go back and remove the team. Users removed for cheating in one competition often have legitimate results in other, older, competitions. Similarly, many users choose to delete their own accounts for a variety of reasons. Removing their teams from long closed leaderboards would artificially inflate the true rankings of the other teams.
", "votes": null, "replies": [ { "id": 1803251, "author_name": "blankaf", "author_url": "", "post_date": "05/27/2022 15:57:27", "content": "@sohier
\nThanks for the first part of your response, though it brings some new mysteries.
\nThe number of teams in the leaderboard is 802, in all other documents the number is 807 (minus 1 from the original). More surprisingly the numbers are different in other competitions as well (H&M 2954 or 2952?, NBME 1472 or 1471?)
\nSimply put all the leaderboards at least of this year's non-lottery type competitions are still full of surprises and miracles.
In the second part of your response you are trying to downplay all the organized cheaters' case by speaking about totally unrelated cases like accounts voluntarily deleted by users or \"occasional small cheaters\" who cheated on their own once but their accounts were not terminated.
\nI write clearly and repeatedly only about apparent members of large organized groups whose accounts were terminated by Kaggle and about their appearance in competitions closely proceeded the ones, where they were convicted of cheating.
\nMoreover their activity in these preceding competitions showed clear characteristics of the systemic common behaviour which they were lately disqualified for. The worst \"spoiled\" leaderboard happens to be the one of Feedback Prize from this March.
\nAt the end of this FP competition only 2 teams were removed from the leaderboard and dozens of others miraculously slipped through the sieve of your anti-cheating algorithm, though their highly suspicious activity could be noticed at first sight. And I wasn't the only one who noticed it, see the discussion here.
The patterns of the behaviour of members of these 2 organized groups can be observed even now by tracing the activity of some still existing accounts apparently belonging to the groups. I hate to be personal but for illustrative purposes I will mention only as an example 2 of them \"efficientalan\" and \"jesusdon\". They typically started in CommonLit competition, \"graduated\" as Experts in Feedback Prize and after reaching the tier they stopped competing, so they were not caught in subsequent competitions.
\nOn the other hand the \"new wave\" of the same group started in Feedback Prize and was mostly but not completely disqualified in 1 of the following 3 competitions (HappyWhale, NBME or H&M).
Conclusion
\nEverything would be easier and clearer if all the members of organized cheaters' groups had been caught at first try.
\nIt is completely unfair and morally unjustifiable that in these very specific cases the terminated accounts still remain in leaderboards as recent as from March.
\nIt is sad and dangerous for future that not all the members of those organized cheaters' groups have yet been identified and disqualified and they can continue their activity. I can already observe the consequences in my currently active 2 competitions - Image Matching and US Patent.
I still hope that Kaggle team will take more pro-active approach against these negative phenomena. The first step would of course be an introduction of really functional anti-cheating algorithm. Isn't it ridiculous that a platform hosting the best and smartest doesn't have any?
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1802283": "One week ago I asked @addisonhoward about removing the 2 teams whose accounts were recently terminated (private LB places 253 and 332) from the leaderboard. The answer was:\n> We do not remove teams until after the competition has concluded.\n\nApparently Kaggle team doesn't remove some properly convicted cheaters **even after the competitions** have concluded.\nIn this leaderboard the teams with terminated accounts fortunately don't occupy the medal zone which can't be said for example about Feedback Prize competition (still 14 \"dead\" teams in medal zones).\nThis way the already convicted cheaters continue \"stealing\" medals or at least competition points from fairly competing participants.\nOn the other hand leaving lately terminated accounts of convicted cheaters in older leaderboards does not give the best picture of Kaggle anti-cheating effort as the teams were showing the typical group behavior (for which they were lately disqualified) already there.\n\nOr @wcukierski is there any other special meaningful reason for keeping teams of convicted cheaters in leaderboards which I cannot see?", "1802594": "Thanks for flagging this, we're making a bug fix as a result. Teams that contain no active users when the leaderboard is finalized should be removed and our filter was missing some relevant cases. I've removed the two teams you mentioned plus a handful of others.\n\nI do want to note that if a user account is closed after a leaderboard is finalized we do not go back and remove the team. Users removed for cheating in one competition often have legitimate results in other, older, competitions. Similarly, many users choose to delete their own accounts for a variety of reasons. Removing their teams from long closed leaderboards would artificially inflate the true rankings of the other teams.", "1803251": "sohier \nThanks for the first part of your response, though it brings some new mysteries.\nThe number of teams in the leaderboard is 802, in all other documents the number is 807 (minus 1 from the original). More surprisingly the numbers are different in other competitions as well (H&M 2954 or 2952?, NBME 1472 or 1471?) \nSimply put all the leaderboards at least of this year's non-lottery type competitions are still full of surprises and miracles.\n\nIn the second part of your response you are trying to downplay all the organized cheaters' case by speaking about totally unrelated cases like accounts voluntarily deleted by users or \"occasional small cheaters\" who cheated on their own once but their accounts were not terminated.\nI write clearly and repeatedly only about apparent **members of large organized groups whose accounts were terminated by Kaggle** and about their appearance in **competitions closely proceeded** the ones, where they were convicted of cheating.\nMoreover their activity in these preceding competitions showed clear characteristics of the systemic common behaviour which they were lately disqualified for. The worst \"spoiled\" leaderboard happens to be the one of Feedback Prize from this March.\nAt the end of this FP competition only 2 teams were removed from the leaderboard and dozens of others miraculously slipped through the sieve of your anti-cheating algorithm, though their highly suspicious activity could be noticed at first sight. And I wasn't the only one who noticed it, see the discussion here. \n\nThe patterns of the behaviour of members of these 2 organized groups can be observed even now by tracing the activity of some still existing accounts apparently belonging to the groups. I hate to be personal but for illustrative purposes I will mention only as an example 2 of them \"efficientalan\" and \"jesusdon\". They typically started in CommonLit competition, \"graduated\" as Experts in Feedback Prize and after reaching the tier they stopped competing, so they were not caught in subsequent competitions. \nOn the other hand the \"new wave\" of the same group started in Feedback Prize and was mostly but not completely disqualified in 1 of the following 3 competitions (HappyWhale, NBME or H&M).\n\n**Conclusion**\nEverything would be easier and clearer if all the members of organized cheaters' groups had been caught at first try.\nIt is completely unfair and morally unjustifiable that **in these very specific cases** the terminated accounts still remain in leaderboards as recent as from March.\nIt is sad and dangerous for future that not all the members of those organized cheaters' groups have yet been identified and disqualified and they can continue their activity. I can already observe the consequences in my currently active 2 competitions - Image Matching and US Patent.\n\nI still hope that Kaggle team will take more pro-active approach against these negative phenomena. The first step would of course be an introduction of really functional anti-cheating algorithm. Isn't it ridiculous that a platform hosting the best and smartest doesn't have any?" }, "source": "meta" }