I was shocked :(
", "messages": [ { "id": "1804937", "postDate": "05/29/2022 15:22:28", "content": "I was shocked :(
", "rawMarkdown": "I was shocked :(", "votes": null }, { "id": "1805083", "postDate": "05/29/2022 19:04:50", "content": "Dear @abuduori,
\nCan you please tell us more?
\nLukas
", "rawMarkdown": "Dear @abuduori,\n\nCan you please tell us more?\n\nLukas", "votes": null }, { "id": "1814757", "postDate": "06/08/2022 09:05:28", "content": "Hi guys,
\nI accept the truth that our team was disqualified and removed from the leaderboard because of using external data, but I can't stand that an obviously cheated team ARM is still there.
\nI was always keeping my eyes on the discussion and leaderboard sections of FungiCLEF 2022 and SnakeCLEF 2022.
\nHere I just provide some observations:
\na. One team [Deleted] 6e08db24-b4aa-40d3-8197-b00bc3d1ed0d got an improvement from 0.87817 to 0.93932 on public leaderboard in just once submission. Fortunately, he posted his concerns as well as the method to crawl label from inaturalist in discussion section (but deleted soon) and deleted his account.
\nb. Another team ARM got an improvement from 0.85863 to 0.92778 on public leaderboard in just once submission, nearly 8 hours before the deadline.
\nAll of you know what I mean.
\nLet's remember, the team members of ARM are YongHuang (aderonhuang) and Henry (henryhuang18).
\nI will not let such a severe academic misconduct happen.
\nIf they can't provide a reproducible training code in 24 hours, please ask the host to disqualify them from the community. BTW, We have enough GPUs to verify the training code.
\nSorry for my tone, but I need all of your help.
\nBest,
\nAbuduory
I didn't know that any team was disqualified and removed. And I have no clue how to detect that teams are cheating.
\nI think only hosts and Kaggle team can say something about Abuduory observations.
@picekl
\nAny idea Lukas?
Thanks in advance.
", "rawMarkdown": "I didn't know that any team was disqualified and removed. And I have no clue how to detect that teams are cheating. \nI think only hosts and Kaggle team can say something about Abuduory observations. \n\n@picekl \nAny idea Lukas?\n\nThanks in advance.", "votes": null }, { "id": "1815386", "postDate": "06/09/2022 02:18:11", "content": "It was almost the worst race experience I've ever had.
", "rawMarkdown": "It was almost the worst race experience I've ever had.", "votes": null }, { "id": "1816287", "postDate": "06/10/2022 04:12:42", "content": "We have been hosting a simultaneous competition in CVPR 2022 Workshop recently. To be honest, hosting a competition like this is not easy, everything should be taken into account to keep from cheating, but it's the host's duty to ensure fairness.
\nWe did the following to guarantee fairness,
\n1) Top 10 (out of 600+) teams were asked to submit their report, codes and models.
\n2) Teams that refused to submit codes or models were disqualified immediately.
\n3) For each team,
\n 3.a) the report was first checked to find obvious violations, such as forbidden data usage.
\n 3.b) the provided models were run to reproduce their submission score, this score should be strictly the same.
\n 3.c) the training codes were run to reproduce models with the help of its authors. Interestingly, nearly all the teams used ensemble strategies, so all of the models were trained. If the result can't be reproduced, i.e. performance gap larger than 1-2%, the team would be disqualified.
Not until we found the real top 3 teams was this procedure stopped. Yes, it costs a lot, but it's our duty.
\nA competition with label leakage should be much more strict, right? @picekl ❤️
\nIn all circumstances, talk is cheap, code tells the truth.
Anyone who is interested in our hosted competition please contact me. 😃😃😃
", "rawMarkdown": "We have been hosting a simultaneous competition in CVPR 2022 Workshop recently. To be honest, hosting a competition like this is not easy, everything should be taken into account to keep from cheating, but it's the host's duty to ensure fairness. \n\nWe did the following to guarantee fairness,\n1) Top 10 (out of 600+) teams were asked to submit their report, codes and models.\n2) Teams that refused to submit codes or models were disqualified immediately.\n3) For each team,\n\t3.a) the report was first checked to find obvious violations, such as forbidden data usage.\n\t3.b) the provided models were run to reproduce their submission score, this score should be strictly the same.\n\t3.c) the training codes were run to reproduce models with the help of its authors. Interestingly, nearly all the teams used ensemble strategies, so all of the models were trained. If the result can't be reproduced, i.e. performance gap larger than 1-2%, the team would be disqualified.\n\nNot until we found the real top 3 teams was this procedure stopped. Yes, it costs a lot, but it's our duty.\n\nA competition with label leakage should be much more strict, right? @picekl ❤️\nIn all circumstances, talk is cheap, code tells the truth.\n\nAnyone who is interested in our hosted competition please contact me. 😃😃😃", "votes": null }, { "id": "1818768", "postDate": "06/13/2022 06:23:48", "content": "this is my code: https://github.com/wanghao19930804/cls_sn, Looking forward to your validation of my training code, thanks!
", "rawMarkdown": "this is my code: https://github.com/wanghao19930804/cls_sn, Looking forward to your validation of my training code, thanks!", "votes": null } ], "comments": [ { "id": 1805083, "author_name": "picekl", "author_url": "", "post_date": "05/29/2022 19:04:50", "content": "Dear @abuduori,
\nCan you please tell us more?
\nLukas
", "votes": null, "replies": [] }, { "id": 1814757, "author_name": "abuduori", "author_url": "", "post_date": "06/08/2022 09:05:28", "content": "Hi guys,
\nI accept the truth that our team was disqualified and removed from the leaderboard because of using external data, but I can't stand that an obviously cheated team ARM is still there.
\nI was always keeping my eyes on the discussion and leaderboard sections of FungiCLEF 2022 and SnakeCLEF 2022.
\nHere I just provide some observations:
\na. One team [Deleted] 6e08db24-b4aa-40d3-8197-b00bc3d1ed0d got an improvement from 0.87817 to 0.93932 on public leaderboard in just once submission. Fortunately, he posted his concerns as well as the method to crawl label from inaturalist in discussion section (but deleted soon) and deleted his account.
\nb. Another team ARM got an improvement from 0.85863 to 0.92778 on public leaderboard in just once submission, nearly 8 hours before the deadline.
\nAll of you know what I mean.
\nLet's remember, the team members of ARM are YongHuang (aderonhuang) and Henry (henryhuang18).
\nI will not let such a severe academic misconduct happen.
\nIf they can't provide a reproducible training code in 24 hours, please ask the host to disqualify them from the community. BTW, We have enough GPUs to verify the training code.
\nSorry for my tone, but I need all of your help.
\nBest,
\nAbuduory
I didn't know that any team was disqualified and removed. And I have no clue how to detect that teams are cheating.
\nI think only hosts and Kaggle team can say something about Abuduory observations.
@picekl
\nAny idea Lukas?
Thanks in advance.
", "votes": null, "replies": [] }, { "id": 1815386, "author_name": "zimoqingfeng", "author_url": "", "post_date": "06/09/2022 02:18:11", "content": "It was almost the worst race experience I've ever had.
", "votes": null, "replies": [] }, { "id": 1816287, "author_name": "wuyouzc", "author_url": "", "post_date": "06/10/2022 04:12:42", "content": "We have been hosting a simultaneous competition in CVPR 2022 Workshop recently. To be honest, hosting a competition like this is not easy, everything should be taken into account to keep from cheating, but it's the host's duty to ensure fairness.
\nWe did the following to guarantee fairness,
\n1) Top 10 (out of 600+) teams were asked to submit their report, codes and models.
\n2) Teams that refused to submit codes or models were disqualified immediately.
\n3) For each team,
\n 3.a) the report was first checked to find obvious violations, such as forbidden data usage.
\n 3.b) the provided models were run to reproduce their submission score, this score should be strictly the same.
\n 3.c) the training codes were run to reproduce models with the help of its authors. Interestingly, nearly all the teams used ensemble strategies, so all of the models were trained. If the result can't be reproduced, i.e. performance gap larger than 1-2%, the team would be disqualified.
Not until we found the real top 3 teams was this procedure stopped. Yes, it costs a lot, but it's our duty.
\nA competition with label leakage should be much more strict, right? @picekl ❤️
\nIn all circumstances, talk is cheap, code tells the truth.
Anyone who is interested in our hosted competition please contact me. 😃😃😃
", "votes": null, "replies": [] }, { "id": 1818768, "author_name": "wanghao1993", "author_url": "", "post_date": "06/13/2022 06:23:48", "content": "this is my code: https://github.com/wanghao19930804/cls_sn, Looking forward to your validation of my training code, thanks!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1804937": "I was shocked :(", "1805083": "Dear @abuduori,\n\nCan you please tell us more?\n\nLukas", "1814757": "Hi guys,\n\nI accept the truth that our team was disqualified and removed from the leaderboard because of using external data, but I can't stand that an obviously cheated team ARM is still there.\n\nI was always keeping my eyes on the discussion and leaderboard sections of FungiCLEF 2022 and SnakeCLEF 2022.\n\nHere I just provide some observations:\n\na. One team [Deleted] 6e08db24-b4aa-40d3-8197-b00bc3d1ed0d got an improvement from 0.87817 to 0.93932 on public leaderboard in just once submission. Fortunately, he posted his concerns as well as the method to crawl label from inaturalist in discussion section (but deleted soon) and deleted his account.\n\nb. Another team ARM got an improvement from 0.85863 to 0.92778 on public leaderboard in just once submission, nearly 8 hours before the deadline.\n\nAll of you know what I mean.\n\nLet's remember, the team members of ARM are YongHuang (aderonhuang) and Henry (henryhuang18). \n\nI will not let such a severe academic misconduct happen. \n\nIf they can't provide a reproducible training code in 24 hours, please ask the host to disqualify them from the community. BTW, We have enough GPUs to verify the training code.\n\nSorry for my tone, but I need all of your help.\n\n\nBest,\nAbuduory", "1815186": "I didn't know that any team was disqualified and removed. And I have no clue how to detect that teams are cheating. \nI think only hosts and Kaggle team can say something about Abuduory observations. \n\n@picekl \nAny idea Lukas?\n\nThanks in advance.", "1815386": "It was almost the worst race experience I've ever had.", "1816287": "We have been hosting a simultaneous competition in CVPR 2022 Workshop recently. To be honest, hosting a competition like this is not easy, everything should be taken into account to keep from cheating, but it's the host's duty to ensure fairness. \n\nWe did the following to guarantee fairness,\n1) Top 10 (out of 600+) teams were asked to submit their report, codes and models.\n2) Teams that refused to submit codes or models were disqualified immediately.\n3) For each team,\n\t3.a) the report was first checked to find obvious violations, such as forbidden data usage.\n\t3.b) the provided models were run to reproduce their submission score, this score should be strictly the same.\n\t3.c) the training codes were run to reproduce models with the help of its authors. Interestingly, nearly all the teams used ensemble strategies, so all of the models were trained. If the result can't be reproduced, i.e. performance gap larger than 1-2%, the team would be disqualified.\n\nNot until we found the real top 3 teams was this procedure stopped. Yes, it costs a lot, but it's our duty.\n\nA competition with label leakage should be much more strict, right? @picekl ❤️\nIn all circumstances, talk is cheap, code tells the truth.\n\nAnyone who is interested in our hosted competition please contact me. 😃😃😃", "1818768": "this is my code: https://github.com/wanghao19930804/cls_sn, Looking forward to your validation of my training code, thanks!" }, "source": "meta" }