![](\"https://storage.googleapis.com/kaggle-forum-message-attachments/64967/2013/GBM_All_Submissions.png?sv=2012-02-12&se=2015-03-01T17:28:46Z&sr=b&sp=r&sig=NwQIC3hewuhkO9iMxvyKigpg%2FBVnu75NkKEsxU0%2Fi%2Bw%3D\")
We have just removed some cheaters. The results on the leaderboard is now final.
\nBig congratulations to the winners! The competition hosts will be in touch with the ones interested in the conference soon.
", "messages": [ { "id": "64899", "postDate": "02/26/2015 00:51:59", "content": "We have just removed some cheaters. The results on the leaderboard is now final.
\nBig congratulations to the winners! The competition hosts will be in touch with the ones interested in the conference soon.
", "rawMarkdown": "", "votes": null }, { "id": "64902", "postDate": "02/26/2015 01:38:14", "content": "Wendy,
\n\nAre you able to provide an explanation as to why the following competitors still exist after the removal of cheaters: GBM_2015_A, GBM_2015_B, GBM_2015_C, GBM_2015_D, GBM_2015_E, GBM_2015_F, and GBM_2015_G? These competitors are largely clustered together on the leaderboard which makes them look even further suspect.
\nThanks,
\nTrent
", "rawMarkdown": "", "votes": null }, { "id": "64915", "postDate": "02/26/2015 07:38:27", "content": "Trent,
\nI agree they look suspicious, but after cross referencing user logs and submission logs, it's not obvious that those are cheaters (unlike the other ones that we removed). We think the GBM_2015_x groups might be a class of students using similar team names.
", "rawMarkdown": "", "votes": null }, { "id": "64922", "postDate": "02/26/2015 08:35:25", "content": "Wendy,
\n\nNone of the three first teams have included the "NER" keyword in their names. Does it mean that they will not receive the prize? (That's what the rules say)
\nI did not include the suffix because I hadn't big expectations. But now that I've finished 5th I could consider submitting a paper if the organizers were interested. But honestly, I'm so far away from the winners in performance that I don't think anybody would be interested...
", "rawMarkdown": "", "votes": null }, { "id": "64924", "postDate": "02/26/2015 09:59:48", "content": "[quote=Wendy Kan;64915]
\nTrent,
\nI agree they look suspicious, but after cross referencing user logs and submission logs, it's not obvious that those are cheaters (unlike the other ones that we removed). We think the GBM_2015_x groups might be a class of students using similar team names.
\n[/quote]
\nWendy,
\nThank you for your response. I would agree that it is possible that GBM_2015_x groups could be a class of students, but it does not sound like there is any positive evidence for that, only no negative evidence against it. I would caution that it is also possible that posing as a class could be a sneaky new way of beating the system. But, assuming that GBM_2015_x groups are legitimately a class of students, it still makes me uneasy that all seven teams landed in the top 20 (top 8%). The class could have a fantastic instructor which I hope is the case, but the class could also be "reading from the same sheet of music" to which other competitors did not have access (e.g., some starter code provided by the instructor), which is expressly disallowed by the rules. I think I have sufficiently expressed my concerns, so I will be silent now and let other competitors respond if they share similar concerns and think further investigation is warranted.
\n\nThank you.
", "rawMarkdown": "", "votes": null }, { "id": "64926", "postDate": "02/26/2015 10:51:26", "content": "[quote=Wendy Kan;64915]
\nTrent,
\nI agree they look suspicious, but after cross referencing user logs and submission logs, it's not obvious that those are cheaters (unlike the other ones that we removed). We think the GBM_2015_x groups might be a class of students using similar team names.
\n[/quote]
\nThat looks really strange - even if they are a class and different people, they have suspiciously pretty close scores - to the extend that some collaboration was unavoidable !
", "rawMarkdown": "", "votes": null }, { "id": "64928", "postDate": "02/26/2015 10:58:18", "content": "I challenge each one of GBM_2015_x teams to describe their approach!!! What's your CV approach? What is your CV score? Preprocess? trainning method? Post-process?
\nIn a competition with a small dataset, small number of instances in public LB, small number of subjects, thousand of features, time dependente of variables and dissimilarity intersubject, it's very hard to generalize well. I say that based in previous similar competitions.
\nWell, they are 7 teams, under top 20, in their first Kaggle competition, performing better than many Kaggle masters and with a very close score. It's at least suspicious...
", "rawMarkdown": "", "votes": null }, { "id": "64929", "postDate": "02/26/2015 11:01:57", "content": "Trent,
\nthanks for bringing this up. I also have an uneasy feeling in this case (my apologies to you guys GBM_2015_A, GBM_2015_B, GBM_2015_C, GBM_2015_D, GBM_2015_E, GBM_2015_F, and GBM_2015_G in case you are really individual people who came up with individual solutions).
\nGiven how discouraging it can be to the honest competitors if cheating does prevail at a significant level, I would appreciate any extra effort that Kaggle can do to establish that these submissions are indeed in accordance with the rules.
\nCheers
", "rawMarkdown": "", "votes": null }, { "id": "64930", "postDate": "02/26/2015 11:02:20", "content": "[quote=Gilberto Titericz Junior;64928]
\nI challenge each one of GBM_2015_x teams to describe their approach!!! What's your CV approach? What is your CV score? Preprocess? trainning method? Post-process?
\nIn a competition with a small dataset, small number of instances in public LB, small number of subjects, thousand of features, time dependente of variables and dissimilarity intersubject, it's very hard to generalize well. I say that based in previous similar competitions.
\nWell, they are 7 teams, under top 20, in their first Kaggle competition, performing better than many Kaggle masters and with a very close score. It's at least suspicious...
\n[/quote]
\n\nWell said. And it is not about the handful of extra kaggle points we are going to get from their removal, but more like what is right do...
", "rawMarkdown": "", "votes": null }, { "id": "64937", "postDate": "02/26/2015 12:06:38", "content": "Agree!!!
\nThe scores are too close to be unrelated. No matter if it's a one individual or a group from the same class, the scores are suspiciously too correlated.
", "rawMarkdown": "", "votes": null }, { "id": "64939", "postDate": "02/26/2015 12:12:12", "content": "The sad thing about this issue is that the fraud wouldn't have been detected if they had chosen a different, less suspicious, set of names.
", "rawMarkdown": "", "votes": null }, { "id": "64942", "postDate": "02/26/2015 12:49:40", "content": "I think the GBM_2015 teams are PhD. students from a university in Nice, France. I think we foiled their plan to spell out the alphabet on the leaderboard. Well done to the all-French cluster!
", "rawMarkdown": "", "votes": null }, { "id": "64950", "postDate": "02/26/2015 15:15:07", "content": "The team GBM_2015_G have 2 brazilian names: "Yuri Oliveira" and "Mariana Rufino de Andrade", other GBM_2015_* has a batman and other the enemy of batman rasal ghul.
\nLook at the timestamps, for me it is clear now:
\n138821 GBM_2015_A 15/01/2015 10:36 0.68875
138824 GBM_2015_E 15/01/2015 10:55 0.69993
144137 GBM_2015_C 05/02/2015 08:27 0.68516
144139 GBM_2015_B 05/02/2015 08:32 0.48596
144137 GBM_2015_C 05/02/2015 08:39 0.68559
144144 GBM_2015_D 05/02/2015 08:40 0.47906
138819 GBM_2015_G 05/02/2015 08:49 0.50779
138819 GBM_2015_G 05/02/2015 09:09 0.64594
138824 GBM_2015_E 05/02/2015 09:16 0.71364
144144 GBM_2015_D 05/02/2015 09:19 0.71478
138824 GBM_2015_E 05/02/2015 09:23 0.73945
144148 GBM_2015_F 05/02/2015 09:31 0.73434
144137 GBM_2015_C 05/02/2015 09:53 0.74987
144144 GBM_2015_D 11/02/2015 14:20 0.74657
144148 GBM_2015_F 11/02/2015 14:30 0.73945
144139 GBM_2015_B 11/02/2015 18:46 0.71413
144139 GBM_2015_B 11/02/2015 19:22 0.72468
138819 GBM_2015_G 12/02/2015 10:56 0.73134
144148 GBM_2015_F 12/02/2015 11:05 0.74296
138819 GBM_2015_G 19/02/2015 09:59 0.73978
144148 GBM_2015_F 19/02/2015 10:22 0.74872
144137 GBM_2015_C 19/02/2015 10:55 0.75091
Certainly looks suspicious. How did you access their submission timestamps / performance?
", "rawMarkdown": "", "votes": null }, { "id": "64961", "postDate": "02/26/2015 16:49:34", "content": "@Maineiac, on the bottom of the leaderboards there is a link to download the CSV. The limitation there is that it only shows you the improvements made. If one was really motivated, you can use the asOf feature for the URL to see when submissions were made (though that's a bit overkill).
", "rawMarkdown": "", "votes": null }, { "id": "64962", "postDate": "02/26/2015 16:56:38", "content": "[quote=mlandry;64961]
\n@Maineiac, on the bottom of the leaderboards there is a link to download the CSV. The limitation there is that it only shows you the improvements made. If one was really motivated, you can use the asOf feature for the URL to see when submissions were made (though that's a bit overkill).
\n[/quote]
\n\nCool. Thanks!
", "rawMarkdown": "", "votes": null }, { "id": "64963", "postDate": "02/26/2015 17:17:14", "content": "I'd be shocked if those GBM... teams didn't have damn near identical code, which violates the no private sharing of code tenant.
\nIt doesn't affect me much, but it looks shady. Similar scores,names, and they all joined at the same time.
\nThanks for a great competition. Really learned a ton on this one.
", "rawMarkdown": "", "votes": null }, { "id": "64967", "postDate": "02/26/2015 17:28:30", "content": "Here's a timeseries graph of the data Gilberto posted (using the CSV from the leaderboard). See for yourself:
\n
![](\"https://storage.googleapis.com/kaggle-forum-message-attachments/64967/2014/GBM_After_Feb1.png?sv=2012-02-12&se=2015-03-01T17:29:18Z&sr=b&sp=r&sig=uGn6jPoFGxjwB1O8YHF9HIwOM5ezV1CmLLlX2nP%2FIAY%3D\")
[quote=Gilberto Titericz Junior;64950]
\nGBM_2015_* has a batman and other the enemy of batman rasal ghul.
\n[/quote]
\nLooks like the entire Justice League is present.
", "rawMarkdown": "", "votes": null }, { "id": "64978", "postDate": "02/26/2015 18:05:32", "content": "I wrote that I would be silent and let others voice their thoughts about GBM_2015_x, but the comment above by Triskelion led me to find some interesting links that I think may help us get to the bottom of this. The Justice_League_of_BCI team is also clustered together with the GBM_2015_x teams. The Justlice_League_of_BCI team is composed of Flam and alain rakotomamonjy. On the website of Alain Rakotomamony (https://sites.google.com/site/alainrakotomamonjy/), it is stated that Rémi Flamary is a past PhD student of his. On the website of Rémi Flamary (http://remi.flamary.com/teaching.html) it is stated that he is teaching a course called "Machine Learning and Brain-computer Interfaces - Master 1 GBM." Those initials GBM look awfully familiar. Dr. Flamary, is there a connection between the GBM_2015_x entries and your class?
", "rawMarkdown": "", "votes": null }, { "id": "64979", "postDate": "02/26/2015 18:17:06", "content": "@Trent: gotcha!
", "rawMarkdown": "", "votes": null }, { "id": "64980", "postDate": "02/26/2015 18:17:32", "content": "Wow! The Justice League of Kaggle is on the case! Kagglelock Holmes may be on to something here....
", "rawMarkdown": "", "votes": null }, { "id": "64987", "postDate": "02/26/2015 20:01:30", "content": "It could be that they all started from this code:
\nhttp://asi.insa-rouen.fr/enseignants/~arakoto/code/bciindex.html
\nNot sure if that's a violation since it's freely available to all prior to the contest.
", "rawMarkdown": "", "votes": null }, { "id": "64988", "postDate": "02/26/2015 20:02:38", "content": "Trent, this reminds me that I've been teaching in a Master program in the past and sometimes we used a Kaggle competition as a final project in the course on data processing. And also, the names of the teams included some keyword related to the course. But they were graded in a competitive way, based on their relative positions in the leaderboard, there was never collaboration among teams. We were always aware of the rules.
\nBy the way, these guys at Lyon are the hosts of the NER conference, I'm I right?
", "rawMarkdown": "", "votes": null }, { "id": "64992", "postDate": "02/26/2015 20:15:31", "content": "Re: belonging to a class, using names to identify, etc., this all makes sense. But why were so many submissions happening very close together in time? Perhaps a lab / discussion section?
", "rawMarkdown": "", "votes": null }, { "id": "65002", "postDate": "02/27/2015 00:44:35", "content": "The timing does seem suspicious, and I share others' concerns about possible cheating. I'm glad Kaggle is being proactive about removing cheaters. I'd also hate to see someone disqualified if they weren't actually cheating.
\nI think it's worth pointing out that because the public leaderboard had such a small sample and overcoming overfitting was one of the primary challenges in the competition, it was a little harder to cheat by take advantage of multiple submissions. Overfitting to the public leaderboard score would likely lead you the wrong way.
\nIf there was cheating, I would guess it would be non-public algorithm/solution/code sharing.
\nAnother way Kaggle might examine cheating would be to examine the correlation of residuals across the submissions. Unlikely high correlations might indicate some sort of shared method.
\nEDIT: You could also see if solutions are weighted ensembles of solutions from other teams.
", "rawMarkdown": "", "votes": null }, { "id": "65031", "postDate": "02/27/2015 14:27:22", "content": "Although we've gone off on the possibility of cheating here, I think it's great that classes are using Kaggle as a testing ground for the things they are learning. We don't want to discourage that.
\nTo avoid this sort of ambiguity in the future, perhaps it would make sense for the instructors of these classes to post something on the forum. For instance, they could indicate how (or if) the students are working together, and ensure the rest of the Kaggle community that the students have all read and understand the competition rules about sharing code, ideas, etc. outside of their team.
", "rawMarkdown": "", "votes": null }, { "id": "65032", "postDate": "02/27/2015 14:42:41", "content": "@Maineaic
\n\nThat's a great point. This group competed as multiple teams, but it appears to have operated more like one big team working from some benchmark code and tweaking params (due to the closeness in scores)
\nIf in fact some baseline code was given, it probably should have been made public.
\nIn some sense this approach allows one general model to make 2 final submissions per team, instead of just 2 overall
\nTo their credit - they did pretty well though.
", "rawMarkdown": "", "votes": null }, { "id": "65214", "postDate": "03/02/2015 04:01:01", "content": "Wendy,
\nAny information on when this competition's results will be validated and officially complete?
\nThanks!
", "rawMarkdown": "", "votes": null }, { "id": "65470", "postDate": "03/05/2015 02:49:53", "content": "Something's up . . . this is taking forever to finalize. Still can't see private scores for individual submissions. Still haven't updated kaggle points.
", "rawMarkdown": "", "votes": null }, { "id": "65472", "postDate": "03/05/2015 03:57:47", "content": "[quote=Jose M.;65367]
\nDear Wendy,
\nIn the thread you are mentioning, there's a long discussion regarding the teams named GBM_2015_X. There's evidence enough about they broke the rules. What is your position about that?
\n[/quote]
\nI hope I am wrong, but my sense is that silence from the administrator(s)/organizer(s) about what I believe are legitimate concerns raised about teams GBM_2015_X and the Justice League of BCI means that they are dismissive of our concerns. If this is the case, it is unfortunate since the success of Kaggle competitions is based on the participation of highly competitive-minded people. And for a competitive person, it is discouraging to compete when one feels like a reasonable effort is not made to follow up on questions about adherence to the rules raised by multiple competitors.
", "rawMarkdown": "", "votes": null }, { "id": "65481", "postDate": "03/05/2015 08:29:39", "content": "It is really disappointing. None of the important questions raised regarding to the cheaters removal or the submission of invited papers to NER are being answered. It looks like there are no competition admins at all.
", "rawMarkdown": "", "votes": null }, { "id": "65487", "postDate": "03/05/2015 10:51:27", "content": "I will be critical here, not because certain action has or not been taken for a matter which I think many people here have shared good insight about, but because the feedback from the admin has been inaccurate (e.g. when things will be ready) ,poor (e.g. no decent updates ) and out of place and time (as I see in another thread about "great the competition will finalize soon", it does not matter that so many people have concerns, we will ignore them).
\nAnyway, I do not think an update would have been such a difficult thing to do, even something like "we are working on this right now, have looked at the stuff you've posted and try to see whether there is any violation of the rules happening here".
", "rawMarkdown": "", "votes": null }, { "id": "65501", "postDate": "03/05/2015 12:36:15", "content": "A word from the organizers (apologizing for the delay in getting in touch)
\nAfter contacting the teams GBM_2015_A ... GBM_2015_F, it appears that they consist of 6 pairs of students from the same class, who participated in the contest for educational purposes. They worked independently on their submissions, but were provided with the same pre-processed data. If the pre-processing functions had been made public on the forum, they would not have been violating the rules. Since this was not the case, I suggest that these teams be withdrawn from the final leaderboard. The "Justice League of BCI" has also accepted to be withdrawn, because they were using the same pre-processing as the GBM_2015 groups.
\nWe acknowledge the nice work they produced to achieve these results.
\nAs a wrapup: could Kaggle please remove "GBM_2015_A", "GBM_2015_B", "GBM_2015_C", "GBM_2015_D", "GBM_2015_E", "GBM_2015_F" and "Justice League of BCI" from the final leaderboard ?
", "rawMarkdown": "", "votes": null }, { "id": "65529", "postDate": "03/05/2015 18:58:03", "content": "[quote=maucle;65501]
\nA word from the organizers (apologizing for the delay in getting in touch)
\nAfter contacting the teams GBM_2015_A ... GBM_2015_F, it appears that they consist of 6 pairs of students from the same class, who participated in the contest for educational purposes. They worked independently on their submissions, but were provided with the same pre-processed data. If the pre-processing functions had been made public on the forum, they would not have been violating the rules. Since this was not the case, I suggest that these teams be withdrawn from the final leaderboard. The "Justice League of BCI" has also accepted to be withdrawn, because they were using the same pre-processing as the GBM_2015 groups.
\nWe acknowledge the nice work they produced to achieve these results.
\nAs a wrapup: could Kaggle please remove "GBM_2015_A", "GBM_2015_B", "GBM_2015_C", "GBM_2015_D", "GBM_2015_E", "GBM_2015_F" and "Justice League of BCI" from the final leaderboard ?
\n[/quote]
\nWhat about GBM_2015_G?
", "rawMarkdown": "", "votes": null }, { "id": "65530", "postDate": "03/05/2015 19:07:36", "content": "[quote=KazAnova;65487]
\nI will be critical here, not because certain action has or not been taken for a matter which I think many people here have shared good insight about, but because the feedback from the admin has been inaccurate (e.g. when things will be ready) ,poor (e.g. no decent updates ) and out of place and time (as I see in another thread about "great the competition will finalize soon", it does not matter that so many people have concerns, we will ignore them).
\nAnyway, I do not think an update would have been such a difficult thing to do, even something like "we are working on this right now, have looked at the stuff you've posted and try to see whether there is any violation of the rules happening here".
\n[/quote]
\nI agree on everything. I think Kaggle is currently just short staffed.
\nhttp://www.wired.com/2015/02/data-science-darling-kaggle-cuts-one-third-staff/
", "rawMarkdown": "", "votes": null }, { "id": "65535", "postDate": "03/05/2015 20:12:09", "content": "[quote=Giulio;65530]
\nI agree on everything. I think Kaggle is currently just short staffed.
\nhttp://www.wired.com/2015/02/data-science-darling-kaggle-cuts-one-third-staff/
\n[/quote]
\nOh no no no! I would volunteer to help if I was considering skipping a competition (by not participating) , But I wont'!
", "rawMarkdown": "", "votes": null }, { "id": "65576", "postDate": "03/06/2015 08:33:14", "content": "[quote=Giulio;65530]
\nI agree on everything. I think Kaggle is currently just short staffed.
\nhttp://www.wired.com/2015/02/data-science-darling-kaggle-cuts-one-third-staff/
\n[/quote]
\nMany of the Competition Administrators don't belong to Kaggle's staff, they are rather on the organizer's side, so I don't think we can blame them for this lack of feedback, answers, actions, etc.
\n[quote=maucle;65530]
\nAs a wrapup: could Kaggle please remove "GBM_2015_A", "GBM_2015_B", "GBM_2015_C", "GBM_2015_D", "GBM_2015_E", "GBM_2015_F" and "Justice League of BCI" from the final leaderboard ?
\n[/quote]
\nSeriosly? Is the forum the communication channel between competition admins and Kaggle admins?
", "rawMarkdown": "", "votes": null }, { "id": "65626", "postDate": "03/06/2015 19:32:05", "content": "[quote=Jose M.;65576]
\nMany of the Competition Administrators don't belong to Kaggle's staff, they are rather on the organizer's side, so I don't think we can blame them for this lack of feedback, answers, actions, etc.
\n[/quote]
\nYes, but I highly doubt that the organizer's admins are allowed to push LB updated, remove cheaters, update Kaggle points. That's still something Kaggle does.
", "rawMarkdown": "", "votes": null }, { "id": "65652", "postDate": "03/06/2015 23:36:40", "content": "It looks like the competition is finalized now. The GBM_2015_X teams are removed but not "Justice League of BCI."
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 64902, "author_name": "trentb", "author_url": "", "post_date": "02/26/2015 01:38:14", "content": "Wendy,
\n\nAre you able to provide an explanation as to why the following competitors still exist after the removal of cheaters: GBM_2015_A, GBM_2015_B, GBM_2015_C, GBM_2015_D, GBM_2015_E, GBM_2015_F, and GBM_2015_G? These competitors are largely clustered together on the leaderboard which makes them look even further suspect.
\nThanks,
\nTrent
", "votes": null, "replies": [] }, { "id": 64915, "author_name": "wendykan", "author_url": "", "post_date": "02/26/2015 07:38:27", "content": "Trent,
\nI agree they look suspicious, but after cross referencing user logs and submission logs, it's not obvious that those are cheaters (unlike the other ones that we removed). We think the GBM_2015_x groups might be a class of students using similar team names.
", "votes": null, "replies": [] }, { "id": 64922, "author_name": "joseleiva", "author_url": "", "post_date": "02/26/2015 08:35:25", "content": "Wendy,
\n\nNone of the three first teams have included the "NER" keyword in their names. Does it mean that they will not receive the prize? (That's what the rules say)
\nI did not include the suffix because I hadn't big expectations. But now that I've finished 5th I could consider submitting a paper if the organizers were interested. But honestly, I'm so far away from the winners in performance that I don't think anybody would be interested...
", "votes": null, "replies": [] }, { "id": 64924, "author_name": "trentb", "author_url": "", "post_date": "02/26/2015 09:59:48", "content": "[quote=Wendy Kan;64915]
\nTrent,
\nI agree they look suspicious, but after cross referencing user logs and submission logs, it's not obvious that those are cheaters (unlike the other ones that we removed). We think the GBM_2015_x groups might be a class of students using similar team names.
\n[/quote]
\nWendy,
\nThank you for your response. I would agree that it is possible that GBM_2015_x groups could be a class of students, but it does not sound like there is any positive evidence for that, only no negative evidence against it. I would caution that it is also possible that posing as a class could be a sneaky new way of beating the system. But, assuming that GBM_2015_x groups are legitimately a class of students, it still makes me uneasy that all seven teams landed in the top 20 (top 8%). The class could have a fantastic instructor which I hope is the case, but the class could also be "reading from the same sheet of music" to which other competitors did not have access (e.g., some starter code provided by the instructor), which is expressly disallowed by the rules. I think I have sufficiently expressed my concerns, so I will be silent now and let other competitors respond if they share similar concerns and think further investigation is warranted.
\n\nThank you.
", "votes": null, "replies": [] }, { "id": 64926, "author_name": "kazanova", "author_url": "", "post_date": "02/26/2015 10:51:26", "content": "[quote=Wendy Kan;64915]
\nTrent,
\nI agree they look suspicious, but after cross referencing user logs and submission logs, it's not obvious that those are cheaters (unlike the other ones that we removed). We think the GBM_2015_x groups might be a class of students using similar team names.
\n[/quote]
\nThat looks really strange - even if they are a class and different people, they have suspiciously pretty close scores - to the extend that some collaboration was unavoidable !
", "votes": null, "replies": [] }, { "id": 64928, "author_name": "titericz", "author_url": "", "post_date": "02/26/2015 10:58:18", "content": "I challenge each one of GBM_2015_x teams to describe their approach!!! What's your CV approach? What is your CV score? Preprocess? trainning method? Post-process?
\nIn a competition with a small dataset, small number of instances in public LB, small number of subjects, thousand of features, time dependente of variables and dissimilarity intersubject, it's very hard to generalize well. I say that based in previous similar competitions.
\nWell, they are 7 teams, under top 20, in their first Kaggle competition, performing better than many Kaggle masters and with a very close score. It's at least suspicious...
", "votes": null, "replies": [] }, { "id": 64929, "author_name": "cdjbee", "author_url": "", "post_date": "02/26/2015 11:01:57", "content": "Trent,
\nthanks for bringing this up. I also have an uneasy feeling in this case (my apologies to you guys GBM_2015_A, GBM_2015_B, GBM_2015_C, GBM_2015_D, GBM_2015_E, GBM_2015_F, and GBM_2015_G in case you are really individual people who came up with individual solutions).
\nGiven how discouraging it can be to the honest competitors if cheating does prevail at a significant level, I would appreciate any extra effort that Kaggle can do to establish that these submissions are indeed in accordance with the rules.
\nCheers
", "votes": null, "replies": [] }, { "id": 64930, "author_name": "kazanova", "author_url": "", "post_date": "02/26/2015 11:02:20", "content": "[quote=Gilberto Titericz Junior;64928]
\nI challenge each one of GBM_2015_x teams to describe their approach!!! What's your CV approach? What is your CV score? Preprocess? trainning method? Post-process?
\nIn a competition with a small dataset, small number of instances in public LB, small number of subjects, thousand of features, time dependente of variables and dissimilarity intersubject, it's very hard to generalize well. I say that based in previous similar competitions.
\nWell, they are 7 teams, under top 20, in their first Kaggle competition, performing better than many Kaggle masters and with a very close score. It's at least suspicious...
\n[/quote]
\n\nWell said. And it is not about the handful of extra kaggle points we are going to get from their removal, but more like what is right do...
", "votes": null, "replies": [] }, { "id": 64937, "author_name": "clustifier", "author_url": "", "post_date": "02/26/2015 12:06:38", "content": "Agree!!!
\nThe scores are too close to be unrelated. No matter if it's a one individual or a group from the same class, the scores are suspiciously too correlated.
", "votes": null, "replies": [] }, { "id": 64939, "author_name": "joseleiva", "author_url": "", "post_date": "02/26/2015 12:12:12", "content": "The sad thing about this issue is that the fraud wouldn't have been detected if they had chosen a different, less suspicious, set of names.
", "votes": null, "replies": [] }, { "id": 64942, "author_name": "triskelion", "author_url": "", "post_date": "02/26/2015 12:49:40", "content": "I think the GBM_2015 teams are PhD. students from a university in Nice, France. I think we foiled their plan to spell out the alphabet on the leaderboard. Well done to the all-French cluster!
", "votes": null, "replies": [] }, { "id": 64950, "author_name": "titericz", "author_url": "", "post_date": "02/26/2015 15:15:07", "content": "The team GBM_2015_G have 2 brazilian names: "Yuri Oliveira" and "Mariana Rufino de Andrade", other GBM_2015_* has a batman and other the enemy of batman rasal ghul.
\nLook at the timestamps, for me it is clear now:
\n138821 GBM_2015_A 15/01/2015 10:36 0.68875
138824 GBM_2015_E 15/01/2015 10:55 0.69993
144137 GBM_2015_C 05/02/2015 08:27 0.68516
144139 GBM_2015_B 05/02/2015 08:32 0.48596
144137 GBM_2015_C 05/02/2015 08:39 0.68559
144144 GBM_2015_D 05/02/2015 08:40 0.47906
138819 GBM_2015_G 05/02/2015 08:49 0.50779
138819 GBM_2015_G 05/02/2015 09:09 0.64594
138824 GBM_2015_E 05/02/2015 09:16 0.71364
144144 GBM_2015_D 05/02/2015 09:19 0.71478
138824 GBM_2015_E 05/02/2015 09:23 0.73945
144148 GBM_2015_F 05/02/2015 09:31 0.73434
144137 GBM_2015_C 05/02/2015 09:53 0.74987
144144 GBM_2015_D 11/02/2015 14:20 0.74657
144148 GBM_2015_F 11/02/2015 14:30 0.73945
144139 GBM_2015_B 11/02/2015 18:46 0.71413
144139 GBM_2015_B 11/02/2015 19:22 0.72468
138819 GBM_2015_G 12/02/2015 10:56 0.73134
144148 GBM_2015_F 12/02/2015 11:05 0.74296
138819 GBM_2015_G 19/02/2015 09:59 0.73978
144148 GBM_2015_F 19/02/2015 10:22 0.74872
144137 GBM_2015_C 19/02/2015 10:55 0.75091
Certainly looks suspicious. How did you access their submission timestamps / performance?
", "votes": null, "replies": [] }, { "id": 64961, "author_name": "mlandry", "author_url": "", "post_date": "02/26/2015 16:49:34", "content": "@Maineiac, on the bottom of the leaderboards there is a link to download the CSV. The limitation there is that it only shows you the improvements made. If one was really motivated, you can use the asOf feature for the URL to see when submissions were made (though that's a bit overkill).
", "votes": null, "replies": [] }, { "id": 64962, "author_name": "maineiac", "author_url": "", "post_date": "02/26/2015 16:56:38", "content": "[quote=mlandry;64961]
\n@Maineiac, on the bottom of the leaderboards there is a link to download the CSV. The limitation there is that it only shows you the improvements made. If one was really motivated, you can use the asOf feature for the URL to see when submissions were made (though that's a bit overkill).
\n[/quote]
\n\nCool. Thanks!
", "votes": null, "replies": [] }, { "id": 64963, "author_name": "scharf", "author_url": "", "post_date": "02/26/2015 17:17:14", "content": "I'd be shocked if those GBM... teams didn't have damn near identical code, which violates the no private sharing of code tenant.
\nIt doesn't affect me much, but it looks shady. Similar scores,names, and they all joined at the same time.
\nThanks for a great competition. Really learned a ton on this one.
", "votes": null, "replies": [] }, { "id": 64967, "author_name": "maineiac", "author_url": "", "post_date": "02/26/2015 17:28:30", "content": "Here's a timeseries graph of the data Gilberto posted (using the CSV from the leaderboard). See for yourself:
\n
[quote=Gilberto Titericz Junior;64950]
\nGBM_2015_* has a batman and other the enemy of batman rasal ghul.
\n[/quote]
\nLooks like the entire Justice League is present.
", "votes": null, "replies": [] }, { "id": 64978, "author_name": "trentb", "author_url": "", "post_date": "02/26/2015 18:05:32", "content": "I wrote that I would be silent and let others voice their thoughts about GBM_2015_x, but the comment above by Triskelion led me to find some interesting links that I think may help us get to the bottom of this. The Justice_League_of_BCI team is also clustered together with the GBM_2015_x teams. The Justlice_League_of_BCI team is composed of Flam and alain rakotomamonjy. On the website of Alain Rakotomamony (https://sites.google.com/site/alainrakotomamonjy/), it is stated that Rémi Flamary is a past PhD student of his. On the website of Rémi Flamary (http://remi.flamary.com/teaching.html) it is stated that he is teaching a course called "Machine Learning and Brain-computer Interfaces - Master 1 GBM." Those initials GBM look awfully familiar. Dr. Flamary, is there a connection between the GBM_2015_x entries and your class?
", "votes": null, "replies": [] }, { "id": 64979, "author_name": "titericz", "author_url": "", "post_date": "02/26/2015 18:17:06", "content": "@Trent: gotcha!
", "votes": null, "replies": [] }, { "id": 64980, "author_name": "maineiac", "author_url": "", "post_date": "02/26/2015 18:17:32", "content": "Wow! The Justice League of Kaggle is on the case! Kagglelock Holmes may be on to something here....
", "votes": null, "replies": [] }, { "id": 64987, "author_name": "inversion", "author_url": "", "post_date": "02/26/2015 20:01:30", "content": "It could be that they all started from this code:
\nhttp://asi.insa-rouen.fr/enseignants/~arakoto/code/bciindex.html
\nNot sure if that's a violation since it's freely available to all prior to the contest.
", "votes": null, "replies": [] }, { "id": 64988, "author_name": "joseleiva", "author_url": "", "post_date": "02/26/2015 20:02:38", "content": "Trent, this reminds me that I've been teaching in a Master program in the past and sometimes we used a Kaggle competition as a final project in the course on data processing. And also, the names of the teams included some keyword related to the course. But they were graded in a competitive way, based on their relative positions in the leaderboard, there was never collaboration among teams. We were always aware of the rules.
\nBy the way, these guys at Lyon are the hosts of the NER conference, I'm I right?
", "votes": null, "replies": [] }, { "id": 64992, "author_name": "maineiac", "author_url": "", "post_date": "02/26/2015 20:15:31", "content": "Re: belonging to a class, using names to identify, etc., this all makes sense. But why were so many submissions happening very close together in time? Perhaps a lab / discussion section?
", "votes": null, "replies": [] }, { "id": 65002, "author_name": "danielyoo", "author_url": "", "post_date": "02/27/2015 00:44:35", "content": "The timing does seem suspicious, and I share others' concerns about possible cheating. I'm glad Kaggle is being proactive about removing cheaters. I'd also hate to see someone disqualified if they weren't actually cheating.
\nI think it's worth pointing out that because the public leaderboard had such a small sample and overcoming overfitting was one of the primary challenges in the competition, it was a little harder to cheat by take advantage of multiple submissions. Overfitting to the public leaderboard score would likely lead you the wrong way.
\nIf there was cheating, I would guess it would be non-public algorithm/solution/code sharing.
\nAnother way Kaggle might examine cheating would be to examine the correlation of residuals across the submissions. Unlikely high correlations might indicate some sort of shared method.
\nEDIT: You could also see if solutions are weighted ensembles of solutions from other teams.
", "votes": null, "replies": [] }, { "id": 65031, "author_name": "maineiac", "author_url": "", "post_date": "02/27/2015 14:27:22", "content": "Although we've gone off on the possibility of cheating here, I think it's great that classes are using Kaggle as a testing ground for the things they are learning. We don't want to discourage that.
\nTo avoid this sort of ambiguity in the future, perhaps it would make sense for the instructors of these classes to post something on the forum. For instance, they could indicate how (or if) the students are working together, and ensure the rest of the Kaggle community that the students have all read and understand the competition rules about sharing code, ideas, etc. outside of their team.
", "votes": null, "replies": [] }, { "id": 65032, "author_name": "scharf", "author_url": "", "post_date": "02/27/2015 14:42:41", "content": "@Maineaic
\n\nThat's a great point. This group competed as multiple teams, but it appears to have operated more like one big team working from some benchmark code and tweaking params (due to the closeness in scores)
\nIf in fact some baseline code was given, it probably should have been made public.
\nIn some sense this approach allows one general model to make 2 final submissions per team, instead of just 2 overall
\nTo their credit - they did pretty well though.
", "votes": null, "replies": [] }, { "id": 65214, "author_name": "mlandry", "author_url": "", "post_date": "03/02/2015 04:01:01", "content": "Wendy,
\nAny information on when this competition's results will be validated and officially complete?
\nThanks!
", "votes": null, "replies": [] }, { "id": 65470, "author_name": "inversion", "author_url": "", "post_date": "03/05/2015 02:49:53", "content": "Something's up . . . this is taking forever to finalize. Still can't see private scores for individual submissions. Still haven't updated kaggle points.
", "votes": null, "replies": [] }, { "id": 65472, "author_name": "trentb", "author_url": "", "post_date": "03/05/2015 03:57:47", "content": "[quote=Jose M.;65367]
\nDear Wendy,
\nIn the thread you are mentioning, there's a long discussion regarding the teams named GBM_2015_X. There's evidence enough about they broke the rules. What is your position about that?
\n[/quote]
\nI hope I am wrong, but my sense is that silence from the administrator(s)/organizer(s) about what I believe are legitimate concerns raised about teams GBM_2015_X and the Justice League of BCI means that they are dismissive of our concerns. If this is the case, it is unfortunate since the success of Kaggle competitions is based on the participation of highly competitive-minded people. And for a competitive person, it is discouraging to compete when one feels like a reasonable effort is not made to follow up on questions about adherence to the rules raised by multiple competitors.
", "votes": null, "replies": [] }, { "id": 65481, "author_name": "joseleiva", "author_url": "", "post_date": "03/05/2015 08:29:39", "content": "It is really disappointing. None of the important questions raised regarding to the cheaters removal or the submission of invited papers to NER are being answered. It looks like there are no competition admins at all.
", "votes": null, "replies": [] }, { "id": 65487, "author_name": "kazanova", "author_url": "", "post_date": "03/05/2015 10:51:27", "content": "I will be critical here, not because certain action has or not been taken for a matter which I think many people here have shared good insight about, but because the feedback from the admin has been inaccurate (e.g. when things will be ready) ,poor (e.g. no decent updates ) and out of place and time (as I see in another thread about "great the competition will finalize soon", it does not matter that so many people have concerns, we will ignore them).
\nAnyway, I do not think an update would have been such a difficult thing to do, even something like "we are working on this right now, have looked at the stuff you've posted and try to see whether there is any violation of the rules happening here".
", "votes": null, "replies": [] }, { "id": 65501, "author_name": "maucle", "author_url": "", "post_date": "03/05/2015 12:36:15", "content": "A word from the organizers (apologizing for the delay in getting in touch)
\nAfter contacting the teams GBM_2015_A ... GBM_2015_F, it appears that they consist of 6 pairs of students from the same class, who participated in the contest for educational purposes. They worked independently on their submissions, but were provided with the same pre-processed data. If the pre-processing functions had been made public on the forum, they would not have been violating the rules. Since this was not the case, I suggest that these teams be withdrawn from the final leaderboard. The "Justice League of BCI" has also accepted to be withdrawn, because they were using the same pre-processing as the GBM_2015 groups.
\nWe acknowledge the nice work they produced to achieve these results.
\nAs a wrapup: could Kaggle please remove "GBM_2015_A", "GBM_2015_B", "GBM_2015_C", "GBM_2015_D", "GBM_2015_E", "GBM_2015_F" and "Justice League of BCI" from the final leaderboard ?
", "votes": null, "replies": [] }, { "id": 65529, "author_name": "clustifier", "author_url": "", "post_date": "03/05/2015 18:58:03", "content": "[quote=maucle;65501]
\nA word from the organizers (apologizing for the delay in getting in touch)
\nAfter contacting the teams GBM_2015_A ... GBM_2015_F, it appears that they consist of 6 pairs of students from the same class, who participated in the contest for educational purposes. They worked independently on their submissions, but were provided with the same pre-processed data. If the pre-processing functions had been made public on the forum, they would not have been violating the rules. Since this was not the case, I suggest that these teams be withdrawn from the final leaderboard. The "Justice League of BCI" has also accepted to be withdrawn, because they were using the same pre-processing as the GBM_2015 groups.
\nWe acknowledge the nice work they produced to achieve these results.
\nAs a wrapup: could Kaggle please remove "GBM_2015_A", "GBM_2015_B", "GBM_2015_C", "GBM_2015_D", "GBM_2015_E", "GBM_2015_F" and "Justice League of BCI" from the final leaderboard ?
\n[/quote]
\nWhat about GBM_2015_G?
", "votes": null, "replies": [] }, { "id": 65530, "author_name": "adjgiulio", "author_url": "", "post_date": "03/05/2015 19:07:36", "content": "[quote=KazAnova;65487]
\nI will be critical here, not because certain action has or not been taken for a matter which I think many people here have shared good insight about, but because the feedback from the admin has been inaccurate (e.g. when things will be ready) ,poor (e.g. no decent updates ) and out of place and time (as I see in another thread about "great the competition will finalize soon", it does not matter that so many people have concerns, we will ignore them).
\nAnyway, I do not think an update would have been such a difficult thing to do, even something like "we are working on this right now, have looked at the stuff you've posted and try to see whether there is any violation of the rules happening here".
\n[/quote]
\nI agree on everything. I think Kaggle is currently just short staffed.
\nhttp://www.wired.com/2015/02/data-science-darling-kaggle-cuts-one-third-staff/
", "votes": null, "replies": [] }, { "id": 65535, "author_name": "kazanova", "author_url": "", "post_date": "03/05/2015 20:12:09", "content": "[quote=Giulio;65530]
\nI agree on everything. I think Kaggle is currently just short staffed.
\nhttp://www.wired.com/2015/02/data-science-darling-kaggle-cuts-one-third-staff/
\n[/quote]
\nOh no no no! I would volunteer to help if I was considering skipping a competition (by not participating) , But I wont'!
", "votes": null, "replies": [] }, { "id": 65576, "author_name": "joseleiva", "author_url": "", "post_date": "03/06/2015 08:33:14", "content": "[quote=Giulio;65530]
\nI agree on everything. I think Kaggle is currently just short staffed.
\nhttp://www.wired.com/2015/02/data-science-darling-kaggle-cuts-one-third-staff/
\n[/quote]
\nMany of the Competition Administrators don't belong to Kaggle's staff, they are rather on the organizer's side, so I don't think we can blame them for this lack of feedback, answers, actions, etc.
\n[quote=maucle;65530]
\nAs a wrapup: could Kaggle please remove "GBM_2015_A", "GBM_2015_B", "GBM_2015_C", "GBM_2015_D", "GBM_2015_E", "GBM_2015_F" and "Justice League of BCI" from the final leaderboard ?
\n[/quote]
\nSeriosly? Is the forum the communication channel between competition admins and Kaggle admins?
", "votes": null, "replies": [] }, { "id": 65626, "author_name": "adjgiulio", "author_url": "", "post_date": "03/06/2015 19:32:05", "content": "[quote=Jose M.;65576]
\nMany of the Competition Administrators don't belong to Kaggle's staff, they are rather on the organizer's side, so I don't think we can blame them for this lack of feedback, answers, actions, etc.
\n[/quote]
\nYes, but I highly doubt that the organizer's admins are allowed to push LB updated, remove cheaters, update Kaggle points. That's still something Kaggle does.
", "votes": null, "replies": [] }, { "id": 65652, "author_name": "trentb", "author_url": "", "post_date": "03/06/2015 23:36:40", "content": "It looks like the competition is finalized now. The GBM_2015_X teams are removed but not "Justice League of BCI."
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "64899": "", "64902": "", "64915": "", "64922": "", "64924": "", "64926": "", "64928": "", "64929": "", "64930": "", "64937": "", "64939": "", "64942": "", "64950": "", "64958": "", "64961": "", "64962": "", "64963": "", "64967": "", "64970": "", "64978": "", "64979": "", "64980": "", "64987": "", "64988": "", "64992": "", "65002": "", "65031": "", "65032": "", "65214": "", "65470": "", "65472": "", "65481": "", "65487": "", "65501": "", "65529": "", "65530": "", "65535": "", "65576": "", "65626": "", "65652": "" }, "source": "meta" }