Thank you for your hard work and participation in this competition, and making it a tremendous success!
\r\nWe at Kaggle wanted to acknowledge two participants who built great models, but who we were regretfully unable to recognize more formally with prize placements and press releases, due to errors with their final submissions.
\r\nThe first, dxsx, could have received a score of 8.30650. However, all the predicted times in his final submission were based in the wrong time zone.
\r\nThe second, __mtb__, had a model that could have received a score of 8.40080 on the final leaderboard. However, he had selected an old public leaderboard submission as his final entry, and our system\r\n calculated his score based on this selection.
\r\nWe regret that these two issues prevented \r\ndxsx and __mtb__ from being among the prize winners. Issues like these are unfortunate for everyone involved, but we wanted to be sure we publically acknowledged these two worthwhile contributions. This\r\n isn’t the first time we’ve had participants put in a tremendous amount of effort, but make a mistake that leaves them out of the money.
\r\nWhenever this happens, we make an effort to improve our tools and processes in order to decrease the likelihood that these issues occur in the future.
", "messages": [ { "id": "22679", "postDate": "04/12/2013 05:52:53", "content": "Thank you for your hard work and participation in this competition, and making it a tremendous success!
\r\nWe at Kaggle wanted to acknowledge two participants who built great models, but who we were regretfully unable to recognize more formally with prize placements and press releases, due to errors with their final submissions.
\r\nThe first, dxsx, could have received a score of 8.30650. However, all the predicted times in his final submission were based in the wrong time zone.
\r\nThe second, __mtb__, had a model that could have received a score of 8.40080 on the final leaderboard. However, he had selected an old public leaderboard submission as his final entry, and our system\r\n calculated his score based on this selection.
\r\nWe regret that these two issues prevented \r\ndxsx and __mtb__ from being among the prize winners. Issues like these are unfortunate for everyone involved, but we wanted to be sure we publically acknowledged these two worthwhile contributions. This\r\n isn’t the first time we’ve had participants put in a tremendous amount of effort, but make a mistake that leaves them out of the money.
\r\nWhenever this happens, we make an effort to improve our tools and processes in order to decrease the likelihood that these issues occur in the future.
", "rawMarkdown": "", "votes": null }, { "id": "22686", "postDate": "04/12/2013 11:54:24", "content": "First off, a sincere congrats to all of the other competitors. This was a tough competition!
\r\nBen:
\r\ndxsx’s error, as unfortunate as it was, is a problem with his model. The error made in my case is with *your* software. I followed your directions\r\n exactly:
\r\n\r\n
\r\n\r\nThe final evaluation set is now up on the data page.
\r\nYou may make a submission on the final evaluation set regardless of whether you've submitted a final model or hash. However, you are only eligible for prize money if you submitted a final model and use the same model to make predictions on this new set.
\r\nWhen you make a submission on this new set, a public score of "0.0000" without errors means the submission was processed correctly.
\r\nPlease let us know if you find any issues with the data on this thread.
\r\nGood luck!
\r\n\r\n
As I have been saying over email repeatedly over the past 8 days, and yesterday on the phone with you and the Sorkin, there is no checkbox to check! I scored the final file and submitted it. I received all zero's like your\r\n directions read, and in your own words: this means the '... was processed correctly'. It is completely unambiguous what my final submission is. Any reasonable person understands this.
\r\nAs competition admins, you need to be able to distinguish between errors in your software and errors in the models. It is clear you are not properly doing this here.
\r\nLike I said in my email last night, it is still my expectation that you do the following by 5pm EST this afternoon:
\r\n1. The final leaderboard updated - moving me into 4th place showing my final score of 8.40080
\r\n2. The contact person at GE who I can coordinate with to get my $30,000
\r\n3. The 'winners page' updated, with me moved into 4th place
\r\n\r\n
Do the right thing!
", "rawMarkdown": "", "votes": null }, { "id": "22693", "postDate": "04/12/2013 15:31:27", "content": "Wow, Matt that's hard to believe. As a fellow competitor, I thought the submission process was unclear. I would agree this is a bug with Kaggle's scoring process.
\r\nOne of the real losers here is GE.
\r\nBen, is there a process for competitors to resolve disputes, after Kaggle declares prize winners?
", "rawMarkdown": "", "votes": null }, { "id": "22695", "postDate": "04/12/2013 16:07:52", "content": "I also find all of this hard to believe (not Matt's side of this story but the fact that Kaggle couldn't resolve it).
\r\n
\r\nAs a new member to Kaggle for this project I can certainly say that there have been highs and lows from this competition process. I really enjoy the concept of Kaggle, the competitive nature of the process, and the atmosphere for learning. But it seems to\r\n me that Kaggle still has a lot of kinks to work out in their process and procedures for competitions. I don't know how the other competitions played out, as this is my first, but there were several sets of confusing guidelines and missed deadlines (i.e. promised\r\n deadlines that weren't made by Kaggle).
I can understand that these kind of projects will have hiccups, but if Kaggle and a veteran of Kaggle (Matt is very high in the Kaggle rankings) can't work out something like this I think that is a major flaw in the whole process. What hope would a new\r\n member to the community have if something like this were to happen?
\r\nMatt - if Kaggle/GE can't resolve this, and I were you, I might reach out to someone like Passur http://www.passur.com/ or Honeywell http://www.honeywell.com and explain the situation. I wouldn't be surprised if someone at that company\r\n would be interested in your algorithms. It would seem as though they already have an interest in Flight Management Systems and aircraft arrival predictions. In addition it would probably be a great piece of PR for them.
\r\nAs an aside, I think the fact that the results aren’t released limits users learning experiences. I don’t feel like I can learn from mistakes as well as I had hoped would be possible. As a user who doesn’t have a strong “big data” background, if I can’t\r\n learn there isn’t an incentive to stick around and use the site. Due to the limited number of winners, and the fact that most of the top Kaggle members win multiple prizes, most users will continue to come back for the challenge and learning, not the prizes. \r\n Maybe the Kaggle model is just to hook up a small set of “Kaggle Masters” to companies. Well, I guess then it doesn’t matter if your user base grows and learns. My belief is that having a larger set of continually improving users from various fields will\r\n provide your customers (i.e. companies and people that host competitions) with better and more interesting results.
", "rawMarkdown": "", "votes": null }, { "id": "22700", "postDate": "04/12/2013 17:25:06", "content": "rjm5, ATCGuy - I appreciate the support. I would love to hear what the other competitors think about this topic.
\r\nIt kills me that a bug in Kaggle's own process is what is holding this whole thing up ...
\r\n\r\n
I should also say that I emailed Ben an hour after the offical results were posted. Then I emailed him again in the afternoon. Then I emailed the community manager because Ben never responded. Finally, the first I heard back\r\n from Ben was Friday night telling me the same thing he said above.
\r\nKaggle, here are a few tips for future competitions:
\r\n1. Bad news doesn't get better with age
\r\n2. Respond to the competitors inquiries
\r\n3. *Don't* announce the winners in a joint press release until the competitors have had a chance to dispute the results. You certainly painted yourself into a corner this time
\r\n4. *Don't* offer the competitors money and kaggle points to ''unofficially' resolve the matter. That was an insult to my integrity and is certainly not fair to the Sponser or other competitors.
", "rawMarkdown": "", "votes": null }, { "id": "22705", "postDate": "04/12/2013 17:49:52", "content": "Ben, did you offer money to keep this quite? Please respond.
", "rawMarkdown": "", "votes": null }, { "id": "22715", "postDate": "04/12/2013 20:51:50", "content": "FWIW, the data submission process was indeed fairly convoluted (and I suppose there could´ve been more people who got lost in it along the way). I believe a clarification from the admins on whether there was a software error glitch on top of it, or not,\r\n would benefit all participants. The silence is having an opposite effect.
", "rawMarkdown": "", "votes": null }, { "id": "22810", "postDate": "04/14/2013 23:53:12", "content": "Thanks Ben for your acknowledgment.
\r\nFirst, I was surprised how important is for Kaggle to strictly follow the competition deadlines over having better models (as I corrected my submission few hours after the submission deadline). Not to mention, almost all the\r\n deadlines were missed from the Kaggle side in this competition. Anyways, it was my own fault!
\r\nIt is very sad to hear what happened to _mtb_. It is just unfair to him to lose his position because of some systematic error.
\r\nI can understand that creating a perfect competition platform is extremely hard. I do appreciate your constant efforts in improving the competitions’ setup. Any competition set up has its own issues. As you mentioned, this\r\n is not the first time that you are disappointing competitors.
\r\nHowever, I expected that Kaggle to be more flexible in handing issues like what happened to _mtb_.
\r\n\r\n
\r\n
\r\n
", "rawMarkdown": "", "votes": null }, { "id": "22867", "postDate": "04/15/2013 23:05:25", "content": "
This is a 2 phase competition so there are 2 separate file types that need to be scored - the public leaderboard file and the final leaderboard file. During the public leaderboard portion of the competition, yes I used the checkbox to keep track of my best\r\n submission. Then, following the directions Ben posted, I scored the final leaderboard file and uploaded it. As Ben's direction describe, I received the all zeros which in his own words means the submission was properly received.\r\n
\r\n
\r\nWhat on earth is a checkbox doing on the submission screen of a two phase competition? There is no ambiguity - there is only a single *file type* that is valid for the final leaderboard file. Why on earth is your software not smart enough to figure that out?\r\n
\r\n
\r\nAlso - why didn't the default selection logic kick in? Because only final leaderboard files are valid for the final submission, why didn't your software pick the *only* valid option?
Certainly I am not going to be competing in anymore Kaggle competitions. I do not trust your company to properly host a competition.
\r\nI am done with Kaggle.
", "rawMarkdown": "", "votes": null }, { "id": "22889", "postDate": "04/16/2013 11:55:31", "content": "Anthony:
\r\nHere is my more detailed response. Let's bring some transparency into the equation by keeping the discussion in the forums. I will respond to your email and ask some serious questions that need to be answered. Please do that here instead of over email.
\r\nThe text you have highlighted from the submission page is only valid for a single phase competition.
\r\nThe submission page stated: 'Note: You can select up to 1 submission that will be used to calculate your final leaderboard score. If you do not select them, up to 1 entry will be chosen for you based on your most recent submissions. Your final score\r\n will not be based on the same exact subset data as the public leaderboard, but rather a different private data subset of your full submission. Your public score is only a rough indication of what your final score might be. You should choose an entry that will\r\n most likely be best overall, and not necessarily just on the public subset.'\r\n
This is not accurate text for the two phase submission. There are no subsets, instead there are two separate files. The public leaderboard file and the final leaderboard file. A submission is for exactly one or the other, but not for both. During the final\r\n submission period of a two phase competition, this text should not be displayed because it is not accurate. If you don't understand why this text doesn't make sens for a two phase competition, run it by your employee's that are competitors.
\r\nLet me outline what I did for the final submission again, because from you email I don't believe you are familiar with how the two phase competition works (Rich didn't understand it either)
\r\nI know the wiki page has a forula, but it doesn't include the additional points awarded to keep competitors quiet. It seems like you would want to update it to include this point. Especially now that companies like the NYTimes and American Express are using\r\n Kaggle rankings when hiring data scientists
\r\nAnthony, it is clear to me what ultimately happened here. You painted yourself into a corner because you announced the results in a press release with GE the same day the rest of the competitors saw the results. I know there is quite a marketing push behind\r\n GE's quests and having to change the results after the press releases and magazines articles were written leaves egg on Kaggle's face. The problem is that the competition portion of your business is built on trust and integrity.
\r\nThe obvious analogy is to sporting events - if the referee can not be trusted than the games don't matter.
\r\nThat is why I will not compete in anymore Kaggle competitions.
\r\n__mtb__
\r\np.s. this forum software is terrible. it always seems to mangle the posts.
\r\nWow. I enter the quest forum after a week and what a surprise to read that post. I think you have your point. If you made only 1 final submission there is no ambiguity. It should have been chosen. Kaggle competitions should be closer to real life in my opinion.\r\n Normally after seeing a skyhigh (which is bad) performance you should be able to correct it. Seeing no performance as in your case should give you a hint that something is wrong. By "real life" I mean that such cases should not diminish 3 months of hard work.\r\n When competing I had a nightmarish vision that something like this would happen to us which is not a good thing.
\r\nI kind of didn't like that you have dismissed dxsx's performance. In "real life" (not Kaggle) he was better. The fact that your mistake/bug was different than his is circumstantial.
", "rawMarkdown": "", "votes": null }, { "id": "22900", "postDate": "04/16/2013 15:34:37", "content": "Pawel
\r\nThanks for pointing that out about dxsx - that is certainly not my intention to dismiss his performance. I will post here what I sent to him in a private email a couple days back:
\r\n\r\n\r\ndxsx:
\r\nTo be honest, the mistake you are talking about was keeping me up at night the last couple weeks of the competition. I really think Kaggle needs to do something different in these 2 phase competitions to let the competitors know more about the submission. Maybe\r\n like let them have up to 5 submissions or something, but then show them the score over say 25% of the dataset. Obviously you wouldn't be able to overfit, but you could still prevent what happened to you. I bet they will change this in the future. It sucks\r\n for both the competitors AND the sponsors to have models not considered because of this.
\r\n
Sorry dxsx if I was dismissive of your hard work! That was not my intentions.
\r\nedit: formatting
", "rawMarkdown": "", "votes": null }, { "id": "22921", "postDate": "04/16/2013 21:53:07", "content": "A competitor emailed me a couple of things that I wanted to share. First he was able to summarize my issue with the submission bug very nicely in a couple of statements and 3 bullet points. I have been working for 2 weeks trying to explain this - yet I was\r\n not able to.
\r\nClearly the Kaggle platform is not the most bug free software in the universe. Let me point these events:
\r\nSo your situation should be treated as one of the bugs. I'm sure that knowing what they know now they would correct this in the future. It's costly but still it is not your fault.
\r\nAnd his other point was that 'You should keep your tone less emotional' . I appologize for that. I am not going to go back and edit old posts or anything - what was said is said, but I will make sure I cut down the commentary and focus on\r\n the facts.
\r\nAnthony, I am certainly looking forward to having an open, civil conversation in this forum, with not just you and me, but with all of the competitors. I posted the questions that I feel need to be answered. Please familiarize yourself with the details of\r\n the two-phase competitions and make sure you take a close look at the submission page and understand why the text you quoted does not make sense for a final submission in a two-phase competition.
\r\nI am looking forward for your response.
", "rawMarkdown": "", "votes": null }, { "id": "22923", "postDate": "04/16/2013 23:28:47", "content": "Also, it sounds like similar issues with the submission process for two-phase competitions was encountered for the fast iron competition. This is highly relevant:
\r\nLike Leustagos says, the two stage submission process in it's current state is really poorly implemented. It was clearly duct taped onto the current submission system sloppily and because of that there are huge usability issues with submitting solutions\r\n and models. In the Bulldozer competition we had numerous issues with the submission process and the responses from kaggle were very slow and often not that helpful. I really like kaggle but I think they have some serious usability issues and problems with\r\n their current support process.
\r\n-- pjreddie
\r\nHere is the original \r\nlink.
\r\n\r\n
Anthony, according to the competitors (who are also the users), there are numerous bugs with the submission process and submission page for two-phase competitions. I believe your assertion that 'There was no bug in the\r\n software.' is incorrect.
", "rawMarkdown": "", "votes": null }, { "id": "22932", "postDate": "04/17/2013 00:38:35", "content": "First of all: Kaggle do an amazing job and I have to tell that this site is probably my favorite place in the web.
\r\nI understand that you guys (Kaggle) had an extremely hard competition this time, the dataset was hard to prepare and process correctly and the competition prize pool and media attention was also higher than usual.
\r\nWith all the respect I think you did not choose the best decision after the competition have ended. I think the problems with\r\ndxsx and __mtb__'s solutions should have been discovered during the 1.5 month while you were validating the final results.
\r\nEven if the problems discovered later than the official announcement I think you should have been far more flexible to reward the two competitor who created excellent solutions but slipped on a ( their or yours?) banana peel.
\r\nCreating a forum post is basically free but don't solve the problem. The e-mail received by __mtb__ is also a mixed message (nit-pciking, sympathy, legal shield...).
\r\nI understand that you do not want to hurt the prizewinner competitors who also created amazing solutions and they definetly deserve the prize amount and the ranking they got.\r\nBut I can not beleive that you do not see the huge problem with this situation and you could not solve it.
\r\nps.: @ __mtb__ I also find your tone sometimes too "emotional" and I am afraid that you will loose most Kagglers sympathy if you can not keep calm. (Of course it is easy to say).
", "rawMarkdown": "", "votes": null }, { "id": "22933", "postDate": "04/17/2013 00:44:04", "content": "I think __mtb__ is being a little emotional, and i totally understand that, but he is right.
\r\nKaggle must improve (and automatize) many steps of two stage competitions in order to avoid this. And in this case, i think that at least, both users should receive the ranking and the public leaderboard recognition for their feats.
\r\nRules exists to prevent users from cheating and everything else, but fair competitors shouldnt be punished with some stretching of them. By the way, what is more important, having the best models or what?
\r\nI'm a software architect myself, and in two stage competitions, allowing a invalid entry to be selected is clearly a violation of a business rule. I deal too much with software uses, and they call it a bug. If not in the implementation, a bug in the specification,\r\n but a bug nevertheless.
", "rawMarkdown": "", "votes": null }, { "id": "22934", "postDate": "04/17/2013 01:10:17", "content": "It is a bit confusing.
\r\nIf the purpose of this discussion is to make kaggle better,
\r\nFor the final score,
\r\nhow about calculating on 5% of the test data? instead of 0% of the data. So, the players would know if something is seriously wrong.
\r\nor, show the score accuracy upto the 50% of the public leaderboard score. For example, if public score is 10, it would show the final score is above 20, or below 20.
\r\n\r\n
Either method does not leak much signal, unless something went wrong, right??
\r\nI think what everyone is asking, is something more indicative than 0.0000 as the blind score.
", "rawMarkdown": "", "votes": null }, { "id": "22942", "postDate": "04/17/2013 04:23:07", "content": "\r\n
Hi\r\n all!
First\r\n of all, I want to express my sympathy _mtb_ , and dxsx.
But\r\n nevertheless, you unfortunate mistake was made.
Practically\r\n I do not know English, and I find it hard to understand the rules, and in the description of the data, but on the page "Your\r\n Submissions"\r\n we all see the column "Select?".
Whose\r\n fault is that you do not put a 'bird'?
\r\nBen?\r\n Anthony?
And\r\n it's not a software bug!!!!
As\r\n for bugs dxsx - very sorry for what happened.
On\r\n the software Kaggle.
I\r\n worked for many years in the computer field and has not yet met the perfect program.
Great\r\n Dijkstra once said: Testing programs can very effectively demonstrate the presence of bugs, but is hopelessly inadequate to demonstrate their absence.
So\r\n we wish to colleagues from Kaggle success in their work is not easy.
And\r\n in conclusion I want to say:
Dear\r\n _ mtb _ do not get excited, life goes on, there will be many difficulties and challenges in your way. But there will be joy and good luck!
", "rawMarkdown": "", "votes": null }, { "id": "22944", "postDate": "04/17/2013 05:21:34", "content": "
I'm not sure if I understand this 100%, is the following what happened and all the relevant issues ?
\r\n1. This is a 2-stage competition. During stage 1, __mtb__ made many submissions and selected 3 (what were the purpose of selections for stage 1 ?).
\r\n2. Stage 1 finished and stage 2 started, with a new data set released. Competitors were supposed to make predictions against this new data set, and their scores will determine their final rankings.
\r\n3. __mtb__ made only one submission against the new dataset. He did not select it, nor did he change the selections he made during stage 1.
\r\n4. In the absence of stage 2 selections, the kaggle system picked one of his stage 1 selections to determine his final score and rank, and presumably because it was for stage 1 dataset, it couldn't be scored against the stage 2 answers, or at least resulted\r\n in a very bad score.
", "rawMarkdown": "", "votes": null }, { "id": "22948", "postDate": "04/17/2013 08:43:08", "content": "B Yang:
\r\nYes, those are the relevant issues.
\r\nRE: #1: I used the selections for stage 1 to keep track of my best public leaderboard score. But I do not believe the stage 1 submission selections serve any purpose for the competition (someone please correct me here if I am wrong).
\r\nRE #4: You are correct. Let me add more information so you can visualize what the 'My Submissions' page looks like for the 2 stage competition.
\r\nOn the 'My Submissions' page, the table displayed has 2 columns for scores: 'Public Score' and 'Private Score'. The 'Public Score' corresponds to the score on the stage 1 file, and the 'Private Score' corresponds to the score on the stage 2 file.
\r\nFor the stage 1 submissions, the 'Public Score' cell value has the Kaggle scored value, and the 'Private Score' cell value is 99999999.00000. I have 21 of these such rows in the 'My Submissions' grid.
\r\nFor the stage 2 submissions, the 'Public Score' cell value is 0.00000, and the 'Private Score' cell value is 8.40080. I have 1 of these rows in the 'My Submissions' grid.
\r\nWhen the competition ended, I had final leaderboard score of 99999999.00000, which corresponded to the 'Private Score' of the stage 1 submissions. When I checked the final leaderboard standings two weeks back, this is what tipped me off\r\n that my final submission was not picked up by Kaggle.
\r\nI had the suspicion that my final, stage 2 leaderboard ranking was being determined by a stage 1 file's score. When I discovered this I contacted Kaggle.
\r\nEdit: RE #1: The stage 1 submissions drive the public leaderboard during stage 1, but I am not certain how the stage 1 submission selections (the checking of the checkbox) are actually used.
", "rawMarkdown": "", "votes": null }, { "id": "22949", "postDate": "04/17/2013 08:45:20", "content": "@Alexander: According to wikipedia
\r\n"A software bug is an error, flaw, \r\nfailure, or \r\nfault in a computer program or \r\nsystem that produces an incorrect or unexpected result, or causes it to behave in unintended ways."
\r\nThis event resulted in:
\r\n1. unexpected behaviour - because both __mtb__ and Kaggle admins didn't expect this situation. otherwise we wouldn't have this conversation. The system behaved unexpectedly and against intuition because it didn't take the only submission that could be tested\r\n against the final dataset. Not only it didn't behave unexpectedly but silently passed through an error in an implicit fashion without any meaningful message to the user.
\r\n2. unintended behaviour - because __mtb__ didn't intend to lose a prize of 30k USD by simply not clicking one checkbox. Kaggle didn't intend not to reward one of the best models.
", "rawMarkdown": "", "votes": null }, { "id": "22951", "postDate": "04/17/2013 09:14:38", "content": "@Pawet:
\r\n"A software bug is an error, flaw, \r\nfailure, or \r\nfault in a computer program or \r\nsystem that produces an incorrect or unexpected result, or causes it to behave in unintended ways."
\r\nThat's right!
\r\nBut who's to blame, if you forgot to save the data?
It seems to me the case of _mtb_ can be defined this way, the user forgot to save the data of the program!
\r\n", "rawMarkdown": "", "votes": null }, { "id": "22952", "postDate": "04/17/2013 09:28:18", "content": "
That's hardly the case because he saved the data by making the submission. Eventually the system didn't acknowledge this. Giving a feedback that his submission scored 0.0000 which was according to Ben a situation where everything was processed accordingly.
\r\n"When you make a submission on this new set, a public score of "0.0000" without errors means the submission was processed correctly."
\r\nsource: http://www.gequest.com/c/flight/forums/t/3973/final-leaderboard-set-release/21080
\r\nTo his best knowledge __mtb__ didn't have any information about this pending error.
\r\nDon't get me wrong. I'm taking part in this discussion only because I really care about these competitions. I love competing and it wouldn't be possible without Kaggle. But I'm worried that this situation is unfairly resolved because Kaggle is being the\r\n judge and jury in its own case.
\r\n
\r\nEDIT: sloppy grammar
Pawet@
\r\nThat's hardly the case because he saved the data by making the submission. Eventually the system didn't acknowledge this. Giving a feedback that his submission scored 0.0000 which was according to Ben a situation where everything was processed accordingly.
\r\n"When you make a submission on this new set, a public score of "0.0000" without errors means the\r\nsubmission was processed correctly."
\r\nIt is!
\r\nAnd this applies to the transferred file, and not to the marked!
\r\nBut was selected (selected was) the wrong file!
\r\nSystem handled it as a wrong file.
\r\nAnd that's really the wrong file for the second phase!
\r\nI need to do a newsletter e-mails in this case?
\r\nI do not know. Maybe!
\r\n\r\n
", "rawMarkdown": "", "votes": null }, { "id": "22962", "postDate": "04/17/2013 13:16:47", "content": "
The 2 phases were independent. The results of the first one didn't affect the results of the second one.
\r\nDid he chosen any second phase submission? No. 2 other top 5 competitors did the same. They didn't select any second phase submission. Whether he chose or not any first phase submission is not important. These submissions shouldn't be taken into consideration\r\n when scoring completely different prediction set.
\r\nLet's agree to disagree on this one :). You have right to an opinion. I have mine.
", "rawMarkdown": "", "votes": null }, { "id": "22966", "postDate": "04/17/2013 13:49:05", "content": "Pawet@ You have right to an opinion. I have mine.
\r\nOk!
\r\nI agree with you.
\r\nOnce again, I'm really sorry guys.
\r\nAll the best,
\r\nAlex.
", "rawMarkdown": "", "votes": null }, { "id": "22968", "postDate": "04/17/2013 14:44:37", "content": "Not that we are counting votes, but in this particular situation I would agree with _mtb_ and Pawel. (Discloser: I do not have a dog in this fight).
\r\nIn my personal opinion the best Kaggle can do is: award _mtb_ points, money prize and issue one more press release announcing award of addition prize. (They do not need to admit any mistakes publicly. Just something like: "Because of the great success of\r\n the competition we decided to award one more prize for a model developed by...").
", "rawMarkdown": "", "votes": null }, { "id": "22971", "postDate": "04/17/2013 14:55:50", "content": "Sergey Yurgenson@
\r\nА что, ваше предложение очень симпатичное!!!
\r\nAnd that, yuor offer is very cute!!!
", "rawMarkdown": "", "votes": null }, { "id": "22973", "postDate": "04/17/2013 15:04:13", "content": "[quote=Sergey Yurgenson;22968]
\r\nNot that we are counting votes, but in this particular situation I would agree with _mtb_ and Pawel. (Discloser: I do not have a dog in this fight).
\r\nIn my personal opinion the best Kaggle can do is: award _mtb_ points, money prize and issue one more press release announcing award of addition prize. (They do not need to admit any mistakes publicly. Just something like: "Because of the great success of\r\n the competition we decided to award one more prize for a model developed by...").
\r\n[/quote]
\r\n\r\n
Totally agree!
\r\nI love kaggle, and I love competing here. I also know that is very difficult to handle these cases, and no piece of software is perfect. I just didn't like how this situation was handled. Its very unfair. For many reasons. I bet that kaggle will be modifiying\r\n their software to avoid this in the future, and that shows that it is something the kaggle platform should have done in the first place.
", "rawMarkdown": "", "votes": null }, { "id": "23005", "postDate": "04/17/2013 21:19:34", "content": "I forgot about the competition the next day after submission deadline, but just for future references:
\r\n\r\n
Anthony, can you please confirm that following statements are official Kaggle policies?
\r\n\r\n
If so, it is a source of concern about “there would soon be another platform that would undercut us[[Kaggle]]“ (Dr. Nicholas Gruen, please read\r\n\r\nhere)
\r\nI believe many of us (and also sponsors) prefer to compete on a platform that doesn’t have those policies.
\r\nPS: I have informed anyone, whom I named here, by email about this post.
", "rawMarkdown": "", "votes": null }, { "id": "23057", "postDate": "04/18/2013 11:36:40", "content": "dxsx, your questions to Anthony are spot on.
\r\nIn Anthony's email to me, he pulled a quote from the competition rules:
\r\n\r\n\r\n"By participating in the Competition, each Entrant agrees to release, indemnify and hold harmless GE, Kaggle Inc., and their respective affiliates, subsidiaries, advertising and promotions agencies, as applicable, and each of their respective agents, representatives,\r\n officers, directors, shareholders, and employees from and against any injuries, losses, damages, claims, actions and any liability of any kind resulting from or arising out of your participation in or association with the Competition. GE is not responsible\r\n for any miscommunications such as technical failures related to computer, telephone, cable, and unavailable network or server connections, related technical failures, or other failures related to hardware, software or virus, or incomplete, late or misdirected\r\n Entries. GE reserves the right to cancel, modify or suspend the Competition should any computer virus, bug or other technical difficulty or other causes beyond the control of GE corrupt the administration, security or proper play of the Competition, and to\r\n determine winners from among Entries not affected by the corruption, if any, in its sole discretion."
\r\n
When I first read this piece of the email, I interpreted the last sentence as 'GE reseves the right to determine the winners however it likes (i.e. sole discretion)'. And that is why my first response to his email was to\r\nRageQuit Kaggle. It was my interpretation from how Kaggle/GE had handled this situation\r\n so far that they were more concerned with the FlightQuest marketing push than they were identifing the correct winners and models.
\r\nAlso, I was happy to hear you contacted Dr. Nicholas Gruen. I enjoyed reading his insights, but I was also happy to see that he, at least at some point was a Kaggle shareholder. The issues we are discussing on this thread are core to Kaggle's competition\r\n business model (at least I would think they are as they touch on the integriy of the competition).
", "rawMarkdown": "", "votes": null }, { "id": "23063", "postDate": "04/18/2013 13:18:21", "content": "[quote=Sergey Yurgenson;22968]
\r\nNot that we are counting votes, but in this particular situation I would agree with _mtb_ and Pawel. (Discloser: I do not have a dog in this fight).
\r\nIn my personal opinion the best Kaggle can do is: award _mtb_ points, money prize and issue one more press release announcing award of addition prize. (They do not need to admit any mistakes publicly. Just something like: "Because of the great success of\r\n the competition we decided to award one more prize for a model developed by...").
\r\n[/quote]
\r\nI also agree that this would be a reasonable solution. I would like to see that Kaggle is willing to stand up for its competitors. I understand this is probably an embarrassing situation for Kaggle, but I think most people would appreciate an honest effort\r\n to correct the situation. The silence is deafening.
", "rawMarkdown": "", "votes": null }, { "id": "23078", "postDate": "04/18/2013 15:12:15", "content": "Below is a copy/paste email exchange I just had with an executive at GE.
\r\n
\r\nUnfortunate, but from what I piece together between Anthony's email and the one below is that this competition might not be about identifing the best models, but instead is really just a marketing push. The fact that the results of a competition can not be\r\n changed even though the results were impactedby a bug in Kaggles software is troubling.
\r\n
GE Exec sent me this
\r\nFrom: "xxx, xxx (GE, Corporate)"
\r\nTo: "mtberseth@yahoo.com"
\r\nSent: Thursday, April 18, 2013 10:39 AM
\r\nSubject: Kaggle Follow up
\r\n
\r\n
\r\nDear Matt,
\r\n
\r\nThank you for your email. Please be assured that GE has been made aware of this issue and we have been working with Kaggle to understand the problem that arose with your entry. After reviewing this matter in some detail, we do agree with the conclusion that\r\n Kaggle has reached. While we do sympathize with your situation, GE has determined that Kaggle has handled this matter appropriately and agrees that the final results should not be changed. We understand your frustration, but we hope that you will continue\r\n to participate in future Challenges.
\r\nThanks,
\r\n
\r\n
\r\nxxx,
\r\n
\r\nExecutive Director - xxx xxx
\r\nGeneral Electric
\r\n
\r\n
I replied with this:
\r\n
\r\nFrom: Matthew Berseth
\r\nTo: "xxx, xxx (GE, Corporate)"
\r\n
\r\nSent: Thursday, April 18, 2013 11:03 AM
\r\nSubject: Re: Kaggle Follow up
\r\n
\r\nxxx:
\r\n
\r\nI really appreciate the email.
\r\n
\r\nI am not sure if you received the same explanation I have from Kaggle, but if they denied that it was an issue with their software than you received an incorrect description of what happened. I encourage to read the forum thread over at Kaggle.com: https://www.gequest.com/c/flight/forums/t/4284/acknowledging-two-more-great-competitors\r\n to gain a full understanding into some of the limitations of the Kaggle.com software and how this effected the outcome of the competition.
\r\n
\r\nUnfortunately, at this time it is hard for me to envision a situation where I could compete in another competition. Kaggle has disappointed me in how they have handled this situation, and as a competitor I can not rationalize spending the effort I did on FlightQuest\r\n on another competition having experienced everything I did with this competition.
\r\n
\r\nAlso, I will remove your name and email address, but I need to be fair and open to the other competitors so I will be posting this email and my response to the forums. I am sure you understand.\r\n
\r\n
\r\nThanks,
\r\nMatt
\r\n
Edit: rearranged the order.
", "rawMarkdown": "", "votes": null }, { "id": "23388", "postDate": "04/23/2013 23:07:11", "content": "This thread is about dead, so I just wanted to follow up with everyone who is still interested one last time.
\r\nI emailed Anthony again, pointing out one more time the bugs in the Kaggle.com software that prevented my submission from being selected and responding to the other points in his email. I sent that to him last Friday (4/19/2013), but I have not heard a response.\r\n At this point I am not expecting one. I posted the email over \r\nhere so I didn't have to reformat it here.
\r\nI really appreciate all of the support I received from all of the other Kagglers. I know not everyone felt like they could voice their opinions publicly, so I appreciated the private emails, linkedin conversations and skype sessions. For me, this was the\r\n best part of this whole ordeal.
\r\nI was offended that Kaggle offered me money and Kaggle points to keep quiet because at the time I didn't think that was fair to GE or the other competitors. I am probably an idealist, but it felt like Kaggle was trying to pay me off to cover up their issue.\r\n In the end, I did not receive anything.
\r\nThis was a crushing experience for me. I worked very hard on this competition, just like the rest of you - over the US holidays and into February. Then I was ignored by Kaggle for eight days while GE and Kaggle put out press releases, blog posts etc ...\r\n The only communications I heard from Kaggle went a lot like Anthony's email - demonstrating Kaggle's lack of understanding of the issue. That is why my response to Ben's post (the one that started this thread) was filled with emotion. I should of handled that\r\n differently.
\r\nI am going to take a break from Kaggle. I do not think I will consider competing again unless they introduce a formal dispute process that is transparent and handled by a third party. I don't like that Kaggle serves as both judge and jury. Also, there are\r\n a lot lingering questions in this forum and others that I would have liked to see Kaggle answer.
\r\nThis situation has to be resolved!
\r\nI propose the following:
\r\nAnthony and Matt try to agree on a group of people who will look into the matter and give a final binding judgement.
\r\nThe judgement could be one of three:
\r\n1. We don't know (in which case we are back to square one)
\r\n2. Matt is more "to blame" than Kaggle
\r\n3. Kaggle is more "to blame" for the situation than Matt
\r\nI propose further that if this group rules judgement 2, Matt accepts his misfortune.
\r\nAnd if this group rules judgement 3, Matt is added to the winners page in a shared 4th place,
\r\nKaggle pays Matt the 30000 USD and Matt gets the Kaggle points for a 4th place.
\r\nAnthony, could you agree with this?
\r\nMatt, could you agree with this?
\r\n\r\n
Jules van Ligtenberg
", "rawMarkdown": "", "votes": null }, { "id": "23470", "postDate": "04/25/2013 17:49:11", "content": "Yes, I agree and I would like that opportunity.
\r\nKaggle needs to have a formal, transparent dispute process that includes a 3rd party - someone who would not be influenced by the sponser, competitor or Kaggle. The competitions are becoming more complicated: multiple deadlines,\r\n multiple leaderboard files, uploading model archive, etc ... There is a gray area, and software bugs can, do and will continue to happen.
\r\nThe precedent Kaggle has set with how this has been handled so far is not great from a competitors stand point.
", "rawMarkdown": "", "votes": null }, { "id": "24157", "postDate": "05/10/2013 17:24:04", "content": "matt I have great sympathy for you. I spent over a month and a half working on this contest (nights mostly) and eventually gave up due to lack of sleep, deteriorating health, and generally disintegrating family cohesion (*slight* exaggeration...).
\r\nI am sure kaggle has some clause in their legal documents somewhere that renders them immune from any sort of legal action. In other words, I'm sure they don't *have* to do the right thing award you the money, even if they are completely in the wrong (which\r\n I agree they are). But, I am confused as to why they don't want to do the right thing anyway. They risk alienating present and future kagglers, and tarnishing the integrity of the whole endeavor. Is it a matter of saving face? i.e. now that they have taken\r\n a certain position, it would be "giving in" to reverse that position? Maybe they are afraid that would open the door to all kinds of disputes...?
\r\nin the absence of any such move by kaggle, maybe GE could skim $30k off of the phase 2 prize pool and give this to Matt? clearly the $$ is there...
\r\nor perhaps there is another way... each of the actual cash $$ winners could take matters into their own hands, and in a show of good sportsmanship skim an equal fraction off of their prizes and pool it together for matt? to get to $30k, each of the prizewinners\r\n would need to take off 12% off their prizes. Perhaps this wouldn't seem quite fair, since then matt would receive 100% of his prize, but everyone else would receive 88% of theirs. To even this out (according to my calculations...) if the winners each skimmed\r\n ~10.7% off their prizes and pooled it together for matt, this would sum to ~89.3% of $30k for matt (~$26,785), as well as ~89.3% of their original prize for everyone else.
\r\n\r\n
Prizewinners?
", "rawMarkdown": "", "votes": null }, { "id": "24282", "postDate": "05/14/2013 02:10:18", "content": "Well, prize winners are not really winners money-wise!
\r\n\r\n\r\n
Moneywise, winners of Kaggle’s competitions aren’t the real winners! They just get a small portion of money which is spent by sponsors. Finally, look at this comment on MIT Technology review about this contest (http://www.technologyreview.com/news/513141/a-data-crunching-prize-to-cut-flight-delays/)
\r\n\r\nMarkyMark: "Congrats to GE/Alaska/Kaggle to discover a way to pay engineers less. $230k in prize money, 242 engineers, that's\r\n only $950 per engineer to solve your problems! What a steal!"
\r\n\r\n", "rawMarkdown": "", "votes": null }, { "id": "24300", "postDate": "05/14/2013 13:45:18", "content": "good point dxsx!
\r\n\r\nin that case, Kaggle can't afford *not* to err on the side of humility, and do the right thing by Matt (and every other kaggler), since they risk alienating the kaggle community and thereby diminishing their cheap labor pool. $30k seems a small price to\r\n pay to keep us all happily working for (almost) nothing...
", "rawMarkdown": "", "votes": null }, { "id": "24528", "postDate": "05/20/2013 17:26:59", "content": "Raised a good point. In many ways, these competitions decrease the financial value for the solution providers (the competition participants and the market of providers in general) but increase the value of the solution for the competition sponsors (and\r\n companies in general).
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 22686, "author_name": "mtb26767", "author_url": "", "post_date": "04/12/2013 11:54:24", "content": "First off, a sincere congrats to all of the other competitors. This was a tough competition!
\r\nBen:
\r\ndxsx’s error, as unfortunate as it was, is a problem with his model. The error made in my case is with *your* software. I followed your directions\r\n exactly:
\r\n\r\n
\r\n\r\nThe final evaluation set is now up on the data page.
\r\nYou may make a submission on the final evaluation set regardless of whether you've submitted a final model or hash. However, you are only eligible for prize money if you submitted a final model and use the same model to make predictions on this new set.
\r\nWhen you make a submission on this new set, a public score of "0.0000" without errors means the submission was processed correctly.
\r\nPlease let us know if you find any issues with the data on this thread.
\r\nGood luck!
\r\n\r\n
As I have been saying over email repeatedly over the past 8 days, and yesterday on the phone with you and the Sorkin, there is no checkbox to check! I scored the final file and submitted it. I received all zero's like your\r\n directions read, and in your own words: this means the '... was processed correctly'. It is completely unambiguous what my final submission is. Any reasonable person understands this.
\r\nAs competition admins, you need to be able to distinguish between errors in your software and errors in the models. It is clear you are not properly doing this here.
\r\nLike I said in my email last night, it is still my expectation that you do the following by 5pm EST this afternoon:
\r\n1. The final leaderboard updated - moving me into 4th place showing my final score of 8.40080
\r\n2. The contact person at GE who I can coordinate with to get my $30,000
\r\n3. The 'winners page' updated, with me moved into 4th place
\r\n\r\n
Do the right thing!
", "votes": null, "replies": [] }, { "id": 22693, "author_name": "rjm558666", "author_url": "", "post_date": "04/12/2013 15:31:27", "content": "Wow, Matt that's hard to believe. As a fellow competitor, I thought the submission process was unclear. I would agree this is a bug with Kaggle's scoring process.
\r\nOne of the real losers here is GE.
\r\nBen, is there a process for competitors to resolve disputes, after Kaggle declares prize winners?
", "votes": null, "replies": [] }, { "id": 22695, "author_name": "mikeofalltrades", "author_url": "", "post_date": "04/12/2013 16:07:52", "content": "I also find all of this hard to believe (not Matt's side of this story but the fact that Kaggle couldn't resolve it).
\r\n
\r\nAs a new member to Kaggle for this project I can certainly say that there have been highs and lows from this competition process. I really enjoy the concept of Kaggle, the competitive nature of the process, and the atmosphere for learning. But it seems to\r\n me that Kaggle still has a lot of kinks to work out in their process and procedures for competitions. I don't know how the other competitions played out, as this is my first, but there were several sets of confusing guidelines and missed deadlines (i.e. promised\r\n deadlines that weren't made by Kaggle).
I can understand that these kind of projects will have hiccups, but if Kaggle and a veteran of Kaggle (Matt is very high in the Kaggle rankings) can't work out something like this I think that is a major flaw in the whole process. What hope would a new\r\n member to the community have if something like this were to happen?
\r\nMatt - if Kaggle/GE can't resolve this, and I were you, I might reach out to someone like Passur http://www.passur.com/ or Honeywell http://www.honeywell.com and explain the situation. I wouldn't be surprised if someone at that company\r\n would be interested in your algorithms. It would seem as though they already have an interest in Flight Management Systems and aircraft arrival predictions. In addition it would probably be a great piece of PR for them.
\r\nAs an aside, I think the fact that the results aren’t released limits users learning experiences. I don’t feel like I can learn from mistakes as well as I had hoped would be possible. As a user who doesn’t have a strong “big data” background, if I can’t\r\n learn there isn’t an incentive to stick around and use the site. Due to the limited number of winners, and the fact that most of the top Kaggle members win multiple prizes, most users will continue to come back for the challenge and learning, not the prizes. \r\n Maybe the Kaggle model is just to hook up a small set of “Kaggle Masters” to companies. Well, I guess then it doesn’t matter if your user base grows and learns. My belief is that having a larger set of continually improving users from various fields will\r\n provide your customers (i.e. companies and people that host competitions) with better and more interesting results.
", "votes": null, "replies": [] }, { "id": 22700, "author_name": "mtb26767", "author_url": "", "post_date": "04/12/2013 17:25:06", "content": "rjm5, ATCGuy - I appreciate the support. I would love to hear what the other competitors think about this topic.
\r\nIt kills me that a bug in Kaggle's own process is what is holding this whole thing up ...
\r\n\r\n
I should also say that I emailed Ben an hour after the offical results were posted. Then I emailed him again in the afternoon. Then I emailed the community manager because Ben never responded. Finally, the first I heard back\r\n from Ben was Friday night telling me the same thing he said above.
\r\nKaggle, here are a few tips for future competitions:
\r\n1. Bad news doesn't get better with age
\r\n2. Respond to the competitors inquiries
\r\n3. *Don't* announce the winners in a joint press release until the competitors have had a chance to dispute the results. You certainly painted yourself into a corner this time
\r\n4. *Don't* offer the competitors money and kaggle points to ''unofficially' resolve the matter. That was an insult to my integrity and is certainly not fair to the Sponser or other competitors.
", "votes": null, "replies": [] }, { "id": 22705, "author_name": "rjm558666", "author_url": "", "post_date": "04/12/2013 17:49:52", "content": "Ben, did you offer money to keep this quite? Please respond.
", "votes": null, "replies": [] }, { "id": 22715, "author_name": "konradb", "author_url": "", "post_date": "04/12/2013 20:51:50", "content": "FWIW, the data submission process was indeed fairly convoluted (and I suppose there could´ve been more people who got lost in it along the way). I believe a clarification from the admins on whether there was a software error glitch on top of it, or not,\r\n would benefit all participants. The silence is having an opposite effect.
", "votes": null, "replies": [] }, { "id": 22810, "author_name": "dxsx78425", "author_url": "", "post_date": "04/14/2013 23:53:12", "content": "Thanks Ben for your acknowledgment.
\r\nFirst, I was surprised how important is for Kaggle to strictly follow the competition deadlines over having better models (as I corrected my submission few hours after the submission deadline). Not to mention, almost all the\r\n deadlines were missed from the Kaggle side in this competition. Anyways, it was my own fault!
\r\nIt is very sad to hear what happened to _mtb_. It is just unfair to him to lose his position because of some systematic error.
\r\nI can understand that creating a perfect competition platform is extremely hard. I do appreciate your constant efforts in improving the competitions’ setup. Any competition set up has its own issues. As you mentioned, this\r\n is not the first time that you are disappointing competitors.
\r\nHowever, I expected that Kaggle to be more flexible in handing issues like what happened to _mtb_.
\r\n\r\n
\r\n
\r\n
", "votes": null, "replies": [] }, { "id": 22867, "author_name": "mtb26767", "author_url": "", "post_date": "04/15/2013 23:05:25", "content": "
This is a 2 phase competition so there are 2 separate file types that need to be scored - the public leaderboard file and the final leaderboard file. During the public leaderboard portion of the competition, yes I used the checkbox to keep track of my best\r\n submission. Then, following the directions Ben posted, I scored the final leaderboard file and uploaded it. As Ben's direction describe, I received the all zeros which in his own words means the submission was properly received.\r\n
\r\n
\r\nWhat on earth is a checkbox doing on the submission screen of a two phase competition? There is no ambiguity - there is only a single *file type* that is valid for the final leaderboard file. Why on earth is your software not smart enough to figure that out?\r\n
\r\n
\r\nAlso - why didn't the default selection logic kick in? Because only final leaderboard files are valid for the final submission, why didn't your software pick the *only* valid option?
Certainly I am not going to be competing in anymore Kaggle competitions. I do not trust your company to properly host a competition.
\r\nI am done with Kaggle.
", "votes": null, "replies": [] }, { "id": 22889, "author_name": "mtb26767", "author_url": "", "post_date": "04/16/2013 11:55:31", "content": "Anthony:
\r\nHere is my more detailed response. Let's bring some transparency into the equation by keeping the discussion in the forums. I will respond to your email and ask some serious questions that need to be answered. Please do that here instead of over email.
\r\nThe text you have highlighted from the submission page is only valid for a single phase competition.
\r\nThe submission page stated: 'Note: You can select up to 1 submission that will be used to calculate your final leaderboard score. If you do not select them, up to 1 entry will be chosen for you based on your most recent submissions. Your final score\r\n will not be based on the same exact subset data as the public leaderboard, but rather a different private data subset of your full submission. Your public score is only a rough indication of what your final score might be. You should choose an entry that will\r\n most likely be best overall, and not necessarily just on the public subset.'\r\n
This is not accurate text for the two phase submission. There are no subsets, instead there are two separate files. The public leaderboard file and the final leaderboard file. A submission is for exactly one or the other, but not for both. During the final\r\n submission period of a two phase competition, this text should not be displayed because it is not accurate. If you don't understand why this text doesn't make sens for a two phase competition, run it by your employee's that are competitors.
\r\nLet me outline what I did for the final submission again, because from you email I don't believe you are familiar with how the two phase competition works (Rich didn't understand it either)
\r\nI know the wiki page has a forula, but it doesn't include the additional points awarded to keep competitors quiet. It seems like you would want to update it to include this point. Especially now that companies like the NYTimes and American Express are using\r\n Kaggle rankings when hiring data scientists
\r\nAnthony, it is clear to me what ultimately happened here. You painted yourself into a corner because you announced the results in a press release with GE the same day the rest of the competitors saw the results. I know there is quite a marketing push behind\r\n GE's quests and having to change the results after the press releases and magazines articles were written leaves egg on Kaggle's face. The problem is that the competition portion of your business is built on trust and integrity.
\r\nThe obvious analogy is to sporting events - if the referee can not be trusted than the games don't matter.
\r\nThat is why I will not compete in anymore Kaggle competitions.
\r\n__mtb__
\r\np.s. this forum software is terrible. it always seems to mangle the posts.
\r\nWow. I enter the quest forum after a week and what a surprise to read that post. I think you have your point. If you made only 1 final submission there is no ambiguity. It should have been chosen. Kaggle competitions should be closer to real life in my opinion.\r\n Normally after seeing a skyhigh (which is bad) performance you should be able to correct it. Seeing no performance as in your case should give you a hint that something is wrong. By "real life" I mean that such cases should not diminish 3 months of hard work.\r\n When competing I had a nightmarish vision that something like this would happen to us which is not a good thing.
\r\nI kind of didn't like that you have dismissed dxsx's performance. In "real life" (not Kaggle) he was better. The fact that your mistake/bug was different than his is circumstantial.
", "votes": null, "replies": [] }, { "id": 22900, "author_name": "mtb26767", "author_url": "", "post_date": "04/16/2013 15:34:37", "content": "Pawel
\r\nThanks for pointing that out about dxsx - that is certainly not my intention to dismiss his performance. I will post here what I sent to him in a private email a couple days back:
\r\n\r\n\r\ndxsx:
\r\nTo be honest, the mistake you are talking about was keeping me up at night the last couple weeks of the competition. I really think Kaggle needs to do something different in these 2 phase competitions to let the competitors know more about the submission. Maybe\r\n like let them have up to 5 submissions or something, but then show them the score over say 25% of the dataset. Obviously you wouldn't be able to overfit, but you could still prevent what happened to you. I bet they will change this in the future. It sucks\r\n for both the competitors AND the sponsors to have models not considered because of this.
\r\n
Sorry dxsx if I was dismissive of your hard work! That was not my intentions.
\r\nedit: formatting
", "votes": null, "replies": [] }, { "id": 22921, "author_name": "mtb26767", "author_url": "", "post_date": "04/16/2013 21:53:07", "content": "A competitor emailed me a couple of things that I wanted to share. First he was able to summarize my issue with the submission bug very nicely in a couple of statements and 3 bullet points. I have been working for 2 weeks trying to explain this - yet I was\r\n not able to.
\r\nClearly the Kaggle platform is not the most bug free software in the universe. Let me point these events:
\r\nSo your situation should be treated as one of the bugs. I'm sure that knowing what they know now they would correct this in the future. It's costly but still it is not your fault.
\r\nAnd his other point was that 'You should keep your tone less emotional' . I appologize for that. I am not going to go back and edit old posts or anything - what was said is said, but I will make sure I cut down the commentary and focus on\r\n the facts.
\r\nAnthony, I am certainly looking forward to having an open, civil conversation in this forum, with not just you and me, but with all of the competitors. I posted the questions that I feel need to be answered. Please familiarize yourself with the details of\r\n the two-phase competitions and make sure you take a close look at the submission page and understand why the text you quoted does not make sense for a final submission in a two-phase competition.
\r\nI am looking forward for your response.
", "votes": null, "replies": [] }, { "id": 22923, "author_name": "mtb26767", "author_url": "", "post_date": "04/16/2013 23:28:47", "content": "Also, it sounds like similar issues with the submission process for two-phase competitions was encountered for the fast iron competition. This is highly relevant:
\r\nLike Leustagos says, the two stage submission process in it's current state is really poorly implemented. It was clearly duct taped onto the current submission system sloppily and because of that there are huge usability issues with submitting solutions\r\n and models. In the Bulldozer competition we had numerous issues with the submission process and the responses from kaggle were very slow and often not that helpful. I really like kaggle but I think they have some serious usability issues and problems with\r\n their current support process.
\r\n-- pjreddie
\r\nHere is the original \r\nlink.
\r\n\r\n
Anthony, according to the competitors (who are also the users), there are numerous bugs with the submission process and submission page for two-phase competitions. I believe your assertion that 'There was no bug in the\r\n software.' is incorrect.
", "votes": null, "replies": [] }, { "id": 22932, "author_name": "gaborfodor", "author_url": "", "post_date": "04/17/2013 00:38:35", "content": "First of all: Kaggle do an amazing job and I have to tell that this site is probably my favorite place in the web.
\r\nI understand that you guys (Kaggle) had an extremely hard competition this time, the dataset was hard to prepare and process correctly and the competition prize pool and media attention was also higher than usual.
\r\nWith all the respect I think you did not choose the best decision after the competition have ended. I think the problems with\r\ndxsx and __mtb__'s solutions should have been discovered during the 1.5 month while you were validating the final results.
\r\nEven if the problems discovered later than the official announcement I think you should have been far more flexible to reward the two competitor who created excellent solutions but slipped on a ( their or yours?) banana peel.
\r\nCreating a forum post is basically free but don't solve the problem. The e-mail received by __mtb__ is also a mixed message (nit-pciking, sympathy, legal shield...).
\r\nI understand that you do not want to hurt the prizewinner competitors who also created amazing solutions and they definetly deserve the prize amount and the ranking they got.\r\nBut I can not beleive that you do not see the huge problem with this situation and you could not solve it.
\r\nps.: @ __mtb__ I also find your tone sometimes too "emotional" and I am afraid that you will loose most Kagglers sympathy if you can not keep calm. (Of course it is easy to say).
", "votes": null, "replies": [] }, { "id": 22933, "author_name": "leustagos", "author_url": "", "post_date": "04/17/2013 00:44:04", "content": "I think __mtb__ is being a little emotional, and i totally understand that, but he is right.
\r\nKaggle must improve (and automatize) many steps of two stage competitions in order to avoid this. And in this case, i think that at least, both users should receive the ranking and the public leaderboard recognition for their feats.
\r\nRules exists to prevent users from cheating and everything else, but fair competitors shouldnt be punished with some stretching of them. By the way, what is more important, having the best models or what?
\r\nI'm a software architect myself, and in two stage competitions, allowing a invalid entry to be selected is clearly a violation of a business rule. I deal too much with software uses, and they call it a bug. If not in the implementation, a bug in the specification,\r\n but a bug nevertheless.
", "votes": null, "replies": [] }, { "id": 22934, "author_name": "sigint", "author_url": "", "post_date": "04/17/2013 01:10:17", "content": "It is a bit confusing.
\r\nIf the purpose of this discussion is to make kaggle better,
\r\nFor the final score,
\r\nhow about calculating on 5% of the test data? instead of 0% of the data. So, the players would know if something is seriously wrong.
\r\nor, show the score accuracy upto the 50% of the public leaderboard score. For example, if public score is 10, it would show the final score is above 20, or below 20.
\r\n\r\n
Either method does not leak much signal, unless something went wrong, right??
\r\nI think what everyone is asking, is something more indicative than 0.0000 as the blind score.
", "votes": null, "replies": [] }, { "id": 22942, "author_name": "alexxanderlarko", "author_url": "", "post_date": "04/17/2013 04:23:07", "content": "\r\n
Hi\r\n all!
First\r\n of all, I want to express my sympathy _mtb_ , and dxsx.
But\r\n nevertheless, you unfortunate mistake was made.
Practically\r\n I do not know English, and I find it hard to understand the rules, and in the description of the data, but on the page "Your\r\n Submissions"\r\n we all see the column "Select?".
Whose\r\n fault is that you do not put a 'bird'?
\r\nBen?\r\n Anthony?
And\r\n it's not a software bug!!!!
As\r\n for bugs dxsx - very sorry for what happened.
On\r\n the software Kaggle.
I\r\n worked for many years in the computer field and has not yet met the perfect program.
Great\r\n Dijkstra once said: Testing programs can very effectively demonstrate the presence of bugs, but is hopelessly inadequate to demonstrate their absence.
So\r\n we wish to colleagues from Kaggle success in their work is not easy.
And\r\n in conclusion I want to say:
Dear\r\n _ mtb _ do not get excited, life goes on, there will be many difficulties and challenges in your way. But there will be joy and good luck!
", "votes": null, "replies": [] }, { "id": 22944, "author_name": "byang1", "author_url": "", "post_date": "04/17/2013 05:21:34", "content": "
I'm not sure if I understand this 100%, is the following what happened and all the relevant issues ?
\r\n1. This is a 2-stage competition. During stage 1, __mtb__ made many submissions and selected 3 (what were the purpose of selections for stage 1 ?).
\r\n2. Stage 1 finished and stage 2 started, with a new data set released. Competitors were supposed to make predictions against this new data set, and their scores will determine their final rankings.
\r\n3. __mtb__ made only one submission against the new dataset. He did not select it, nor did he change the selections he made during stage 1.
\r\n4. In the absence of stage 2 selections, the kaggle system picked one of his stage 1 selections to determine his final score and rank, and presumably because it was for stage 1 dataset, it couldn't be scored against the stage 2 answers, or at least resulted\r\n in a very bad score.
", "votes": null, "replies": [] }, { "id": 22948, "author_name": "mtb26767", "author_url": "", "post_date": "04/17/2013 08:43:08", "content": "B Yang:
\r\nYes, those are the relevant issues.
\r\nRE: #1: I used the selections for stage 1 to keep track of my best public leaderboard score. But I do not believe the stage 1 submission selections serve any purpose for the competition (someone please correct me here if I am wrong).
\r\nRE #4: You are correct. Let me add more information so you can visualize what the 'My Submissions' page looks like for the 2 stage competition.
\r\nOn the 'My Submissions' page, the table displayed has 2 columns for scores: 'Public Score' and 'Private Score'. The 'Public Score' corresponds to the score on the stage 1 file, and the 'Private Score' corresponds to the score on the stage 2 file.
\r\nFor the stage 1 submissions, the 'Public Score' cell value has the Kaggle scored value, and the 'Private Score' cell value is 99999999.00000. I have 21 of these such rows in the 'My Submissions' grid.
\r\nFor the stage 2 submissions, the 'Public Score' cell value is 0.00000, and the 'Private Score' cell value is 8.40080. I have 1 of these rows in the 'My Submissions' grid.
\r\nWhen the competition ended, I had final leaderboard score of 99999999.00000, which corresponded to the 'Private Score' of the stage 1 submissions. When I checked the final leaderboard standings two weeks back, this is what tipped me off\r\n that my final submission was not picked up by Kaggle.
\r\nI had the suspicion that my final, stage 2 leaderboard ranking was being determined by a stage 1 file's score. When I discovered this I contacted Kaggle.
\r\nEdit: RE #1: The stage 1 submissions drive the public leaderboard during stage 1, but I am not certain how the stage 1 submission selections (the checking of the checkbox) are actually used.
", "votes": null, "replies": [] }, { "id": 22949, "author_name": "paweljankiewicz", "author_url": "", "post_date": "04/17/2013 08:45:20", "content": "@Alexander: According to wikipedia
\r\n"A software bug is an error, flaw, \r\nfailure, or \r\nfault in a computer program or \r\nsystem that produces an incorrect or unexpected result, or causes it to behave in unintended ways."
\r\nThis event resulted in:
\r\n1. unexpected behaviour - because both __mtb__ and Kaggle admins didn't expect this situation. otherwise we wouldn't have this conversation. The system behaved unexpectedly and against intuition because it didn't take the only submission that could be tested\r\n against the final dataset. Not only it didn't behave unexpectedly but silently passed through an error in an implicit fashion without any meaningful message to the user.
\r\n2. unintended behaviour - because __mtb__ didn't intend to lose a prize of 30k USD by simply not clicking one checkbox. Kaggle didn't intend not to reward one of the best models.
", "votes": null, "replies": [] }, { "id": 22951, "author_name": "alexxanderlarko", "author_url": "", "post_date": "04/17/2013 09:14:38", "content": "@Pawet:
\r\n"A software bug is an error, flaw, \r\nfailure, or \r\nfault in a computer program or \r\nsystem that produces an incorrect or unexpected result, or causes it to behave in unintended ways."
\r\nThat's right!
\r\nBut who's to blame, if you forgot to save the data?
It seems to me the case of _mtb_ can be defined this way, the user forgot to save the data of the program!
\r\n", "votes": null, "replies": [] }, { "id": 22952, "author_name": "paweljankiewicz", "author_url": "", "post_date": "04/17/2013 09:28:18", "content": "
That's hardly the case because he saved the data by making the submission. Eventually the system didn't acknowledge this. Giving a feedback that his submission scored 0.0000 which was according to Ben a situation where everything was processed accordingly.
\r\n"When you make a submission on this new set, a public score of "0.0000" without errors means the submission was processed correctly."
\r\nsource: http://www.gequest.com/c/flight/forums/t/3973/final-leaderboard-set-release/21080
\r\nTo his best knowledge __mtb__ didn't have any information about this pending error.
\r\nDon't get me wrong. I'm taking part in this discussion only because I really care about these competitions. I love competing and it wouldn't be possible without Kaggle. But I'm worried that this situation is unfairly resolved because Kaggle is being the\r\n judge and jury in its own case.
\r\n
\r\nEDIT: sloppy grammar
Pawet@
\r\nThat's hardly the case because he saved the data by making the submission. Eventually the system didn't acknowledge this. Giving a feedback that his submission scored 0.0000 which was according to Ben a situation where everything was processed accordingly.
\r\n"When you make a submission on this new set, a public score of "0.0000" without errors means the\r\nsubmission was processed correctly."
\r\nIt is!
\r\nAnd this applies to the transferred file, and not to the marked!
\r\nBut was selected (selected was) the wrong file!
\r\nSystem handled it as a wrong file.
\r\nAnd that's really the wrong file for the second phase!
\r\nI need to do a newsletter e-mails in this case?
\r\nI do not know. Maybe!
\r\n\r\n
", "votes": null, "replies": [] }, { "id": 22962, "author_name": "paweljankiewicz", "author_url": "", "post_date": "04/17/2013 13:16:47", "content": "
The 2 phases were independent. The results of the first one didn't affect the results of the second one.
\r\nDid he chosen any second phase submission? No. 2 other top 5 competitors did the same. They didn't select any second phase submission. Whether he chose or not any first phase submission is not important. These submissions shouldn't be taken into consideration\r\n when scoring completely different prediction set.
\r\nLet's agree to disagree on this one :). You have right to an opinion. I have mine.
", "votes": null, "replies": [] }, { "id": 22966, "author_name": "alexxanderlarko", "author_url": "", "post_date": "04/17/2013 13:49:05", "content": "Pawet@ You have right to an opinion. I have mine.
\r\nOk!
\r\nI agree with you.
\r\nOnce again, I'm really sorry guys.
\r\nAll the best,
\r\nAlex.
", "votes": null, "replies": [] }, { "id": 22968, "author_name": "ccccat", "author_url": "", "post_date": "04/17/2013 14:44:37", "content": "Not that we are counting votes, but in this particular situation I would agree with _mtb_ and Pawel. (Discloser: I do not have a dog in this fight).
\r\nIn my personal opinion the best Kaggle can do is: award _mtb_ points, money prize and issue one more press release announcing award of addition prize. (They do not need to admit any mistakes publicly. Just something like: "Because of the great success of\r\n the competition we decided to award one more prize for a model developed by...").
", "votes": null, "replies": [] }, { "id": 22971, "author_name": "alexxanderlarko", "author_url": "", "post_date": "04/17/2013 14:55:50", "content": "Sergey Yurgenson@
\r\nА что, ваше предложение очень симпатичное!!!
\r\nAnd that, yuor offer is very cute!!!
", "votes": null, "replies": [] }, { "id": 22973, "author_name": "leustagos", "author_url": "", "post_date": "04/17/2013 15:04:13", "content": "[quote=Sergey Yurgenson;22968]
\r\nNot that we are counting votes, but in this particular situation I would agree with _mtb_ and Pawel. (Discloser: I do not have a dog in this fight).
\r\nIn my personal opinion the best Kaggle can do is: award _mtb_ points, money prize and issue one more press release announcing award of addition prize. (They do not need to admit any mistakes publicly. Just something like: "Because of the great success of\r\n the competition we decided to award one more prize for a model developed by...").
\r\n[/quote]
\r\n\r\n
Totally agree!
\r\nI love kaggle, and I love competing here. I also know that is very difficult to handle these cases, and no piece of software is perfect. I just didn't like how this situation was handled. Its very unfair. For many reasons. I bet that kaggle will be modifiying\r\n their software to avoid this in the future, and that shows that it is something the kaggle platform should have done in the first place.
", "votes": null, "replies": [] }, { "id": 23005, "author_name": "dxsx78425", "author_url": "", "post_date": "04/17/2013 21:19:34", "content": "I forgot about the competition the next day after submission deadline, but just for future references:
\r\n\r\n
Anthony, can you please confirm that following statements are official Kaggle policies?
\r\n\r\n
If so, it is a source of concern about “there would soon be another platform that would undercut us[[Kaggle]]“ (Dr. Nicholas Gruen, please read\r\n\r\nhere)
\r\nI believe many of us (and also sponsors) prefer to compete on a platform that doesn’t have those policies.
\r\nPS: I have informed anyone, whom I named here, by email about this post.
", "votes": null, "replies": [] }, { "id": 23057, "author_name": "mtb26767", "author_url": "", "post_date": "04/18/2013 11:36:40", "content": "dxsx, your questions to Anthony are spot on.
\r\nIn Anthony's email to me, he pulled a quote from the competition rules:
\r\n\r\n\r\n"By participating in the Competition, each Entrant agrees to release, indemnify and hold harmless GE, Kaggle Inc., and their respective affiliates, subsidiaries, advertising and promotions agencies, as applicable, and each of their respective agents, representatives,\r\n officers, directors, shareholders, and employees from and against any injuries, losses, damages, claims, actions and any liability of any kind resulting from or arising out of your participation in or association with the Competition. GE is not responsible\r\n for any miscommunications such as technical failures related to computer, telephone, cable, and unavailable network or server connections, related technical failures, or other failures related to hardware, software or virus, or incomplete, late or misdirected\r\n Entries. GE reserves the right to cancel, modify or suspend the Competition should any computer virus, bug or other technical difficulty or other causes beyond the control of GE corrupt the administration, security or proper play of the Competition, and to\r\n determine winners from among Entries not affected by the corruption, if any, in its sole discretion."
\r\n
When I first read this piece of the email, I interpreted the last sentence as 'GE reseves the right to determine the winners however it likes (i.e. sole discretion)'. And that is why my first response to his email was to\r\nRageQuit Kaggle. It was my interpretation from how Kaggle/GE had handled this situation\r\n so far that they were more concerned with the FlightQuest marketing push than they were identifing the correct winners and models.
\r\nAlso, I was happy to hear you contacted Dr. Nicholas Gruen. I enjoyed reading his insights, but I was also happy to see that he, at least at some point was a Kaggle shareholder. The issues we are discussing on this thread are core to Kaggle's competition\r\n business model (at least I would think they are as they touch on the integriy of the competition).
", "votes": null, "replies": [] }, { "id": 23063, "author_name": "rjm558666", "author_url": "", "post_date": "04/18/2013 13:18:21", "content": "[quote=Sergey Yurgenson;22968]
\r\nNot that we are counting votes, but in this particular situation I would agree with _mtb_ and Pawel. (Discloser: I do not have a dog in this fight).
\r\nIn my personal opinion the best Kaggle can do is: award _mtb_ points, money prize and issue one more press release announcing award of addition prize. (They do not need to admit any mistakes publicly. Just something like: "Because of the great success of\r\n the competition we decided to award one more prize for a model developed by...").
\r\n[/quote]
\r\nI also agree that this would be a reasonable solution. I would like to see that Kaggle is willing to stand up for its competitors. I understand this is probably an embarrassing situation for Kaggle, but I think most people would appreciate an honest effort\r\n to correct the situation. The silence is deafening.
", "votes": null, "replies": [] }, { "id": 23078, "author_name": "mtb26767", "author_url": "", "post_date": "04/18/2013 15:12:15", "content": "Below is a copy/paste email exchange I just had with an executive at GE.
\r\n
\r\nUnfortunate, but from what I piece together between Anthony's email and the one below is that this competition might not be about identifing the best models, but instead is really just a marketing push. The fact that the results of a competition can not be\r\n changed even though the results were impactedby a bug in Kaggles software is troubling.
\r\n
GE Exec sent me this
\r\nFrom: "xxx, xxx (GE, Corporate)"
\r\nTo: "mtberseth@yahoo.com"
\r\nSent: Thursday, April 18, 2013 10:39 AM
\r\nSubject: Kaggle Follow up
\r\n
\r\n
\r\nDear Matt,
\r\n
\r\nThank you for your email. Please be assured that GE has been made aware of this issue and we have been working with Kaggle to understand the problem that arose with your entry. After reviewing this matter in some detail, we do agree with the conclusion that\r\n Kaggle has reached. While we do sympathize with your situation, GE has determined that Kaggle has handled this matter appropriately and agrees that the final results should not be changed. We understand your frustration, but we hope that you will continue\r\n to participate in future Challenges.
\r\nThanks,
\r\n
\r\n
\r\nxxx,
\r\n
\r\nExecutive Director - xxx xxx
\r\nGeneral Electric
\r\n
\r\n
I replied with this:
\r\n
\r\nFrom: Matthew Berseth
\r\nTo: "xxx, xxx (GE, Corporate)"
\r\n
\r\nSent: Thursday, April 18, 2013 11:03 AM
\r\nSubject: Re: Kaggle Follow up
\r\n
\r\nxxx:
\r\n
\r\nI really appreciate the email.
\r\n
\r\nI am not sure if you received the same explanation I have from Kaggle, but if they denied that it was an issue with their software than you received an incorrect description of what happened. I encourage to read the forum thread over at Kaggle.com: https://www.gequest.com/c/flight/forums/t/4284/acknowledging-two-more-great-competitors\r\n to gain a full understanding into some of the limitations of the Kaggle.com software and how this effected the outcome of the competition.
\r\n
\r\nUnfortunately, at this time it is hard for me to envision a situation where I could compete in another competition. Kaggle has disappointed me in how they have handled this situation, and as a competitor I can not rationalize spending the effort I did on FlightQuest\r\n on another competition having experienced everything I did with this competition.
\r\n
\r\nAlso, I will remove your name and email address, but I need to be fair and open to the other competitors so I will be posting this email and my response to the forums. I am sure you understand.\r\n
\r\n
\r\nThanks,
\r\nMatt
\r\n
Edit: rearranged the order.
", "votes": null, "replies": [] }, { "id": 23388, "author_name": "mtb26767", "author_url": "", "post_date": "04/23/2013 23:07:11", "content": "This thread is about dead, so I just wanted to follow up with everyone who is still interested one last time.
\r\nI emailed Anthony again, pointing out one more time the bugs in the Kaggle.com software that prevented my submission from being selected and responding to the other points in his email. I sent that to him last Friday (4/19/2013), but I have not heard a response.\r\n At this point I am not expecting one. I posted the email over \r\nhere so I didn't have to reformat it here.
\r\nI really appreciate all of the support I received from all of the other Kagglers. I know not everyone felt like they could voice their opinions publicly, so I appreciated the private emails, linkedin conversations and skype sessions. For me, this was the\r\n best part of this whole ordeal.
\r\nI was offended that Kaggle offered me money and Kaggle points to keep quiet because at the time I didn't think that was fair to GE or the other competitors. I am probably an idealist, but it felt like Kaggle was trying to pay me off to cover up their issue.\r\n In the end, I did not receive anything.
\r\nThis was a crushing experience for me. I worked very hard on this competition, just like the rest of you - over the US holidays and into February. Then I was ignored by Kaggle for eight days while GE and Kaggle put out press releases, blog posts etc ...\r\n The only communications I heard from Kaggle went a lot like Anthony's email - demonstrating Kaggle's lack of understanding of the issue. That is why my response to Ben's post (the one that started this thread) was filled with emotion. I should of handled that\r\n differently.
\r\nI am going to take a break from Kaggle. I do not think I will consider competing again unless they introduce a formal dispute process that is transparent and handled by a third party. I don't like that Kaggle serves as both judge and jury. Also, there are\r\n a lot lingering questions in this forum and others that I would have liked to see Kaggle answer.
\r\nThis situation has to be resolved!
\r\nI propose the following:
\r\nAnthony and Matt try to agree on a group of people who will look into the matter and give a final binding judgement.
\r\nThe judgement could be one of three:
\r\n1. We don't know (in which case we are back to square one)
\r\n2. Matt is more "to blame" than Kaggle
\r\n3. Kaggle is more "to blame" for the situation than Matt
\r\nI propose further that if this group rules judgement 2, Matt accepts his misfortune.
\r\nAnd if this group rules judgement 3, Matt is added to the winners page in a shared 4th place,
\r\nKaggle pays Matt the 30000 USD and Matt gets the Kaggle points for a 4th place.
\r\nAnthony, could you agree with this?
\r\nMatt, could you agree with this?
\r\n\r\n
Jules van Ligtenberg
", "votes": null, "replies": [] }, { "id": 23470, "author_name": "mtb26767", "author_url": "", "post_date": "04/25/2013 17:49:11", "content": "Yes, I agree and I would like that opportunity.
\r\nKaggle needs to have a formal, transparent dispute process that includes a 3rd party - someone who would not be influenced by the sponser, competitor or Kaggle. The competitions are becoming more complicated: multiple deadlines,\r\n multiple leaderboard files, uploading model archive, etc ... There is a gray area, and software bugs can, do and will continue to happen.
\r\nThe precedent Kaggle has set with how this has been handled so far is not great from a competitors stand point.
", "votes": null, "replies": [] }, { "id": 24157, "author_name": "davidsvaughn", "author_url": "", "post_date": "05/10/2013 17:24:04", "content": "matt I have great sympathy for you. I spent over a month and a half working on this contest (nights mostly) and eventually gave up due to lack of sleep, deteriorating health, and generally disintegrating family cohesion (*slight* exaggeration...).
\r\nI am sure kaggle has some clause in their legal documents somewhere that renders them immune from any sort of legal action. In other words, I'm sure they don't *have* to do the right thing award you the money, even if they are completely in the wrong (which\r\n I agree they are). But, I am confused as to why they don't want to do the right thing anyway. They risk alienating present and future kagglers, and tarnishing the integrity of the whole endeavor. Is it a matter of saving face? i.e. now that they have taken\r\n a certain position, it would be "giving in" to reverse that position? Maybe they are afraid that would open the door to all kinds of disputes...?
\r\nin the absence of any such move by kaggle, maybe GE could skim $30k off of the phase 2 prize pool and give this to Matt? clearly the $$ is there...
\r\nor perhaps there is another way... each of the actual cash $$ winners could take matters into their own hands, and in a show of good sportsmanship skim an equal fraction off of their prizes and pool it together for matt? to get to $30k, each of the prizewinners\r\n would need to take off 12% off their prizes. Perhaps this wouldn't seem quite fair, since then matt would receive 100% of his prize, but everyone else would receive 88% of theirs. To even this out (according to my calculations...) if the winners each skimmed\r\n ~10.7% off their prizes and pooled it together for matt, this would sum to ~89.3% of $30k for matt (~$26,785), as well as ~89.3% of their original prize for everyone else.
\r\n\r\n
Prizewinners?
", "votes": null, "replies": [] }, { "id": 24282, "author_name": "dxsx78425", "author_url": "", "post_date": "05/14/2013 02:10:18", "content": "Well, prize winners are not really winners money-wise!
\r\n\r\n\r\n
Moneywise, winners of Kaggle’s competitions aren’t the real winners! They just get a small portion of money which is spent by sponsors. Finally, look at this comment on MIT Technology review about this contest (http://www.technologyreview.com/news/513141/a-data-crunching-prize-to-cut-flight-delays/)
\r\n\r\nMarkyMark: "Congrats to GE/Alaska/Kaggle to discover a way to pay engineers less. $230k in prize money, 242 engineers, that's\r\n only $950 per engineer to solve your problems! What a steal!"
\r\n\r\n", "votes": null, "replies": [] }, { "id": 24300, "author_name": "davidsvaughn", "author_url": "", "post_date": "05/14/2013 13:45:18", "content": "good point dxsx!
\r\n\r\nin that case, Kaggle can't afford *not* to err on the side of humility, and do the right thing by Matt (and every other kaggler), since they risk alienating the kaggle community and thereby diminishing their cheap labor pool. $30k seems a small price to\r\n pay to keep us all happily working for (almost) nothing...
", "votes": null, "replies": [] }, { "id": 24528, "author_name": "intaka", "author_url": "", "post_date": "05/20/2013 17:26:59", "content": "Raised a good point. In many ways, these competitions decrease the financial value for the solution providers (the competition participants and the market of providers in general) but increase the value of the solution for the competition sponsors (and\r\n companies in general).
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "22679": "", "22686": "", "22693": "", "22695": "", "22700": "", "22705": "", "22715": "", "22810": "", "22867": "", "22868": "", "22889": "", "22896": "", "22900": "", "22921": "", "22923": "", "22932": "", "22933": "", "22934": "", "22942": "", "22944": "", "22948": "", "22949": "", "22951": "", "22952": "", "22960": "", "22962": "", "22966": "", "22968": "", "22971": "", "22973": "", "23005": "", "23057": "", "23063": "", "23078": "", "23388": "", "23451": "", "23470": "", "24157": "", "24282": "", "24300": "", "24528": "" }, "source": "meta" }