{ "id": 12515, "title": "Is anyone actually doing this for the prize money?", "url": "/competitions/malware-classification/discussion/12515", "author_name": "", "post_date": "2015-02-14T19:26:23.523Z", "votes": 7, "comment_count": 24, "views": 10062, "content": "

$16,000 from Microsoft, for a solution that will probably have tremendous economic value to them?  Am I missing something here?  Just curious what reasons people have for competing in some of these competitions if they are already a professional data scientist.  If you are a student, your reasons are obvious. 

\n

Thanks,

\n

Bob

", "messages": [ { "id": "64236", "postDate": "02/14/2015 19:26:23", "content": "

$16,000 from Microsoft, for a solution that will probably have tremendous economic value to them?  Am I missing something here?  Just curious what reasons people have for competing in some of these competitions if they are already a professional data scientist.  If you are a student, your reasons are obvious. 

\n

Thanks,

\n

Bob

", "rawMarkdown": "", "votes": null }, { "id": "64238", "postDate": "02/14/2015 19:57:28", "content": "

It's not just prize money: you learn a lot by working on new types of problems, and by sharing ideas on the forums. Also, good performance in a kaggle competition is a nice way to prove your skills to your employer or to new customers. And the competitive context can be very motivating.

", "rawMarkdown": "", "votes": null }, { "id": "64239", "postDate": "02/14/2015 20:10:23", "content": "

Let me point out another angle to this.  If Microsoft offered more prize money, more people would spend more time on it, and they would likely get a much better result.  Next to the Booz-Allen-Hamilton prize amount $16,000 is actually quite laughable in all honesty.  Especially when you consider the total value of the Microsoft corporation.  This is like all those graphics artists who work for free to "get their work out there."

", "rawMarkdown": "", "votes": null }, { "id": "64242", "postDate": "02/14/2015 20:37:33", "content": "

Here's my take:

\n\n\n

As far as the Microsoft prize money, a couple of thoughts:

\n", "rawMarkdown": "", "votes": null }, { "id": "64244", "postDate": "02/14/2015 20:57:52", "content": "

We have been allowing smaller prize projects under an open source license option, the motivation being that if the IP is so valuable, you are welcome to take it and create value with it. A couple other points worth calling out on this particular situation:

\n
    \n
  1. This was a conference problem for WWW / BIG, which Microsoft agreed to back with a prize.
    2. \n
  2. Kaggle is taking no fee to host this. We think it's a really interesting and novel ML problem and are eating the costs to run it.
    3. \n
  3. We deemed this particular problem to have a public good angle. Malware sucks for everyone except malware authors. Helping stop malware is akin to herd immunity in vaccinations; everybody wins.
    4. \n
  4. Nobody is obligated to be here, obligated to work on this, obligated to accept a prize, or obligated to share their code. If you judge the value of your code to be greater than the value you lose by open sourcing it, don't do it.
    5. \n
\n

We are very tuned in to the complaints Kaggle gets about "honest pay for an honest day's work." I don't want to write an essay right now on markets and crowdsourcing, but we do believe this is very different than, say, a design spec where 20 people make mockups and only one is chosen.

\n

On this note:

\n

Is there another way you'd like to see industry datasets released?

\n

Would you rather this have run without any prize?

\n

Is the distinction of open sourcing meaningful to you?

", "rawMarkdown": "", "votes": null }, { "id": "64248", "postDate": "02/14/2015 21:56:44", "content": "

     Okay, I'll just answer everyone's points all at once.  First of all, I don't disagree with what you guys are saying, so can we be friends again?  I was just curious how many people were here primarily for the prize money, not sure why that seemed to offend. 

\n

      I work as a data scientist already, so I am trying to point out that for some professional data scientists (with families and serious financial commitments) participating in contests with low prizes versus ones with high prizes is an enormous factor in deciding which ones to do.    Also, for those of you who work as data scientists already you know what we get paid, and doing side consulting on top of a day job is quite lucrative. 

\n

     Personally, I would rather compete in competitions like this and have fun and learn new things, but in real life you don't always have that luxury.  All I asked was a simple question which was if anyone was doing this competition simply for the prize money.  Is that morally wrong somehow? 

\n

     As far as "good luck finding somewhere else besides Kaggle."  Now that I think about it, I haven't even searched so I don't have the slightest clue if there are sites out there that compete with Kaggle or not.  If I find any I'm not going to tell anyone here, that's less competition for me.  :)

\n

     BTW, Netflix ended up using their second place answer, so you were wrong on them not getting anything for their money.

\n

Thanks,Bob

", "rawMarkdown": "", "votes": null }, { "id": "64257", "postDate": "02/15/2015 02:58:24", "content": "

Bob,

\n

I took your question completely at face value, and wasn't offended by it at all. It's very rational.

\n

Of course, matter-of-fact answers can come across as cold or even adversarial on the intertubes. I didn't intend that at all.

\n

I put contests in 4 buckets:

\n
    \n
  1. I want to get in the Top 10, and hit money if possible
    2. \n
  2. I want to learn a new tool, and the contest is a good excuse to do so
    3. \n
  3. Might as well get a few extra kaggle points by jumping in
    4. \n
  4. Don't have the time or interest
    5. \n
\n\n

If you do find another machine learning competition site, please, please, please don't share it. I barely have time to work on the ones hosted here.   :-)

", "rawMarkdown": "", "votes": null }, { "id": "64259", "postDate": "02/15/2015 03:46:35", "content": "

Thanks inversion for clarifying your intent and meaning.  I'm looking forward to future competitions we may be in, and may the best model win.  If you Google this particular challenge on the internet I think you will be surprised by the opinions that you read (from data scientists).  People seem to forget that the motivations of some companies are always viewed with skepticism (whether fair or not).  If Halliburton posted a challenge would you view it the same way as a challenge posted by the Red Cross?  Whether that is fair or not, it's the world we live in.

\n

Thanks,

\n

Bob

", "rawMarkdown": "", "votes": null }, { "id": "64281", "postDate": "02/15/2015 19:22:22", "content": "

If you were really interested in solving a problem like this with machine learning, without Microsoft and Kaggle, you probably would have to pay around 100$ to play around with this dataset (if made available at all).

\n

Also haven't seen mentioned "competing on a leaderboard with good competition": More than CV and selecting your own evaluation metrics, Kaggle offers a method to objectively compare your approaches, skills, insights with others. Some people started their own businesses after doing well on Kaggle competitions. Some people wrote papers on good results that got them published. Some people got PhD. positions due to good overall results on Kaggle. Some people got hired. etc. All less likely if they had created the competition task in isolation with only themselves to compete with.

\n

In that sense Kaggle is also harsh and "too fair". A student happening upon good SVM-hyper params can outperform a seasoned data scientist using elaborate algo's and years of domain expertise. Some of that ego-hurt is then reflected in the (fair and unfair) criticism of Kaggle.

\n

In the end it is a sport: Create the best model using the available data. In many competitions better models could be made by crawling the web. But that's not the sport. It's competing on a common ground with others using the same data.

", "rawMarkdown": "", "votes": null }, { "id": "64285", "postDate": "02/15/2015 19:59:11", "content": "

My criticism is of Microsoft here, not Kaggle.  Nor am I criticizing the crowd sourcing problem solving methodology that Kaggle and its 5 other competitors engage in.  All I am saying is Microsoft can and should pay more for a problem of this scope.  They paid more than $16,000 just to put the thing together I'm sure.

", "rawMarkdown": "", "votes": null }, { "id": "64287", "postDate": "02/15/2015 20:11:29", "content": "

The 16k is (in my eyes quite hefty) prize money for a data science competition and this prize money is sponsored by Microsoft. It is not money to engineer a pipeline and implement a model in production for a problem of this scope. Winning models are open-sourced.

", "rawMarkdown": "", "votes": null }, { "id": "64688", "postDate": "02/21/2015 02:43:06", "content": "

If you win this competition, I'm pretty sure the $16,000 will just be the start of the money stream if you play your cards right. This is especially true if you have a novel approach, and you know how to sell your expertise. Good Kagglers know a lot more than one competition worth of insight, and such insight can be leveraged for money if desired. 

", "rawMarkdown": "", "votes": null }, { "id": "64690", "postDate": "02/21/2015 02:55:16", "content": "

@Mike - That is an excellent point.  It is quite possible that the winner of this competition may become a very valuable commodity overnight.  Thank you for offering the relevant insight.

", "rawMarkdown": "", "votes": null }, { "id": "64717", "postDate": "02/21/2015 21:15:03", "content": "

yes, I am. I need to buy some groceries :D

", "rawMarkdown": "", "votes": null }, { "id": "64726", "postDate": "02/22/2015 04:26:30", "content": "

There is one attraction no one has mentioned.  I am a professional data scientist.  Kaggle competitions are often more interesting than the day-to-day work I get paid to do!  

\n

Sure, the money and recognition would be great, but I get pulled in because I like fascinating data problems.  In this case, I've never seen loads of disassembled virus code before, and I enjoy thinking about how this could be attacked.  Nothing like a good challenge, eh?

\n

I only regret that my day job keeps me from devoting more time...

\n

-Tom

", "rawMarkdown": "", "votes": null }, { "id": "64737", "postDate": "02/22/2015 16:13:49", "content": "

@Tom - Good insight.  Sometimes value is measured in things other than dollars.

", "rawMarkdown": "", "votes": null }, { "id": "64751", "postDate": "02/23/2015 02:13:24", "content": "

There is a piece of the puzzle missing in these contests.  

\n

For those of us who can't yet figure out how to load the data let alone worry about winning there needs to be some occasional clues and hints so we can play along in our own way....   Yes I am working through Ng's ML course, redoing my calculus/linear algebra 1&2 and reading along but even the mechanics of downloading and addressing the data can be a puzzle.   Trivial data and real data aren't the same beasts.

\n

The occasional tutorial or detailed walk through of previous Kaggles would be very helpful for those of us who are likely to be more spectators than players in the foreseeable future.

\n

There's lots of value to be had here.   I think Kaggle could do a little more for the learning players. 
From a business perspective there could very well be a model that is self funded through subscription, as well as, advertising and Industry sponsorship.     I of course prefer an industry funded model but if it gets practical tutorials and teachings in data analysis on the site I could probably be persuaded to give up some of the allowance money my wife lets me keep.

", "rawMarkdown": "", "votes": null }, { "id": "64752", "postDate": "02/23/2015 02:26:51", "content": "

@Robert - There are some good tutorials on the Titanic challenge in both R and Python.  In fact, that's how I learned how Kaggle contests work.  I think they may be what you are looking for.

", "rawMarkdown": "", "votes": null }, { "id": "64753", "postDate": "02/23/2015 02:33:06", "content": "

thanks beaker!

", "rawMarkdown": "", "votes": null }, { "id": "66321", "postDate": "03/16/2015 03:52:29", "content": "

Beaker,

\n

I am competing on Kaggle because  (in order)

\n

1)  I am looking to improve my data science skills to become a professional data science.  ( I am in mechanical engineering now )

\n

2)  I am looking to demonstrate to a potential employer that I am good at this, despite not having a C.S. degree

\n

3)  The prize money on the AXA contest is large enough to mean at least something to me

\n

4)  It's fun

\n\n

I'm not competing on this particular Malware contest because ( in reverse order )

\n

1)  I don't have any particular knowledge of malware I could bring to bear

\n

2)  the size of the data set would make it challenging for me, and potentially a cost source instead of a revenue source

\n

3)  After spending the last 3.5 months doing Santa competition & AXA competition in all my spare time, I promised my wife I wouldn't do any more for a while

", "rawMarkdown": "", "votes": null }, { "id": "66323", "postDate": "03/16/2015 04:10:09", "content": "

@Scott

\n

I hear you on the proving you are good without the degree.  My degree is in Chemistry and I taught myself everything computer related over the years.  Some HR folks aren't bright enough to understand we work in a field that is self-driven, not by getting a piece of paper from some random college, but by our love of what we do.  Good insight.

\n

Beaker

", "rawMarkdown": "", "votes": null }, { "id": "66324", "postDate": "03/16/2015 04:27:01", "content": "

[quote=Scott H;66321]

\n

3)  After spending the last 3.5 months doing Santa competition & AXA competition in all my spare time, I promised my wife I wouldn't do any more for a while

\n

[/quote]

\n

Let's connect offline. I have mastered a series of tricks to help with that :-)

", "rawMarkdown": "", "votes": null }, { "id": "66639", "postDate": "03/17/2015 05:19:18", "content": "

Beaker,

It's great to see a fellow chemist here; I am currently finishing up my PhD in theoretical chemistry. 

I use Kaggle competitions primarily to hone and advertise my skills in hopes of getting a job in data science once I graduate. The prize money would be nice (grad school does not pay well), but I am more concerned with the experience.

", "rawMarkdown": "", "votes": null }, { "id": "68829", "postDate": "03/29/2015 08:38:40", "content": "

I couldn't help but add my two cents to this conversation..... First let me say first that I really like the idea of Kaggle and the fact in a relatively short period of time that I have belonged to this group and read the forums I have learned a good number of different approaches people are using to tackle these data mining (machine learning) problems.... my personal preferences is towards the biological inspired methods... aside from learning from each other I will say that there is a real danger of devaluing the worth of a data scientist or machine learning experts in countries like the USA. The $16,000 prize being offered here is very small sum of money to someone living and working as a DS/DM in say some place like the New York City area of the USA. However, in some other part of the world, like certain parts of Eastern Europe, the $16000 could represent a years income and worth having a team of the best and brightest from that area of the world working on these problem...

\n

There was a time when software development required very experience personnel to build systems...I would day between the 1960's mid 2000's. These days are as a result of the evolution numerous wonderful software frameworks and libraries you don't need to be the brightest or most skilled person on the block to deliver a product your boss will not complain about (from experience, IMHO, most upper management are really unknowing and borderline incompetent, most of the time, since, most of time time they are focused on time-lines and deliverables not necessarily quality of the work that most of the time don't understand.)

\n

There was never a shortage of skilled technology  worker in the USA... It was the large consulting firms desire for greater profit that created that illusion of a shortage; along with Allen Greenspan in front of congress saying in was in the best interest of the country to break the back of the lucrative independent software consultation practices... even the IRS got in it lick by changing the corporate laws as to discriminate again the single person corporation... $1400 to $2000 per day practice was readily possible

\n

Now I hear my very talented friends from Eastern Europe complaining about another group saying things like "....having a difficult time finding a contract that pays a reasonable rate.... because of low skilled workers from ...."

\n

I am a very hands on developer. Unless otherwise told/paid, I write everything in C++. All my AI/ML and data-mining tools are written by me and I don't use anyone libraries -- I like to know exactly why and how my ML is doing what it does; besides it is fun visualizing the actual execution of the tools I build and understand the failing of a particular algorithm/method....

\n

What is surprising to me here, in these these competitions, are number of people using over the counter tools/libraries --I am not saying this is a bad thing....

\n

But I have been doing this since this late 1970's and I see the writing on the wall... the $16000 is a lot of money (perhaps a whole years income) to certain groups/people come. With most DM/DS being address by ever improving "STANDARD" set of Data Mining/Scientist tools, it is just a matter of time before we see a nose dive in what companies are willing to pay for DM/DS in the USA and other high compensation states.

\n

Kaggle is a great place to practice conventional DM/DS and even like me learn a few new tricks using conventional methods .... However, I am pretty sure, if someone, knowingly, had a yet to be published, novel/non-standard approach to these problems, would they willingly release exactly how they got there solution, for a relatively small sum compared to the New York City area of the USA income levels.

\n

Just my humble opinion ;)

", "rawMarkdown": "", "votes": null }, { "id": "142570", "postDate": "11/03/2016 03:08:35", "content": "

How can someone earn 5,00,000$, if he's competitive enough to win in Kaggle competition. Would like to know more.

\n

[quote=inversion;64242]

\n

Here's my take:

\n\n

As far as the Microsoft prize money, a couple of thoughts:

\n\n

[/quote]

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 64238, "author_name": "gertjac", "author_url": "", "post_date": "02/14/2015 19:57:28", "content": "

It's not just prize money: you learn a lot by working on new types of problems, and by sharing ideas on the forums. Also, good performance in a kaggle competition is a nice way to prove your skills to your employer or to new customers. And the competitive context can be very motivating.

", "votes": null, "replies": [] }, { "id": 64239, "author_name": "robertbunn", "author_url": "", "post_date": "02/14/2015 20:10:23", "content": "

Let me point out another angle to this.  If Microsoft offered more prize money, more people would spend more time on it, and they would likely get a much better result.  Next to the Booz-Allen-Hamilton prize amount $16,000 is actually quite laughable in all honesty.  Especially when you consider the total value of the Microsoft corporation.  This is like all those graphics artists who work for free to "get their work out there."

", "votes": null, "replies": [] }, { "id": 64242, "author_name": "inversion", "author_url": "", "post_date": "02/14/2015 20:37:33", "content": "

Here's my take:

\n\n\n

As far as the Microsoft prize money, a couple of thoughts:

\n", "votes": null, "replies": [] }, { "id": 64244, "author_name": "wcukierski", "author_url": "", "post_date": "02/14/2015 20:57:52", "content": "

We have been allowing smaller prize projects under an open source license option, the motivation being that if the IP is so valuable, you are welcome to take it and create value with it. A couple other points worth calling out on this particular situation:

\n
    \n
  1. This was a conference problem for WWW / BIG, which Microsoft agreed to back with a prize.
    2. \n
  2. Kaggle is taking no fee to host this. We think it's a really interesting and novel ML problem and are eating the costs to run it.
    3. \n
  3. We deemed this particular problem to have a public good angle. Malware sucks for everyone except malware authors. Helping stop malware is akin to herd immunity in vaccinations; everybody wins.
    4. \n
  4. Nobody is obligated to be here, obligated to work on this, obligated to accept a prize, or obligated to share their code. If you judge the value of your code to be greater than the value you lose by open sourcing it, don't do it.
    5. \n
\n

We are very tuned in to the complaints Kaggle gets about "honest pay for an honest day's work." I don't want to write an essay right now on markets and crowdsourcing, but we do believe this is very different than, say, a design spec where 20 people make mockups and only one is chosen.

\n

On this note:

\n

Is there another way you'd like to see industry datasets released?

\n

Would you rather this have run without any prize?

\n

Is the distinction of open sourcing meaningful to you?

", "votes": null, "replies": [] }, { "id": 64248, "author_name": "robertbunn", "author_url": "", "post_date": "02/14/2015 21:56:44", "content": "

     Okay, I'll just answer everyone's points all at once.  First of all, I don't disagree with what you guys are saying, so can we be friends again?  I was just curious how many people were here primarily for the prize money, not sure why that seemed to offend. 

\n

      I work as a data scientist already, so I am trying to point out that for some professional data scientists (with families and serious financial commitments) participating in contests with low prizes versus ones with high prizes is an enormous factor in deciding which ones to do.    Also, for those of you who work as data scientists already you know what we get paid, and doing side consulting on top of a day job is quite lucrative. 

\n

     Personally, I would rather compete in competitions like this and have fun and learn new things, but in real life you don't always have that luxury.  All I asked was a simple question which was if anyone was doing this competition simply for the prize money.  Is that morally wrong somehow? 

\n

     As far as "good luck finding somewhere else besides Kaggle."  Now that I think about it, I haven't even searched so I don't have the slightest clue if there are sites out there that compete with Kaggle or not.  If I find any I'm not going to tell anyone here, that's less competition for me.  :)

\n

     BTW, Netflix ended up using their second place answer, so you were wrong on them not getting anything for their money.

\n

Thanks,Bob

", "votes": null, "replies": [] }, { "id": 64257, "author_name": "inversion", "author_url": "", "post_date": "02/15/2015 02:58:24", "content": "

Bob,

\n

I took your question completely at face value, and wasn't offended by it at all. It's very rational.

\n

Of course, matter-of-fact answers can come across as cold or even adversarial on the intertubes. I didn't intend that at all.

\n

I put contests in 4 buckets:

\n
    \n
  1. I want to get in the Top 10, and hit money if possible
    2. \n
  2. I want to learn a new tool, and the contest is a good excuse to do so
    3. \n
  3. Might as well get a few extra kaggle points by jumping in
    4. \n
  4. Don't have the time or interest
    5. \n
\n\n

If you do find another machine learning competition site, please, please, please don't share it. I barely have time to work on the ones hosted here.   :-)

", "votes": null, "replies": [] }, { "id": 64259, "author_name": "robertbunn", "author_url": "", "post_date": "02/15/2015 03:46:35", "content": "

Thanks inversion for clarifying your intent and meaning.  I'm looking forward to future competitions we may be in, and may the best model win.  If you Google this particular challenge on the internet I think you will be surprised by the opinions that you read (from data scientists).  People seem to forget that the motivations of some companies are always viewed with skepticism (whether fair or not).  If Halliburton posted a challenge would you view it the same way as a challenge posted by the Red Cross?  Whether that is fair or not, it's the world we live in.

\n

Thanks,

\n

Bob

", "votes": null, "replies": [] }, { "id": 64281, "author_name": "triskelion", "author_url": "", "post_date": "02/15/2015 19:22:22", "content": "

If you were really interested in solving a problem like this with machine learning, without Microsoft and Kaggle, you probably would have to pay around 100$ to play around with this dataset (if made available at all).

\n

Also haven't seen mentioned "competing on a leaderboard with good competition": More than CV and selecting your own evaluation metrics, Kaggle offers a method to objectively compare your approaches, skills, insights with others. Some people started their own businesses after doing well on Kaggle competitions. Some people wrote papers on good results that got them published. Some people got PhD. positions due to good overall results on Kaggle. Some people got hired. etc. All less likely if they had created the competition task in isolation with only themselves to compete with.

\n

In that sense Kaggle is also harsh and "too fair". A student happening upon good SVM-hyper params can outperform a seasoned data scientist using elaborate algo's and years of domain expertise. Some of that ego-hurt is then reflected in the (fair and unfair) criticism of Kaggle.

\n

In the end it is a sport: Create the best model using the available data. In many competitions better models could be made by crawling the web. But that's not the sport. It's competing on a common ground with others using the same data.

", "votes": null, "replies": [] }, { "id": 64285, "author_name": "robertbunn", "author_url": "", "post_date": "02/15/2015 19:59:11", "content": "

My criticism is of Microsoft here, not Kaggle.  Nor am I criticizing the crowd sourcing problem solving methodology that Kaggle and its 5 other competitors engage in.  All I am saying is Microsoft can and should pay more for a problem of this scope.  They paid more than $16,000 just to put the thing together I'm sure.

", "votes": null, "replies": [] }, { "id": 64287, "author_name": "triskelion", "author_url": "", "post_date": "02/15/2015 20:11:29", "content": "

The 16k is (in my eyes quite hefty) prize money for a data science competition and this prize money is sponsored by Microsoft. It is not money to engineer a pipeline and implement a model in production for a problem of this scope. Winning models are open-sourced.

", "votes": null, "replies": [] }, { "id": 64688, "author_name": "mikeskim", "author_url": "", "post_date": "02/21/2015 02:43:06", "content": "

If you win this competition, I'm pretty sure the $16,000 will just be the start of the money stream if you play your cards right. This is especially true if you have a novel approach, and you know how to sell your expertise. Good Kagglers know a lot more than one competition worth of insight, and such insight can be leveraged for money if desired. 

", "votes": null, "replies": [] }, { "id": 64690, "author_name": "robertbunn", "author_url": "", "post_date": "02/21/2015 02:55:16", "content": "

@Mike - That is an excellent point.  It is quite possible that the winner of this competition may become a very valuable commodity overnight.  Thank you for offering the relevant insight.

", "votes": null, "replies": [] }, { "id": 64717, "author_name": "jiweiliu", "author_url": "", "post_date": "02/21/2015 21:15:03", "content": "

yes, I am. I need to buy some groceries :D

", "votes": null, "replies": [] }, { "id": 64726, "author_name": "", "author_url": "", "post_date": "02/22/2015 04:26:30", "content": "

There is one attraction no one has mentioned.  I am a professional data scientist.  Kaggle competitions are often more interesting than the day-to-day work I get paid to do!  

\n

Sure, the money and recognition would be great, but I get pulled in because I like fascinating data problems.  In this case, I've never seen loads of disassembled virus code before, and I enjoy thinking about how this could be attacked.  Nothing like a good challenge, eh?

\n

I only regret that my day job keeps me from devoting more time...

\n

-Tom

", "votes": null, "replies": [] }, { "id": 64737, "author_name": "robertbunn", "author_url": "", "post_date": "02/22/2015 16:13:49", "content": "

@Tom - Good insight.  Sometimes value is measured in things other than dollars.

", "votes": null, "replies": [] }, { "id": 64751, "author_name": "robertfontaine", "author_url": "", "post_date": "02/23/2015 02:13:24", "content": "

There is a piece of the puzzle missing in these contests.  

\n

For those of us who can't yet figure out how to load the data let alone worry about winning there needs to be some occasional clues and hints so we can play along in our own way....   Yes I am working through Ng's ML course, redoing my calculus/linear algebra 1&2 and reading along but even the mechanics of downloading and addressing the data can be a puzzle.   Trivial data and real data aren't the same beasts.

\n

The occasional tutorial or detailed walk through of previous Kaggles would be very helpful for those of us who are likely to be more spectators than players in the foreseeable future.

\n

There's lots of value to be had here.   I think Kaggle could do a little more for the learning players. 
From a business perspective there could very well be a model that is self funded through subscription, as well as, advertising and Industry sponsorship.     I of course prefer an industry funded model but if it gets practical tutorials and teachings in data analysis on the site I could probably be persuaded to give up some of the allowance money my wife lets me keep.

", "votes": null, "replies": [] }, { "id": 64752, "author_name": "robertbunn", "author_url": "", "post_date": "02/23/2015 02:26:51", "content": "

@Robert - There are some good tutorials on the Titanic challenge in both R and Python.  In fact, that's how I learned how Kaggle contests work.  I think they may be what you are looking for.

", "votes": null, "replies": [] }, { "id": 64753, "author_name": "robertfontaine", "author_url": "", "post_date": "02/23/2015 02:33:06", "content": "

thanks beaker!

", "votes": null, "replies": [] }, { "id": 66321, "author_name": "scottdh", "author_url": "", "post_date": "03/16/2015 03:52:29", "content": "

Beaker,

\n

I am competing on Kaggle because  (in order)

\n

1)  I am looking to improve my data science skills to become a professional data science.  ( I am in mechanical engineering now )

\n

2)  I am looking to demonstrate to a potential employer that I am good at this, despite not having a C.S. degree

\n

3)  The prize money on the AXA contest is large enough to mean at least something to me

\n

4)  It's fun

\n\n

I'm not competing on this particular Malware contest because ( in reverse order )

\n

1)  I don't have any particular knowledge of malware I could bring to bear

\n

2)  the size of the data set would make it challenging for me, and potentially a cost source instead of a revenue source

\n

3)  After spending the last 3.5 months doing Santa competition & AXA competition in all my spare time, I promised my wife I wouldn't do any more for a while

", "votes": null, "replies": [] }, { "id": 66323, "author_name": "robertbunn", "author_url": "", "post_date": "03/16/2015 04:10:09", "content": "

@Scott

\n

I hear you on the proving you are good without the degree.  My degree is in Chemistry and I taught myself everything computer related over the years.  Some HR folks aren't bright enough to understand we work in a field that is self-driven, not by getting a piece of paper from some random college, but by our love of what we do.  Good insight.

\n

Beaker

", "votes": null, "replies": [] }, { "id": 66324, "author_name": "adjgiulio", "author_url": "", "post_date": "03/16/2015 04:27:01", "content": "

[quote=Scott H;66321]

\n

3)  After spending the last 3.5 months doing Santa competition & AXA competition in all my spare time, I promised my wife I wouldn't do any more for a while

\n

[/quote]

\n

Let's connect offline. I have mastered a series of tricks to help with that :-)

", "votes": null, "replies": [] }, { "id": 66639, "author_name": "thekannman", "author_url": "", "post_date": "03/17/2015 05:19:18", "content": "

Beaker,

It's great to see a fellow chemist here; I am currently finishing up my PhD in theoretical chemistry. 

I use Kaggle competitions primarily to hone and advertise my skills in hopes of getting a job in data science once I graduate. The prize money would be nice (grad school does not pay well), but I am more concerned with the experience.

", "votes": null, "replies": [] }, { "id": 68829, "author_name": "spaceman", "author_url": "", "post_date": "03/29/2015 08:38:40", "content": "

I couldn't help but add my two cents to this conversation..... First let me say first that I really like the idea of Kaggle and the fact in a relatively short period of time that I have belonged to this group and read the forums I have learned a good number of different approaches people are using to tackle these data mining (machine learning) problems.... my personal preferences is towards the biological inspired methods... aside from learning from each other I will say that there is a real danger of devaluing the worth of a data scientist or machine learning experts in countries like the USA. The $16,000 prize being offered here is very small sum of money to someone living and working as a DS/DM in say some place like the New York City area of the USA. However, in some other part of the world, like certain parts of Eastern Europe, the $16000 could represent a years income and worth having a team of the best and brightest from that area of the world working on these problem...

\n

There was a time when software development required very experience personnel to build systems...I would day between the 1960's mid 2000's. These days are as a result of the evolution numerous wonderful software frameworks and libraries you don't need to be the brightest or most skilled person on the block to deliver a product your boss will not complain about (from experience, IMHO, most upper management are really unknowing and borderline incompetent, most of the time, since, most of time time they are focused on time-lines and deliverables not necessarily quality of the work that most of the time don't understand.)

\n

There was never a shortage of skilled technology  worker in the USA... It was the large consulting firms desire for greater profit that created that illusion of a shortage; along with Allen Greenspan in front of congress saying in was in the best interest of the country to break the back of the lucrative independent software consultation practices... even the IRS got in it lick by changing the corporate laws as to discriminate again the single person corporation... $1400 to $2000 per day practice was readily possible

\n

Now I hear my very talented friends from Eastern Europe complaining about another group saying things like "....having a difficult time finding a contract that pays a reasonable rate.... because of low skilled workers from ...."

\n

I am a very hands on developer. Unless otherwise told/paid, I write everything in C++. All my AI/ML and data-mining tools are written by me and I don't use anyone libraries -- I like to know exactly why and how my ML is doing what it does; besides it is fun visualizing the actual execution of the tools I build and understand the failing of a particular algorithm/method....

\n

What is surprising to me here, in these these competitions, are number of people using over the counter tools/libraries --I am not saying this is a bad thing....

\n

But I have been doing this since this late 1970's and I see the writing on the wall... the $16000 is a lot of money (perhaps a whole years income) to certain groups/people come. With most DM/DS being address by ever improving "STANDARD" set of Data Mining/Scientist tools, it is just a matter of time before we see a nose dive in what companies are willing to pay for DM/DS in the USA and other high compensation states.

\n

Kaggle is a great place to practice conventional DM/DS and even like me learn a few new tricks using conventional methods .... However, I am pretty sure, if someone, knowingly, had a yet to be published, novel/non-standard approach to these problems, would they willingly release exactly how they got there solution, for a relatively small sum compared to the New York City area of the USA income levels.

\n

Just my humble opinion ;)

", "votes": null, "replies": [] }, { "id": 142570, "author_name": "sohilgupta", "author_url": "", "post_date": "11/03/2016 03:08:35", "content": "

How can someone earn 5,00,000$, if he's competitive enough to win in Kaggle competition. Would like to know more.

\n

[quote=inversion;64242]

\n

Here's my take:

\n\n

As far as the Microsoft prize money, a couple of thoughts:

\n\n

[/quote]

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "64236": "", "64238": "", "64239": "", "64242": "", "64244": "", "64248": "", "64257": "", "64259": "", "64281": "", "64285": "", "64287": "", "64688": "", "64690": "", "64717": "", "64726": "", "64737": "", "64751": "", "64752": "", "64753": "", "66321": "", "66323": "", "66324": "", "66639": "", "68829": "", "142570": "" }, "source": "meta" }