{ "id": 13291, "title": "Amazing performance", "url": "/competitions/malware-classification/discussion/13291", "author_name": "", "post_date": "2015-04-09T05:06:05.527Z", "votes": null, "comment_count": 35, "views": 7245, "content": "

Although private LB may tell a different story, it is still amazing that top 2 teams (almost) get all answers correct in public LB. I have only seen 0 loss or 100% accuracy in Knowledge contests.  Great work!

", "messages": [ { "id": "70087", "postDate": "04/09/2015 05:06:05", "content": "

Although private LB may tell a different story, it is still amazing that top 2 teams (almost) get all answers correct in public LB. I have only seen 0 loss or 100% accuracy in Knowledge contests.  Great work!

", "rawMarkdown": "", "votes": null }, { "id": "70091", "postDate": "04/09/2015 06:48:31", "content": "

0.0000000 log loss. That's something you don't see every day.

", "rawMarkdown": "", "votes": null }, { "id": "70092", "postDate": "04/09/2015 06:59:40", "content": "

Hand labelling, I bet!!!

", "rawMarkdown": "", "votes": null }, { "id": "70093", "postDate": "04/09/2015 07:04:19", "content": "

Can we have the hand labelling classification rules please :)

", "rawMarkdown": "", "votes": null }, { "id": "70094", "postDate": "04/09/2015 07:06:39", "content": "

With so few outliers like 15 or 20, hand labeling would be no different from calibration. So I don't think it is against any rules. or I can not tell the difference.

", "rawMarkdown": "", "votes": null }, { "id": "70095", "postDate": "04/09/2015 07:13:01", "content": "

the rules do say:

\n

"Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records."

\n

I don't know why it says 'may' instead of 'should/shall' 

", "rawMarkdown": "", "votes": null }, { "id": "70096", "postDate": "04/09/2015 07:21:59", "content": "

how is hand labeling defined? If, for example, I look through the dataset and see that some class contains a string in the asm files that is unique to it. Is labeling all files that contain that string as belonging to that class considered hand labeling or not?

", "rawMarkdown": "", "votes": null }, { "id": "70098", "postDate": "04/09/2015 07:29:08", "content": "

Think this is called "signature based detection" not hand labeling :)

", "rawMarkdown": "", "votes": null }, { "id": "70113", "postDate": "04/09/2015 13:40:57", "content": "

I'm going to bet there will be a massive shift from public to private.

", "rawMarkdown": "", "votes": null }, { "id": "70125", "postDate": "04/09/2015 15:20:55", "content": "

I bet the top 2 will remain their super performance in private, and they will tie in 0.000001! 

", "rawMarkdown": "", "votes": null }, { "id": "70129", "postDate": "04/09/2015 17:38:01", "content": "

what about the 2 after top2 ? :P

", "rawMarkdown": "", "votes": null }, { "id": "70142", "postDate": "04/09/2015 19:12:18", "content": "

[quote=Abhishek;70129]

\n

what about the 2 after top2 ? :P

\n

[/quote]

\n

I have no idea. Maybe we could benefit from a new round of CV sharing. Our best signal model achieves 4-fold cv of 0.0051 and 0.0044 public LB. What about yours? :P

", "rawMarkdown": "", "votes": null }, { "id": "70145", "postDate": "04/09/2015 19:46:46", "content": "

I'm in the same ballpark -- .0052 10-fold CV, .0041 LB.

", "rawMarkdown": "", "votes": null }, { "id": "70146", "postDate": "04/09/2015 19:51:33", "content": "

Thank you all for sharing :D

", "rawMarkdown": "", "votes": null }, { "id": "70175", "postDate": "04/10/2015 06:13:51", "content": "", "rawMarkdown": "", "votes": null }, { "id": "70193", "postDate": "04/10/2015 09:25:21", "content": "

Lol

", "rawMarkdown": "", "votes": null }, { "id": "70194", "postDate": "04/10/2015 09:38:38", "content": "

Don't be disappointed soon Sachin, Go on.

\n

There are still 7 days to the end ;-)

", "rawMarkdown": "", "votes": null }, { "id": "71271", "postDate": "04/11/2015 07:37:37", "content": "

[quote=Abhishek;70193]

\n

Lol

\n

[/quote]

\n

Hi Abhishek,

\n

Can you please explain if someone has got a score of 0 on the public leaderboard and he chooses 2 best entries with scores 0 and 0 how can private leaderboard score be different. Since private data is never revealed to the participants and we are not asked to submit our code, how can they determine the results on private data if it is not a subset of public data with a different mix. Or am I totally missing something?

", "rawMarkdown": "", "votes": null }, { "id": "71273", "postDate": "04/11/2015 08:17:43", "content": "

@Sachin: you submit 10k predictions. Some 3k (fixed on kaggle side) used for calculate Public LB. Other 7k will be used for calculating Private score.

\n

So, there is no problem.

", "rawMarkdown": "", "votes": null }, { "id": "71275", "postDate": "04/11/2015 08:38:32", "content": "

[quote=Mikhail Trofimov;71273]

\n

@Sachin: you submit 10k predictions. Some 3k (fixed on kaggle side) used for calculate Public LB. Other 7k will be used for calculating Private score.

\n

So, there is no problem.

\n

[/quote]

\n

Thanks Mikhail

", "rawMarkdown": "", "votes": null }, { "id": "71679", "postDate": "04/15/2015 11:39:11", "content": "

[quote=Abhishek;70092]

\n

Hand labelling, I bet!!!

\n

[/quote]

\n\n

Do you still bet on that one? :)

", "rawMarkdown": "", "votes": null }, { "id": "71680", "postDate": "04/15/2015 11:52:42", "content": "

[quote=Stergios;71679]

\n

Do you still bet on that one? :)

\n

[/quote]

\n

Abhishek would never do something like that :p . It is not in his mentality . 

", "rawMarkdown": "", "votes": null }, { "id": "71681", "postDate": "04/15/2015 11:55:37", "content": "

[quote=Stergios;71679]

\n

[quote=Abhishek;70092]

\n

Hand labelling, I bet!!!

\n

[/quote]

\n

Do you still bet on that one? :)

\n

[/quote]

\n

$10 bucks says its handlabeling for all teams above us.

", "rawMarkdown": "", "votes": null }, { "id": "71684", "postDate": "04/15/2015 12:29:02", "content": "

Not necessarily (---with the exception of  two submission I did my accident and one that was in the incorrect format--) every submission I have had a significant jump in my ranking ...

\n

eg.  my initial post, I had a 98.34% accuracy on my classifier and was place 9th from last place.

\n

I was able to move to 98.74% accuracy on my classifier and jump only an additional 2 levels in my ranking...

\n

My most recent submission I had a little better than 98.8% on my classifier and at my submission I was surprised to find out I had jumped in ranking by, if I recall correctly, 97 point in ranking....

\n

So my guess that the people/group in the top ranking is in the 99% in whatever classification method they are using...

\n

I have been using an unproven theory about ML that seems to be working for me.

\n

A few hours ago I realized, in my rush, to join the competition, I had introduced a lot of noise into my testing and training datasets.... It could be the reason I could not seem to get very far beyond 98.8% on my classification system. --I am in the process of removing the noise and hopefully push beyond the 98.8% accuracy

\n

This is my first competition ever, I am blown away and humbled at the fact that all the people above my ranking probably have classification system doing better that 98.8% and closely packed together if someone was to actually look at the raw numbers...

\n

I think if I or anyone else can get to the 99.9% they maybe able to take over the top spot...

", "rawMarkdown": "", "votes": null }, { "id": "71686", "postDate": "04/15/2015 12:33:31", "content": "

I think everyone in top 10 already achieved 99.9% accuracy.

", "rawMarkdown": "", "votes": null }, { "id": "71687", "postDate": "04/15/2015 12:44:07", "content": "

Ok, so our numbers are, don't think it hurts to share:

\n

CV 10 folds, overall accuracy 0.9993558,  avg log loss 0.0015 ... 0.00291786719343801 depending on the seed...

\n

Rounded high probs (>0.9999) to 1 not to loose on log loss.

", "rawMarkdown": "", "votes": null }, { "id": "71693", "postDate": "04/15/2015 14:45:55", "content": "

@Michael, the ranking has everything to do with how the evaluation metric, the average log loss, is being calculated. If I'm doing my math right, even if you have 100.00% accuracy, you only get  a LB score of 1.0 of you only put 0.367 probabilities on the correct class.  To get a LB score of 0.01, you would need to put 0.99 probabilities on the correct class. The trick then is hedging against the misclassifications, because you're heavily penalized for putting too little on the correct class.

", "rawMarkdown": "", "votes": null }, { "id": "71717", "postDate": "04/15/2015 15:54:07", "content": "

[quote=Μαριος Μιχαηλιδης KazAnova;71680]

\n

[quote=Stergios;71679]

\n

Do you still bet on that one? :)

\n

[/quote]

\n

Abhishek would never do something like that :p . It is not in his mentality . 

\n

[/quote]

\n

Here's a question: is hand labeling or non algorithmic ex-post processing of submission files inherently cheating? Or is it only if you end up choosing that submission as your final model? I mean, would it be wrong to hand label some observations, get LB feedback, and use that information to legitimately (aka algorithmically) tune your model? Furthermore, how is hand labeling of a specific observation different from hand labeling of clusters of observations? Take for example competitions where the test set is chronologically following the train set. There is usually a trend component that is difficult to model. In KDD 2014 for example, many teams, including mine, multiplied chunks of predictions by certain factors to account for time trends that would have otherwise been impossible to model. That is actually a very common Kaggle trick, used in many competitions, including SeeClick, KDD, Influenza,... In those cases the output of a base model is hand crafted before submission. Is that cheating? It hasn't been in the past.

\n

I think hand labeling in this competition is a touchy topic. The accuracy is so high (and models so good) that you can legitimately hone down the problematic observations by pure modelling. It is not like Cats & Dogs where you could score a perfect submission w/o having any model. So, with probably a handful of uncertain observations, would it be considered cheating to just try and manually change predictions to 1's and 0's and gather LB feedback?

\n

Don't want to seem arrogant or anything, especially because I'm far from the top. But if I had a model that allowed me to hone down the remainder logloss to a handful of observations, I would hand label those for the sake of observing LB feedback. Would I choose one of those submissions as my final 2? Heck no. But would I use that information to improve my models if possible? Upvote this post if you want the answer (but you should know the answer :-) ).

", "rawMarkdown": "", "votes": null }, { "id": "71765", "postDate": "04/15/2015 18:43:13", "content": "

[quote=Giulio;71717]

\n

Don't want to seem arrogant or anything, especially because I'm far from the top. But if I had a model that allowed me to hone down the remainder logloss to a handful of observations, I would hand label those for the sake of observing LB feedback. Would I choose one of those submissions as my final 2? Heck no. But would I use that information to improve my models if possible? Upvote this post if you want the answer (but you should know the answer :-) ).

\n

[/quote]

\n

For me, that is cheating (aka I would never do that). Given the rather small size of the data set, including 30-40 difficult-to-score hand-labelled observation can make some impact. Anyway, I believe there will be shake-up due to this kind of over fitting- we'll see .  

\n

I think there is a clear distinction in getting leaderboard feedback (even probing) and (-while having a relatively small training set-) to increase your set of known labels artificially like that. 

", "rawMarkdown": "", "votes": null }, { "id": "71779", "postDate": "04/15/2015 20:12:42", "content": "

For me, this is a grey area.  The top 3 teams do have an upper hand now. Predicting outliers largely depend on outliers we know. Now they know outliers in public LB and we don't yet. I expect them will benefit from this information.

", "rawMarkdown": "", "votes": null }, { "id": "72093", "postDate": "04/17/2015 15:22:36", "content": "

@David Shinn

\n

I am using the softmax function across the output of my classifier so those .999 in the right place you mentioned is automatically done for me.

\n

My one problem is that I did not stick with what I know and used gradient decent algorithm to optimize my system ..... I am really having a hard time getting beyond 98.8.....

\n

I truly believe when I finally properly tune the gradient decent optimizer and get into the 99.9xyz I will surely blow the top of the leaders....

\n

Hopefully I can do it before the end of the competition that occurs in about 8 hours.....

\n

It was only yesterday I was able to get the noise out the the datasets; something I introduced by accident :(

\n

All I need is time .... It is really unfortunately be at Kaggle for two week and just one week ago decide to participate.....

\n

Here is to pulling something out of my hat and get beyond the 98.8% on the cleaned up datasets

\n

;)

", "rawMarkdown": "", "votes": null }, { "id": "72106", "postDate": "04/17/2015 17:01:09", "content": "

@Michael,

\n

Good luck to you.  Even if you achieved 99.9% accuracy on the Public LB, if you're putting too little probabilities on your misclassifications, you still won't break the top 10.  I estimate that at best one can achieve an average log_loss of around 0.0069 on the Public LB if one has a 99.9% accuracy  with 1.00 probabilities on the correct classes but put 0.001 probabilities on the misclassifications.  See my calcs below.  I've learned a lot about the log_loss metric this competition.

\n

In [22]: import math

\n

In [23]: (1 - 0.999)*3000 # Get estimated counts of misclassified at 99.9% accuracy
Out[23]: 3.0000000000000027

\n

In [24]: -math.log(0.001)*3 / 3000 # Get estimated impact on score if put 0.001 prob on those misclassifications
Out[24]: 0.006907755278982137

", "rawMarkdown": "", "votes": null }, { "id": "72126", "postDate": "04/17/2015 18:37:41", "content": "

Okay, your calculated probability seems justified...

\n

That said, do you think it is by pure luck the leader have such good scores....

\n

Actually I just notice that half the top 10 have more than 128 submission and realized once you get to certain level of of classification score say 98.8% accuracy in classification on the training data you could actually use the submission process to weed out incorrect classification within each submission (classic probability/statistical analysis) .... I think seem like a possible way to boost ones score beyond brute force training of ones system....

\n

I think this maybe a reasonable thing I/we/competitors need to invest some time in automating. Does such a system as describe exist?

\n

Is this what people mean by labelling?  It is the first time I have heard of it. Is/are there any papers on the subject?

", "rawMarkdown": "", "votes": null }, { "id": "72141", "postDate": "04/17/2015 20:03:46", "content": "

[quote=Michael George Hart;72126]

\n

Okay, your calculated probability seems justified...

\n

That said, do you think it is by pure luck the leader have such good scores....

\n

Actually I just notice that half the top 10 have more than 128 submission and realized once you get to certain level of of classification score say 98.8% accuracy in classification on the training data you could actually use the submission process to weed out incorrect classification within each submission (classic probability/statistical analysis) .... I think seem like a possible way to boost ones score beyond brute force training of ones system....

\n

I think this maybe a reasonable thing I/we/competitors need to invest some time in automating. Does such a system as describe exist?

\n

Is this what people mean by labelling?  It is the first time I have heard of it. Is/are there any papers on the subject?

\n

[/quote]

\n

You can easily get to 99% accuracy without any calibration or hand labelling. Just a single model with some fair features can do it. The problem here is not accuracy, but how to optimize the logloss with great generalization. And the reason we have many submissions is mainly because we entered early and felt very comfortable to see even tiny improvement on the LB, even though we know it most likely may not show in the private board.

", "rawMarkdown": "", "votes": null }, { "id": "72211", "postDate": "04/18/2015 00:31:51", "content": "

I knew when I saw the 0.0000 logloss there would be massive shift between public and private. It's not arrogance. It's just very easy to get one mistake and be pounded hard (not infinitely but very hard) for it. Calibration is hard. After Otto I'd like to release some secret sauce with respect to this.

", "rawMarkdown": "", "votes": null }, { "id": "72217", "postDate": "04/18/2015 00:45:09", "content": "

Its not about 0.00. We never used 0.0. It was done only to show that its possible. There were a lot of other reasons for the drop. And we will release everything soom

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 70091, "author_name": "davidtran", "author_url": "", "post_date": "04/09/2015 06:48:31", "content": "

0.0000000 log loss. That's something you don't see every day.

", "votes": null, "replies": [] }, { "id": 70092, "author_name": "abhishek", "author_url": "", "post_date": "04/09/2015 06:59:40", "content": "

Hand labelling, I bet!!!

", "votes": null, "replies": [] }, { "id": 70093, "author_name": "drpatrickchan", "author_url": "", "post_date": "04/09/2015 07:04:19", "content": "

Can we have the hand labelling classification rules please :)

", "votes": null, "replies": [] }, { "id": 70094, "author_name": "jiweiliu", "author_url": "", "post_date": "04/09/2015 07:06:39", "content": "

With so few outliers like 15 or 20, hand labeling would be no different from calibration. So I don't think it is against any rules. or I can not tell the difference.

", "votes": null, "replies": [] }, { "id": 70095, "author_name": "", "author_url": "", "post_date": "04/09/2015 07:13:01", "content": "

the rules do say:

\n

"Submissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records."

\n

I don't know why it says 'may' instead of 'should/shall' 

", "votes": null, "replies": [] }, { "id": 70096, "author_name": "izikgo", "author_url": "", "post_date": "04/09/2015 07:21:59", "content": "

how is hand labeling defined? If, for example, I look through the dataset and see that some class contains a string in the asm files that is unique to it. Is labeling all files that contain that string as belonging to that class considered hand labeling or not?

", "votes": null, "replies": [] }, { "id": 70098, "author_name": "gmilosev", "author_url": "", "post_date": "04/09/2015 07:29:08", "content": "

Think this is called "signature based detection" not hand labeling :)

", "votes": null, "replies": [] }, { "id": 70113, "author_name": "mikeskim", "author_url": "", "post_date": "04/09/2015 13:40:57", "content": "

I'm going to bet there will be a massive shift from public to private.

", "votes": null, "replies": [] }, { "id": 70125, "author_name": "xiaozhouwang", "author_url": "", "post_date": "04/09/2015 15:20:55", "content": "

I bet the top 2 will remain their super performance in private, and they will tie in 0.000001! 

", "votes": null, "replies": [] }, { "id": 70129, "author_name": "abhishek", "author_url": "", "post_date": "04/09/2015 17:38:01", "content": "

what about the 2 after top2 ? :P

", "votes": null, "replies": [] }, { "id": 70142, "author_name": "jiweiliu", "author_url": "", "post_date": "04/09/2015 19:12:18", "content": "

[quote=Abhishek;70129]

\n

what about the 2 after top2 ? :P

\n

[/quote]

\n

I have no idea. Maybe we could benefit from a new round of CV sharing. Our best signal model achieves 4-fold cv of 0.0051 and 0.0044 public LB. What about yours? :P

", "votes": null, "replies": [] }, { "id": 70145, "author_name": "psilogram", "author_url": "", "post_date": "04/09/2015 19:46:46", "content": "

I'm in the same ballpark -- .0052 10-fold CV, .0041 LB.

", "votes": null, "replies": [] }, { "id": 70146, "author_name": "jiweiliu", "author_url": "", "post_date": "04/09/2015 19:51:33", "content": "

Thank you all for sharing :D

", "votes": null, "replies": [] }, { "id": 70175, "author_name": "sachinvernekar", "author_url": "", "post_date": "04/10/2015 06:13:51", "content": "", "votes": null, "replies": [] }, { "id": 70193, "author_name": "abhishek", "author_url": "", "post_date": "04/10/2015 09:25:21", "content": "

Lol

", "votes": null, "replies": [] }, { "id": 70194, "author_name": "mahmadi", "author_url": "", "post_date": "04/10/2015 09:38:38", "content": "

Don't be disappointed soon Sachin, Go on.

\n

There are still 7 days to the end ;-)

", "votes": null, "replies": [] }, { "id": 71271, "author_name": "sachinvernekar", "author_url": "", "post_date": "04/11/2015 07:37:37", "content": "

[quote=Abhishek;70193]

\n

Lol

\n

[/quote]

\n

Hi Abhishek,

\n

Can you please explain if someone has got a score of 0 on the public leaderboard and he chooses 2 best entries with scores 0 and 0 how can private leaderboard score be different. Since private data is never revealed to the participants and we are not asked to submit our code, how can they determine the results on private data if it is not a subset of public data with a different mix. Or am I totally missing something?

", "votes": null, "replies": [] }, { "id": 71273, "author_name": "mikhailtrofimov", "author_url": "", "post_date": "04/11/2015 08:17:43", "content": "

@Sachin: you submit 10k predictions. Some 3k (fixed on kaggle side) used for calculate Public LB. Other 7k will be used for calculating Private score.

\n

So, there is no problem.

", "votes": null, "replies": [] }, { "id": 71275, "author_name": "sachinvernekar", "author_url": "", "post_date": "04/11/2015 08:38:32", "content": "

[quote=Mikhail Trofimov;71273]

\n

@Sachin: you submit 10k predictions. Some 3k (fixed on kaggle side) used for calculate Public LB. Other 7k will be used for calculating Private score.

\n

So, there is no problem.

\n

[/quote]

\n

Thanks Mikhail

", "votes": null, "replies": [] }, { "id": 71679, "author_name": "asterios", "author_url": "", "post_date": "04/15/2015 11:39:11", "content": "

[quote=Abhishek;70092]

\n

Hand labelling, I bet!!!

\n

[/quote]

\n\n

Do you still bet on that one? :)

", "votes": null, "replies": [] }, { "id": 71680, "author_name": "kazanova", "author_url": "", "post_date": "04/15/2015 11:52:42", "content": "

[quote=Stergios;71679]

\n

Do you still bet on that one? :)

\n

[/quote]

\n

Abhishek would never do something like that :p . It is not in his mentality . 

", "votes": null, "replies": [] }, { "id": 71681, "author_name": "abhishek", "author_url": "", "post_date": "04/15/2015 11:55:37", "content": "

[quote=Stergios;71679]

\n

[quote=Abhishek;70092]

\n

Hand labelling, I bet!!!

\n

[/quote]

\n

Do you still bet on that one? :)

\n

[/quote]

\n

$10 bucks says its handlabeling for all teams above us.

", "votes": null, "replies": [] }, { "id": 71684, "author_name": "spaceman", "author_url": "", "post_date": "04/15/2015 12:29:02", "content": "

Not necessarily (---with the exception of  two submission I did my accident and one that was in the incorrect format--) every submission I have had a significant jump in my ranking ...

\n

eg.  my initial post, I had a 98.34% accuracy on my classifier and was place 9th from last place.

\n

I was able to move to 98.74% accuracy on my classifier and jump only an additional 2 levels in my ranking...

\n

My most recent submission I had a little better than 98.8% on my classifier and at my submission I was surprised to find out I had jumped in ranking by, if I recall correctly, 97 point in ranking....

\n

So my guess that the people/group in the top ranking is in the 99% in whatever classification method they are using...

\n

I have been using an unproven theory about ML that seems to be working for me.

\n

A few hours ago I realized, in my rush, to join the competition, I had introduced a lot of noise into my testing and training datasets.... It could be the reason I could not seem to get very far beyond 98.8% on my classification system. --I am in the process of removing the noise and hopefully push beyond the 98.8% accuracy

\n

This is my first competition ever, I am blown away and humbled at the fact that all the people above my ranking probably have classification system doing better that 98.8% and closely packed together if someone was to actually look at the raw numbers...

\n

I think if I or anyone else can get to the 99.9% they maybe able to take over the top spot...

", "votes": null, "replies": [] }, { "id": 71686, "author_name": "abhishek", "author_url": "", "post_date": "04/15/2015 12:33:31", "content": "

I think everyone in top 10 already achieved 99.9% accuracy.

", "votes": null, "replies": [] }, { "id": 71687, "author_name": "gmilosev", "author_url": "", "post_date": "04/15/2015 12:44:07", "content": "

Ok, so our numbers are, don't think it hurts to share:

\n

CV 10 folds, overall accuracy 0.9993558,  avg log loss 0.0015 ... 0.00291786719343801 depending on the seed...

\n

Rounded high probs (>0.9999) to 1 not to loose on log loss.

", "votes": null, "replies": [] }, { "id": 71693, "author_name": "davidshinn", "author_url": "", "post_date": "04/15/2015 14:45:55", "content": "

@Michael, the ranking has everything to do with how the evaluation metric, the average log loss, is being calculated. If I'm doing my math right, even if you have 100.00% accuracy, you only get  a LB score of 1.0 of you only put 0.367 probabilities on the correct class.  To get a LB score of 0.01, you would need to put 0.99 probabilities on the correct class. The trick then is hedging against the misclassifications, because you're heavily penalized for putting too little on the correct class.

", "votes": null, "replies": [] }, { "id": 71717, "author_name": "adjgiulio", "author_url": "", "post_date": "04/15/2015 15:54:07", "content": "

[quote=Μαριος Μιχαηλιδης KazAnova;71680]

\n

[quote=Stergios;71679]

\n

Do you still bet on that one? :)

\n

[/quote]

\n

Abhishek would never do something like that :p . It is not in his mentality . 

\n

[/quote]

\n

Here's a question: is hand labeling or non algorithmic ex-post processing of submission files inherently cheating? Or is it only if you end up choosing that submission as your final model? I mean, would it be wrong to hand label some observations, get LB feedback, and use that information to legitimately (aka algorithmically) tune your model? Furthermore, how is hand labeling of a specific observation different from hand labeling of clusters of observations? Take for example competitions where the test set is chronologically following the train set. There is usually a trend component that is difficult to model. In KDD 2014 for example, many teams, including mine, multiplied chunks of predictions by certain factors to account for time trends that would have otherwise been impossible to model. That is actually a very common Kaggle trick, used in many competitions, including SeeClick, KDD, Influenza,... In those cases the output of a base model is hand crafted before submission. Is that cheating? It hasn't been in the past.

\n

I think hand labeling in this competition is a touchy topic. The accuracy is so high (and models so good) that you can legitimately hone down the problematic observations by pure modelling. It is not like Cats & Dogs where you could score a perfect submission w/o having any model. So, with probably a handful of uncertain observations, would it be considered cheating to just try and manually change predictions to 1's and 0's and gather LB feedback?

\n

Don't want to seem arrogant or anything, especially because I'm far from the top. But if I had a model that allowed me to hone down the remainder logloss to a handful of observations, I would hand label those for the sake of observing LB feedback. Would I choose one of those submissions as my final 2? Heck no. But would I use that information to improve my models if possible? Upvote this post if you want the answer (but you should know the answer :-) ).

", "votes": null, "replies": [] }, { "id": 71765, "author_name": "kazanova", "author_url": "", "post_date": "04/15/2015 18:43:13", "content": "

[quote=Giulio;71717]

\n

Don't want to seem arrogant or anything, especially because I'm far from the top. But if I had a model that allowed me to hone down the remainder logloss to a handful of observations, I would hand label those for the sake of observing LB feedback. Would I choose one of those submissions as my final 2? Heck no. But would I use that information to improve my models if possible? Upvote this post if you want the answer (but you should know the answer :-) ).

\n

[/quote]

\n

For me, that is cheating (aka I would never do that). Given the rather small size of the data set, including 30-40 difficult-to-score hand-labelled observation can make some impact. Anyway, I believe there will be shake-up due to this kind of over fitting- we'll see .  

\n

I think there is a clear distinction in getting leaderboard feedback (even probing) and (-while having a relatively small training set-) to increase your set of known labels artificially like that. 

", "votes": null, "replies": [] }, { "id": 71779, "author_name": "jiweiliu", "author_url": "", "post_date": "04/15/2015 20:12:42", "content": "

For me, this is a grey area.  The top 3 teams do have an upper hand now. Predicting outliers largely depend on outliers we know. Now they know outliers in public LB and we don't yet. I expect them will benefit from this information.

", "votes": null, "replies": [] }, { "id": 72093, "author_name": "spaceman", "author_url": "", "post_date": "04/17/2015 15:22:36", "content": "

@David Shinn

\n

I am using the softmax function across the output of my classifier so those .999 in the right place you mentioned is automatically done for me.

\n

My one problem is that I did not stick with what I know and used gradient decent algorithm to optimize my system ..... I am really having a hard time getting beyond 98.8.....

\n

I truly believe when I finally properly tune the gradient decent optimizer and get into the 99.9xyz I will surely blow the top of the leaders....

\n

Hopefully I can do it before the end of the competition that occurs in about 8 hours.....

\n

It was only yesterday I was able to get the noise out the the datasets; something I introduced by accident :(

\n

All I need is time .... It is really unfortunately be at Kaggle for two week and just one week ago decide to participate.....

\n

Here is to pulling something out of my hat and get beyond the 98.8% on the cleaned up datasets

\n

;)

", "votes": null, "replies": [] }, { "id": 72106, "author_name": "davidshinn", "author_url": "", "post_date": "04/17/2015 17:01:09", "content": "

@Michael,

\n

Good luck to you.  Even if you achieved 99.9% accuracy on the Public LB, if you're putting too little probabilities on your misclassifications, you still won't break the top 10.  I estimate that at best one can achieve an average log_loss of around 0.0069 on the Public LB if one has a 99.9% accuracy  with 1.00 probabilities on the correct classes but put 0.001 probabilities on the misclassifications.  See my calcs below.  I've learned a lot about the log_loss metric this competition.

\n

In [22]: import math

\n

In [23]: (1 - 0.999)*3000 # Get estimated counts of misclassified at 99.9% accuracy
Out[23]: 3.0000000000000027

\n

In [24]: -math.log(0.001)*3 / 3000 # Get estimated impact on score if put 0.001 prob on those misclassifications
Out[24]: 0.006907755278982137

", "votes": null, "replies": [] }, { "id": 72126, "author_name": "spaceman", "author_url": "", "post_date": "04/17/2015 18:37:41", "content": "

Okay, your calculated probability seems justified...

\n

That said, do you think it is by pure luck the leader have such good scores....

\n

Actually I just notice that half the top 10 have more than 128 submission and realized once you get to certain level of of classification score say 98.8% accuracy in classification on the training data you could actually use the submission process to weed out incorrect classification within each submission (classic probability/statistical analysis) .... I think seem like a possible way to boost ones score beyond brute force training of ones system....

\n

I think this maybe a reasonable thing I/we/competitors need to invest some time in automating. Does such a system as describe exist?

\n

Is this what people mean by labelling?  It is the first time I have heard of it. Is/are there any papers on the subject?

", "votes": null, "replies": [] }, { "id": 72141, "author_name": "xiaozhouwang", "author_url": "", "post_date": "04/17/2015 20:03:46", "content": "

[quote=Michael George Hart;72126]

\n

Okay, your calculated probability seems justified...

\n

That said, do you think it is by pure luck the leader have such good scores....

\n

Actually I just notice that half the top 10 have more than 128 submission and realized once you get to certain level of of classification score say 98.8% accuracy in classification on the training data you could actually use the submission process to weed out incorrect classification within each submission (classic probability/statistical analysis) .... I think seem like a possible way to boost ones score beyond brute force training of ones system....

\n

I think this maybe a reasonable thing I/we/competitors need to invest some time in automating. Does such a system as describe exist?

\n

Is this what people mean by labelling?  It is the first time I have heard of it. Is/are there any papers on the subject?

\n

[/quote]

\n

You can easily get to 99% accuracy without any calibration or hand labelling. Just a single model with some fair features can do it. The problem here is not accuracy, but how to optimize the logloss with great generalization. And the reason we have many submissions is mainly because we entered early and felt very comfortable to see even tiny improvement on the LB, even though we know it most likely may not show in the private board.

", "votes": null, "replies": [] }, { "id": 72211, "author_name": "mikeskim", "author_url": "", "post_date": "04/18/2015 00:31:51", "content": "

I knew when I saw the 0.0000 logloss there would be massive shift between public and private. It's not arrogance. It's just very easy to get one mistake and be pounded hard (not infinitely but very hard) for it. Calibration is hard. After Otto I'd like to release some secret sauce with respect to this.

", "votes": null, "replies": [] }, { "id": 72217, "author_name": "abhishek", "author_url": "", "post_date": "04/18/2015 00:45:09", "content": "

Its not about 0.00. We never used 0.0. It was done only to show that its possible. There were a lot of other reasons for the drop. And we will release everything soom

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "70087": "", "70091": "", "70092": "", "70093": "", "70094": "", "70095": "", "70096": "", "70098": "", "70113": "", "70125": "", "70129": "", "70142": "", "70145": "", "70146": "", "70175": "", "70193": "", "70194": "", "71271": "", "71273": "", "71275": "", "71679": "", "71680": "", "71681": "", "71684": "", "71686": "", "71687": "", "71693": "", "71717": "", "71765": "", "71779": "", "72093": "", "72106": "", "72126": "", "72141": "", "72211": "", "72217": "" }, "source": "meta" }