{ "id": 13248, "title": "Some results", "url": "/competitions/diabetic-retinopathy-detection/discussion/13248", "author_name": "", "post_date": "2015-04-06T04:21:10.687Z", "votes": 7, "comment_count": 16, "views": 6322, "content": "

Hi guys - I thought I'd share some results.

\n

Confusion matrix, rows = human label, columns = predicted.

\n

[[ 1175 90 23  3  0 ]
 [   72 37 12  1  0 ]
 [   55 53 98 58  1 ]
 [    0  1  4 38  1 ]
 [    0  3  2 28  3 ]]

\n


Kappa ~0.781

\n

Binary classification, none or mild (0-1) vs moderate and above (2+):

\n

ROC AUC ~0.934
73% TPR at 5% FPR
84% TPR at 10% FPR

\n

ROC curve attached.

\n

I'm wondering if anyone has a model that would be good to combine with this?

\n

Regards,

\n

Alex

", "messages": [ { "id": "69765", "postDate": "04/06/2015 04:21:10", "content": "

Hi guys - I thought I'd share some results.

\n

Confusion matrix, rows = human label, columns = predicted.

\n

[[ 1175 90 23  3  0 ]
 [   72 37 12  1  0 ]
 [   55 53 98 58  1 ]
 [    0  1  4 38  1 ]
 [    0  3  2 28  3 ]]

\n


Kappa ~0.781

\n

Binary classification, none or mild (0-1) vs moderate and above (2+):

\n

ROC AUC ~0.934
73% TPR at 5% FPR
84% TPR at 10% FPR

\n

ROC curve attached.

\n

I'm wondering if anyone has a model that would be good to combine with this?

\n

Regards,

\n

Alex

", "rawMarkdown": "", "votes": null }, { "id": "69811", "postDate": "04/06/2015 16:56:31", "content": "

Dude, there's still 3 months to go and you are already 0.77. Take it easy :) jk! Awesome!

\n

What's the mystery though, seriously? Are you just doing a combination of binary classifications or ensemble them with multiclass?

", "rawMarkdown": "", "votes": null }, { "id": "69926", "postDate": "04/07/2015 19:06:37", "content": "

Alex,

\n

Congrats for these results, 

\n

I'm surprised to know how you would expect one's model to be good to combine with yours?

\n

Maybe any model with a score of 0.50 would add you around 0.01 benefit, but it's difficult to predict the benefit with just some ROC curves, isn't it?

\n

I'd be pleased to combine my model with yours but I'm not sure it will be of any benefit for you :-) 

", "rawMarkdown": "", "votes": null }, { "id": "69935", "postDate": "04/07/2015 20:33:12", "content": "

@Deep: Thank you!  I can't say much about what I'm doing yet, but I hope the way it works will be an interesting surprise :)

", "rawMarkdown": "", "votes": null }, { "id": "69937", "postDate": "04/07/2015 20:54:57", "content": "

Hi Zidmie,

\n

I think it is possible: for example if there was someone who was really great at finding 4s, that could significantly improve my model.  They would not necessarily have a high score: correct prediction of all 4s gives you 0.3-something.  Anyway, I thought I'd offer.  

\n

I admit I'm also just curious, esp whether anyone has a much better ROC curve for very low false positives area.

\n

Best,

\n

Alex

\n

[quote=Zidmie;69926]

\n

Alex,

\n

Congrats for these results, 

\n

I'm surprised to know how you would expect one's model to be good to combine with yours?

\n

Maybe any model with a score of 0.50 would add you around 0.01 benefit, but it's difficult to predict the benefit with just some ROC curves, isn't it?

\n

I'd be pleased to combine my model with yours but I'm not sure it will be of any benefit for you :-) 

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "69944", "postDate": "04/07/2015 21:12:17", "content": "

[quote=Alexander Izvorski;69935]

\n

@Deep: Thank you!  I can't say much about what I'm doing yet, but I hope the way it works will be an interesting surprise :)

\n

[/quote]

\n

Congratulations on your results! :)

\n

I am curious: do you combine different methods (neural networks, classic detection algorithms etc.) or do you have some other kind of magic way?

", "rawMarkdown": "", "votes": null }, { "id": "69946", "postDate": "04/07/2015 21:23:43", "content": "

I understand you may not feel comfortable to share your methods (even at very high level discussion) at this point.

\n

So, just curious, what is the nature of this collaboration say if someone has a model that is really good at predicting level 4 vs. level 3 really well (you seem to struggle with it :) )? Are you looking for team merger?

\n

Also, is the confusion matrix for a stratified sample?

", "rawMarkdown": "", "votes": null }, { "id": "69949", "postDate": "04/07/2015 22:33:49", "content": "

Hi Deep:

\n

I wouldn't mind a team merger if that's what it takes.  This is a really hard problem, model diversity is always a good thing.  This might also spark a bit of discussion about which parts of it are hard and why.

\n

Yes, it's a stratified sample.

\n

-Alex

", "rawMarkdown": "", "votes": null }, { "id": "71262", "postDate": "04/11/2015 01:56:14", "content": "

Hi Zoltan - Thanks!  I appreciate your curiosity!  It's a hybrid approach.  I guess I'd better keep the details "top secret" for now :)

\n

-Alex

\n\n

[quote=Zoltan Fegyver;69944]

\n

Congratulations on your results! :)

\n

I am curious: do you combine different methods (neural networks, classic detection algorithms etc.) or do you have some other kind of magic way?

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "72671", "postDate": "04/20/2015 17:03:41", "content": "

Alex,

\n

Thanks for posting this; it's interesting to see how another approach is doing. Just to reciprocate, here's my confusion matrix, good for a kappa of ~0.63

\n

[[1099 161 27  2  2]
 [ 100  25  3  0  0]
 [  93  89 51 27  0]
 [   1  11 17 20  0]
 [   0   2 11  3 18]]

\n

Looks like I'm doing better with class 4, and considerably worse everywhere else. I'll be very curious to see what kind of secret sauce you are using when this is over.

", "rawMarkdown": "", "votes": null }, { "id": "73010", "postDate": "04/21/2015 19:30:43", "content": "

Hi Tim,

\n

Thanks!  That's really solid.

\n

It looks like the "confusability" of things in your model is about the same as in mine, qualitatively speaking.  For example, there just doesn't seem to be a great way to distinguish 1s from 0s, and real 2s get spread somewhat evenly between predicted 0-3s.  That's what I was curious about.

\n

I wonder, how much of this is limitations in our models, and how much is inherent "softness" of the category boundaries?  I mean in theory just one microaneurysm (MA) is enough to make a 0 into a 1, and I think humans are not much better than (say) 50% recall/80% precision on identifying those.

\n

Likewise :)

\n

Best,

\n

Alex

\n\n

[quote=Tim Hochberg;72671]

\n

Alex,

\n

Thanks for posting this; it's interesting to see how another approach is doing. Just to reciprocate, here's my confusion matrix, good for a kappa of ~0.63

\n

[[1099 161 27 2 2]
[ 100 25 3 0 0]
[ 93 89 51 27 0]
[ 1 11 17 20 0]
[ 0 2 11 3 18]]

\n

Looks like I'm doing better with class 4, and considerably worse everywhere else. I'll be very curious to see what kind of secret sauce you are using when this is over.

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "73012", "postDate": "04/21/2015 19:41:52", "content": "

Hi Alex,

\n

Didn't you use test set for improving results?

", "rawMarkdown": "", "votes": null }, { "id": "74308", "postDate": "04/23/2015 15:34:13", "content": "

Hi Alex,

\n

I agree that the softness of the categories is going to be a limiting factor. Presumably the progression of the disease is more or less continuous. (In fact, different facets of the disease might proceed at different rates in different patients, but that's a whole 'nother level of complication) and mapping that into discrete bins is bound to result in problems near the bin borders. Add to that these are rated by different doctors using different equipment and there's bound to be a limit to how close of an agreement can be achieved. 

\n

It doesn't add to my confidence any that a significant fraction of the images are either extremely over or under exposed and seem unlikely to actually have been used in the ratings of the patients! Still, it may be possible to reach a useful level of accuracy even given these limitations.

", "rawMarkdown": "", "votes": null }, { "id": "74865", "postDate": "04/26/2015 15:03:07", "content": "

Hi, I am a new member. I am just wondering about these results. Are these results obtained by applying your method on the test set or are these obtained from for e.g. cross validation on the training set?

", "rawMarkdown": "", "votes": null }, { "id": "74875", "postDate": "04/26/2015 16:15:21", "content": "

I can't speak for Alex, but I'm reserving 5% of the data for cross validation. I compute the predicted values using the CV data and compute kappa and the confusion matrix based on the predictions and the provided labels.

", "rawMarkdown": "", "votes": null }, { "id": "77402", "postDate": "05/06/2015 21:09:48", "content": "

How do you determine your ROC curve and other info when Kaggle doesn't give you any statistical information about the test set other than the kappa?

", "rawMarkdown": "", "votes": null }, { "id": "87314", "postDate": "07/28/2015 15:54:49", "content": "

[quote=Alexander Izvorski;71262]

\n

Hi Zoltan - Thanks!  I appreciate your curiosity!  It's a hybrid approach.  I guess I'd better keep the details "top secret" for now :)

\n

-Alex

\n

[/quote]

\n

Hi Alex.

\n

I am very interesting (hope not only me) , how you got the 19th final result 3 month ago. I am feel, you used rather simple and shallow solution (comparing with large deep convolution net)

\n

It's seems this is not "top secret" for now and you can provide us some more details about your solution, at last :-)

\n

Thanks

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 69811, "author_name": "deepcnn", "author_url": "", "post_date": "04/06/2015 16:56:31", "content": "

Dude, there's still 3 months to go and you are already 0.77. Take it easy :) jk! Awesome!

\n

What's the mystery though, seriously? Are you just doing a combination of binary classifications or ensemble them with multiclass?

", "votes": null, "replies": [] }, { "id": 69926, "author_name": "zidmie", "author_url": "", "post_date": "04/07/2015 19:06:37", "content": "

Alex,

\n

Congrats for these results, 

\n

I'm surprised to know how you would expect one's model to be good to combine with yours?

\n

Maybe any model with a score of 0.50 would add you around 0.01 benefit, but it's difficult to predict the benefit with just some ROC curves, isn't it?

\n

I'd be pleased to combine my model with yours but I'm not sure it will be of any benefit for you :-) 

", "votes": null, "replies": [] }, { "id": 69935, "author_name": "aizvorski", "author_url": "", "post_date": "04/07/2015 20:33:12", "content": "

@Deep: Thank you!  I can't say much about what I'm doing yet, but I hope the way it works will be an interesting surprise :)

", "votes": null, "replies": [] }, { "id": 69937, "author_name": "aizvorski", "author_url": "", "post_date": "04/07/2015 20:54:57", "content": "

Hi Zidmie,

\n

I think it is possible: for example if there was someone who was really great at finding 4s, that could significantly improve my model.  They would not necessarily have a high score: correct prediction of all 4s gives you 0.3-something.  Anyway, I thought I'd offer.  

\n

I admit I'm also just curious, esp whether anyone has a much better ROC curve for very low false positives area.

\n

Best,

\n

Alex

\n

[quote=Zidmie;69926]

\n

Alex,

\n

Congrats for these results, 

\n

I'm surprised to know how you would expect one's model to be good to combine with yours?

\n

Maybe any model with a score of 0.50 would add you around 0.01 benefit, but it's difficult to predict the benefit with just some ROC curves, isn't it?

\n

I'd be pleased to combine my model with yours but I'm not sure it will be of any benefit for you :-) 

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 69944, "author_name": "zoltanfegyver", "author_url": "", "post_date": "04/07/2015 21:12:17", "content": "

[quote=Alexander Izvorski;69935]

\n

@Deep: Thank you!  I can't say much about what I'm doing yet, but I hope the way it works will be an interesting surprise :)

\n

[/quote]

\n

Congratulations on your results! :)

\n

I am curious: do you combine different methods (neural networks, classic detection algorithms etc.) or do you have some other kind of magic way?

", "votes": null, "replies": [] }, { "id": 69946, "author_name": "deepcnn", "author_url": "", "post_date": "04/07/2015 21:23:43", "content": "

I understand you may not feel comfortable to share your methods (even at very high level discussion) at this point.

\n

So, just curious, what is the nature of this collaboration say if someone has a model that is really good at predicting level 4 vs. level 3 really well (you seem to struggle with it :) )? Are you looking for team merger?

\n

Also, is the confusion matrix for a stratified sample?

", "votes": null, "replies": [] }, { "id": 69949, "author_name": "aizvorski", "author_url": "", "post_date": "04/07/2015 22:33:49", "content": "

Hi Deep:

\n

I wouldn't mind a team merger if that's what it takes.  This is a really hard problem, model diversity is always a good thing.  This might also spark a bit of discussion about which parts of it are hard and why.

\n

Yes, it's a stratified sample.

\n

-Alex

", "votes": null, "replies": [] }, { "id": 71262, "author_name": "aizvorski", "author_url": "", "post_date": "04/11/2015 01:56:14", "content": "

Hi Zoltan - Thanks!  I appreciate your curiosity!  It's a hybrid approach.  I guess I'd better keep the details "top secret" for now :)

\n

-Alex

\n\n

[quote=Zoltan Fegyver;69944]

\n

Congratulations on your results! :)

\n

I am curious: do you combine different methods (neural networks, classic detection algorithms etc.) or do you have some other kind of magic way?

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 72671, "author_name": "bitsofbits", "author_url": "", "post_date": "04/20/2015 17:03:41", "content": "

Alex,

\n

Thanks for posting this; it's interesting to see how another approach is doing. Just to reciprocate, here's my confusion matrix, good for a kappa of ~0.63

\n

[[1099 161 27  2  2]
 [ 100  25  3  0  0]
 [  93  89 51 27  0]
 [   1  11 17 20  0]
 [   0   2 11  3 18]]

\n

Looks like I'm doing better with class 4, and considerably worse everywhere else. I'll be very curious to see what kind of secret sauce you are using when this is over.

", "votes": null, "replies": [] }, { "id": 73010, "author_name": "aizvorski", "author_url": "", "post_date": "04/21/2015 19:30:43", "content": "

Hi Tim,

\n

Thanks!  That's really solid.

\n

It looks like the "confusability" of things in your model is about the same as in mine, qualitatively speaking.  For example, there just doesn't seem to be a great way to distinguish 1s from 0s, and real 2s get spread somewhat evenly between predicted 0-3s.  That's what I was curious about.

\n

I wonder, how much of this is limitations in our models, and how much is inherent "softness" of the category boundaries?  I mean in theory just one microaneurysm (MA) is enough to make a 0 into a 1, and I think humans are not much better than (say) 50% recall/80% precision on identifying those.

\n

Likewise :)

\n

Best,

\n

Alex

\n\n

[quote=Tim Hochberg;72671]

\n

Alex,

\n

Thanks for posting this; it's interesting to see how another approach is doing. Just to reciprocate, here's my confusion matrix, good for a kappa of ~0.63

\n

[[1099 161 27 2 2]
[ 100 25 3 0 0]
[ 93 89 51 27 0]
[ 1 11 17 20 0]
[ 0 2 11 3 18]]

\n

Looks like I'm doing better with class 4, and considerably worse everywhere else. I'll be very curious to see what kind of secret sauce you are using when this is over.

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 73012, "author_name": "", "author_url": "", "post_date": "04/21/2015 19:41:52", "content": "

Hi Alex,

\n

Didn't you use test set for improving results?

", "votes": null, "replies": [] }, { "id": 74308, "author_name": "bitsofbits", "author_url": "", "post_date": "04/23/2015 15:34:13", "content": "

Hi Alex,

\n

I agree that the softness of the categories is going to be a limiting factor. Presumably the progression of the disease is more or less continuous. (In fact, different facets of the disease might proceed at different rates in different patients, but that's a whole 'nother level of complication) and mapping that into discrete bins is bound to result in problems near the bin borders. Add to that these are rated by different doctors using different equipment and there's bound to be a limit to how close of an agreement can be achieved. 

\n

It doesn't add to my confidence any that a significant fraction of the images are either extremely over or under exposed and seem unlikely to actually have been used in the ratings of the patients! Still, it may be possible to reach a useful level of accuracy even given these limitations.

", "votes": null, "replies": [] }, { "id": 74865, "author_name": "thewinner", "author_url": "", "post_date": "04/26/2015 15:03:07", "content": "

Hi, I am a new member. I am just wondering about these results. Are these results obtained by applying your method on the test set or are these obtained from for e.g. cross validation on the training set?

", "votes": null, "replies": [] }, { "id": 74875, "author_name": "bitsofbits", "author_url": "", "post_date": "04/26/2015 16:15:21", "content": "

I can't speak for Alex, but I'm reserving 5% of the data for cross validation. I compute the predicted values using the CV data and compute kappa and the confusion matrix based on the predictions and the provided labels.

", "votes": null, "replies": [] }, { "id": 77402, "author_name": "dannn314472", "author_url": "", "post_date": "05/06/2015 21:09:48", "content": "

How do you determine your ROC curve and other info when Kaggle doesn't give you any statistical information about the test set other than the kappa?

", "votes": null, "replies": [] }, { "id": 87314, "author_name": "goodok", "author_url": "", "post_date": "07/28/2015 15:54:49", "content": "

[quote=Alexander Izvorski;71262]

\n

Hi Zoltan - Thanks!  I appreciate your curiosity!  It's a hybrid approach.  I guess I'd better keep the details "top secret" for now :)

\n

-Alex

\n

[/quote]

\n

Hi Alex.

\n

I am very interesting (hope not only me) , how you got the 19th final result 3 month ago. I am feel, you used rather simple and shallow solution (comparing with large deep convolution net)

\n

It's seems this is not "top secret" for now and you can provide us some more details about your solution, at last :-)

\n

Thanks

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "69765": "", "69811": "", "69926": "", "69935": "", "69937": "", "69944": "", "69946": "", "69949": "", "71262": "", "72671": "", "73010": "", "73012": "", "74308": "", "74865": "", "74875": "", "77402": "", "87314": "" }, "source": "meta" }