Kaggle update on LinkedIn claims that "Winning models from @CHCFNews Diabetic Retinopathy comp are on par with human performance!" No further details provided. Has human performance score been already posted somewhere?
", "messages": [ { "id": "87357", "postDate": "07/29/2015 00:24:15", "content": "Kaggle update on LinkedIn claims that "Winning models from @CHCFNews Diabetic Retinopathy comp are on par with human performance!" No further details provided. Has human performance score been already posted somewhere?
", "rawMarkdown": "Kaggle update on LinkedIn claims that \"Winning models from @CHCFNews Diabetic Retinopathy comp are on par with human performance!\" No further details provided. Has human performance score been already posted somewhere?", "votes": null }, { "id": "87365", "postDate": "07/29/2015 01:12:13", "content": "E.g. http://casemed.case.edu/cpcpold/students/module4/Reeves/Acta.pdf\nsays a general physician and an ophthalmologist have a kappa of 0.838.
\n\nhttp://visionquest-bio.com/VQ_pdf/SBarriga_05-01-2014web.pdf\nsays an optometrist and an ophthalmologist have a kappa of 0.72.
\n\nThese studies are on different datasets, and may be using slightly different classifications, e,g. scales of 0-3 instead of 0-4.
\n\nAnother point of reference: the kappa between left and right eyes in this dataset was 0.85, so the winning algorithm is as good as the ophthalmologist looking at the wrong eye.
\n\nI would argue the winning algorithm's true kappa is better than 0.85, because many of the images were very poor in quality, an an algorithm that was fed only well-prepared images would have probably done even better.
", "rawMarkdown": "E.g. http://casemed.case.edu/cpcpold/students/module4/Reeves/Acta.pdf\r\nsays a general physician and an ophthalmologist have a kappa of 0.838.\r\n\r\nhttp://visionquest-bio.com/VQ_pdf/SBarriga_05-01-2014web.pdf\r\nsays an optometrist and an ophthalmologist have a kappa of 0.72.\r\n\r\nThese studies are on different datasets, and may be using slightly different classifications, e,g. scales of 0-3 instead of 0-4. \r\n\r\nAnother point of reference: the kappa between left and right eyes in this dataset was 0.85, so the winning algorithm is as good as the ophthalmologist looking at the wrong eye. \r\n\r\nI would argue the winning algorithm's true kappa is better than 0.85, because many of the images were very poor in quality, an an algorithm that was fed only well-prepared images would have probably done even better.", "votes": null }, { "id": "87383", "postDate": "07/29/2015 04:07:58", "content": "Here some more pointers.
\nThis is the confusion matrix on one of our last submissions.
\nWhen you take a look at, for instance, doctor labeled 4 and computer predicted 0 you see at least 2 of the cases are very discussable IMO.
\n
Thanks all and congratulations!\nJulian, how did you obtain doctor's labeling?
", "rawMarkdown": "Thanks all and congratulations!\r\nJulian, how did you obtain doctor's labeling?", "votes": null }, { "id": "87395", "postDate": "07/29/2015 09:31:43", "content": "Hello,
\nThe confusion matrix is from the trainset and holdout set.
\nThere the doctor labels are given.
\nThe difference in predictions between train/test set is not big so it's good enough for illustrational purposes.
Ah thanks. I should suspect this obvious explanation :)
", "rawMarkdown": "Ah thanks. I should suspect this obvious explanation :)", "votes": null } ], "comments": [ { "id": 87365, "author_name": "hallayang", "author_url": "", "post_date": "07/29/2015 01:12:13", "content": "E.g. http://casemed.case.edu/cpcpold/students/module4/Reeves/Acta.pdf\nsays a general physician and an ophthalmologist have a kappa of 0.838.
\n\nhttp://visionquest-bio.com/VQ_pdf/SBarriga_05-01-2014web.pdf\nsays an optometrist and an ophthalmologist have a kappa of 0.72.
\n\nThese studies are on different datasets, and may be using slightly different classifications, e,g. scales of 0-3 instead of 0-4.
\n\nAnother point of reference: the kappa between left and right eyes in this dataset was 0.85, so the winning algorithm is as good as the ophthalmologist looking at the wrong eye.
\n\nI would argue the winning algorithm's true kappa is better than 0.85, because many of the images were very poor in quality, an an algorithm that was fed only well-prepared images would have probably done even better.
", "votes": null, "replies": [] }, { "id": 87383, "author_name": "juliandewit", "author_url": "", "post_date": "07/29/2015 04:07:58", "content": "Here some more pointers.
\nThis is the confusion matrix on one of our last submissions.
\nWhen you take a look at, for instance, doctor labeled 4 and computer predicted 0 you see at least 2 of the cases are very discussable IMO.
\n
Thanks all and congratulations!\nJulian, how did you obtain doctor's labeling?
", "votes": null, "replies": [] }, { "id": 87395, "author_name": "juliandewit", "author_url": "", "post_date": "07/29/2015 09:31:43", "content": "Hello,
\nThe confusion matrix is from the trainset and holdout set.
\nThere the doctor labels are given.
\nThe difference in predictions between train/test set is not big so it's good enough for illustrational purposes.
Ah thanks. I should suspect this obvious explanation :)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "87357": "Kaggle update on LinkedIn claims that \"Winning models from @CHCFNews Diabetic Retinopathy comp are on par with human performance!\" No further details provided. Has human performance score been already posted somewhere?", "87365": "E.g. http://casemed.case.edu/cpcpold/students/module4/Reeves/Acta.pdf\r\nsays a general physician and an ophthalmologist have a kappa of 0.838.\r\n\r\nhttp://visionquest-bio.com/VQ_pdf/SBarriga_05-01-2014web.pdf\r\nsays an optometrist and an ophthalmologist have a kappa of 0.72.\r\n\r\nThese studies are on different datasets, and may be using slightly different classifications, e,g. scales of 0-3 instead of 0-4. \r\n\r\nAnother point of reference: the kappa between left and right eyes in this dataset was 0.85, so the winning algorithm is as good as the ophthalmologist looking at the wrong eye. \r\n\r\nI would argue the winning algorithm's true kappa is better than 0.85, because many of the images were very poor in quality, an an algorithm that was fed only well-prepared images would have probably done even better.", "87383": "Here some more pointers.