{ "id": 12612, "title": "Outliers and expert ratings", "url": "/competitions/diabetic-retinopathy-detection/discussion/12612", "author_name": "", "post_date": "2015-02-25T23:12:16.747Z", "votes": 5, "comment_count": 6, "views": 3067, "content": "

Are the human ratings based on a human rating the image, or a human rating the patient? (for example during an exam where a doctor can look around, and also takes a picture along the way)

\n

Could you provide some cross-validation data from multiple human expert ratings of the images only?

\n

I mention this because of of some images in the training set like: 679_right, 21345_left.  These are both rated as... 2, apparently.

", "messages": [ { "id": "64895", "postDate": "02/25/2015 23:12:16", "content": "

Are the human ratings based on a human rating the image, or a human rating the patient? (for example during an exam where a doctor can look around, and also takes a picture along the way)

\n

Could you provide some cross-validation data from multiple human expert ratings of the images only?

\n

I mention this because of of some images in the training set like: 679_right, 21345_left.  These are both rated as... 2, apparently.

", "rawMarkdown": "", "votes": null }, { "id": "64896", "postDate": "02/25/2015 23:40:59", "content": "

In some cases the raters had access to multiple fields of the same eye (i.e. they will take two images of the eye if one does not come out right). Since the dataset only includes one field per eye, there are a few (but not many) images like this.

", "rawMarkdown": "", "votes": null }, { "id": "64900", "postDate": "02/26/2015 01:10:43", "content": "

Ah, okay, good to know, it's not magic :)

", "rawMarkdown": "", "votes": null }, { "id": "66905", "postDate": "03/18/2015 04:35:25", "content": "

I have a follow-up question...

\n

Did the doctor have other information while rating the patient? i.e. The doctor knew the patient has diabetes and/or some other factors that would influence the patient's retina health. 

\n

Or is it that the expert looked only at the picture(s) of the patient and made the judgement?

", "rawMarkdown": "", "votes": null }, { "id": "67502", "postDate": "03/21/2015 05:46:05", "content": "

William - a few more questions about this:  

\n

Do you have any human vs human cross-validation results?  (and can you share them? or at least a qualitative description: something like "the ratings look pretty reliable" :)

\n

Are the ratings by a single expert each, or the consensus of a panel?  (or perhaps a single expert who could get a second opinion if in doubt)

\n

How many different raters were there?

\n

Best regards,

\n

Alex

\n\n

[quote=William Cukierski;64896]

\n

In some cases the raters had access to multiple fields of the same eye (i.e. they will take two images of the eye if one does not come out right). Since the dataset only includes one field per eye, there are a few (but not many) images like this.

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "67533", "postDate": "03/21/2015 16:23:08", "content": "

There was some discussion of that in this thread.

", "rawMarkdown": "", "votes": null }, { "id": "67955", "postDate": "03/24/2015 12:44:19", "content": "

Following up on the expert ratings, could the organizers provide any details about the grading protocols / guidelines followed by the experts while performing the grading?

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 64896, "author_name": "wcukierski", "author_url": "", "post_date": "02/25/2015 23:40:59", "content": "

In some cases the raters had access to multiple fields of the same eye (i.e. they will take two images of the eye if one does not come out right). Since the dataset only includes one field per eye, there are a few (but not many) images like this.

", "votes": null, "replies": [] }, { "id": 64900, "author_name": "aizvorski", "author_url": "", "post_date": "02/26/2015 01:10:43", "content": "

Ah, okay, good to know, it's not magic :)

", "votes": null, "replies": [] }, { "id": 66905, "author_name": "desmondiwang", "author_url": "", "post_date": "03/18/2015 04:35:25", "content": "

I have a follow-up question...

\n

Did the doctor have other information while rating the patient? i.e. The doctor knew the patient has diabetes and/or some other factors that would influence the patient's retina health. 

\n

Or is it that the expert looked only at the picture(s) of the patient and made the judgement?

", "votes": null, "replies": [] }, { "id": 67502, "author_name": "aizvorski", "author_url": "", "post_date": "03/21/2015 05:46:05", "content": "

William - a few more questions about this:  

\n

Do you have any human vs human cross-validation results?  (and can you share them? or at least a qualitative description: something like "the ratings look pretty reliable" :)

\n

Are the ratings by a single expert each, or the consensus of a panel?  (or perhaps a single expert who could get a second opinion if in doubt)

\n

How many different raters were there?

\n

Best regards,

\n

Alex

\n\n

[quote=William Cukierski;64896]

\n

In some cases the raters had access to multiple fields of the same eye (i.e. they will take two images of the eye if one does not come out right). Since the dataset only includes one field per eye, there are a few (but not many) images like this.

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 67533, "author_name": "alexcoventry", "author_url": "", "post_date": "03/21/2015 16:23:08", "content": "

There was some discussion of that in this thread.

", "votes": null, "replies": [] }, { "id": 67955, "author_name": "markvangrinsven", "author_url": "", "post_date": "03/24/2015 12:44:19", "content": "

Following up on the expert ratings, could the organizers provide any details about the grading protocols / guidelines followed by the experts while performing the grading?

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "64895": "", "64896": "", "64900": "", "66905": "", "67502": "", "67533": "", "67955": "" }, "source": "meta" }