{ "id": 14402, "title": "A rogue's gallery of training cases", "url": "/competitions/diabetic-retinopathy-detection/discussion/14402", "author_name": "", "post_date": "2015-05-27T17:47:58.573Z", "votes": 8, "comment_count": 10, "views": 3950, "content": "

The data description page is not kidding when it states:

\n

Like any real-world data set, you will encounter noise in both the images and labels. Images may contain artifacts, be out of focus, underexposed, or overexposed.

\n

For your amusement I have attached images of a set of shall-we-say challenging cases I ran across in my wanderings through the training data set.

\n

In a more serious vein, can anyone give a rationale for the severity ratings in any of these cases? Also, feel free to add your own examples.

", "messages": [ { "id": "80163", "postDate": "05/27/2015 17:47:58", "content": "

The data description page is not kidding when it states:

\n

Like any real-world data set, you will encounter noise in both the images and labels. Images may contain artifacts, be out of focus, underexposed, or overexposed.

\n

For your amusement I have attached images of a set of shall-we-say challenging cases I ran across in my wanderings through the training data set.

\n

In a more serious vein, can anyone give a rationale for the severity ratings in any of these cases? Also, feel free to add your own examples.

", "rawMarkdown": "", "votes": null }, { "id": "80229", "postDate": "05/28/2015 09:23:30", "content": "

I'm no expert but I wonder if the evaluation of this one was made by someone suffering from the affliction we are trying to diagnose :) 

\n

It gets a '0' but I'd say at least a '2' or '3'.
Or am I missing something here ?

", "rawMarkdown": "", "votes": null }, { "id": "80264", "postDate": "05/28/2015 16:49:03", "content": "

[quote=Julian de Wit;80229]

\n

I'm no expert but I wonder if the evaluation of this one was made by someone suffering from the affliction we are trying to diagnose :) 

\n

[/quote]

\n\n

I was particularly amused by 31202_right getting a 3 rating for a blank image.

", "rawMarkdown": "", "votes": null }, { "id": "80307", "postDate": "05/28/2015 21:03:20", "content": "

bobalv,

\n

where did you take all these grey images?

", "rawMarkdown": "", "votes": null }, { "id": "80378", "postDate": "05/29/2015 15:28:39", "content": "

[quote=soreshn;80307]

\n

bobalv,

\n

where did you take all these grey images?

\n

[/quote]

\n

I computed them from the RGB planes of the color images.

\n

gray =  0.299R + 0.587G + 0.114B

\n


cf. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3046650/

", "rawMarkdown": "", "votes": null }, { "id": "80412", "postDate": "05/29/2015 19:51:41", "content": "

Thank you.

", "rawMarkdown": "", "votes": null }, { "id": "81071", "postDate": "06/06/2015 00:00:13", "content": "

I'm new here and I wonder if there is a mechanism to request or propose changes to the dataset, which would remove the obvious errors.

", "rawMarkdown": "", "votes": null }, { "id": "81072", "postDate": "06/06/2015 00:10:46", "content": "

[quote=PaulJurczak;81071]

\n

I'm new here and I wonder if there is a mechanism to request or propose changes to the dataset, which would remove the obvious errors.

\n

[/quote]

\n

There generally will not be any changes to the dataset unless something is severally broken(the labels are leaked into the training set or half of one of the data sets is missing.) Every time there is a change everyone needs to redownload, and if it is in the test set it will invalidate the whole leaderboard.

", "rawMarkdown": "", "votes": null }, { "id": "81073", "postDate": "06/06/2015 00:15:32", "content": "

Hi Paul,

\n

We only update datasets if the amount of noise jeopardizes the competition. Otherwise, it's there for everybody, hurts everybody equally (in the test set), and becomes part of the challenge (in the train set). These errors happen in real data, so part of a good algorithm should be insensitivity to noise.

\n

https://www.kaggle.com/wiki/ANoteOnDataQuality

\n

Hope this helps to clarify our stance. Given Kaggle's popularity, fixing a small number of errors and releasing new data is generally not worth the confusion, nor is it a small undertaking (on this competition we've already served hundreds of terabytes of copies of the data).

", "rawMarkdown": "", "votes": null }, { "id": "81079", "postDate": "06/06/2015 02:26:06", "content": "

Hi Devin and William,

\n

Thank you for clarification. I agree on the noise insensitivity, but I would hate to see an algorithm win, which is not the best performer on "good" samples, but compensates by better performance on "junk" samples. OTOH, this happens in real life quite often...

\n

On releasing new data: only a list of files to remove and updated train labels would have to be published - the rest stays the same. Optionally, "bad sample" category could be introduced.

\n

On the workload required to cleanup the data: this could be crowdsourced to competition participants with some sort of voting mechanism and veto power granted to administrators.  

", "rawMarkdown": "", "votes": null }, { "id": "361275", "postDate": "07/24/2018 06:30:39", "content": "

I wonder whether there is image quality assessment in real practice. Should bad image really be used for diagnose?

", "rawMarkdown": "I wonder whether there is image quality assessment in real practice. Should bad image really be used for diagnose?", "votes": null } ], "comments": [ { "id": 80229, "author_name": "juliandewit", "author_url": "", "post_date": "05/28/2015 09:23:30", "content": "

I'm no expert but I wonder if the evaluation of this one was made by someone suffering from the affliction we are trying to diagnose :) 

\n

It gets a '0' but I'd say at least a '2' or '3'.
Or am I missing something here ?

", "votes": null, "replies": [] }, { "id": 80264, "author_name": "bobalv", "author_url": "", "post_date": "05/28/2015 16:49:03", "content": "

[quote=Julian de Wit;80229]

\n

I'm no expert but I wonder if the evaluation of this one was made by someone suffering from the affliction we are trying to diagnose :) 

\n

[/quote]

\n\n

I was particularly amused by 31202_right getting a 3 rating for a blank image.

", "votes": null, "replies": [] }, { "id": 80307, "author_name": "soreshn", "author_url": "", "post_date": "05/28/2015 21:03:20", "content": "

bobalv,

\n

where did you take all these grey images?

", "votes": null, "replies": [] }, { "id": 80378, "author_name": "bobalv", "author_url": "", "post_date": "05/29/2015 15:28:39", "content": "

[quote=soreshn;80307]

\n

bobalv,

\n

where did you take all these grey images?

\n

[/quote]

\n

I computed them from the RGB planes of the color images.

\n

gray =  0.299R + 0.587G + 0.114B

\n


cf. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3046650/

", "votes": null, "replies": [] }, { "id": 80412, "author_name": "soreshn", "author_url": "", "post_date": "05/29/2015 19:51:41", "content": "

Thank you.

", "votes": null, "replies": [] }, { "id": 81071, "author_name": "pauljurczak", "author_url": "", "post_date": "06/06/2015 00:00:13", "content": "

I'm new here and I wonder if there is a mechanism to request or propose changes to the dataset, which would remove the obvious errors.

", "votes": null, "replies": [] }, { "id": 81072, "author_name": "devinanzelmo", "author_url": "", "post_date": "06/06/2015 00:10:46", "content": "

[quote=PaulJurczak;81071]

\n

I'm new here and I wonder if there is a mechanism to request or propose changes to the dataset, which would remove the obvious errors.

\n

[/quote]

\n

There generally will not be any changes to the dataset unless something is severally broken(the labels are leaked into the training set or half of one of the data sets is missing.) Every time there is a change everyone needs to redownload, and if it is in the test set it will invalidate the whole leaderboard.

", "votes": null, "replies": [] }, { "id": 81073, "author_name": "wcukierski", "author_url": "", "post_date": "06/06/2015 00:15:32", "content": "

Hi Paul,

\n

We only update datasets if the amount of noise jeopardizes the competition. Otherwise, it's there for everybody, hurts everybody equally (in the test set), and becomes part of the challenge (in the train set). These errors happen in real data, so part of a good algorithm should be insensitivity to noise.

\n

https://www.kaggle.com/wiki/ANoteOnDataQuality

\n

Hope this helps to clarify our stance. Given Kaggle's popularity, fixing a small number of errors and releasing new data is generally not worth the confusion, nor is it a small undertaking (on this competition we've already served hundreds of terabytes of copies of the data).

", "votes": null, "replies": [] }, { "id": 81079, "author_name": "pauljurczak", "author_url": "", "post_date": "06/06/2015 02:26:06", "content": "

Hi Devin and William,

\n

Thank you for clarification. I agree on the noise insensitivity, but I would hate to see an algorithm win, which is not the best performer on "good" samples, but compensates by better performance on "junk" samples. OTOH, this happens in real life quite often...

\n

On releasing new data: only a list of files to remove and updated train labels would have to be published - the rest stays the same. Optionally, "bad sample" category could be introduced.

\n

On the workload required to cleanup the data: this could be crowdsourced to competition participants with some sort of voting mechanism and veto power granted to administrators.  

", "votes": null, "replies": [] }, { "id": 361275, "author_name": "samjia", "author_url": "", "post_date": "07/24/2018 06:30:39", "content": "

I wonder whether there is image quality assessment in real practice. Should bad image really be used for diagnose?

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "80163": "", "80229": "", "80264": "", "80307": "", "80378": "", "80412": "", "81071": "", "81072": "", "81073": "", "81079": "", "361275": "I wonder whether there is image quality assessment in real practice. Should bad image really be used for diagnose?" }, "source": "meta" }