I look through some of the cases that were rated as "0". However, there are definitely bleeding spots there. This incorrect rating will cause problems for the machine.
", "messages": [ { "id": "65261", "postDate": "03/02/2015 16:52:02", "content": "I look through some of the cases that were rated as "0". However, there are definitely bleeding spots there. This incorrect rating will cause problems for the machine.
", "rawMarkdown": "", "votes": null }, { "id": "65266", "postDate": "03/02/2015 17:07:12", "content": "I strongly suggested that the interested guys take a look at the images quickly. If there are some incorrect labeling with the benchmark, it will be tough to let the machine to learn correctly.
", "rawMarkdown": "", "votes": null }, { "id": "65268", "postDate": "03/02/2015 17:08:08", "content": "it may be better for the organizer to refine the labeling results.
", "rawMarkdown": "", "votes": null }, { "id": "65269", "postDate": "03/02/2015 17:12:08", "content": "Why do you think seeing a red spot means that a label of zero is incorrect? Not all bleeding in the eye is caused by DR.
", "rawMarkdown": "", "votes": null }, { "id": "65275", "postDate": "03/02/2015 18:29:29", "content": "Please send case numbers to the moderator and we can verify.
", "rawMarkdown": "", "votes": null }, { "id": "65341", "postDate": "03/03/2015 16:20:04", "content": "Your task is to agree as much as possible with the human labeler. Human labeling is hardly ever 100% correct, and thus the ground truth on most Kaggle competitions is hardly ever 100%. In theory your algorithm can only score as well as the human labeler, but not better: It could do better, but it would receive a lower score.
\nThen there is a maximum on how well your algorithms can model the data in practice. It is rare for a model to score the best possible score among all known algorithms.
\nIn the context of a spam problem: a human labeler may have 98% accuracy labeling spam. A model with the best possible hyperparams may score 97%. In the presence of such noise, robust algorithms are still able to learn.
\nI fully expect some labels to disagree with reality in this competition. This is a "problem" I'll have to work around. You can also use this problem to get better accuracies: instead of feeding 5 label vectors like [0,0,1,0,0] (human target label is "2"), you can try a model with gradient vectors [0,0.25,1,0.25,0] (human labeled as "2", so there is a small chance it could be "1" or 3"). In head pose estimation datasets (where the ground truth is also somewhat fuzzy) this gradient labeling is used to aid regression and make models more robust.
", "rawMarkdown": "", "votes": null }, { "id": "67028", "postDate": "03/18/2015 18:16:29", "content": "[quote=Triskelion;65341]
\nYour task is to agree as much as possible with the human labeler. Human labeling is hardly ever 100% correct, and thus the ground truth on most Kaggle competitions is hardly ever 100%. In theory your algorithm can only score as well as the human labeler, but not better: It could do better, but it would receive a lower score.
\nThen there is a maximum on how well your algorithms can model the data in practice. It is rare for a model to score the best possible score among all known algorithms.
\nIn the context of a spam problem: a human labeler may have 98% accuracy labeling spam. A model with the best possible hyperparams may score 97%. In the presence of such noise, robust algorithms are still able to learn.
\nI fully expect some labels to disagree with reality in this competition. This is a "problem" I'll have to work around. You can also use this problem to get better accuracies: instead of feeding 5 label vectors like [0,0,1,0,0] (human target label is "2"), you can try a model with gradient vectors [0,0.25,1,0.25,0] (human labeled as "2", so there is a small chance it could be "1" or 3"). In head pose estimation datasets (where the ground truth is also somewhat fuzzy) this gradient labeling is used to aid regression and make models more robust.
\n[/quote]
\nIt's kind of like overfitting, right? we shouldn't overfit our model with training set; otherwise, it has terrible performance on test set.
", "rawMarkdown": "", "votes": null }, { "id": "67033", "postDate": "03/18/2015 18:33:55", "content": "No I don't think that has to do with fitting. But agreed: overfitting usually is bad. The correct term for my "gradient labeling" approach is "soft labeling".
", "rawMarkdown": "", "votes": null }, { "id": "67075", "postDate": "03/18/2015 19:54:18", "content": "The goal of this competition is to find a robust algorithm for such a problem. If your results fluctuate too much under the effect of mislabeling then it won't do any good in actually practice. At least you don't have to deal with missing data and typos.
\nAs stated in the dataset page:
\n"Like any real-world data set, you will encounter noise in both the images and labels. Images may contain artifacts, be out of focus, underexposed, or overexposed. A major aim of this competition is to develop robust algorithms that can function in the presence of noise and variation."
", "rawMarkdown": "", "votes": null }, { "id": "71340", "postDate": "04/12/2015 20:21:32", "content": "35199_right is labeled 4 but, in my layman eyes, there not a single bleed.
\nThe other "4" samples look rather obvious.
\nIs it possible to know if it is a mislabeling or if it is really a "Proliferative DR" picture?
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 65266, "author_name": "xinmeng2", "author_url": "", "post_date": "03/02/2015 17:07:12", "content": "I strongly suggested that the interested guys take a look at the images quickly. If there are some incorrect labeling with the benchmark, it will be tough to let the machine to learn correctly.
", "votes": null, "replies": [] }, { "id": 65268, "author_name": "xinmeng2", "author_url": "", "post_date": "03/02/2015 17:08:08", "content": "it may be better for the organizer to refine the labeling results.
", "votes": null, "replies": [] }, { "id": 65269, "author_name": "jfkingiii", "author_url": "", "post_date": "03/02/2015 17:12:08", "content": "Why do you think seeing a red spot means that a label of zero is incorrect? Not all bleeding in the eye is caused by DR.
", "votes": null, "replies": [] }, { "id": 65275, "author_name": "jorge9", "author_url": "", "post_date": "03/02/2015 18:29:29", "content": "Please send case numbers to the moderator and we can verify.
", "votes": null, "replies": [] }, { "id": 65341, "author_name": "triskelion", "author_url": "", "post_date": "03/03/2015 16:20:04", "content": "Your task is to agree as much as possible with the human labeler. Human labeling is hardly ever 100% correct, and thus the ground truth on most Kaggle competitions is hardly ever 100%. In theory your algorithm can only score as well as the human labeler, but not better: It could do better, but it would receive a lower score.
\nThen there is a maximum on how well your algorithms can model the data in practice. It is rare for a model to score the best possible score among all known algorithms.
\nIn the context of a spam problem: a human labeler may have 98% accuracy labeling spam. A model with the best possible hyperparams may score 97%. In the presence of such noise, robust algorithms are still able to learn.
\nI fully expect some labels to disagree with reality in this competition. This is a "problem" I'll have to work around. You can also use this problem to get better accuracies: instead of feeding 5 label vectors like [0,0,1,0,0] (human target label is "2"), you can try a model with gradient vectors [0,0.25,1,0.25,0] (human labeled as "2", so there is a small chance it could be "1" or 3"). In head pose estimation datasets (where the ground truth is also somewhat fuzzy) this gradient labeling is used to aid regression and make models more robust.
", "votes": null, "replies": [] }, { "id": 67028, "author_name": "zhouyang2", "author_url": "", "post_date": "03/18/2015 18:16:29", "content": "[quote=Triskelion;65341]
\nYour task is to agree as much as possible with the human labeler. Human labeling is hardly ever 100% correct, and thus the ground truth on most Kaggle competitions is hardly ever 100%. In theory your algorithm can only score as well as the human labeler, but not better: It could do better, but it would receive a lower score.
\nThen there is a maximum on how well your algorithms can model the data in practice. It is rare for a model to score the best possible score among all known algorithms.
\nIn the context of a spam problem: a human labeler may have 98% accuracy labeling spam. A model with the best possible hyperparams may score 97%. In the presence of such noise, robust algorithms are still able to learn.
\nI fully expect some labels to disagree with reality in this competition. This is a "problem" I'll have to work around. You can also use this problem to get better accuracies: instead of feeding 5 label vectors like [0,0,1,0,0] (human target label is "2"), you can try a model with gradient vectors [0,0.25,1,0.25,0] (human labeled as "2", so there is a small chance it could be "1" or 3"). In head pose estimation datasets (where the ground truth is also somewhat fuzzy) this gradient labeling is used to aid regression and make models more robust.
\n[/quote]
\nIt's kind of like overfitting, right? we shouldn't overfit our model with training set; otherwise, it has terrible performance on test set.
", "votes": null, "replies": [] }, { "id": 67033, "author_name": "triskelion", "author_url": "", "post_date": "03/18/2015 18:33:55", "content": "No I don't think that has to do with fitting. But agreed: overfitting usually is bad. The correct term for my "gradient labeling" approach is "soft labeling".
", "votes": null, "replies": [] }, { "id": 67075, "author_name": "shaynewang", "author_url": "", "post_date": "03/18/2015 19:54:18", "content": "The goal of this competition is to find a robust algorithm for such a problem. If your results fluctuate too much under the effect of mislabeling then it won't do any good in actually practice. At least you don't have to deal with missing data and typos.
\nAs stated in the dataset page:
\n"Like any real-world data set, you will encounter noise in both the images and labels. Images may contain artifacts, be out of focus, underexposed, or overexposed. A major aim of this competition is to develop robust algorithms that can function in the presence of noise and variation."
", "votes": null, "replies": [] }, { "id": 71340, "author_name": "oliviers", "author_url": "", "post_date": "04/12/2015 20:21:32", "content": "35199_right is labeled 4 but, in my layman eyes, there not a single bleed.
\nThe other "4" samples look rather obvious.
\nIs it possible to know if it is a mislabeling or if it is really a "Proliferative DR" picture?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "65261": "", "65266": "", "65268": "", "65269": "", "65275": "", "65341": "", "67028": "", "67033": "", "67075": "", "71340": "" }, "source": "meta" }