![\"fundus](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F479538%2F4e4289f3e3482027732e1b8ef9e2d201%2Fimg1.png?generation=1562465525301620&alt=media\")
Although I could not yet manged to build a good model to predict the retinopathic severity, guessing my model seems to be more accurate in some cases ( I'm not an ophthalmologist, it's my intuition ;) ). Here is the cases I observed.
\n\nFundus images
\n\n
After taking a quick check on the Internet, I found out that the label inaccuracy has been an issue in this field. For example, the label inconsistency among ophthalmologists is introduced in the following movie.
\n\nTensorFlow Dev Summit 2017 Retinal Imaging\nhttps://youtu.be/oOeZ7IgEN4o?t=156
\n\nI suppose, to tackle this issue, we must implement a robust loss function considering abnormal losses or other remedies as seen in the previous competition (DRD).
\nAny opinions are welcome.
Updated Jul.10.2019
\n\nThe paper of GoogleAI related to above movie
\nDeep Learning vs. Human Graders for Classifying Severity Levels of Diabetic Retinopathy in a Real-World Nationwide Screening Program
https://arxiv.org/abs/1810.08290
", "messages": [ { "id": 569611, "postDate": "2019-07-07T02:40:02.707Z", "content": "Although I could not yet manged to build a good model to predict the retinopathic severity, guessing my model seems to be more accurate in some cases ( I'm not an ophthalmologist, it's my intuition ;) ). Here is the cases I observed.
\n\nFundus images
\n\n
After taking a quick check on the Internet, I found out that the label inaccuracy has been an issue in this field. For example, the label inconsistency among ophthalmologists is introduced in the following movie.
\n\nTensorFlow Dev Summit 2017 Retinal Imaging\nhttps://youtu.be/oOeZ7IgEN4o?t=156
\n\nI suppose, to tackle this issue, we must implement a robust loss function considering abnormal losses or other remedies as seen in the previous competition (DRD).
\nAny opinions are welcome.
Updated Jul.10.2019
\n\nThe paper of GoogleAI related to above movie
\nDeep Learning vs. Human Graders for Classifying Severity Levels of Diabetic Retinopathy in a Real-World Nationwide Screening Program
https://arxiv.org/abs/1810.08290
", "rawMarkdown": "Although I could not yet manged to build a good model to predict the retinopathic severity, guessing my model seems to be more accurate in some cases ( I'm not an ophthalmologist, it's my intuition ;) ). Here is the cases I observed. \n\n**Fundus images**\n\n\n\n \nAfter taking a quick check on the Internet, I found out that the label inaccuracy has been an issue in this field. For example, the label inconsistency among ophthalmologists is introduced in the following movie. \n\nTensorFlow Dev Summit 2017 Retinal Imaging\n[https://youtu.be/oOeZ7IgEN4o?t=156](https://youtu.be/oOeZ7IgEN4o?t=156)\n\nI suppose, to tackle this issue, we must implement a robust loss function considering abnormal losses or other remedies as seen in the previous competition (DRD). \nAny opinions are welcome.\n\n---\nUpdated Jul.10.2019 \n\n**The paper of GoogleAI related to above movie** \nDeep Learning vs. Human Graders for Classifying Severity Levels of Diabetic Retinopathy in a Real-World Nationwide Screening Program\n\nhttps://arxiv.org/abs/1810.08290", "votes": 53 }, { "id": 569759, "postDate": "2019-07-07T09:14:44.780Z", "content": "As Maxwell said, this is the specific slide showing inconsistency of label estimation:\nNote that each row illustrates a single patient, and each column represents a single doctor's severity estimation of that patient case.
\n\n![\"inconsistent](\"https://i.ibb.co/6rQ2sFG/inconsistent-estimation.png\")
I feel the same. \nI found 268 duplicated images in train data, and 64 among them have different labels (e.g. 14e3f84445f7 and f0f89314e860). \nDuplication also occurs between train and test, and I guess some of them have different labels.
", "rawMarkdown": "I feel the same. \nI found 268 duplicated images in train data, and 64 among them have different labels (e.g. 14e3f84445f7 and f0f89314e860). \nDuplication also occurs between train and test, and I guess some of them have different labels.", "votes": 13, "replies": [ { "id": 569648, "postDate": "2019-07-07T05:13:22.670Z", "content": "@naka2ka \nGreat insight.\nSo our models are forced to learn the different labels on the same image😣...
\n\nIn this kernel, as you did, the duplication was checked by @miklgr500 .\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/98856#latest-569611
\n\nI felt the models tend to overfit, one of the reason of that will be this...
", "rawMarkdown": "@naka2ka \nGreat insight.\nSo our models are forced to learn the different labels on the same image😣... \n\nIn this kernel, as you did, the duplication was checked by @miklgr500 .\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/98856#latest-569611\n\nI felt the models tend to overfit, one of the reason of that will be this...", "votes": 2 }, { "id": 569662, "postDate": "2019-07-07T05:48:37.777Z", "content": "Yeah. @sohier\nid_code: 14e3f84445f7 diagnosis: 3
\n\nid_code: f0f89314e860 diagnosis: 4
\n\nThese two photos are the same but have different labels.
", "rawMarkdown": "Yeah. @sohier\nid_code: 14e3f84445f7 diagnosis: 3\n\nid_code: f0f89314e860 diagnosis: 4\n\nThese two photos are the same but have different labels.", "votes": 3 }, { "id": 569666, "postDate": "2019-07-07T05:55:16.280Z", "content": "Are any ophthalmologists on board?\nAre any ophthalmologists on board?
", "rawMarkdown": "Are any ophthalmologists on board?\nAre any ophthalmologists on board?", "votes": 3 } ] }, { "id": 570171, "postDate": "2019-07-08T00:10:26.410Z", "content": "Updated:
\nI published a little bit modified version of below kernel.
\nYou can download duplicated list of train data.
https://www.kaggle.com/maxwell110/duplicated-list-csv-file
\n\nTo detect the duplicated images, this awesome kernel will be a help.
\nhttps://www.kaggle.com/manojprabhaakr/similar-duplicate-images-in-aptos-data
Thanks @manojprabhaakr !
", "rawMarkdown": "Updated: \nI published a little bit modified version of below kernel. \nYou can download duplicated list of train data. \n\nhttps://www.kaggle.com/maxwell110/duplicated-list-csv-file\n\n---\n\nTo detect the duplicated images, this awesome kernel will be a help. \nhttps://www.kaggle.com/manojprabhaakr/similar-duplicate-images-in-aptos-data\n\nThanks @manojprabhaakr !", "votes": 5 }, { "id": 598054, "postDate": "2019-08-13T04:01:59Z", "content": "@maxwell110, I also see there is label inaccuracy. Below severity stage differences could help. I feel Data Cleaning is must but how would that be accounted on the hidden test dataset?\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F43269%2Ff37410a72422e57d78e9ea5c7eb2d024%2Fretinopathy.png?generation=1565668605750117&alt=media\")
@hmnshu \nI'm not sure Data Cleansing works well or not for private test dataset. But I think private test dataset will also include inaccurate labels.
\n\nFor example, DRD2015(previous compettition) dataset seems to include ambiguous labels.
\nIn this paper, the author said that 75% images are gradable.
\nhttps://www.biorxiv.org/content/biorxiv/early/2018/06/19/225508.full.pdf
I have used the cleaned dataset to train, better cv : 0.9189->0.9284, but lower lb: 0.782->0.769; maybe those duplicate images could work like regularization, I'm a noob 😯
", "rawMarkdown": "I have used the cleaned dataset to train, better cv : 0.9189->0.9284, but lower lb: 0.782->0.769; maybe those duplicate images could work like regularization, I'm a noob 😯 ", "votes": 3, "replies": [ { "id": 571019, "postDate": "2019-07-09T04:17:27.060Z", "content": "I recommend you forget current test data with public lb.
", "rawMarkdown": "I recommend you forget current test data with public lb.", "votes": 3 }, { "id": 571035, "postDate": "2019-07-09T04:52:57.397Z", "content": "Thanks for your recommendation, I have read your kernel about the data distribution, seems the training data has a different distribution with testing data, so maybe cv is not compromising. How about the relationship between public test data and private test data? Based on your kernel, cv should be not promising right?
", "rawMarkdown": "Thanks for your recommendation, I have read your kernel about the data distribution, seems the training data has a different distribution with testing data, so maybe cv is not compromising. How about the relationship between public test data and private test data? Based on your kernel, cv should be not promising right?", "votes": 1 }, { "id": 571068, "postDate": "2019-07-09T06:00:43.243Z", "content": "Current public lb calculated on 15%(~300 img) of test data, which have 8% of leaks. Also we know that final decision must have possibility run on 20GB test dataset, that meaning that test dataset changed in the end of competition. So all above make this competition without real public lb.
", "rawMarkdown": "Current public lb calculated on 15%(~300 img) of test data, which have 8% of leaks. Also we know that final decision must have possibility run on 20GB test dataset, that meaning that test dataset changed in the end of competition. So all above make this competition without real public lb.", "votes": 2 }, { "id": 571074, "postDate": "2019-07-09T06:12:27.400Z", "content": "But since we have stage 2 running (a new test set), I think the public lb may contain whole 1928 test images that we can see in test.csv.
", "rawMarkdown": "But since we have stage 2 running (a new test set), I think the public lb may contain whole 1928 test images that we can see in test.csv.", "votes": 1 }, { "id": 571082, "postDate": "2019-07-09T06:23:25.180Z", "content": "Ok, if it's true, then we have 300/13000=3/130~1/40 -> 2.5%. Are you really believe that 2.5% of final test dataset can influence on your choice better decision?:)
", "rawMarkdown": "Ok, if it's true, then we have 300/13000=3/130~1/40 -> 2.5%. Are you really believe that 2.5% of final test dataset can influence on your choice better decision?:)" }, { "id": 572069, "postDate": "2019-07-10T12:10:28.573Z", "content": "Do you use external dataset or you just use the official dataset to achieve 7th?
", "rawMarkdown": "Do you use external dataset or you just use the official dataset to achieve 7th?" }, { "id": 598799, "postDate": "2019-08-14T03:34:21.933Z", "content": "Hi @jionie , by \"cleaned dataset\" did you mean you had also removed images in training set that seem to have unreliable labels or you just removed duplicates?
", "rawMarkdown": "Hi @jionie , by \"cleaned dataset\" did you mean you had also removed images in training set that seem to have unreliable labels or you just removed duplicates?", "votes": 1 }, { "id": 598800, "postDate": "2019-08-14T03:38:27.607Z", "content": "@miklgr500 Isn't the public lb calculated based on 1928 images(out of 1928 / 15% test images)? They won't use only 300 images for the public lb... Also if the 15% are randomly sampled(instead of not sampling evenly on purpose) from the whole test set, then public lb would be much more reliable than cv.
", "rawMarkdown": "@miklgr500 Isn't the public lb calculated based on 1928 images(out of 1928 / 15% test images)? They won't use only 300 images for the public lb... Also if the 15% are randomly sampled(instead of not sampling evenly on purpose) from the whole test set, then public lb would be much more reliable than cv." }, { "id": 599436, "postDate": "2019-08-15T00:50:13.553Z", "content": "I removed all samples based on the public csv, but now I just keep all.
", "rawMarkdown": "I removed all samples based on the public csv, but now I just keep all." } ] }, { "id": 570168, "postDate": "2019-07-08T00:06:07.260Z", "content": "Well, I guess this is another evidence suggesting that some hard work needs to be done on data cleaning.
", "rawMarkdown": "Well, I guess this is another evidence suggesting that some hard work needs to be done on data cleaning. ", "votes": 1, "replies": [ { "id": 570350, "postDate": "2019-07-08T07:05:07.760Z", "content": "But what if test lables are also mislabled and cleaning the training set would result in poor performance in model?
\n\nThis manual cleaning can also result in overfitting of training set, right?
", "rawMarkdown": "But what if test lables are also mislabled and cleaning the training set would result in poor performance in model?\n\nThis manual cleaning can also result in overfitting of training set, right?", "votes": 1 } ] }, { "id": 570098, "postDate": "2019-07-07T19:13:32.523Z", "content": "Emm, may need to remove those images with different labels in the training set, just view them as unlabeled data.
", "rawMarkdown": "Emm, may need to remove those images with different labels in the training set, just view them as unlabeled data.", "votes": 2, "replies": [ { "id": 570170, "postDate": "2019-07-08T00:08:26.007Z", "content": "Thanks to @manojprabhaakr ,\nhttps://www.kaggle.com/manojprabhaakr/similar-duplicate-images-in-aptos-data\nwe can detect the duplicated images.
\n\nAccording to this kernel, there are 143 duplicated images in train data 😩 .
\nI will exclude these extra images not to overfit when training models.
I believe there may be manual errors leading to this result
", "rawMarkdown": "I believe there may be manual errors leading to this result" }, { "id": 571264, "postDate": "2019-07-09T12:08:49.540Z", "content": "agree with you
", "rawMarkdown": "agree with you" }, { "id": 571046, "postDate": "2019-07-09T05:25:30.813Z", "content": "This is cool. Did you consider not using accuracy at all, but something like precision or recall? f-score (a little of both)?
", "rawMarkdown": "This is cool. Did you consider not using accuracy at all, but something like precision or recall? f-score (a little of both)?", "replies": [ { "id": 571159, "postDate": "2019-07-09T08:49:24.773Z", "content": "@kbridge14 \nActually, I'm using MSE. I have tried to solve this task as a regression problem.
", "rawMarkdown": "@kbridge14 \nActually, I'm using MSE. I have tried to solve this task as a regression problem.", "votes": 2 }, { "id": 598143, "postDate": "2019-08-13T07:39:13.720Z", "content": "@maxwell110 Given you are solving this task as a regression problem, What are your thoughts on Mixup Data augmentation?
", "rawMarkdown": "@maxwell110 Given you are solving this task as a regression problem, What are your thoughts on Mixup Data augmentation? " }, { "id": 598658, "postDate": "2019-08-13T21:05:14.580Z", "content": "@monsterspy \nIMO, Mixup will not work well due to the nonlinearity of diabetic retinopathy grading. I mean, ophthalmologists grade images with some non-linear thresholds.
\nFor example, imagine when you mix grade 1 and 3 images with equivalent probability(= 0.5), the target value would be 2(= 1 * 0.5 + 3 * 0.5). But I suppose ophthalmologists will make a diagnosis as grade 3 or higher. Because they will think the image has a lot of blurred but serious findings.
For more detail about diagnosis criteria, check the following material.
\n(Sorry, I forgot who introduced this in the discussion thread.)\nhttp://eyesteve.com/diabetic-retinopathy-grading/
Thanks @maxwell110 for sharing your thoughts.
", "rawMarkdown": "Thanks @maxwell110 for sharing your thoughts. ", "votes": 1 } ] }, { "id": 570270, "postDate": "2019-07-08T04:28:42.027Z", "content": "up
", "rawMarkdown": "up" } ], "comments": [ { "id": 569759, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-07-07T09:14:44.780000", "content": "As Maxwell said, this is the specific slide showing inconsistency of label estimation:\nNote that each row illustrates a single patient, and each column represents a single doctor's severity estimation of that patient case.
\n\n
I feel the same. \nI found 268 duplicated images in train data, and 64 among them have different labels (e.g. 14e3f84445f7 and f0f89314e860). \nDuplication also occurs between train and test, and I guess some of them have different labels.
", "votes": 13, "replies": [ { "id": 569648, "author_name": "Maxwell", "author_url": "", "post_date": "2019-07-07T05:13:22.670000", "content": "@naka2ka \nGreat insight.\nSo our models are forced to learn the different labels on the same image😣...
\n\nIn this kernel, as you did, the duplication was checked by @miklgr500 .\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/98856#latest-569611
\n\nI felt the models tend to overfit, one of the reason of that will be this...
", "votes": 2, "replies": [] }, { "id": 569662, "author_name": "Zheng Li", "author_url": "", "post_date": "2019-07-07T05:48:37.777000", "content": "Yeah. @sohier\nid_code: 14e3f84445f7 diagnosis: 3
\n\nid_code: f0f89314e860 diagnosis: 4
\n\nThese two photos are the same but have different labels.
", "votes": 3, "replies": [] }, { "id": 569666, "author_name": "Maxwell", "author_url": "", "post_date": "2019-07-07T05:55:16.280000", "content": "Are any ophthalmologists on board?\nAre any ophthalmologists on board?
", "votes": 3, "replies": [] } ] }, { "id": 570171, "author_name": "Maxwell", "author_url": "", "post_date": "2019-07-08T00:10:26.410000", "content": "Updated:
\nI published a little bit modified version of below kernel.
\nYou can download duplicated list of train data.
https://www.kaggle.com/maxwell110/duplicated-list-csv-file
\n\nTo detect the duplicated images, this awesome kernel will be a help.
\nhttps://www.kaggle.com/manojprabhaakr/similar-duplicate-images-in-aptos-data
Thanks @manojprabhaakr !
", "votes": 5, "replies": [] }, { "id": 598054, "author_name": "Himanshu Gamit", "author_url": "", "post_date": "2019-08-13T04:01:59", "content": "@maxwell110, I also see there is label inaccuracy. Below severity stage differences could help. I feel Data Cleaning is must but how would that be accounted on the hidden test dataset?\n
@hmnshu \nI'm not sure Data Cleansing works well or not for private test dataset. But I think private test dataset will also include inaccurate labels.
\n\nFor example, DRD2015(previous compettition) dataset seems to include ambiguous labels.
\nIn this paper, the author said that 75% images are gradable.
\nhttps://www.biorxiv.org/content/biorxiv/early/2018/06/19/225508.full.pdf
I have used the cleaned dataset to train, better cv : 0.9189->0.9284, but lower lb: 0.782->0.769; maybe those duplicate images could work like regularization, I'm a noob 😯
", "votes": 3, "replies": [ { "id": 571019, "author_name": "Welf Crozzo", "author_url": "", "post_date": "2019-07-09T04:17:27.060000", "content": "I recommend you forget current test data with public lb.
", "votes": 3, "replies": [] }, { "id": 571035, "author_name": "jionie", "author_url": "", "post_date": "2019-07-09T04:52:57.397000", "content": "Thanks for your recommendation, I have read your kernel about the data distribution, seems the training data has a different distribution with testing data, so maybe cv is not compromising. How about the relationship between public test data and private test data? Based on your kernel, cv should be not promising right?
", "votes": 1, "replies": [] }, { "id": 571068, "author_name": "Welf Crozzo", "author_url": "", "post_date": "2019-07-09T06:00:43.243000", "content": "Current public lb calculated on 15%(~300 img) of test data, which have 8% of leaks. Also we know that final decision must have possibility run on 20GB test dataset, that meaning that test dataset changed in the end of competition. So all above make this competition without real public lb.
", "votes": 2, "replies": [] }, { "id": 571074, "author_name": "jionie", "author_url": "", "post_date": "2019-07-09T06:12:27.400000", "content": "But since we have stage 2 running (a new test set), I think the public lb may contain whole 1928 test images that we can see in test.csv.
", "votes": 1, "replies": [] }, { "id": 571082, "author_name": "Welf Crozzo", "author_url": "", "post_date": "2019-07-09T06:23:25.180000", "content": "Ok, if it's true, then we have 300/13000=3/130~1/40 -> 2.5%. Are you really believe that 2.5% of final test dataset can influence on your choice better decision?:)
", "votes": 0, "replies": [] }, { "id": 572069, "author_name": "å“ˆå°”çš„ç§»åŠ¨åŸŽå ¡", "author_url": "", "post_date": "2019-07-10T12:10:28.573000", "content": "Do you use external dataset or you just use the official dataset to achieve 7th?
", "votes": 0, "replies": [] }, { "id": 598799, "author_name": "Homoalways", "author_url": "", "post_date": "2019-08-14T03:34:21.933000", "content": "Hi @jionie , by \"cleaned dataset\" did you mean you had also removed images in training set that seem to have unreliable labels or you just removed duplicates?
", "votes": 1, "replies": [] }, { "id": 598800, "author_name": "Homoalways", "author_url": "", "post_date": "2019-08-14T03:38:27.607000", "content": "@miklgr500 Isn't the public lb calculated based on 1928 images(out of 1928 / 15% test images)? They won't use only 300 images for the public lb... Also if the 15% are randomly sampled(instead of not sampling evenly on purpose) from the whole test set, then public lb would be much more reliable than cv.
", "votes": 0, "replies": [] }, { "id": 599436, "author_name": "jionie", "author_url": "", "post_date": "2019-08-15T00:50:13.553000", "content": "I removed all samples based on the public csv, but now I just keep all.
", "votes": 0, "replies": [] } ] }, { "id": 570168, "author_name": "Xuan Cao", "author_url": "", "post_date": "2019-07-08T00:06:07.260000", "content": "Well, I guess this is another evidence suggesting that some hard work needs to be done on data cleaning.
", "votes": 1, "replies": [ { "id": 570350, "author_name": "Rahul", "author_url": "", "post_date": "2019-07-08T07:05:07.760000", "content": "But what if test lables are also mislabled and cleaning the training set would result in poor performance in model?
\n\nThis manual cleaning can also result in overfitting of training set, right?
", "votes": 1, "replies": [] } ] }, { "id": 570098, "author_name": "jionie", "author_url": "", "post_date": "2019-07-07T19:13:32.523000", "content": "Emm, may need to remove those images with different labels in the training set, just view them as unlabeled data.
", "votes": 2, "replies": [ { "id": 570170, "author_name": "Maxwell", "author_url": "", "post_date": "2019-07-08T00:08:26.007000", "content": "Thanks to @manojprabhaakr ,\nhttps://www.kaggle.com/manojprabhaakr/similar-duplicate-images-in-aptos-data\nwe can detect the duplicated images.
\n\nAccording to this kernel, there are 143 duplicated images in train data 😩 .
\nI will exclude these extra images not to overfit when training models.
I believe there may be manual errors leading to this result
", "votes": 0, "replies": [] }, { "id": 571264, "author_name": "Wangchentong", "author_url": "", "post_date": "2019-07-09T12:08:49.540000", "content": "agree with you
", "votes": 0, "replies": [] }, { "id": 571046, "author_name": "Kerry", "author_url": "", "post_date": "2019-07-09T05:25:30.813000", "content": "This is cool. Did you consider not using accuracy at all, but something like precision or recall? f-score (a little of both)?
", "votes": 0, "replies": [ { "id": 571159, "author_name": "Maxwell", "author_url": "", "post_date": "2019-07-09T08:49:24.773000", "content": "@kbridge14 \nActually, I'm using MSE. I have tried to solve this task as a regression problem.
", "votes": 2, "replies": [] }, { "id": 598143, "author_name": "Arjun", "author_url": "", "post_date": "2019-08-13T07:39:13.720000", "content": "@maxwell110 Given you are solving this task as a regression problem, What are your thoughts on Mixup Data augmentation?
", "votes": 0, "replies": [] }, { "id": 598658, "author_name": "Maxwell", "author_url": "", "post_date": "2019-08-13T21:05:14.580000", "content": "@monsterspy \nIMO, Mixup will not work well due to the nonlinearity of diabetic retinopathy grading. I mean, ophthalmologists grade images with some non-linear thresholds.
\nFor example, imagine when you mix grade 1 and 3 images with equivalent probability(= 0.5), the target value would be 2(= 1 * 0.5 + 3 * 0.5). But I suppose ophthalmologists will make a diagnosis as grade 3 or higher. Because they will think the image has a lot of blurred but serious findings.
For more detail about diagnosis criteria, check the following material.
\n(Sorry, I forgot who introduced this in the discussion thread.)\nhttp://eyesteve.com/diabetic-retinopathy-grading/
Thanks @maxwell110 for sharing your thoughts.
", "votes": 1, "replies": [] } ] }, { "id": 570270, "author_name": "aisyah depalan", "author_url": "", "post_date": "2019-07-08T04:28:42.027000", "content": "up
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "569611": "Although I could not yet manged to build a good model to predict the retinopathic severity, guessing my model seems to be more accurate in some cases ( I'm not an ophthalmologist, it's my intuition ;) ). Here is the cases I observed. \n\n**Fundus images**\n\n\n\n \nAfter taking a quick check on the Internet, I found out that the label inaccuracy has been an issue in this field. For example, the label inconsistency among ophthalmologists is introduced in the following movie. \n\nTensorFlow Dev Summit 2017 Retinal Imaging\n[https://youtu.be/oOeZ7IgEN4o?t=156](https://youtu.be/oOeZ7IgEN4o?t=156)\n\nI suppose, to tackle this issue, we must implement a robust loss function considering abnormal losses or other remedies as seen in the previous competition (DRD). \nAny opinions are welcome.\n\n---\nUpdated Jul.10.2019 \n\n**The paper of GoogleAI related to above movie** \nDeep Learning vs. Human Graders for Classifying Severity Levels of Diabetic Retinopathy in a Real-World Nationwide Screening Program\n\nhttps://arxiv.org/abs/1810.08290", "569759": "As Maxwell said, this is the specific slide showing inconsistency of label estimation:\nNote that each row illustrates a single patient, and each column represents a single doctor's severity estimation of that patient case.\n\n", "569636": "I feel the same. \nI found 268 duplicated images in train data, and 64 among them have different labels (e.g. 14e3f84445f7 and f0f89314e860). \nDuplication also occurs between train and test, and I guess some of them have different labels.", "570171": "Updated: \nI published a little bit modified version of below kernel. \nYou can download duplicated list of train data. \n\nhttps://www.kaggle.com/maxwell110/duplicated-list-csv-file\n\n---\n\nTo detect the duplicated images, this awesome kernel will be a help. \nhttps://www.kaggle.com/manojprabhaakr/similar-duplicate-images-in-aptos-data\n\nThanks @manojprabhaakr !", "598054": "@maxwell110, I also see there is label inaccuracy. Below severity stage differences could help. I feel Data Cleaning is must but how would that be accounted on the hidden test dataset?\n\n\n", "570766": "I have used the cleaned dataset to train, better cv : 0.9189->0.9284, but lower lb: 0.782->0.769; maybe those duplicate images could work like regularization, I'm a noob 😯 ", "570168": "Well, I guess this is another evidence suggesting that some hard work needs to be done on data cleaning. ", "570098": "Emm, may need to remove those images with different labels in the training set, just view them as unlabeled data.", "571308": "I believe there may be manual errors leading to this result", "571264": "agree with you", "571046": "This is cool. Did you consider not using accuracy at all, but something like precision or recall? f-score (a little of both)?", "570270": "up" } }