I am collecting some papers on the subject apart from the 6 references. Feel free to add more here starting with:
\nI am collecting some papers on the subject apart from the 6 references. Feel free to add more here starting with:
\nSo the Lupascu et al. paper mentions that an independent human observer scores a kappa of 0.7589 and Staal et al. score 0.7345 (Table 1). It's said that the independent human observer has been trained to detect diabetic retinopathy but there is not much more information in that paper or on the website mentioned (www.isi.uu.nl/Research/Databases/DRIVE/ ). Is this independent human observer assumed to be equivalent to the first observer making the classifications? If so wouldn't that mean a kappa of greater than 0.7589 would essentially just be overfitting the first observer's classifications? Also, I'm not sure I understand the overall goal of the competition then; is it to bridge the gap and to create an algorithm that scores between 0.7345 and 0.7589 kappa or is it something along the lines of a faster algorithm or discovering new features etc.?
\nIt's quite possible I'm misunderstanding the results from the paper so please let me know your opinions.
", "rawMarkdown": "", "votes": null }, { "id": "67492", "postDate": "03/21/2015 04:38:06", "content": "Accuracy aside, there are a lot of ancillary reasons to work on computer-aided diagnosis (these may be obvious to you, but I think they worth stating):
\nGood accuracy is necessary to use the algorithm, but it's more than just pushing the decimal places at the frontier of human diagnostic skill.
", "rawMarkdown": "", "votes": null }, { "id": "67499", "postDate": "03/21/2015 05:39:45", "content": "@J Kolb: The kappa cited in the Lupascu 2010 paper refers to vessel segmentation accuracy, ie rating individual pixels as vessel or non-vessel. It is a long way from there to detecting pathological lesions, and an even longer way to assigning accurate diagnostic ratings to whole images.
\nThere are systems (both experimental and in daily clinical use) which are used for DR screening today; but it is my belief that none of them would score much higher than kappa 0.30 on this dataset. I am fairly sure they are looking at it, but they have not chosen to enter the competition yet, draw your own conclusions :)
\nA system which got everything correct except it classified half of the true 0's as 1's would get a kappa of >0.80 here; so would a system that classified half the true 2's as 0's (ouch?). A practically useful system should (ideally) have kappa that is much higher than that, or a particular shape to the confusion matrix.
\nThe question of inter-human-observer agreement is quite interesting.
", "rawMarkdown": "", "votes": null }, { "id": "67536", "postDate": "03/21/2015 16:32:57", "content": "It would be great if there were some longitudinal data on actual outcomes, to minimize or at least change observer effects.
", "rawMarkdown": "", "votes": null }, { "id": "67604", "postDate": "03/22/2015 11:10:56", "content": "OK, this makes sense to me now. I understand there are many benefits of computer-aided-diagnosis as stated on the introduction to this competition. I, however, read a couple of these papers and falsely thought that based on those kappa values that Staal et al. had essentially achieved the same accuracy as human observers on the type of dataset we are working on. I was then wondering: if they had already achieved that accuracy which part of their algorithm we were attempting to improve? Didn't realize it was only for the vessel segmentation classification. Thanks for the clarification.
", "rawMarkdown": "", "votes": null }, { "id": "71442", "postDate": "04/14/2015 04:05:23", "content": "I found many research papers claiming to have sensitivity and specificity in the range of 90%. But it seems most of them are from some seemingly fake journals. So how can we find some credible papers?
", "rawMarkdown": "", "votes": null }, { "id": "74567", "postDate": "04/24/2015 20:02:11", "content": "Hi guys! Interesting discussion!
\nAcharya et al reports in the paper "Application of Higher Order Spectra for the Identification of Diabetes Retinopathy Stages" an accuracy of 82% on the same classification problem we are dealing with here.
\nWe have tried to replicate his results following his reported method with this competitions' data set, but the results we obtained were disastrous (kappa=0). I know the image set of the kaggle competition are very poor quality, which could explain to certain degree the divergence in results. Nonetheless we were surprised to find such discrepancy.
\nIn our limited experience the H.O.S. method gives no classification discrimination with these images.
", "rawMarkdown": "", "votes": null }, { "id": "74570", "postDate": "04/24/2015 20:15:34", "content": "Without knowing the details, are you sure you don't have a bug? If something works on one dataset but doesn't work as well on another, I tend to blame the algorithm. If it works on one dataset but has no performance on another, I'd go looking for places I messed up.
", "rawMarkdown": "", "votes": null }, { "id": "74587", "postDate": "04/24/2015 22:38:51", "content": "No, can´t be 100% sure we didn´t mess up. We've checked it thoroughly but couldn't find anything wrong. Now we´ve moved on to try other methods, but what you say is certainly reasonable. Maybe I'll give it another look before discarding it, given the time and effort we've put into it.
\nDeciding when to stop betting on a particular line of approach when results are not conforming is always a delicate matter.
\nThank you for your comment!!!
", "rawMarkdown": "", "votes": null }, { "id": "74901", "postDate": "04/26/2015 19:34:05", "content": "with respect to the medical value of the competition I do question whether better performance would be more easily achieved by a better image capture mechanism (eg standardising colour ranges etc) rather than trying sophisticated image processing techniques after the image has been captured poorly.
", "rawMarkdown": "", "votes": null }, { "id": "78130", "postDate": "05/12/2015 21:25:49", "content": "here is a recent paper on vessel segmentation: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7055281 Automated Vessel Segmentation Using Infinite
Perimeter Active Contour Model with Hybrid
Region Information with Application to Retinal
Images
Yitian Zhao, Lavdie Rada, Ke Chen, Simon P Harding, Yalin Zheng, Member, IEEE
@Sean, with respect to the medical value of the competition, I do question whether it would lead to greater benefit to predict actual outcomes (of disease progression, therapy effectiveness, etc), than trying to split hairs on a somewhat arbitrary and seemingly useless classification scheme.
\nFrom https://nei.nih.gov/health/diabetic/retinopathy:
\nHow is diabetic retinopathy treated?
\nDuring the first three stages of diabetic retinopathy, no treatment is needed, unless you have macular edema.
", "rawMarkdown": "", "votes": null }, { "id": "178537", "postDate": "04/28/2017 05:51:48", "content": "Diabetic retinopathy detection dataset\ni wanted to know the total number of images in the dataset of diabetic retinopathy detetction dataset???? kindly reply
", "rawMarkdown": "Diabetic retinopathy detection dataset\ni wanted to know the total number of images in the dataset of diabetic retinopathy detetction dataset???? kindly reply", "votes": null }, { "id": "178578", "postDate": "04/28/2017 08:32:36", "content": "Approximately 80.000
", "rawMarkdown": "Approximately 80.000", "votes": null }, { "id": "688552", "postDate": "12/05/2019 18:20:45", "content": "can someone help me how to unzip this test.zip.000 file
", "rawMarkdown": "can someone help me how to unzip this test.zip.000 file", "votes": null }, { "id": "756766", "postDate": "02/26/2020 03:40:23", "content": "To my understanding, one cannot get HOS from normal images. It requires special equipment...
", "rawMarkdown": "To my understanding, one cannot get HOS from normal images. It requires special equipment...", "votes": null } ], "comments": [ { "id": 66827, "author_name": "deepcnn", "author_url": "", "post_date": "03/17/2015 21:15:29", "content": "", "votes": null, "replies": [] }, { "id": 67491, "author_name": "jessekolb", "author_url": "", "post_date": "03/21/2015 04:04:19", "content": "So the Lupascu et al. paper mentions that an independent human observer scores a kappa of 0.7589 and Staal et al. score 0.7345 (Table 1). It's said that the independent human observer has been trained to detect diabetic retinopathy but there is not much more information in that paper or on the website mentioned (www.isi.uu.nl/Research/Databases/DRIVE/ ). Is this independent human observer assumed to be equivalent to the first observer making the classifications? If so wouldn't that mean a kappa of greater than 0.7589 would essentially just be overfitting the first observer's classifications? Also, I'm not sure I understand the overall goal of the competition then; is it to bridge the gap and to create an algorithm that scores between 0.7345 and 0.7589 kappa or is it something along the lines of a faster algorithm or discovering new features etc.?
\nIt's quite possible I'm misunderstanding the results from the paper so please let me know your opinions.
", "votes": null, "replies": [] }, { "id": 67492, "author_name": "wcukierski", "author_url": "", "post_date": "03/21/2015 04:38:06", "content": "Accuracy aside, there are a lot of ancillary reasons to work on computer-aided diagnosis (these may be obvious to you, but I think they worth stating):
\nGood accuracy is necessary to use the algorithm, but it's more than just pushing the decimal places at the frontier of human diagnostic skill.
", "votes": null, "replies": [] }, { "id": 67499, "author_name": "aizvorski", "author_url": "", "post_date": "03/21/2015 05:39:45", "content": "@J Kolb: The kappa cited in the Lupascu 2010 paper refers to vessel segmentation accuracy, ie rating individual pixels as vessel or non-vessel. It is a long way from there to detecting pathological lesions, and an even longer way to assigning accurate diagnostic ratings to whole images.
\nThere are systems (both experimental and in daily clinical use) which are used for DR screening today; but it is my belief that none of them would score much higher than kappa 0.30 on this dataset. I am fairly sure they are looking at it, but they have not chosen to enter the competition yet, draw your own conclusions :)
\nA system which got everything correct except it classified half of the true 0's as 1's would get a kappa of >0.80 here; so would a system that classified half the true 2's as 0's (ouch?). A practically useful system should (ideally) have kappa that is much higher than that, or a particular shape to the confusion matrix.
\nThe question of inter-human-observer agreement is quite interesting.
", "votes": null, "replies": [] }, { "id": 67536, "author_name": "alexcoventry", "author_url": "", "post_date": "03/21/2015 16:32:57", "content": "It would be great if there were some longitudinal data on actual outcomes, to minimize or at least change observer effects.
", "votes": null, "replies": [] }, { "id": 67604, "author_name": "jessekolb", "author_url": "", "post_date": "03/22/2015 11:10:56", "content": "OK, this makes sense to me now. I understand there are many benefits of computer-aided-diagnosis as stated on the introduction to this competition. I, however, read a couple of these papers and falsely thought that based on those kappa values that Staal et al. had essentially achieved the same accuracy as human observers on the type of dataset we are working on. I was then wondering: if they had already achieved that accuracy which part of their algorithm we were attempting to improve? Didn't realize it was only for the vessel segmentation classification. Thanks for the clarification.
", "votes": null, "replies": [] }, { "id": 71442, "author_name": "akilaw", "author_url": "", "post_date": "04/14/2015 04:05:23", "content": "I found many research papers claiming to have sensitivity and specificity in the range of 90%. But it seems most of them are from some seemingly fake journals. So how can we find some credible papers?
", "votes": null, "replies": [] }, { "id": 74567, "author_name": "damianfondevila", "author_url": "", "post_date": "04/24/2015 20:02:11", "content": "Hi guys! Interesting discussion!
\nAcharya et al reports in the paper "Application of Higher Order Spectra for the Identification of Diabetes Retinopathy Stages" an accuracy of 82% on the same classification problem we are dealing with here.
\nWe have tried to replicate his results following his reported method with this competitions' data set, but the results we obtained were disastrous (kappa=0). I know the image set of the kaggle competition are very poor quality, which could explain to certain degree the divergence in results. Nonetheless we were surprised to find such discrepancy.
\nIn our limited experience the H.O.S. method gives no classification discrimination with these images.
", "votes": null, "replies": [ { "id": 756766, "author_name": "rocreguant", "author_url": "", "post_date": "02/26/2020 03:40:23", "content": "To my understanding, one cannot get HOS from normal images. It requires special equipment...
", "votes": null, "replies": [] } ] }, { "id": 74570, "author_name": "wcukierski", "author_url": "", "post_date": "04/24/2015 20:15:34", "content": "Without knowing the details, are you sure you don't have a bug? If something works on one dataset but doesn't work as well on another, I tend to blame the algorithm. If it works on one dataset but has no performance on another, I'd go looking for places I messed up.
", "votes": null, "replies": [] }, { "id": 74587, "author_name": "damianfondevila", "author_url": "", "post_date": "04/24/2015 22:38:51", "content": "No, can´t be 100% sure we didn´t mess up. We've checked it thoroughly but couldn't find anything wrong. Now we´ve moved on to try other methods, but what you say is certainly reasonable. Maybe I'll give it another look before discarding it, given the time and effort we've put into it.
\nDeciding when to stop betting on a particular line of approach when results are not conforming is always a delicate matter.
\nThank you for your comment!!!
", "votes": null, "replies": [] }, { "id": 74901, "author_name": "seanv507", "author_url": "", "post_date": "04/26/2015 19:34:05", "content": "with respect to the medical value of the competition I do question whether better performance would be more easily achieved by a better image capture mechanism (eg standardising colour ranges etc) rather than trying sophisticated image processing techniques after the image has been captured poorly.
", "votes": null, "replies": [] }, { "id": 78130, "author_name": "siddjain", "author_url": "", "post_date": "05/12/2015 21:25:49", "content": "here is a recent paper on vessel segmentation: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7055281 Automated Vessel Segmentation Using Infinite
Perimeter Active Contour Model with Hybrid
Region Information with Application to Retinal
Images
Yitian Zhao, Lavdie Rada, Ke Chen, Simon P Harding, Yalin Zheng, Member, IEEE
@Sean, with respect to the medical value of the competition, I do question whether it would lead to greater benefit to predict actual outcomes (of disease progression, therapy effectiveness, etc), than trying to split hairs on a somewhat arbitrary and seemingly useless classification scheme.
\nFrom https://nei.nih.gov/health/diabetic/retinopathy:
\nHow is diabetic retinopathy treated?
\nDuring the first three stages of diabetic retinopathy, no treatment is needed, unless you have macular edema.
", "votes": null, "replies": [] }, { "id": 178537, "author_name": "tahira", "author_url": "", "post_date": "04/28/2017 05:51:48", "content": "Diabetic retinopathy detection dataset\ni wanted to know the total number of images in the dataset of diabetic retinopathy detetction dataset???? kindly reply
", "votes": null, "replies": [ { "id": 178578, "author_name": "ioannischatzis", "author_url": "", "post_date": "04/28/2017 08:32:36", "content": "Approximately 80.000
", "votes": null, "replies": [] } ] }, { "id": 688552, "author_name": "", "author_url": "", "post_date": "12/05/2019 18:20:45", "content": "can someone help me how to unzip this test.zip.000 file
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "66679": "", "66827": "", "67491": "", "67492": "", "67499": "", "67536": "", "67604": "", "71442": "", "74567": "", "74570": "", "74587": "", "74901": "", "78130": "", "81881": "", "178537": "Diabetic retinopathy detection dataset\ni wanted to know the total number of images in the dataset of diabetic retinopathy detetction dataset???? kindly reply", "178578": "Approximately 80.000", "688552": "can someone help me how to unzip this test.zip.000 file", "756766": "To my understanding, one cannot get HOS from normal images. It requires special equipment..." }, "source": "meta" }