{ "id": 199606, "title": "What do you think about the quality of the labeling (or how many errors are in labels)?", "url": "/competitions/cassava-leaf-disease-classification/discussion/199606", "author_name": "", "post_date": "2020-11-26T12:17:34.573407600Z", "votes": 45, "comment_count": 21, "views": 0, "content": "

I have started looking at the images and see a lot of examples when cassava with yellow leaves is marked as healthy, even though it seems to have disease.

\n

What is your opinion about the quality of the labels in this dataset?

", "messages": [ { "id": "1091953", "postDate": "11/26/2020 12:17:34", "content": "

I have started looking at the images and see a lot of examples when cassava with yellow leaves is marked as healthy, even though it seems to have disease.

\n

What is your opinion about the quality of the labels in this dataset?

", "rawMarkdown": "I have started looking at the images and see a lot of examples when cassava with yellow leaves is marked as healthy, even though it seems to have disease.\n\nWhat is your opinion about the quality of the labels in this dataset?", "votes": null }, { "id": "1092141", "postDate": "11/26/2020 15:11:07", "content": "

Im not plant expert and I'm not sure, but what I want to ask is, can we change the label by ourself? Is this a illegal operation?

", "rawMarkdown": "Im not plant expert and I'm not sure, but what I want to ask is, can we change the label by ourself? Is this a illegal operation?", "votes": null }, { "id": "1092146", "postDate": "11/26/2020 15:13:36", "content": "

We could change (not sure, but I think we have to share a newly labeled dataset), but the main problem is that test set will have the same errors as train data.

", "rawMarkdown": "We could change (not sure, but I think we have to share a newly labeled dataset), but the main problem is that test set will have the same errors as train data.", "votes": null }, { "id": "1092169", "postDate": "11/26/2020 15:23:33", "content": "

I'm not sure but did you try doing any form of outlier detection for the image set? Maybe those show you these images as results and you can proceed to clean up the respective labels.

", "rawMarkdown": "I'm not sure but did you try doing any form of outlier detection for the image set? Maybe those show you these images as results and you can proceed to clean up the respective labels.", "votes": null }, { "id": "1092193", "postDate": "11/26/2020 15:34:57", "content": "

Other competiton - Plant Pathology 2020 haved same issues.

\n

p.s. noisy labels

", "rawMarkdown": "Other competiton - [Plant Pathology 2020](https://www.kaggle.com/c/plant-pathology-2020-fgvc7) haved same issues. \n\np.s. noisy labels", "votes": null }, { "id": "1092516", "postDate": "11/26/2020 23:15:09", "content": "

I think the main problem here is that this is not \"leaf disease classification\" because almost every image has a lot of leaves. In order for this to be classification problem it needs one leaf per image and annotation which disease is it. In this way there are images with trees. Hey, trees????? Take a look at images, please.

", "rawMarkdown": "I think the main problem here is that this is not \"leaf disease classification\" because almost every image has a lot of leaves. In order for this to be classification problem it needs one leaf per image and annotation which disease is it. In this way there are images with trees. Hey, trees????? Take a look at images, please.", "votes": null }, { "id": "1092525", "postDate": "11/26/2020 23:30:17", "content": "

I think having noise in the train set is fine (there were similar challanges in competitions), but looking at LB, the test set seems to have noise as well(e.g. strong CV corrrlation). That makes fair competition impossible 🤣

", "rawMarkdown": "I think having noise in the train set is fine (there were similar challanges in competitions), but looking at LB, the test set seems to have noise as well(e.g. strong CV corrrlation). That makes fair competition impossible 🤣", "votes": null }, { "id": "1092533", "postDate": "11/27/2020 00:00:33", "content": "

We have seen similar cases like yours. After training the model, we found that it predicts healthy labelled ~500 images as diseased. We checked out the most of the images and they were containing yellow or dotted leaves like explained in this discussion.

\n

We are not sure about how the relabelling will effect the results since the private test set might contain noisy data as well. We will probably burn some submissions on that to understand how the LB will reflect on those changes.

", "rawMarkdown": "We have seen similar cases like yours. After training the model, we found that it predicts healthy labelled ~500 images as diseased. We checked out the most of the images and they were containing yellow or dotted leaves like explained in [this](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/198143) discussion.\n\nWe are not sure about how the relabelling will effect the results since the private test set might contain noisy data as well. We will probably burn some submissions on that to understand how the LB will reflect on those changes.", "votes": null }, { "id": "1093142", "postDate": "11/27/2020 13:34:42", "content": "

No matter how many labels we changed during the train phase.it will be tested with incorrectly labeled data when submitting for prediction?

", "rawMarkdown": "No matter how many labels we changed during the train phase.it will be tested with incorrectly labeled data when submitting for prediction?", "votes": null }, { "id": "1094087", "postDate": "11/28/2020 10:01:14", "content": "

This doesn't really look like a healthy plant to me, but it is labeled with 4 (healthy class).
\n\"\"

", "rawMarkdown": "This doesn't really look like a healthy plant to me, but it is labeled with 4 (healthy class).\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1696976%2F36c685c3f5ed482a82159032ef5c90c6%2F.png?generation=1606557578656671&alt=media)", "votes": null }, { "id": "1094423", "postDate": "11/28/2020 15:52:31", "content": "

Just because leaves are stained, broken etc doesn't mean it has disease. It could be just rotting. Just like banana can have stains, rotting or non edible etc and still not have disease. Correct?

", "rawMarkdown": "Just because leaves are stained, broken etc doesn't mean it has disease. It could be just rotting. Just like banana can have stains, rotting or non edible etc and still not have disease. Correct?", "votes": null }, { "id": "1094583", "postDate": "11/28/2020 18:33:10", "content": "

Are you saying rotting is a healthy condition?

", "rawMarkdown": "Are you saying rotting is a healthy condition?", "votes": null }, { "id": "1094658", "postDate": "11/28/2020 19:37:59", "content": "

Well. Rotting is not edible or beneficial, but is not a disease. Competition is to find the disease.

", "rawMarkdown": "Well. Rotting is not edible or beneficial, but is not a disease. Competition is to find the disease.", "votes": null }, { "id": "1094991", "postDate": "11/29/2020 06:54:49", "content": "

here are some ground truth which i suspect to be wrong (I think there is quite a lot of them)

\n

\"\"

\n

i am less worried about the train samples, because we can actually filter the noisy samples away.

\n
\n

i wonder if the hidden test samples are cleanly labeled? or are they as noisy?

", "rawMarkdown": "here are some ground truth which i suspect to be wrong (I think there is quite a lot of them)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F30fe231f433d4997ec0fa43e16b0818f%2FSelection_090.png?generation=1606632813567504&alt=media)\n\ni am less worried about the train samples, because we can actually filter the noisy samples away.\n\n---\n**i wonder if the hidden test samples are cleanly labeled? or are they as noisy?**", "votes": null }, { "id": "1095023", "postDate": "11/29/2020 07:34:34", "content": "

\"\"

\n

red: what i think should be the correct label

", "rawMarkdown": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F8bbfe890497bb0f8fd56a56490bd8dff%2FSelection_094.png?generation=1606635238499943&alt=media)\n\nred: what i think should be the correct label", "votes": null }, { "id": "1095190", "postDate": "11/29/2020 10:41:16", "content": "

How do you solve it? Lable Smoothing?

", "rawMarkdown": "How do you solve it? Lable Smoothing?", "votes": null }, { "id": "1095333", "postDate": "11/29/2020 13:54:09", "content": "

well there is nothing to be done if the test labels are corrupted..

", "rawMarkdown": "well there is nothing to be done if the test labels are corrupted..", "votes": null }, { "id": "1095422", "postDate": "11/29/2020 15:32:36", "content": "

I think @kyoshioka47 is right, there are two scenarios:
\n-The test set has similar label noise and we keep the noise in the training
\n-The test set doesnt have this label noise and we have to find a way to solve the label noise in the training set

", "rawMarkdown": "I think @kyoshioka47 is right, there are two scenarios: \n-The test set has similar label noise and we keep the noise in the training\n-The test set doesnt have this label noise and we have to find a way to solve the label noise in the training set", "votes": null }, { "id": "1095802", "postDate": "11/30/2020 01:07:23", "content": "

@yannmajewski

\n
\n

The test set doesnt have this label noise and we have to find a way to solve the label noise in the training set

\n
\n

I tried label denoising with the similar method in the 1st place solution of the PANDA challange. In PANDA, the trainsets were noisy but the test sets were insured that the label noise is low.
\nIn the cassava case, even with denoising, the LB did not improve at all. Meaning..?

", "rawMarkdown": "yannmajewski \n> The test set doesnt have this label noise and we have to find a way to solve the label noise in the training set\n\nI tried label denoising with the similar method in the [1st place solution of the PANDA challange](https://www.kaggle.com/c/prostate-cancer-grade-assessment/discussion/169143). In PANDA, the trainsets were noisy but the test sets were insured that the label noise is low.\nIn the cassava case, even with denoising, the LB did not improve at all. Meaning..?", "votes": null }, { "id": "1095922", "postDate": "11/30/2020 04:27:16", "content": "

@kyoshioka47 No way we had the same idea haha! I did the same and i think we have the same conclusion. There is noise in the test set because I had 90.8 CV 1 fold and got 89.3 LB. CV and LB didnt correlate as much

", "rawMarkdown": "kyoshioka47 No way we had the same idea haha! I did the same and i think we have the same conclusion. There is noise in the test set because I had 90.8 CV 1 fold and got 89.3 LB. CV and LB didnt correlate as much", "votes": null }, { "id": "1096824", "postDate": "11/30/2020 19:59:58", "content": "

there is a \"stupid\" way to test or probe:

\n
    \n
  1. submit all zero
    2. \n
  2. make prediction with your model, for the most confidence \"0-class\", submit as zero
    3. \n
  3. same as 2,but for the most confidence \"0-class\", submit as one
    4. \n
  4. same as 2,but for the most confidence \"0-class\", submit as two
    5. \n
  5. same as 2,but for the most confidence \"0-class\", submit as three …
    6. \n
\n

this is assumed that your model is trained on clean labels and detect only clean test samples.
\nby doing sub probing, you can reverse engineer a bit and measure the confusion matrix of the LB test samples

", "rawMarkdown": "there is a \"stupid\" way to test or probe:\n1. submit all zero \n2. make prediction with your model, for the most confidence \"0-class\", submit as zero\n3. same as 2,but for the most confidence \"0-class\", submit as one\n4. same as 2,but for the most confidence \"0-class\", submit as two\n5. same as 2,but for the most confidence \"0-class\", submit as three ...\n\nthis is assumed that your model is trained on clean labels and detect only clean test samples.\nby doing sub probing, you can reverse engineer a bit and measure the confusion matrix of the LB test samples", "votes": null }, { "id": "1100337", "postDate": "12/03/2020 02:26:15", "content": "

The labels are surely noisy. However, I think part of the unhealthy looking samples in the healthy category do not have the diseases we are given in the label set. For instance the yellow leaves shown in this notebook can be just due to poor drainage.

\n

Edit: Furthermore, CV and LB values reported in the discussions are very close to each other. I don't think the noise in the labels is a big matter.

", "rawMarkdown": "The labels are surely noisy. However, I think part of the unhealthy looking samples in the healthy category do not have the diseases we are given in the label set. For instance the yellow leaves shown in [this notebook](https://www.kaggle.com/tanulsingh077/how-to-become-leaf-doctor-with-deep-learning/notebook) can be just due to poor drainage.\n\nEdit: Furthermore, CV and LB values reported in the discussions are very close to each other. I don't think the noise in the labels is a big matter.", "votes": null } ], "comments": [ { "id": 1092141, "author_name": "frkaka", "author_url": "", "post_date": "11/26/2020 15:11:07", "content": "

Im not plant expert and I'm not sure, but what I want to ask is, can we change the label by ourself? Is this a illegal operation?

", "votes": null, "replies": [ { "id": 1092146, "author_name": "artgor", "author_url": "", "post_date": "11/26/2020 15:13:36", "content": "

We could change (not sure, but I think we have to share a newly labeled dataset), but the main problem is that test set will have the same errors as train data.

", "votes": null, "replies": [] } ] }, { "id": 1092169, "author_name": "namansood", "author_url": "", "post_date": "11/26/2020 15:23:33", "content": "

I'm not sure but did you try doing any form of outlier detection for the image set? Maybe those show you these images as results and you can proceed to clean up the respective labels.

", "votes": null, "replies": [] }, { "id": 1092193, "author_name": "piantic", "author_url": "", "post_date": "11/26/2020 15:34:57", "content": "

Other competiton - Plant Pathology 2020 haved same issues.

\n

p.s. noisy labels

", "votes": null, "replies": [] }, { "id": 1092516, "author_name": "alem88", "author_url": "", "post_date": "11/26/2020 23:15:09", "content": "

I think the main problem here is that this is not \"leaf disease classification\" because almost every image has a lot of leaves. In order for this to be classification problem it needs one leaf per image and annotation which disease is it. In this way there are images with trees. Hey, trees????? Take a look at images, please.

", "votes": null, "replies": [] }, { "id": 1092525, "author_name": "kyoshioka47", "author_url": "", "post_date": "11/26/2020 23:30:17", "content": "

I think having noise in the train set is fine (there were similar challanges in competitions), but looking at LB, the test set seems to have noise as well(e.g. strong CV corrrlation). That makes fair competition impossible 🤣

", "votes": null, "replies": [] }, { "id": 1092533, "author_name": "snnclsr", "author_url": "", "post_date": "11/27/2020 00:00:33", "content": "

We have seen similar cases like yours. After training the model, we found that it predicts healthy labelled ~500 images as diseased. We checked out the most of the images and they were containing yellow or dotted leaves like explained in this discussion.

\n

We are not sure about how the relabelling will effect the results since the private test set might contain noisy data as well. We will probably burn some submissions on that to understand how the LB will reflect on those changes.

", "votes": null, "replies": [] }, { "id": 1093142, "author_name": "emreulgac", "author_url": "", "post_date": "11/27/2020 13:34:42", "content": "

No matter how many labels we changed during the train phase.it will be tested with incorrectly labeled data when submitting for prediction?

", "votes": null, "replies": [] }, { "id": 1094087, "author_name": "nroman", "author_url": "", "post_date": "11/28/2020 10:01:14", "content": "

This doesn't really look like a healthy plant to me, but it is labeled with 4 (healthy class).
\n\"\"

", "votes": null, "replies": [ { "id": 1094423, "author_name": "roadrunner0", "author_url": "", "post_date": "11/28/2020 15:52:31", "content": "

Just because leaves are stained, broken etc doesn't mean it has disease. It could be just rotting. Just like banana can have stains, rotting or non edible etc and still not have disease. Correct?

", "votes": null, "replies": [] }, { "id": 1094583, "author_name": "nroman", "author_url": "", "post_date": "11/28/2020 18:33:10", "content": "

Are you saying rotting is a healthy condition?

", "votes": null, "replies": [] }, { "id": 1094658, "author_name": "roadrunner0", "author_url": "", "post_date": "11/28/2020 19:37:59", "content": "

Well. Rotting is not edible or beneficial, but is not a disease. Competition is to find the disease.

", "votes": null, "replies": [] } ] }, { "id": 1094991, "author_name": "hengck23", "author_url": "", "post_date": "11/29/2020 06:54:49", "content": "

here are some ground truth which i suspect to be wrong (I think there is quite a lot of them)

\n

\"\"

\n

i am less worried about the train samples, because we can actually filter the noisy samples away.

\n
\n

i wonder if the hidden test samples are cleanly labeled? or are they as noisy?

", "votes": null, "replies": [ { "id": 1095023, "author_name": "hengck23", "author_url": "", "post_date": "11/29/2020 07:34:34", "content": "

\"\"

\n

red: what i think should be the correct label

", "votes": null, "replies": [] }, { "id": 1095190, "author_name": "chenbaoying", "author_url": "", "post_date": "11/29/2020 10:41:16", "content": "

How do you solve it? Lable Smoothing?

", "votes": null, "replies": [] }, { "id": 1095333, "author_name": "kyoshioka47", "author_url": "", "post_date": "11/29/2020 13:54:09", "content": "

well there is nothing to be done if the test labels are corrupted..

", "votes": null, "replies": [] }, { "id": 1095422, "author_name": "yannmajewski", "author_url": "", "post_date": "11/29/2020 15:32:36", "content": "

I think @kyoshioka47 is right, there are two scenarios:
\n-The test set has similar label noise and we keep the noise in the training
\n-The test set doesnt have this label noise and we have to find a way to solve the label noise in the training set

", "votes": null, "replies": [] }, { "id": 1095802, "author_name": "kyoshioka47", "author_url": "", "post_date": "11/30/2020 01:07:23", "content": "

@yannmajewski

\n
\n

The test set doesnt have this label noise and we have to find a way to solve the label noise in the training set

\n
\n

I tried label denoising with the similar method in the 1st place solution of the PANDA challange. In PANDA, the trainsets were noisy but the test sets were insured that the label noise is low.
\nIn the cassava case, even with denoising, the LB did not improve at all. Meaning..?

", "votes": null, "replies": [] }, { "id": 1095922, "author_name": "yannmajewski", "author_url": "", "post_date": "11/30/2020 04:27:16", "content": "

@kyoshioka47 No way we had the same idea haha! I did the same and i think we have the same conclusion. There is noise in the test set because I had 90.8 CV 1 fold and got 89.3 LB. CV and LB didnt correlate as much

", "votes": null, "replies": [] }, { "id": 1096824, "author_name": "hengck23", "author_url": "", "post_date": "11/30/2020 19:59:58", "content": "

there is a \"stupid\" way to test or probe:

\n
    \n
  1. submit all zero
    2. \n
  2. make prediction with your model, for the most confidence \"0-class\", submit as zero
    3. \n
  3. same as 2,but for the most confidence \"0-class\", submit as one
    4. \n
  4. same as 2,but for the most confidence \"0-class\", submit as two
    5. \n
  5. same as 2,but for the most confidence \"0-class\", submit as three …
    6. \n
\n

this is assumed that your model is trained on clean labels and detect only clean test samples.
\nby doing sub probing, you can reverse engineer a bit and measure the confusion matrix of the LB test samples

", "votes": null, "replies": [] } ] }, { "id": 1100337, "author_name": "tolgadincer", "author_url": "", "post_date": "12/03/2020 02:26:15", "content": "

The labels are surely noisy. However, I think part of the unhealthy looking samples in the healthy category do not have the diseases we are given in the label set. For instance the yellow leaves shown in this notebook can be just due to poor drainage.

\n

Edit: Furthermore, CV and LB values reported in the discussions are very close to each other. I don't think the noise in the labels is a big matter.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1091953": "I have started looking at the images and see a lot of examples when cassava with yellow leaves is marked as healthy, even though it seems to have disease.\n\nWhat is your opinion about the quality of the labels in this dataset?", "1092141": "Im not plant expert and I'm not sure, but what I want to ask is, can we change the label by ourself? Is this a illegal operation?", "1092146": "We could change (not sure, but I think we have to share a newly labeled dataset), but the main problem is that test set will have the same errors as train data.", "1092169": "I'm not sure but did you try doing any form of outlier detection for the image set? Maybe those show you these images as results and you can proceed to clean up the respective labels.", "1092193": "Other competiton - [Plant Pathology 2020](https://www.kaggle.com/c/plant-pathology-2020-fgvc7) haved same issues. \n\np.s. noisy labels", "1092516": "I think the main problem here is that this is not \"leaf disease classification\" because almost every image has a lot of leaves. In order for this to be classification problem it needs one leaf per image and annotation which disease is it. In this way there are images with trees. Hey, trees????? Take a look at images, please.", "1092525": "I think having noise in the train set is fine (there were similar challanges in competitions), but looking at LB, the test set seems to have noise as well(e.g. strong CV corrrlation). That makes fair competition impossible 🤣", "1092533": "We have seen similar cases like yours. After training the model, we found that it predicts healthy labelled ~500 images as diseased. We checked out the most of the images and they were containing yellow or dotted leaves like explained in [this](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/198143) discussion.\n\nWe are not sure about how the relabelling will effect the results since the private test set might contain noisy data as well. We will probably burn some submissions on that to understand how the LB will reflect on those changes.", "1093142": "No matter how many labels we changed during the train phase.it will be tested with incorrectly labeled data when submitting for prediction?", "1094087": "This doesn't really look like a healthy plant to me, but it is labeled with 4 (healthy class).\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1696976%2F36c685c3f5ed482a82159032ef5c90c6%2F.png?generation=1606557578656671&alt=media)", "1094423": "Just because leaves are stained, broken etc doesn't mean it has disease. It could be just rotting. Just like banana can have stains, rotting or non edible etc and still not have disease. Correct?", "1094583": "Are you saying rotting is a healthy condition?", "1094658": "Well. Rotting is not edible or beneficial, but is not a disease. Competition is to find the disease.", "1094991": "here are some ground truth which i suspect to be wrong (I think there is quite a lot of them)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F30fe231f433d4997ec0fa43e16b0818f%2FSelection_090.png?generation=1606632813567504&alt=media)\n\ni am less worried about the train samples, because we can actually filter the noisy samples away.\n\n---\n**i wonder if the hidden test samples are cleanly labeled? or are they as noisy?**", "1095023": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F8bbfe890497bb0f8fd56a56490bd8dff%2FSelection_094.png?generation=1606635238499943&alt=media)\n\nred: what i think should be the correct label", "1095190": "How do you solve it? Lable Smoothing?", "1095333": "well there is nothing to be done if the test labels are corrupted..", "1095422": "I think @kyoshioka47 is right, there are two scenarios: \n-The test set has similar label noise and we keep the noise in the training\n-The test set doesnt have this label noise and we have to find a way to solve the label noise in the training set", "1095802": "yannmajewski \n> The test set doesnt have this label noise and we have to find a way to solve the label noise in the training set\n\nI tried label denoising with the similar method in the [1st place solution of the PANDA challange](https://www.kaggle.com/c/prostate-cancer-grade-assessment/discussion/169143). In PANDA, the trainsets were noisy but the test sets were insured that the label noise is low.\nIn the cassava case, even with denoising, the LB did not improve at all. Meaning..?", "1095922": "kyoshioka47 No way we had the same idea haha! I did the same and i think we have the same conclusion. There is noise in the test set because I had 90.8 CV 1 fold and got 89.3 LB. CV and LB didnt correlate as much", "1096824": "there is a \"stupid\" way to test or probe:\n1. submit all zero \n2. make prediction with your model, for the most confidence \"0-class\", submit as zero\n3. same as 2,but for the most confidence \"0-class\", submit as one\n4. same as 2,but for the most confidence \"0-class\", submit as two\n5. same as 2,but for the most confidence \"0-class\", submit as three ...\n\nthis is assumed that your model is trained on clean labels and detect only clean test samples.\nby doing sub probing, you can reverse engineer a bit and measure the confusion matrix of the LB test samples", "1100337": "The labels are surely noisy. However, I think part of the unhealthy looking samples in the healthy category do not have the diseases we are given in the label set. For instance the yellow leaves shown in [this notebook](https://www.kaggle.com/tanulsingh077/how-to-become-leaf-doctor-with-deep-learning/notebook) can be just due to poor drainage.\n\nEdit: Furthermore, CV and LB values reported in the discussions are very close to each other. I don't think the noise in the labels is a big matter." }, "source": "meta" }