\nHere are some examples. First one is labeled as cgm which stand for Cassava Green Mottle however there are so many leaves on the image that I see some healthy ones even though label says cgm.
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1696976%2F39cd9b1d6a8ddfe40d434bced2fddbe9%2Fmultiple_1.png?generation=1607676555901854&alt=media\")
There are already multiple discussion threads about noisy labels in this competitions dataset: one, two, three.
\nBut what I want to share is samples which are mispredicted by my model that scored 0.902 in LB.
I have devided those noisy samples into four categories: multilabeled, confusing, questionable and other.
\nEach image that will be shown in this thread has 3 lines on the title: image id, true label and probabilities of my model prediction.
\nThose are the images that contain samples with different labels whilst we are to predict only one label per image.
\nHere are some examples. First one is labeled as cgm which stand for Cassava Green Mottle however there are so many leaves on the image that I see some healthy ones even though label says cgm.
\n
And another two examples. By the way when I first took a look at the first image below I though that this is a cbb (Cassava Bacterial Blight) sample and, as you can see, my model agrees with me. But the label says that this is an example of Cassava Brown Streak Disease.
\nAnyway - there are also a plenty of healthy leaves on the image.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1696976%2F308aa7468761e60a7dca934eebe05fd3%2Fmultiple_2.png?generation=1607677013241260&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1696976%2F6443781a4588e5ca1c5674255673dbc2%2Fmultiple_3.png?generation=1607677024196284&alt=media\")
This kind of noise would only teach your model a \"bad things\". Lets take a look at the example below.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1696976%2F9b9c37f6b9598da0792e196f01a57c48%2Fconfusing_1.png?generation=1607677296154632&alt=media\")
\nLeaves that are closer to the camera are definitely healthy. But the label is cbsd, which is, probably, about that guy in the background. This image would only \"confuse\" you model, showing it a good sample of healthy leaves but saying that they are not healthy.
I am no expert in the field by any means but during my analysis of the dataset I have seend tens of examples of Bacterial Blight and the following image, in my opition, demonstrates exactly this kind of disease (and my model agrees with me). But somehow this is labeled as healthy, which is questionable.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1696976%2F4b9636c1785b212f76e480ae0310d110%2Fquestion_1.png?generation=1607677632349805&alt=media\")
Finally there are some images that I don't even know how to categorise because they don't have much to do with leaves disease classification.
\nThe label for the next image is \"healthy\". Well, I am really glad that this guy is ok. But how about the plant?
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1696976%2Fe69d3a97f4e2d033fc02f3a698138bdf%2Fother_1.png?generation=1607677788211732&alt=media\")
There are already multiple discussion threads about noisy labels in this competitions dataset: one, two, three.
\nBut what I want to share is samples which are mispredicted by my model that scored 0.902 in LB.
I have devided those noisy samples into four categories: multilabeled, confusing, questionable and other.
\nEach image that will be shown in this thread has 3 lines on the title: image id, true label and probabilities of my model prediction.
\nThose are the images that contain samples with different labels whilst we are to predict only one label per image.
\nHere are some examples. First one is labeled as cgm which stand for Cassava Green Mottle however there are so many leaves on the image that I see some healthy ones even though label says cgm.
\n
And another two examples. By the way when I first took a look at the first image below I though that this is a cbb (Cassava Bacterial Blight) sample and, as you can see, my model agrees with me. But the label says that this is an example of Cassava Brown Streak Disease.
\nAnyway - there are also a plenty of healthy leaves on the image.
\n
This kind of noise would only teach your model a \"bad things\". Lets take a look at the example below.
\n
\nLeaves that are closer to the camera are definitely healthy. But the label is cbsd, which is, probably, about that guy in the background. This image would only \"confuse\" you model, showing it a good sample of healthy leaves but saying that they are not healthy.
I am no expert in the field by any means but during my analysis of the dataset I have seend tens of examples of Bacterial Blight and the following image, in my opition, demonstrates exactly this kind of disease (and my model agrees with me). But somehow this is labeled as healthy, which is questionable.
\n
Finally there are some images that I don't even know how to categorise because they don't have much to do with leaves disease classification.
\nThe label for the next image is \"healthy\". Well, I am really glad that this guy is ok. But how about the plant?
\n
Interesting findings. Thanks to a healthy guy I now know how the Cassava roots look like!
\nI guess handling label noise will become an important element of the pipeline. Of course we can expect similar noise in the test set, but that does not mean correcting it on the training stage is not needed.
", "rawMarkdown": "Interesting findings. Thanks to a healthy guy I now know how the Cassava roots look like!\n\nI guess handling label noise will become an important element of the pipeline. Of course we can expect similar noise in the test set, but that does not mean correcting it on the training stage is not needed.", "votes": null }, { "id": "1109226", "postDate": "12/11/2020 12:57:09", "content": "Not all image are leaf image, some are roots image and some are stem image. which explain the photo with the guy holding the roots of the plant
", "rawMarkdown": "Not all image are leaf image, some are roots image and some are stem image. which explain the photo with the guy holding the roots of the plant", "votes": null }, { "id": "1109294", "postDate": "12/11/2020 14:10:04", "content": "Those are all valid points @nroman , I wonder how are you dealing or plan to deal with that noise? I have done only a few experiments on that, but got nothing useful yet.
", "rawMarkdown": "Those are all valid points @nroman , I wonder how are you dealing or plan to deal with that noise? I have done only a few experiments on that, but got nothing useful yet.", "votes": null }, { "id": "1109348", "postDate": "12/11/2020 15:09:59", "content": "from the application point of view, the framer should have placed the object of interest in the center of the image.
\nif you only take center crop or put a center focused attention mask, maybe the results will be better?
https://ipsj.ixsq.nii.ac.jp/ej/?action=pages_view_main&active_action=repository_view_main_item_detail&item_id=205376&item_no=1&page_id=13&block_id=8
\n中央領域に注目する Center Attention による頑健性の高い植物病害診断装置の構築
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fef908e44aa455ed0e41f0bc8505be269%2FSelection_195.png?generation=1607699600870085&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F65535d8b8bf637932963bb15383304a9%2FSelection_193.png?generation=1607699627776548&alt=media\")
Currently, I am also thinking about the above problem.
\nIf we look for some improvements related to this, both cv and lb are likely to improve significantly.
p.s. The dataset has many noisy, but cv and lb are more stable than we think.
", "rawMarkdown": "Currently, I am also thinking about the above problem.\nIf we look for some improvements related to this, both cv and lb are likely to improve significantly.\n\np.s. The dataset has many noisy, but cv and lb are more stable than we think.", "votes": null }, { "id": "1109397", "postDate": "12/11/2020 16:18:35", "content": "I would add this discussion giving a list of mislabeled images :
\n['2782668721.jpg','3238704279.jpg','1365612235.jpg','1649500149.jpg','1236952675.jpg','3085440105.jpg',\n '2929245875.jpg','2509491848.jpg','1227531167.jpg']
\nHere is a look on the 5 first :
![\"2782668721.jpg'\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5972996%2F9f47fe8edc169610ef873295eb06338f%2F0.png?generation=1607696456508076&alt=media\")
![\"3238704279.jpg\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5972996%2F0874283c9fb4ad1b4b48c2cd423bd946%2F1.png?generation=1607696499134585&alt=media\")
![\"1365612235.jpg\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5972996%2Fef00d17ce427531854ecb0c4e0a8d9ee%2F2.png?generation=1607696506680598&alt=media\")
![\"1649500149.jpg\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5972996%2F9b9f49a4a360dd245a9dc95bf49b39c9%2F3.png?generation=1607696519232780&alt=media\")
![\"1236952675.jpg\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5972996%2F11692b00ea67de08391bdb98c2904d57%2F4.png?generation=1607696525487998&alt=media\")
I can add some more :
\n1300599354.jpg, labeled 4, what we see : 0 ? (not sure here) : ![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5972996%2F439419666dccd2e6d8a1bd446156696c%2Flabeled4_but_0_0.png?generation=1607700066550324&alt=media\")
2715221153.jpg, labeled 4, what we see : 2 or 3 : ![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5972996%2Fe697c92896fba194fe6cfe56bc2bdd2a%2Flabeled4_but_3_1.png?generation=1607700132435798&alt=media\")
also : there are confusions on 2-3 labels as they show very similar features. Some images are labeled as 2 but look like 3 :
\n1492444202.jpg
\n1456881000.jpg
\n1654084150.jpg (see image below)
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5972996%2F6f15190bc00a2c8ea540057d5792a120%2Flabel2_but3_2.png?generation=1607703243704012&alt=media\")
Relabeling all of those images by hand is fastidious and would take too much time and effort, nevertheless training a model on good labels is the most important, even if the test set is also noisy.
", "rawMarkdown": "I would add [this discussion](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/200901) giving a list of mislabeled images : \n```['2782668721.jpg','3238704279.jpg','1365612235.jpg','1649500149.jpg','1236952675.jpg','3085440105.jpg',\n '2929245875.jpg','2509491848.jpg','1227531167.jpg']```\nHere is a look on the 5 first : \n\n\n\n\n\n\n\n\n\n\n\n\n0. 3238704279.jpg labeled : 4, What we see : 3\n1. 2782668721.jpg, labeled : 4, what we see : 3\n2. 1365612235.jpg, labeled : 4, what we see : 3\n3. 1649500149.jpg, labeled : 4, what we see : 3\n4. 1236952675.jpg, labeled : 4, what we see : 3\n5. 3085440105.jpg, labeled : 4, what we see : 3\n6. 2929245875.jpg, labeled : 4 , what we see : 3 (and a foot)\n7. 2509491848.jpg, labeled : 4, what we see : 3\n8. 1227531167.jpg, labeled : 4, what we see : 3 or 2\n\nI can add some more : \n\n9. 1300599354.jpg, labeled 4, what we see : 0 ? (not sure here) : \n\n10. 2715221153.jpg, labeled 4, what we see : 2 or 3 : \n\n\nalso : there are confusions on 2-3 labels as they show very similar features. Some images are labeled as 2 but look like 3 :\n1492444202.jpg\n1456881000.jpg\n1654084150.jpg (see image below)\n\n\n\n\nRelabeling all of those images by hand is fastidious and would take too much time and effort, nevertheless training a model on good labels is the most important, even if the test set is also noisy.", "votes": null }, { "id": "1109917", "postDate": "12/12/2020 08:02:04", "content": "Perhaps the reason why they have asked us to clean this mess first and then go for the training!!😅
", "rawMarkdown": "Perhaps the reason why they have asked us to clean this mess first and then go for the training!!😅", "votes": null }, { "id": "1110652", "postDate": "12/12/2020 23:34:56", "content": "Thanks for bring this topic up, I am thinking to drop those images in training. It should cause less confusions to the model
", "rawMarkdown": "Thanks for bring this topic up, I am thinking to drop those images in training. It should cause less confusions to the model", "votes": null }, { "id": "1112114", "postDate": "12/14/2020 09:41:01", "content": "I guess one way will be using a semi supervised learning approach (or like Snorkel to write some labeling function) to quickly relabel those mislabeled images. Perhaps, we can have our first model to detected samples that model prediction strongly disagree with label. Based on the threshold, we relabel those examples. Then we can train the second model on these corrected labels.
", "rawMarkdown": "I guess one way will be using a semi supervised learning approach (or like Snorkel to write some labeling function) to quickly relabel those mislabeled images. Perhaps, we can have our first model to detected samples that model prediction strongly disagree with label. Based on the threshold, we relabel those examples. Then we can train the second model on these corrected labels.", "votes": null } ], "comments": [ { "id": 1109100, "author_name": "kozodoi", "author_url": "", "post_date": "12/11/2020 10:27:39", "content": "Interesting findings. Thanks to a healthy guy I now know how the Cassava roots look like!
\nI guess handling label noise will become an important element of the pipeline. Of course we can expect similar noise in the test set, but that does not mean correcting it on the training stage is not needed.
", "votes": null, "replies": [] }, { "id": 1109226, "author_name": "thomasgaltier2", "author_url": "", "post_date": "12/11/2020 12:57:09", "content": "Not all image are leaf image, some are roots image and some are stem image. which explain the photo with the guy holding the roots of the plant
", "votes": null, "replies": [] }, { "id": 1109294, "author_name": "dimitreoliveira", "author_url": "", "post_date": "12/11/2020 14:10:04", "content": "Those are all valid points @nroman , I wonder how are you dealing or plan to deal with that noise? I have done only a few experiments on that, but got nothing useful yet.
", "votes": null, "replies": [] }, { "id": 1109348, "author_name": "hengck23", "author_url": "", "post_date": "12/11/2020 15:09:59", "content": "from the application point of view, the framer should have placed the object of interest in the center of the image.
\nif you only take center crop or put a center focused attention mask, maybe the results will be better?
https://ipsj.ixsq.nii.ac.jp/ej/?action=pages_view_main&active_action=repository_view_main_item_detail&item_id=205376&item_no=1&page_id=13&block_id=8
\n中央領域に注目する Center Attention による頑健性の高い植物病害診断装置の構築
Perhaps the reason why they have asked us to clean this mess first and then go for the training!!😅
", "votes": null, "replies": [] } ] }, { "id": 1109351, "author_name": "piantic", "author_url": "", "post_date": "12/11/2020 15:18:55", "content": "Currently, I am also thinking about the above problem.
\nIf we look for some improvements related to this, both cv and lb are likely to improve significantly.
p.s. The dataset has many noisy, but cv and lb are more stable than we think.
", "votes": null, "replies": [] }, { "id": 1109397, "author_name": "lizardkingdom", "author_url": "", "post_date": "12/11/2020 16:18:35", "content": "I would add this discussion giving a list of mislabeled images :
\n['2782668721.jpg','3238704279.jpg','1365612235.jpg','1649500149.jpg','1236952675.jpg','3085440105.jpg',\n '2929245875.jpg','2509491848.jpg','1227531167.jpg']
\nHere is a look on the 5 first :
I can add some more :
\n1300599354.jpg, labeled 4, what we see : 0 ? (not sure here) :
2715221153.jpg, labeled 4, what we see : 2 or 3 :
also : there are confusions on 2-3 labels as they show very similar features. Some images are labeled as 2 but look like 3 :
\n1492444202.jpg
\n1456881000.jpg
\n1654084150.jpg (see image below)
Relabeling all of those images by hand is fastidious and would take too much time and effort, nevertheless training a model on good labels is the most important, even if the test set is also noisy.
", "votes": null, "replies": [] }, { "id": 1110652, "author_name": "luqing2", "author_url": "", "post_date": "12/12/2020 23:34:56", "content": "Thanks for bring this topic up, I am thinking to drop those images in training. It should cause less confusions to the model
", "votes": null, "replies": [] }, { "id": 1112114, "author_name": "louis925", "author_url": "", "post_date": "12/14/2020 09:41:01", "content": "I guess one way will be using a semi supervised learning approach (or like Snorkel to write some labeling function) to quickly relabel those mislabeled images. Perhaps, we can have our first model to detected samples that model prediction strongly disagree with label. Based on the threshold, we relabel those examples. Then we can train the second model on these corrected labels.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1109079": "There are already multiple discussion threads about noisy labels in this competitions dataset: [one](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/199606), [two](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/201471), [three](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/202206).\nBut what I want to share is samples which are mispredicted by my model that scored 0.902 in LB.\n\nI have devided those noisy samples into four categories: **multilabeled**, **confusing**, **questionable** and **other**.\n\nEach image that will be shown in this thread has 3 lines on the title: image id, true label and probabilities of my model prediction.\n\n## Multilabeled\nThose are the images that contain samples with different labels whilst we are to predict only one label per image.\nHere are some examples. First one is labeled as **cgm** which stand for Cassava Green Mottle however there are so many leaves on the image that I see some healthy ones even though label says cgm.\n\n\nAnd another two examples. By the way when I first took a look at the first image below I though that this is a cbb (Cassava Bacterial Blight) sample and, as you can see, my model agrees with me. But the label says that this is an example of Cassava Brown Streak Disease.\nAnyway - there are also a plenty of healthy leaves on the image.\n\n\n\n\n## Confusing\nThis kind of noise would only teach your model a \"bad things\". Lets take a look at the example below.\n\nLeaves that are closer to the camera are definitely healthy. But the label is **cbsd**, which is, probably, about that guy in the background. This image would only \"confuse\" you model, showing it a good sample of healthy leaves but saying that they are not healthy.\n\n## Questionable\nI am no expert in the field by any means but during my analysis of the dataset I have seend tens of examples of Bacterial Blight and the following image, in my opition, demonstrates exactly this kind of disease (and my model agrees with me). But somehow this is labeled as **healthy**, which is questionable.\n\n\n## Other\nFinally there are some images that I don't even know how to categorise because they don't have much to do with leaves disease classification.\n\nThe label for the next image is \"healthy\". Well, I am really glad that this guy is ok. But how about the plant?\n", "1109100": "Interesting findings. Thanks to a healthy guy I now know how the Cassava roots look like!\n\nI guess handling label noise will become an important element of the pipeline. Of course we can expect similar noise in the test set, but that does not mean correcting it on the training stage is not needed.", "1109226": "Not all image are leaf image, some are roots image and some are stem image. which explain the photo with the guy holding the roots of the plant", "1109294": "Those are all valid points @nroman , I wonder how are you dealing or plan to deal with that noise? I have done only a few experiments on that, but got nothing useful yet.", "1109348": "from the application point of view, the framer should have placed the object of interest in the center of the image.\nif you only take center crop or put a center focused attention mask, maybe the results will be better?\n\nhttps://ipsj.ixsq.nii.ac.jp/ej/?action=pages_view_main&active_action=repository_view_main_item_detail&item_id=205376&item_no=1&page_id=13&block_id=8\n中央領域に注目する Center Attention による頑健性の高い植物病害診断装置の構築\n\n\n\n", "1109351": "Currently, I am also thinking about the above problem.\nIf we look for some improvements related to this, both cv and lb are likely to improve significantly.\n\np.s. The dataset has many noisy, but cv and lb are more stable than we think.", "1109397": "I would add [this discussion](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/200901) giving a list of mislabeled images : \n```['2782668721.jpg','3238704279.jpg','1365612235.jpg','1649500149.jpg','1236952675.jpg','3085440105.jpg',\n '2929245875.jpg','2509491848.jpg','1227531167.jpg']```\nHere is a look on the 5 first : \n\n\n\n\n\n\n\n\n\n\n\n\n0. 3238704279.jpg labeled : 4, What we see : 3\n1. 2782668721.jpg, labeled : 4, what we see : 3\n2. 1365612235.jpg, labeled : 4, what we see : 3\n3. 1649500149.jpg, labeled : 4, what we see : 3\n4. 1236952675.jpg, labeled : 4, what we see : 3\n5. 3085440105.jpg, labeled : 4, what we see : 3\n6. 2929245875.jpg, labeled : 4 , what we see : 3 (and a foot)\n7. 2509491848.jpg, labeled : 4, what we see : 3\n8. 1227531167.jpg, labeled : 4, what we see : 3 or 2\n\nI can add some more : \n\n9. 1300599354.jpg, labeled 4, what we see : 0 ? (not sure here) : \n\n10. 2715221153.jpg, labeled 4, what we see : 2 or 3 : \n\n\nalso : there are confusions on 2-3 labels as they show very similar features. Some images are labeled as 2 but look like 3 :\n1492444202.jpg\n1456881000.jpg\n1654084150.jpg (see image below)\n\n\n\n\nRelabeling all of those images by hand is fastidious and would take too much time and effort, nevertheless training a model on good labels is the most important, even if the test set is also noisy.", "1109917": "Perhaps the reason why they have asked us to clean this mess first and then go for the training!!😅", "1110652": "Thanks for bring this topic up, I am thinking to drop those images in training. It should cause less confusions to the model", "1112114": "I guess one way will be using a semi supervised learning approach (or like Snorkel to write some labeling function) to quickly relabel those mislabeled images. Perhaps, we can have our first model to detected samples that model prediction strongly disagree with label. Based on the threshold, we relabel those examples. Then we can train the second model on these corrected labels." }, "source": "meta" }