![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4795157%2F0770905613063c921d9bc0573dfd4e8d%2FUnknown.png?generation=1606108288273135&alt=media\")
Hi, this is my first time post. And I'm a beginner at kaggle.
\nI'm sorry if someone have posted in the past.
\nThis photo shows cassavas, not cassava leaves.
How should I handle it?
\nSorry for my English.
", "messages": [ { "id": "1087824", "postDate": "11/23/2020 05:12:49", "content": "Hi, this is my first time post. And I'm a beginner at kaggle.
\nI'm sorry if someone have posted in the past.
\nThis photo shows cassavas, not cassava leaves.
How should I handle it?
\nSorry for my English.
", "rawMarkdown": "Hi, this is my first time post. And I'm a beginner at kaggle.\n\nI'm sorry if someone have posted in the past.\nThis photo shows cassavas, not cassava leaves.\n\n\n\nHow should I handle it?\n\nSorry for my English.", "votes": null }, { "id": "1088110", "postDate": "11/23/2020 10:47:05", "content": "Nothing to do for you, just drop it out and dont use it for training.
\nDont be sorry for you English at all, be confident using it and dont feel ashamed in any way, we are all here to learn in different ways :)
", "rawMarkdown": "Nothing to do for you, just drop it out and dont use it for training.\n\nDont be sorry for you English at all, be confident using it and dont feel ashamed in any way, we are all here to learn in different ways :)", "votes": null }, { "id": "1088247", "postDate": "11/23/2020 13:20:17", "content": "Thank you for your kindness :)
\nWe can't deal with it if the test data contains images other than leaves. Therefore, should we drop images other than leaves if they are found?
I would drop it, because you will \"manipulate\" all the other entries in Healthy class and at the end, it would not learn it because its just one image like that or did you find multiple of those without leaves?
I just did some research, so I feel like there is a little more.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4795157%2F96315ee9515c872ad967c28ccaa4dd58%2FUnknown-3.png?generation=1606139525199100&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4795157%2F2169e8db4bed6030e6b8dbde0a1c46ab%2FUnknown-4.png?generation=1606139538898045&alt=media\")
Indeed, the problem then is, that we dont know if the testset also includes these kind of pictures.
\nThe only way to find that out would be to train once a classifier with these images and once without, unfortunately you would have to submit with both of them. Then you would know if these images are needed or not. You also have to find all relevant images and remove them, maybe by using consine similarity on the features of a pretrained CNN.
", "rawMarkdown": "Indeed, the problem then is, that we dont know if the testset also includes these kind of pictures. \n\nThe only way to find that out would be to train once a classifier with these images and once without, unfortunately you would have to submit with both of them. Then you would know if these images are needed or not. You also have to find all relevant images and remove them, maybe by using consine similarity on the features of a pretrained CNN.", "votes": null }, { "id": "1088342", "postDate": "11/23/2020 14:33:14", "content": "Thank you for sharing your information. I've never heard of cosine similarity. I'll try it.
", "rawMarkdown": "Thank you for sharing your information. I've never heard of cosine similarity. I'll try it.", "votes": null }, { "id": "1088367", "postDate": "11/23/2020 15:00:55", "content": "I have wrote a little notebook which uses cosine similarity, I have found 4 such images within the first 100 CSV lines.
\nTheir index in the CSV is:
\n[22, 42, 60, 87]
I have to optimise memory usage and then I will try over a bigger subset.
", "rawMarkdown": "I have wrote a little notebook which uses cosine similarity, I have found 4 such images within the first 100 CSV lines.\n\nTheir index in the CSV is:\n`[22, 42, 60, 87]`\n\nI have to optimise memory usage and then I will try over a bigger subset.", "votes": null }, { "id": "1088473", "postDate": "11/23/2020 16:29:59", "content": "I checked the whole trainset and there are 147 images with a cosine similarity higher than 0.65 thats around 0.6% of the whole trainset and should therefore only affect our training a bit.
Keeping them in your training process should not have a big effect on the model.
\nI have made the notebook public: https://www.kaggle.com/aliabdin1/find-similar-images-with-cosine-similarity
", "rawMarkdown": "I checked the whole trainset and there are `147` images with a cosine similarity higher than `0.65` thats around `0.6%` of the whole trainset and should therefore only affect our training a bit. \n\nKeeping them in your training process should not have a big effect on the model.\n\nI have made the notebook public: https://www.kaggle.com/aliabdin1/find-similar-images-with-cosine-similarity", "votes": null }, { "id": "1088700", "postDate": "11/23/2020 21:32:46", "content": "I suppose this represents the real-life scenario where farmers might not follow instructions properly and submit images of the roots rather than (or in addition to-) leaves. A small \"domain knowledge\" note regarding Cassava Brown Streak Disease: unlike the more common Cassava Mosaic Disease, it also causes the root itself to rot (rather than just limiting root growth by damaging the leaves), which is probably why some farmers have felt inclined to include root photos in this case.
", "rawMarkdown": "I suppose this represents the real-life scenario where farmers might not follow instructions properly and submit images of the roots rather than (or in addition to-) leaves. A small \"domain knowledge\" note regarding Cassava Brown Streak Disease: unlike the more common Cassava Mosaic Disease, it also causes the root itself to rot (rather than just limiting root growth by damaging the leaves), which is probably why some farmers have felt inclined to include root photos in this case.", "votes": null }, { "id": "1089087", "postDate": "11/24/2020 07:38:56", "content": "maybe pay more attention on green space , or drop it out from training stage;
", "rawMarkdown": "maybe pay more attention on green space , or drop it out from training stage;", "votes": null } ], "comments": [ { "id": 1088110, "author_name": "aliabdin1", "author_url": "", "post_date": "11/23/2020 10:47:05", "content": "Nothing to do for you, just drop it out and dont use it for training.
\nDont be sorry for you English at all, be confident using it and dont feel ashamed in any way, we are all here to learn in different ways :)
", "votes": null, "replies": [ { "id": 1088247, "author_name": "hattyoriiiiiii", "author_url": "", "post_date": "11/23/2020 13:20:17", "content": "Thank you for your kindness :)
\nWe can't deal with it if the test data contains images other than leaves. Therefore, should we drop images other than leaves if they are found?
I would drop it, because you will \"manipulate\" all the other entries in Healthy class and at the end, it would not learn it because its just one image like that or did you find multiple of those without leaves?
I just did some research, so I feel like there is a little more.
\n
\n
Indeed, the problem then is, that we dont know if the testset also includes these kind of pictures.
\nThe only way to find that out would be to train once a classifier with these images and once without, unfortunately you would have to submit with both of them. Then you would know if these images are needed or not. You also have to find all relevant images and remove them, maybe by using consine similarity on the features of a pretrained CNN.
", "votes": null, "replies": [] }, { "id": 1088342, "author_name": "hattyoriiiiiii", "author_url": "", "post_date": "11/23/2020 14:33:14", "content": "Thank you for sharing your information. I've never heard of cosine similarity. I'll try it.
", "votes": null, "replies": [] }, { "id": 1088367, "author_name": "aliabdin1", "author_url": "", "post_date": "11/23/2020 15:00:55", "content": "I have wrote a little notebook which uses cosine similarity, I have found 4 such images within the first 100 CSV lines.
\nTheir index in the CSV is:
\n[22, 42, 60, 87]
I have to optimise memory usage and then I will try over a bigger subset.
", "votes": null, "replies": [] }, { "id": 1088473, "author_name": "aliabdin1", "author_url": "", "post_date": "11/23/2020 16:29:59", "content": "I checked the whole trainset and there are 147 images with a cosine similarity higher than 0.65 thats around 0.6% of the whole trainset and should therefore only affect our training a bit.
Keeping them in your training process should not have a big effect on the model.
\nI have made the notebook public: https://www.kaggle.com/aliabdin1/find-similar-images-with-cosine-similarity
", "votes": null, "replies": [] } ] }, { "id": 1088700, "author_name": "thomasbrekkunnvik", "author_url": "", "post_date": "11/23/2020 21:32:46", "content": "I suppose this represents the real-life scenario where farmers might not follow instructions properly and submit images of the roots rather than (or in addition to-) leaves. A small \"domain knowledge\" note regarding Cassava Brown Streak Disease: unlike the more common Cassava Mosaic Disease, it also causes the root itself to rot (rather than just limiting root growth by damaging the leaves), which is probably why some farmers have felt inclined to include root photos in this case.
", "votes": null, "replies": [] }, { "id": 1089087, "author_name": "buctl2013014093", "author_url": "", "post_date": "11/24/2020 07:38:56", "content": "maybe pay more attention on green space , or drop it out from training stage;
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1087824": "Hi, this is my first time post. And I'm a beginner at kaggle.\n\nI'm sorry if someone have posted in the past.\nThis photo shows cassavas, not cassava leaves.\n\n\n\nHow should I handle it?\n\nSorry for my English.", "1088110": "Nothing to do for you, just drop it out and dont use it for training.\n\nDont be sorry for you English at all, be confident using it and dont feel ashamed in any way, we are all here to learn in different ways :)", "1088247": "Thank you for your kindness :)\nWe can't deal with it if the test data contains images other than leaves. Therefore, should we drop images other than leaves if they are found?", "1088253": "I would drop it, because you will \"manipulate\" all the other entries in `Healthy` class and at the end, it would not learn it because its just one image like that or did you find multiple of those without leaves?", "1088288": "I just did some research, so I feel like there is a little more.\n\n\n", "1088306": "Indeed, the problem then is, that we dont know if the testset also includes these kind of pictures. \n\nThe only way to find that out would be to train once a classifier with these images and once without, unfortunately you would have to submit with both of them. Then you would know if these images are needed or not. You also have to find all relevant images and remove them, maybe by using consine similarity on the features of a pretrained CNN.", "1088342": "Thank you for sharing your information. I've never heard of cosine similarity. I'll try it.", "1088367": "I have wrote a little notebook which uses cosine similarity, I have found 4 such images within the first 100 CSV lines.\n\nTheir index in the CSV is:\n`[22, 42, 60, 87]`\n\nI have to optimise memory usage and then I will try over a bigger subset.", "1088473": "I checked the whole trainset and there are `147` images with a cosine similarity higher than `0.65` thats around `0.6%` of the whole trainset and should therefore only affect our training a bit. \n\nKeeping them in your training process should not have a big effect on the model.\n\nI have made the notebook public: https://www.kaggle.com/aliabdin1/find-similar-images-with-cosine-similarity", "1088700": "I suppose this represents the real-life scenario where farmers might not follow instructions properly and submit images of the roots rather than (or in addition to-) leaves. A small \"domain knowledge\" note regarding Cassava Brown Streak Disease: unlike the more common Cassava Mosaic Disease, it also causes the root itself to rot (rather than just limiting root growth by damaging the leaves), which is probably why some farmers have felt inclined to include root photos in this case.", "1089087": "maybe pay more attention on green space , or drop it out from training stage;" }, "source": "meta" }