{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"---------------------------------------\n# Plant Pathology 2021 FGVC8 - training dataset review\n\nThe Plant Pathology 2021 FGVC8 dataset contains about 18k images of Apple leaf diseases. This current version considers six classes, namely 'healthy', 'scab', 'rust', 'powdery mildew', 'frog_eye_leaf_spot', 'complex'. \n\nDuring training session, it has been noted that a few predicted labels matched with disease in the images but not with the actual labels. Certainly, such cases compromised the model training and induced a wrong classification in the validation session. So, in an attempt to check the dataset for better training, a large sample (~15k) of the images have been reviewed an some findings have been displayed in this notebook to promote the discussion about it.\n\n**Some important considerations**\n* I´m no botanist, so my conclusions are derived from reading some specialized sites, comparing it with google images and looking at the dataset trying to find patterns;\n* The images shown here are just selected images. There might be many more misclassfied images but, again, I´m no authority in the field;\n* If, in the end, it comes to a conclusion that the dataset might have a large portion of compromised labels, the implication is that the Test Dataset and the Model Scoring might also be compromised;  \n\n**Conclusions and further considerations on the labels mismatch**\n\nOverall, whats been found is that:\n* Some images are not related to leaf diseases which should be removed from the dataset;\n* Some images have been clearly misclassified;\n* Some leaf diseases are very tricky to find on the images. A small, white spot might be an indicative of Powdery Mildew while an olive green patch might be a Scab. So probably discussion may occur as to define if a leaf is healthy or there is some disease.\n* The above point is probably related to the life cycle of the disease. An early stage might have just a weak sympton on the leaves and might go unnoticed. But a neural network could easily identify.\n* The *complex* class contains a large subset of images that might be classified differently as, for instance, a 'two label' classification. This might be a source of error in the training/testing step.","metadata":{}},{"cell_type":"markdown","source":"____________________________________________________\n# Summary:\n0. Typical images for each disease\n1. Images not related to Leaf Diseases\n2. Misclassification\n\n        a. In Healthy subset \n        b. In Complex subset\n        c. In Rust subset\n        \n        \n3. Disease stage of contamination\n\n        3.a Scab - Possible early contamination\n        3.b Powdery Mildew - Possible early contamination\n___________________________________________________","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport os\nimport cv2\nimport matplotlib.pyplot as plt\nfrom plotly import figure_factory as ff\n\nimages_path = '/kaggle/input/plant-pathology-2021-fgvc8/train_images/'\n\ndef plot_images(images_list, title_list, images_path, nx, ny, width, height):\n    \n    fig=plt.figure(figsize=(width, height))\n    \n    for idx, (image, title) in enumerate(zip(images_list, title_list)):\n       \n        img_file = os.path.join(images_path, image)\n        img = cv2.imread(img_file)\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        \n        ax = fig.add_subplot(nx, ny, idx+1)\n        ax.title.set_text(title+'\\n'+image)\n        plt.imshow(img)\n    plt.show()\n    \ndef plot_table(image_list, orig_class, corr_class):\n    \n    data_matrix = [image_list, orig_class, corr_class]\n    \n    fig = ff.create_table(data_matrix)\n    fig.show()","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-05-19T23:32:34.075776Z","iopub.execute_input":"2021-05-19T23:32:34.076247Z","iopub.status.idle":"2021-05-19T23:32:34.082226Z","shell.execute_reply.started":"2021-05-19T23:32:34.076219Z","shell.execute_reply":"2021-05-19T23:32:34.081542Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 0. Typical images for each disease\nFirst, let´s just take a look at typical images for each disease so there is a base for comparison. Each disease as its own characteristics:\n* **Rust**: Small, pale yellow spots on the upper surfaces. Eventually, tiny, black, fruiting bodies (pycnia) become visible.\n* **Powdery mildew**: Small, whitish felt-like patches of fungal growth appear and quickly cover the entire leaf.\n* **Scab**: The fungal disease forms pale yellow or olive-green spots on the upper surface of leaves. Dark, velvety spots may appear on the lower surface.\n* **Frog eye spot**: Small, purple specks appear on infected leaves. The round to irregularly lobed spots develop a light brown-to-gray center surrounded by one or more dark-brown concentric rings and a purple margin.\n* **Complex**: As defined by the competition. Unhealthy leaves with too many diseases to classify visually will have the complex class.\n","metadata":{}},{"cell_type":"code","source":"image_list = ['809e3fc0cbac7c59.jpg', '808f897fa9e2adb0.jpg', '828d2f4a91754ddb.jpg', '80a047cd9aa9fb4f.jpg', '80bf21a07dfcf047.jpg', '9090b5d5aba53cd3.jpg']\ntitle_list = ['Healthy', 'Rust', 'Powdery Mildew', 'Scab', 'Frog Eye Leaf Spot', 'Complex']\nplot_images(image_list, title_list, images_path, nx=3, ny=2, width=23, height=23)","metadata":{"execution":{"iopub.status.busy":"2021-05-20T02:56:31.046660Z","iopub.execute_input":"2021-05-20T02:56:31.046986Z","iopub.status.idle":"2021-05-20T02:56:38.941901Z","shell.execute_reply.started":"2021-05-20T02:56:31.046958Z","shell.execute_reply":"2021-05-20T02:56:38.940820Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"_________________________________\n## 1. Images not related to Leaf Diseases\nSome images have been found that are not related to Apple leaf diseases. I´ve particularly found just these four but there might be more.\nThose images should simply be removed from the dataset.\n\n    95276ccd226ad933.jpg - Sign\n    ead085dfac287263.jpg - Nest\n    ccec54723ff91860.jpg - Sign\n    da8770e819d2696d.jpg - Sign\n","metadata":{}},{"cell_type":"code","source":"image_list = ['95276ccd226ad933.jpg', 'ead085dfac287263.jpg', 'ccec54723ff91860.jpg', 'da8770e819d2696d.jpg']\ntitle_list = ['Random sign', 'Nest (lol)', 'Another sign', 'Another sign']\nplot_images(image_list, title_list, images_path, nx=1, ny=4, width=23, height=23)","metadata":{"execution":{"iopub.status.busy":"2021-05-20T02:56:42.753275Z","iopub.execute_input":"2021-05-20T02:56:42.753586Z","iopub.status.idle":"2021-05-20T02:56:47.599202Z","shell.execute_reply.started":"2021-05-20T02:56:42.753560Z","shell.execute_reply":"2021-05-20T02:56:47.598437Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"_______________________________________________\n## 2. Misclassification\nAnother set of images have been found to be misclassified. Below there are some examples of misclassification. Reminder: these are just some selected images and does not compreheend all the images that are set with the wrong labels.\n\n## **a. Healthy subset**\nThe following images have been found in the healthy sub dataset. Each image title is the possible classification.\n##### Some images have only a trace of the disease. For instance, the images:\n\n    bdd290c8cbc9aca9.jpg - small yellow spot (rust) and white material on the leaf tip\n    aa896f86f43a38b1.jpg - has a few olive-green patches \n    efbcc09086859ebc.jpg - has a white silky spot near the stem.\n    ad8addd454e14a6c.jpg - has a very light clearer spot near the stem on the upper side of the leaf.\n    \nOn these images it can be argued that the disease lifecycle matter for the classification. Image *ce5b7057c3259c1c.jpg* has a very obvious Powdery Mildew infection but image *bdd290c8cbc9aca9.jpg* is not that clear. In fact a large portion of images has a light powdery mildew infection on the leaf tip that went unnoticed (of course if that actually is powdery mildew, I´m no specialist). Some other cases are similar to picture *efbcc09086859ebc.jpg* which a small white patch is somewhere on the blade of the leaf.\nThe question put here is, in which stage if any the leaf should be classified as healthy or with any contamination. Take, as a matter of example the picture below. Should a coverage less than 1.5% be considered?\n\n![% of leaf with disease sympton](https://www.scielo.br/img/revistas/aabc/v92s1//0001-3765-aabc-92-s1-e20180889-f1.jpg)","metadata":{"execution":{"iopub.status.busy":"2021-05-20T00:11:13.410356Z","iopub.execute_input":"2021-05-20T00:11:13.410824Z","iopub.status.idle":"2021-05-20T00:11:13.41706Z","shell.execute_reply.started":"2021-05-20T00:11:13.410772Z","shell.execute_reply":"2021-05-20T00:11:13.415448Z"}}},{"cell_type":"code","source":"image_list_healthy = ['ce5b7057c3259c1c.jpg', \n              'bdd290c8cbc9aca9.jpg', \n              'ad8addd454e14a6c.jpg']\n\ntitle_list_healthy = ['Powdery Mildew', \n              'Rust + Powdery Mildew', \n              'Scab?']\nplot_images(image_list_healthy, title_list_healthy, images_path, nx=3, ny=1, width=35, height=35)","metadata":{"execution":{"iopub.status.busy":"2021-05-20T02:53:04.844373Z","iopub.execute_input":"2021-05-20T02:53:04.844728Z","iopub.status.idle":"2021-05-20T02:53:09.913164Z","shell.execute_reply.started":"2021-05-20T02:53:04.844690Z","shell.execute_reply":"2021-05-20T02:53:09.911572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_list_healthy = ['aa896f86f43a38b1.jpg',\n                      'efbcc09086859ebc.jpg']\n\ntitle_list_healthy = ['Scab',\n                      'Powdery Mildew']\nplot_images(image_list_healthy, title_list_healthy, images_path, nx=2, ny=1, width=35, height=35)","metadata":{"execution":{"iopub.status.busy":"2021-05-20T02:53:41.596433Z","iopub.execute_input":"2021-05-20T02:53:41.596825Z","iopub.status.idle":"2021-05-20T02:53:46.119320Z","shell.execute_reply.started":"2021-05-20T02:53:41.596777Z","shell.execute_reply":"2021-05-20T02:53:46.118509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **b. Complex subset**\nThe following images have been found in the Complex sub dataset. Each image title is the possible classification","metadata":{}},{"cell_type":"code","source":"image_list_complex = ['8cc32cc7f82f81f8.jpg',\n                     '90df66a17f502e13.jpg',\n                     'a0d79b991b79a0b8.jpg']\n\ntitle_list_complex = ['Scab',\n                     'Rust + Frog eye',\n                     'Scab + Frog eye']\n\nplot_images(image_list_complex, title_list_complex, images_path, nx=3, ny=1, width=35, height=35)","metadata":{"execution":{"iopub.status.busy":"2021-05-20T02:54:51.972515Z","iopub.execute_input":"2021-05-20T02:54:51.972861Z","iopub.status.idle":"2021-05-20T02:54:56.245173Z","shell.execute_reply.started":"2021-05-20T02:54:51.972830Z","shell.execute_reply":"2021-05-20T02:54:56.243922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_list_complex = ['a2be96953443c7e4.jpg',\n                     'aab096b7030ae3f9.jpg',\n                     'a7b0921acf317ce1.jpg']\n\ntitle_list_complex = ['Rust',\n                     'Rust + Scab',\n                     'Frog eye + Powdery Mildew']\n\nplot_images(image_list_complex, title_list_complex, images_path, nx=3, ny=1, width=35, height=35)","metadata":{"execution":{"iopub.status.busy":"2021-05-20T02:55:01.533792Z","iopub.execute_input":"2021-05-20T02:55:01.534148Z","iopub.status.idle":"2021-05-20T02:55:06.514903Z","shell.execute_reply.started":"2021-05-20T02:55:01.534117Z","shell.execute_reply":"2021-05-20T02:55:06.513890Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **c. Rust subset**","metadata":{}},{"cell_type":"code","source":"image_list_rust = ['d3945c098edc9dd1.jpg']\n\ntitle_list_rust = ['Powdery Mildew + Frog Eye']\n\nplot_images(image_list_rust, title_list_rust, images_path, nx=1, ny=1, width=14, height=14)","metadata":{"execution":{"iopub.status.busy":"2021-05-20T02:55:13.351080Z","iopub.execute_input":"2021-05-20T02:55:13.351606Z","iopub.status.idle":"2021-05-20T02:55:14.312155Z","shell.execute_reply.started":"2021-05-20T02:55:13.351567Z","shell.execute_reply":"2021-05-20T02:55:14.311035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **3. Disease stage of contamination**\n\nAs mentioned previously, due to the nature of the disease life cycles, some leaves present an earlier stage of contamination than others. This might be hard for the annotation procedure an also for the model training. Some examples are shown below.","metadata":{}},{"cell_type":"markdown","source":"## 3.a Scab - Possible early contamination\n\nThe images below have been found both on the Healthy and Scab subdataset, which shows how hard it is to determine manually the real state of the disease.\nSo, the question here is the same as posed before. How to create a rule to determined when there is a early scab contamination and when its a healthy leaf?\n\n**The image title shows which label it originally belongs.**","metadata":{}},{"cell_type":"code","source":"image_list_escab = ['b9909e99fd34210f.jpg',\n                     'dc7fe1898d402277.jpg',\n                     'cff02d717850c0b7.jpg']\n\ntitle_list_escab = ['healthy',\n                     'scab',\n                     'scab']\n\nplot_images(image_list_escab, title_list_escab, images_path, nx=3, ny=1, width=35, height=35)","metadata":{"execution":{"iopub.status.busy":"2021-05-20T02:55:33.076718Z","iopub.execute_input":"2021-05-20T02:55:33.077068Z","iopub.status.idle":"2021-05-20T02:55:38.088586Z","shell.execute_reply.started":"2021-05-20T02:55:33.077038Z","shell.execute_reply":"2021-05-20T02:55:38.086552Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_list_escab = ['ab1981fd06e23f8a.jpg',\n                     '8ab57810f73ec568.jpg',\n                     'f758af92944f9282.jpg']\n\ntitle_list_escab = ['healthy',\n                     'scab',\n                     'healthy']\n\nplot_images(image_list_escab, title_list_escab, images_path, nx=3, ny=1, width=35, height=35)","metadata":{"execution":{"iopub.status.busy":"2021-05-20T02:55:52.704101Z","iopub.execute_input":"2021-05-20T02:55:52.704453Z","iopub.status.idle":"2021-05-20T02:55:57.594937Z","shell.execute_reply.started":"2021-05-20T02:55:52.704425Z","shell.execute_reply":"2021-05-20T02:55:57.593200Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.b Powdery Mildew - Possible early contamination\nAs previously stated for Scab, here as some examples for powdery mildew.\nThe firts 3 cases presented here show a indicative of powdery mildew on the leaf tip. The last image shows some white-fluffly material on the blade.","metadata":{}},{"cell_type":"code","source":"image_list_epow = ['da9bcaa9a6d088c7.jpg',\n                  'f1bb28fee48284c3.jpg']\ntitle_list_epow = ['healthy',\n                  'scab']\nplot_images(image_list_epow, title_list_epow, images_path, nx=2, ny=1, width=24, height=24)","metadata":{"execution":{"iopub.status.busy":"2021-05-20T02:56:11.584515Z","iopub.execute_input":"2021-05-20T02:56:11.584859Z","iopub.status.idle":"2021-05-20T02:56:14.941172Z","shell.execute_reply.started":"2021-05-20T02:56:11.584828Z","shell.execute_reply":"2021-05-20T02:56:14.938975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_list_epow = ['d54a6887653e273c.jpg',\n                  '90ece9d647c23ca6.jpg']\ntitle_list_epow = ['healthy',\n                  'healthy']\nplot_images(image_list_epow, title_list_epow, images_path, nx=2, ny=1, width=24, height=24)","metadata":{"execution":{"iopub.status.busy":"2021-05-20T02:56:19.692889Z","iopub.execute_input":"2021-05-20T02:56:19.693238Z","iopub.status.idle":"2021-05-20T02:56:23.103038Z","shell.execute_reply.started":"2021-05-20T02:56:19.693205Z","shell.execute_reply":"2021-05-20T02:56:23.102261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}