{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"In this article I'll talk about one Multi-Label Image Classification competition: https://www.kaggle.com/competitions/plant-pathology-2021-fgvc8\n\nI began to deal with this dataset when the competition was already finished. And as I see in \"Discussions\" tab there were some changes in dataset. That could be a problem why I can't make same successful results as other members. But let's discuss current dataset.\n\nFirst of all if you check amount of different classes (labels) you will get something like this:","metadata":{}},{"cell_type":"code","source":"import pandas as pd\ndf = pd.read_csv('../input/plant-pathology-2021-fgvc8/train.csv')\ndf['labels'].value_counts()","metadata":{"tags":[],"execution":{"iopub.status.busy":"2022-06-17T16:56:43.951750Z","iopub.execute_input":"2022-06-17T16:56:43.952147Z","iopub.status.idle":"2022-06-17T16:56:43.988240Z","shell.execute_reply.started":"2022-06-17T16:56:43.952116Z","shell.execute_reply":"2022-06-17T16:56:43.987238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Imagine that we already have a Multi-Label Classification script that is capable to train a model. Let's launch it and get the model. And when we get the model we will use that model to predict labels the same training (+validation) dataset. In that case you'd probably say: \"We will get ~100% prediction rate results because we already used this dataset to make training!\". Yes, this is obvious. And by means of that we'll check whether everything is OK with model/dataset.\n\nWe are expecting to get very hisgh accuracy (90% +), but we'll get something like this:","metadata":{}},{"cell_type":"code","source":"                    Real  Predicted  amount %  correct %\nscab                4826       4577      95.0       90.0\nrust                1860         94       5.0        0.0\npowdery_mildew      1184          7       1.0        0.0\nhealthy             4624        423       9.0        0.0\nfrog_eye_leaf_spot  3181          9       0.0        0.0\ncomplex             1602          1       0.0        0.0","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**I used MobileNetV2 and source images resized to 600x400 px.**\n\nLook at our results (most importand column is the last one \"correct %\"). The model `couldn't predict` most of labels. Why?! There are some reasons for that bad results. Let's check images properly. For example let's pick some with 'rust' label. Unfortunately, I'm not an agronomist and I can't determine plants diseases by photo. But I can compare few photos and say that diseases with same label are different:","metadata":{}},{"cell_type":"code","source":"from matplotlib import pyplot as plt\nfrom matplotlib.image import imread\n\ndef show_img(img, text=''):\n    img = imread(img)\n    plt.figure(figsize=(8, 12), )\n    plt.imshow(img, cmap='gray')\n    plt.title(text)\n    plt.axis('off')\n    plt.show()\n\nimgs = ['82f535ab5a31a5ca.jpg',\n        '82add70df6ab2854.jpg',\n        '839d6e94c94d3659.jpg']\n\nda = df[df['labels'] == 'rust']\nda = da[da['image'].isin(imgs)]\n\nfor index, row in da.iterrows():\n    path, label = row\n    show_img('../input/plant-pathology-2021-fgvc8/train_images/' + path, f'{label}: {path}')","metadata":{"tags":[],"execution":{"iopub.status.busy":"2022-06-17T16:59:45.602408Z","iopub.execute_input":"2022-06-17T16:59:45.603651Z","iopub.status.idle":"2022-06-17T16:59:49.851583Z","shell.execute_reply.started":"2022-06-17T16:59:45.603593Z","shell.execute_reply":"2022-06-17T16:59:49.850788Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Take a look at that three different examples of 'rust'. As for me there are two infected/ill leafs. And one is quite healthy! Now we can see why our model cant predict 'rust' - because it can't understand what it is. And the worst thing that these examples of 'rust' makes impact on our model when we try to fit all labels semustaniously.\n\nTo solve that problem lets devide labels into groups to make a lot of models that can predict only few labels. For examples it could be pairs: \"healthy - scab\", \"healthy - frog_eye_leaf_spot\" etc. Every model will be able to separate healthy leafs and infected by only one disease.\n\nAs a dataset we'll use only pure data (not mixed), pure labels. I'll not do any augmentation. Let's think that 1000+ pics is enough for labels.","metadata":{}},{"cell_type":"code","source":"[scab]                                 4826\n[healthy]                              4624\n[frog_eye_leaf_spot]                   3181\n[rust]                                 1860\n[complex]                              1602\n[powdery_mildew]                       1184","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After models fitting we'll apply them to the same dataset as used for fitting (we already did something like that in the beginning of this article). And we'll get that results:","metadata":{}},{"cell_type":"code","source":"                    Real  Predicted  amount %  correct %\nfrog_eye_leaf_spot  3181       3902     123.0       98.0\nhealthy             4624       3899      84.0       83.0\n========================================================\ncomplex             1602       2249     140.0       99.0\nhealthy             4624       3969      86.0       86.0\n========================================================\nscab                4826       3652      76.0       75.0\nhealthy             4624       5794     125.0       99.0\n========================================================\nrust                1860          1       0.0        0.0\nhealthy             4624       6484     140.0      100.0\n========================================================\npowdery_mildew      1184       4465     377.0        0.0\nhealthy             4624       5808     126.0        4.0\n========================================================","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"These 5 models can predict their illnesses with different probabilities. We can predict \"frog_eye_leaf_spot\" with 98% rate and in 83% cases can separate \"healthy\".\n\nNow look at the bottom of this table. These two models are not capable to understand whether it is \"powdery_mildew\" or \"healthy\", \"rust\" or \"healthy\". It means that the dataset for \"powdery_mildew\" and \"rust\" is not clear (there is a \"noise\"). It contains combination of images (not only with \"powdery_mildew\" and \"rust\").\n\nBut we can make it clear by means of applying successful model to bad labels (\"powdery_mildew\" and \"rust\") and deleting images with recognized deseases. Whole process of data cleaning for \"rust\" dataset will look like that:\n\n1. Use \"frog_eye_leaf_spot / healthy' model to predict on \"rust\" label list.\n2. Leave only \"unknown\" labels in the results (in tensorflow - [UNK] labels): delete all images with successfully detected \"frog_eye_leaf_spot\" or \"healthy\" labels.\n3. Use dataset from stage 2 (with \"unknown\" images). Apply \"scab / healthy\" model to predict. Leave only \"unknown\" labels in the results: delete all images with successfully detected \"scab\" or \"healthy\" labels.\n4. Use dataset from stage 3 (with \"unknown\" images). Apply \"complex / healthy\" model to predict. Leave only \"unknown\" labels in the results: delete all images with successfully detected \"complex\" or \"healthy\" labels.\n\nFinnaly you'll get list of images that was not recognized by any other model. All of them have \"unknown\" label. It is really a (rust) disease that you are looking for. If you do that process properly then you will leave only ~20% of dataset of each: \"rust\" and \"powdery_mildew\". You will delete about 80% and the rest will be not enough to make a good fitting.\n\nThe second problem is that you have two \"bad\" labels. So \"unknown\" label contains \"rust\" OR/AND \"powdery_mildew\" (could be even mixed). And you can separate it only with \"cross labels\" data that we didn't use before:","metadata":{}},{"cell_type":"code","source":"[rust, frog_eye_leaf_spot]              120\n[rust, complex]                          97\n[powdery_mildew, complex]                87","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Look at amount of that images. It's uncomparable with thousands images of pure classes....\n\nAnd here I could make some good augmentation to increase that amount of images. But I decided not to make a kind of limited model that can predict only \"frog_eye_leaf_spot\", \"complex\", \"scab\", \"healthy\". And that model is quite successful:","metadata":{}},{"cell_type":"code","source":"                    Real  Predicted  amount %  correct %\nscab                4826       4857     101.0       92.0\nhealthy             4624       4738     102.0       96.0\nfrog_eye_leaf_spot  3181       3216     101.0       92.0\ncomplex             1602       1351      84.0       68.0","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"At that point I stoped my attempts to make it better. Because as I said in the beggining of that article the competition is already finished. And there were changes in dataset. Probably I got data with mixed labels (other members use to talk about that in comments). Or my programming code contains a kind of mistakes... Anyway I'm done. Thank you for readnig! Hope you could get some interesting ideas from that article.\n\nP.S.\nI've programmed final model (that can predict 4 labels: \"frog_eye_leaf_spot\", \"complex\", \"scab\", \"healthy\") as RestAPI service (using Flask), and packed in docker container. You can find whole code on my GIT: [plant-pathology-2021](https://github.com/AlekseyGur/Education/tree/master/Kaggle/plant-pathology-2021)","metadata":{}}]}