{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Understanding the evaluation metric & CV for this competion\n\n**Update:\nThe code to calculate CV for all mentioned submissions is added at the end of this notebook.**\n\nIt might be somewhat unclear how the results are evaluated in this competition.\n\nThe official explanation presents Mean F1-Score metric as a function of true positives (tp) and false negatives (fn). However, a multi-label task introduces some uncertainty into the definition of tp and fn. For example, if the true label is `'scab frog_eye_leaf_spot'`, and the submitted prediction is `'scab'`, will this prediction be considered *partially correct* or just *incorrect* for the purpose of evaluation? In [this discussion thread](https://www.kaggle.com/c/plant-pathology-2021-fgvc8/discussion/227237) a few related questions were raised, and they became an inspiration for this notebook.\n\nFirst of all, I would like to test:\n1. Whether there is some value in predicting correctly only some labels, but not all labels, for a particular picture (if the true label is `'scab frog_eye_leaf_spot'`, will we get some credit for predicting just `'scab'` or just `'frog_eye_leaf_spot'`?).\n2. Whether the order of predicted classes matters (`'scab frog_eye_leaf_spot'` vs `'frog_eye_leaf_spot scab'`).\n\nWith the code in this notebook I made a few submissions in which predictions for every image in the test were the same:\n\n`Prediction     Public score`\n\n`'healthy'            0.251`\n\n`'scab'               0.292`\n\n`'scab healthy'       0.365`\n\n`'healthy scab'       0.365`\n\n`all possible labels  0.265` \n\nThe implication of this:\n1. **There is a benefit in partially predicting the correct answer.** Otherwise nonsense prediction `'healthy scab'` would score zero.\n\n2. Looks like **the order of labels does not matter** because the scores for `'healthy scab'` and `'scab healthy'` are the same.\n\nYou might want to use this approach to test other hypotheses about the evaluation."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's list all the possible metrics in the train set."},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/plant-pathology-2021-fgvc8/train.csv')\ntrain.labels.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"A prediction with all possible labels would look like this:"},{"metadata":{"trusted":true},"cell_type":"code","source":"s = set()\nfor k in train.labels.value_counts().index:\n    for c in k.split(' '):\n        s.add(c)\n' '.join(sorted(s))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's create a few submissions with the same prediction for every image.\n* Submission1: `'healthy'`\n* Submission2: `'scab'`\n* Submission3: `'scab healthy'`\n* Submission4: `'healthy scab'`\n* Submission4: `'cider_apple_rust complex frog_eye_leaf_spot healthy powdery_mildew rust scab'`"},{"metadata":{"trusted":true},"cell_type":"code","source":"TEST_FOLDER = '../input/plant-pathology-2021-fgvc8/test_images/'\nimages = os.listdir(TEST_FOLDER)\nsub = pd.DataFrame(images, columns=['image'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# sub['labels'] = 'healthy' # score 0.251\n# sub['labels'] = 'scab' # score 0.292\n# sub['labels'] = 'scab healthy' # score 0.365\nsub['labels'] = 'healthy scab' # score 0.365\n# sub['labels'] = 'cider_apple_rust complex frog_eye_leaf_spot healthy powdery_mildew rust scab' # score 0.265\n\nsub.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Calculating CV score from train"},{"metadata":{},"cell_type":"markdown","source":"Here is [an article](https://medium.com/synthesio-engineering/precision-accuracy-and-f1-score-for-multi-label-classification-34ac6bdfb404) that explains application of Mean F1-Score metric for multi-label classification. It explains the difference between 'macro' and 'micro' averaging. It is claimed that macro-averaging is to be preferred over micro-averaging.\n\nAt the moment, I am not sure which version ('macro' or 'micro') is used in the evaluation. But I find it problematic to apply macro-averaging because it is not defined for some cases mentioned in this workbook. Consider Submission1 when predictions for every image are 'healthy'. Macro-averaging requires calculating precision and recall for every label. But all labels except 'healthy' have tp=0 and fp=0, thus p=tp/(tp+fp) is undefined.\n\nThat is why I try to use micro-averaging here, which is properly defined for all cases mentioned above. Micro-averaging requires calculating the sum of tp, fp and fn across all classes. Then the formula described in the competition overview is applied: \n$$F1 = 2\\frac{p \\cdot r}{p+r}\\ \\ \\mathrm{where}\\ \\ p = \\frac{tp}{tp+fp},\\ \\ r = \\frac{tp}{tp+fn}$$\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = [\n    'healthy',\n    'scab', \n    'scab healthy',\n    'healthy scab',\n    'cider_apple_rust complex frog_eye_leaf_spot healthy powdery_mildew rust scab',\n]\nfor prediction in predictions:\n    prediction_labels = set(prediction.split(' '))\n    tp = sum([sum(train.labels.map(lambda x: c in x.split(' '))) for c in prediction_labels])\n    fp = sum([sum(train.labels.map(lambda x: c not in x.split(' '))) for c in prediction_labels])\n    fn = sum([sum(train.labels.map(lambda x: c in x.split(' '))) for c in s if c not in prediction_labels])\n    p = tp/(tp+fp)\n    r = tp/(tp+fn)\n    f1 = 2*p*r/(p+r)\n    print('{}: tp={}, fp={}, fn={}, p={:.3f}, r={:.3f}, f1={:.3f}'.format(\n        prediction, tp, fp, fn, p, r, f1\n    ))\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's put the results into a table:\n\n`Prediction       Public score        CV`\n\n`'healthy'              0.251      0.238`\n\n`'scab'                 0.292      0.294`\n\n`'scab healthy'         0.365      0.360`\n\n`'healthy scab'         0.365      0.360`\n\n`all possible labels    0.265      0.268`\n\nThis looks not bad at all. Probably even too good to be true."},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}