@fruitpathology, @sohier, we need your help.
\nFrom the recent comments on the CV-LB Inconcistency topic it's still unclear whether the evaluation metric for the competition is a row-wise F1, i.e.:
\nscore = f1_score(y_true, y_pred, average='micro')\nor a column-wise (or label-wise) F1, i.e.:
\nscore = f1_score(y_true, y_pred, average='macro')\nInformation on the Evaluation page is not full to make sure whether it is former or latter (and has a broken link). As there's up to a 10% difference between them clarification of the metric is also necessary for understanding the reasons for inconsistency.
", "messages": [ { "id": "1251267", "postDate": "03/24/2021 16:21:36", "content": "@fruitpathology, @sohier, we need your help.
\nFrom the recent comments on the CV-LB Inconcistency topic it's still unclear whether the evaluation metric for the competition is a row-wise F1, i.e.:
\nscore = f1_score(y_true, y_pred, average='micro')\nor a column-wise (or label-wise) F1, i.e.:
\nscore = f1_score(y_true, y_pred, average='macro')\nInformation on the Evaluation page is not full to make sure whether it is former or latter (and has a broken link). As there's up to a 10% difference between them clarification of the metric is also necessary for understanding the reasons for inconsistency.
", "rawMarkdown": "fruitpathology, @sohier, we need your help.\n\nFrom the recent comments on the **[CV-LB Inconcistency](https://www.kaggle.com/c/plant-pathology-2021-fgvc8/discussion/227237)** topic it's still unclear whether the evaluation metric for the competition is a row-wise F1, i.e.:\n```\nscore = f1_score(y_true, y_pred, average='micro')\n```\nor a column-wise (or label-wise) F1, i.e.:\n```\nscore = f1_score(y_true, y_pred, average='macro')\n```\nInformation on the **Evaluation** page is not full to make sure whether it is former or latter (and has a broken link). As there's up to a 10% difference between them clarification of the metric is also necessary for understanding the reasons for inconsistency.", "votes": null }, { "id": "1253022", "postDate": "03/26/2021 09:50:10", "content": "I believe it's 'macro' as in many of competitions, it's default.
", "rawMarkdown": "I believe it's 'macro' as in many of competitions, it's default.", "votes": null }, { "id": "1253705", "postDate": "03/27/2021 01:09:55", "content": "I agree, there is still and abnormally large gap between LB scores and CV scores. Fingers crossed that we hear back from the competition organizers!
", "rawMarkdown": "I agree, there is still and abnormally large gap between LB scores and CV scores. Fingers crossed that we hear back from the competition organizers!", "votes": null }, { "id": "1264107", "postDate": "04/05/2021 22:01:03", "content": "It's closest to sklearn's average='samples', though there might be some small differences in our implementations.
So strange
", "rawMarkdown": "So strange", "votes": null }, { "id": "1264256", "postDate": "04/06/2021 02:50:45", "content": "Hi @sohier Thank you for this clarification. Could you please specify the details of your implementation? For example, consider 3 images with labels ['scab', 'healthy', 'powdery_mildew complex'] and the predictions ['rust complex', 'healthy', 'powdery_mildew']. What is f1 score for this case?
I believe it's 'macro' as in many of competitions, it's default.
", "votes": null, "replies": [] }, { "id": 1253705, "author_name": "brendanartley", "author_url": "", "post_date": "03/27/2021 01:09:55", "content": "I agree, there is still and abnormally large gap between LB scores and CV scores. Fingers crossed that we hear back from the competition organizers!
", "votes": null, "replies": [] }, { "id": 1264107, "author_name": "sohier", "author_url": "", "post_date": "04/05/2021 22:01:03", "content": "It's closest to sklearn's average='samples', though there might be some small differences in our implementations.
Hi @sohier Thank you for this clarification. Could you please specify the details of your implementation? For example, consider 3 images with labels ['scab', 'healthy', 'powdery_mildew complex'] and the predictions ['rust complex', 'healthy', 'powdery_mildew']. What is f1 score for this case?
So strange
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1251267": "fruitpathology, @sohier, we need your help.\n\nFrom the recent comments on the **[CV-LB Inconcistency](https://www.kaggle.com/c/plant-pathology-2021-fgvc8/discussion/227237)** topic it's still unclear whether the evaluation metric for the competition is a row-wise F1, i.e.:\n```\nscore = f1_score(y_true, y_pred, average='micro')\n```\nor a column-wise (or label-wise) F1, i.e.:\n```\nscore = f1_score(y_true, y_pred, average='macro')\n```\nInformation on the **Evaluation** page is not full to make sure whether it is former or latter (and has a broken link). As there's up to a 10% difference between them clarification of the metric is also necessary for understanding the reasons for inconsistency.", "1253022": "I believe it's 'macro' as in many of competitions, it's default.", "1253705": "I agree, there is still and abnormally large gap between LB scores and CV scores. Fingers crossed that we hear back from the competition organizers!", "1264107": "It's closest to sklearn's `average='samples'`, though there might be some small differences in our implementations.", "1264197": "So strange", "1264256": "Hi @sohier Thank you for this clarification. Could you please specify the details of your implementation? For example, consider 3 images with labels `['scab', 'healthy', 'powdery_mildew complex']` and the predictions `['rust complex', 'healthy', 'powdery_mildew']`. What is f1 score for this case?" }, "source": "meta" }