From the provided link of the competition metric:
\n\n\nAccuracy alone doesn't tell the full story when you're working with a class-imbalanced data set, like this one, where there is a significant disparity between the number of positive and negative labels.
\n
Then why using this metric?
", "messages": [ { "id": "1084532", "postDate": "11/20/2020 05:56:27", "content": "From the provided link of the competition metric:
\n\n\nAccuracy alone doesn't tell the full story when you're working with a class-imbalanced data set, like this one, where there is a significant disparity between the number of positive and negative labels.
\n
Then why using this metric?
", "rawMarkdown": "From the provided link of the competition metric: \n> **Accuracy** alone doesn't tell the full story when you're working with a **class-imbalanced** data set, like this one, where there is a significant disparity between the number of positive and negative labels.\n\nThen why using this metric?", "votes": null }, { "id": "1084589", "postDate": "11/20/2020 07:37:02", "content": "To tackle the imbalance task is our job alone. If they use this metric, possibly there will be different distribution between train and test set.
", "rawMarkdown": "To tackle the imbalance task is our job alone. If they use this metric, possibly there will be different distribution between train and test set.", "votes": null }, { "id": "1084717", "postDate": "11/20/2020 10:23:01", "content": "To my surprise, it looks like it is not sensitive to class imbalance. This notebook actually demonstrates this as it got a score of 0.614 just by categorising everything as the most present class (class 3). Maybe the organiser wants to give more importance to the classes that are the most seen in real-life (assuming the dataset is representative of the natural distribution), as they might be the ones giving farmers the most pain.
", "rawMarkdown": "To my surprise, it looks like it is not sensitive to class imbalance. This [notebook](https://www.kaggle.com/ihelon/cassava-leaf-disease-exploratory-data-analysis) actually demonstrates this as it got a score of 0.614 just by categorising everything as the most present class (class 3). Maybe the organiser wants to give more importance to the classes that are the most seen in real-life (assuming the dataset is representative of the natural distribution), as they might be the ones giving farmers the most pain.", "votes": null }, { "id": "1084766", "postDate": "11/20/2020 11:11:47", "content": "Isn't that exactly why it is not a good metric? A constant classifier which predicts label 3 for each sample is able to achieve a competition metric of 0.614 while a different metric e.g. f1-score would take the overall classifier performance into account and result in a worse score.
Accuracy is indeed not a good metric for imbalance dataset, but it is only if the dataset is really imbalance. We are provided with only the imbalance train set, but the test set is unknown.
", "rawMarkdown": "Accuracy is indeed not a good metric for imbalance dataset, but it is only if the dataset is really imbalance. We are provided with only the imbalance train set, but the test set is unknown.", "votes": null }, { "id": "1086248", "postDate": "11/21/2020 14:03:37", "content": "Check following discussion: https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/198135
", "rawMarkdown": "Check following discussion: https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/198135", "votes": null } ], "comments": [ { "id": 1084589, "author_name": "aeryss", "author_url": "", "post_date": "11/20/2020 07:37:02", "content": "To tackle the imbalance task is our job alone. If they use this metric, possibly there will be different distribution between train and test set.
", "votes": null, "replies": [] }, { "id": 1084717, "author_name": "frlemarchand", "author_url": "", "post_date": "11/20/2020 10:23:01", "content": "To my surprise, it looks like it is not sensitive to class imbalance. This notebook actually demonstrates this as it got a score of 0.614 just by categorising everything as the most present class (class 3). Maybe the organiser wants to give more importance to the classes that are the most seen in real-life (assuming the dataset is representative of the natural distribution), as they might be the ones giving farmers the most pain.
", "votes": null, "replies": [ { "id": 1084766, "author_name": "aliabdin1", "author_url": "", "post_date": "11/20/2020 11:11:47", "content": "Isn't that exactly why it is not a good metric? A constant classifier which predicts label 3 for each sample is able to achieve a competition metric of 0.614 while a different metric e.g. f1-score would take the overall classifier performance into account and result in a worse score.
Accuracy is indeed not a good metric for imbalance dataset, but it is only if the dataset is really imbalance. We are provided with only the imbalance train set, but the test set is unknown.
", "votes": null, "replies": [] }, { "id": 1086248, "author_name": "aliabdin1", "author_url": "", "post_date": "11/21/2020 14:03:37", "content": "Check following discussion: https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/198135
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1084532": "From the provided link of the competition metric: \n> **Accuracy** alone doesn't tell the full story when you're working with a **class-imbalanced** data set, like this one, where there is a significant disparity between the number of positive and negative labels.\n\nThen why using this metric?", "1084589": "To tackle the imbalance task is our job alone. If they use this metric, possibly there will be different distribution between train and test set.", "1084717": "To my surprise, it looks like it is not sensitive to class imbalance. This [notebook](https://www.kaggle.com/ihelon/cassava-leaf-disease-exploratory-data-analysis) actually demonstrates this as it got a score of 0.614 just by categorising everything as the most present class (class 3). Maybe the organiser wants to give more importance to the classes that are the most seen in real-life (assuming the dataset is representative of the natural distribution), as they might be the ones giving farmers the most pain.", "1084766": "Isn't that exactly why it is not a good metric? A constant classifier which predicts label `3` for each sample is able to achieve a competition metric of `0.614` while a different metric e.g. f1-score would take the overall classifier performance into account and result in a worse score.", "1084783": "Accuracy is indeed not a good metric for imbalance dataset, but it is only if the dataset is really imbalance. We are provided with only the imbalance train set, but the test set is unknown.", "1086248": "Check following discussion: https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/198135" }, "source": "meta" }