Did you think about using another metric in this competition?
\nFor example categorical log loss.
\nBecause I think many players will be able to achieve a very high accuracy score, and for many, this indicator will be equal. And if your dataset has an imbalance, then the accuracy metric is not the best choice for evaluating models.
", "messages": [ { "id": 1084305, "postDate": "2020-11-19T23:01:20.613Z", "content": "Did you think about using another metric in this competition?
\nFor example categorical log loss.
\nBecause I think many players will be able to achieve a very high accuracy score, and for many, this indicator will be equal. And if your dataset has an imbalance, then the accuracy metric is not the best choice for evaluating models.
", "rawMarkdown": "Did you think about using another metric in this competition?\n\nFor example categorical log loss.\n\nBecause I think many players will be able to achieve a very high accuracy score, and for many, this indicator will be equal. And if your dataset has an imbalance, then the accuracy metric is not the best choice for evaluating models.", "votes": 56 }, { "id": 1084737, "postDate": "2020-11-20T10:42:51.350Z", "content": "Agree, F1 or (even better) the Matthews Correlation Coefficient might be better alternatives
", "rawMarkdown": "Agree, F1 or (even better) the Matthews Correlation Coefficient might be better alternatives", "votes": 6 }, { "id": 1087385, "postDate": "2020-11-22T16:55:47.270Z", "content": "whatever the metric is, who gets the most accurate predictions wins. so I don't think it matters much
", "rawMarkdown": "whatever the metric is, who gets the most accurate predictions wins. so I don't think it matters much", "votes": 5, "replies": [ { "id": 1087413, "postDate": "2020-11-22T17:28:05.927Z", "content": "@moewie94 That's correct. It doesn't matter to us but assume a scenario that the winner's model is working very well on imbalanced class (which have very high number of data points) but performs average on other classes. This will impact to organizers.
", "rawMarkdown": "@moewie94 That's correct. It doesn't matter to us but assume a scenario that the winner's model is working very well on imbalanced class (which have very high number of data points) but performs average on other classes. This will impact to organizers.", "votes": 3 }, { "id": 1087694, "postDate": "2020-11-23T02:20:10.700Z", "content": "this can even be the way the organizer wants it. they surely aware of this matters when they split the test set (they can just exclude some class 3 samples in the test set to make to test set class balanced) but chose to not do it so it might not be a big problem. furthermore, this can even be the way this problem work in real-life scenario (for example the third class appears most in nature). so just split everything to be balanced is not a good way sometimes.
", "rawMarkdown": "this can even be the way the organizer wants it. they surely aware of this matters when they split the test set (they can just exclude some class 3 samples in the test set to make to test set class balanced) but chose to not do it so it might not be a big problem. furthermore, this can even be the way this problem work in real-life scenario (for example the third class appears most in nature). so just split everything to be balanced is not a good way sometimes.", "votes": 2 }, { "id": 1088086, "postDate": "2020-11-23T10:14:03.263Z", "content": "@moewie94 It matters because finally we have to classify all the 4 diseases correctly. As there is a less data for remaining 3 diseases accuracy score will give less weightage to those 3 diseases. Our model wont get enough data to train properly for these 3 diseases compared to the class 3 disease(higher one)
", "rawMarkdown": "@moewie94 It matters because finally we have to classify all the 4 diseases correctly. As there is a less data for remaining 3 diseases accuracy score will give less weightage to those 3 diseases. Our model wont get enough data to train properly for these 3 diseases compared to the class 3 disease(higher one)", "votes": 2 }, { "id": 1088338, "postDate": "2020-11-23T14:30:54.187Z", "content": "my point is the organizers are fully aware of this matter and they let it happen this way so this imbalance can be their very intention. every organizer has to pay tens of thousands of dollars for kaggle consultant services and every dataset must be analyzed by the kaggle ds team before being accepted for a competition. so i think it's unlikely they are naive enough to not know this class imbalance and how it affects the scoring system.
\nsometimes the test set distribution should be imbalanced like the way they are in real life, not like the way they are in MNIST.
", "rawMarkdown": "my point is the organizers are fully aware of this matter and they let it happen this way so this imbalance can be their very intention. every organizer has to pay tens of thousands of dollars for kaggle consultant services and every dataset must be analyzed by the kaggle ds team before being accepted for a competition. so i think it's unlikely they are naive enough to not know this class imbalance and how it affects the scoring system.\n\nsometimes the test set distribution should be imbalanced like the way they are in real life, not like the way they are in MNIST.", "votes": 8 }, { "id": 1088344, "postDate": "2020-11-23T14:36:12.837Z", "content": "@moewie94 I totally agree with you (I don't know why you got so many dislikes)
\nMy goal was to open this discussion to make sure that the metric will not be changed in the middle of the competition.
Agreed! Accuracy is a very bad choice for this dataset. However I've never seen a change of metrics in the middle of a competition.
", "rawMarkdown": "Agreed! Accuracy is a very bad choice for this dataset. However I've never seen a change of metrics in the middle of a competition. ", "votes": 3, "replies": [ { "id": 1085900, "postDate": "2020-11-21T09:11:17.543Z", "content": "It happened many times (first Jigsaw competition , Ion Switching etc. )
\nFor Jigsaw , the metric was changed about a month after the beginning. But Kaggle extended the deadline thereafter.
", "rawMarkdown": "It happened many times (first Jigsaw competition , Ion Switching etc. ) \n\nFor [Jigsaw](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/48639) , the metric was changed about a month after the beginning. But Kaggle extended the deadline thereafter.", "votes": 5 }, { "id": 1085954, "postDate": "2020-11-21T09:50:34.197Z", "content": "Thanks for the info. I hope this accuracy metrics can change! Otherwise I don't even know which loss function to choose.
", "rawMarkdown": "Thanks for the info. I hope this accuracy metrics can change! Otherwise I don't even know which loss function to choose.", "votes": 2 } ] }, { "id": 1084365, "postDate": "2020-11-20T01:03:58.410Z", "content": "Categorical log loss metric holds good for imbalance classification @ihelon
", "rawMarkdown": "Categorical log loss metric holds good for imbalance classification @ihelon ", "votes": 3 }, { "id": 1085132, "postDate": "2020-11-20T17:35:28.750Z", "content": "From my perspective, I am not sure how the competition would improve with a different metric. Accuracy always increases as the number of correct predictions increase, and the competition is won by the submission with the highest number of correct predictions.
", "rawMarkdown": "From my perspective, I am not sure how the competition would improve with a different metric. Accuracy always increases as the number of correct predictions increase, and the competition is won by the submission with the highest number of correct predictions.", "votes": 2, "replies": [ { "id": 1085135, "postDate": "2020-11-20T17:38:26.397Z", "content": "@jeffalltogether Yes, your point of view may also be correct. In this case, we need to think over an approach for fitting the imbalance
", "rawMarkdown": "@jeffalltogether Yes, your point of view may also be correct. In this case, we need to think over an approach for fitting the imbalance", "votes": 3 }, { "id": 1088022, "postDate": "2020-11-23T08:47:33.197Z", "content": "@jeffalltogether Main aim of this competition is to classify these 4 diseases correctly because farmers has to use the fertilizers for that particular disease.
\n As there are more class 3 diseases our model will learn perfectly for class 3 but for remaining class it will fail. As the data for remaining 3 diseases is less, in accuracy score they will have less weightage.
\n So along with accuracy score we also have to look about how accurately we are predicting for each class of disease
I think ROC-AUC score is best fit for imbalance dataset. You can calculate ROC-AUC via sklearn.metrics.roc_auc_score on the test set.
", "rawMarkdown": "I think ROC-AUC score is best fit for imbalance dataset. You can calculate ROC-AUC via sklearn.metrics.roc_auc_score on the test set.", "votes": 2 }, { "id": 1090333, "postDate": "2020-11-25T09:03:38.980Z", "content": "@ihelon Thanks for creating this post on metric Yaroslav . I enjoyed going through the discussion .
", "rawMarkdown": "@ihelon Thanks for creating this post on metric Yaroslav . I enjoyed going through the discussion . ", "votes": 1, "replies": [ { "id": 1095307, "postDate": "2020-11-29T13:17:25.253Z", "content": "@usharengaraju u are welcome)
", "rawMarkdown": "@usharengaraju u are welcome)", "votes": 1 } ] }, { "id": 1087967, "postDate": "2020-11-23T07:47:03.670Z", "content": "Agree. Since at least the public dataset has imbalanced classes (https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/198410 ), other evaluation metrics (e.g. F1) would be suitable for this competition.
", "rawMarkdown": "Agree. Since at least the public dataset has imbalanced classes (https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/198410 ), other evaluation metrics (e.g. F1) would be suitable for this competition.", "votes": 1, "replies": [ { "id": 1088041, "postDate": "2020-11-23T09:08:38.500Z", "content": "Yes you are right
\nI showed this fact first in the notebook
\nhttps://www.kaggle.com/ihelon/cassava-leaf-disease-exploratory-data-analysis
The imbalance in the dataset seems to reflect the proportions one would encounter in a practical scenario, based on domain research, as I discuss in the post linked below, so I think we can safely assume that the private test set is likely to be similar.
\nhttps://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/198584
Good point! I would like to see an answer to this!
", "rawMarkdown": "Good point! I would like to see an answer to this!", "votes": 2, "replies": [ { "id": 1095306, "postDate": "2020-11-29T13:17:00.043Z", "content": "@aldi1t I am waiting too)
", "rawMarkdown": "@aldi1t I am waiting too)" } ] }, { "id": 1088438, "postDate": "2020-11-23T15:52:52.440Z", "content": "I think this is a good point! Seems like competition hosts should reply here.
", "rawMarkdown": "I think this is a good point! Seems like competition hosts should reply here.", "votes": 2, "replies": [ { "id": 1095305, "postDate": "2020-11-29T13:16:16.200Z", "content": "@bayartsogtya agree
", "rawMarkdown": "@bayartsogtya agree" } ] }, { "id": 1087066, "postDate": "2020-11-22T10:03:21.363Z", "content": "The organizer seems too silent on this until now even it's a serious matter.
", "rawMarkdown": "The organizer seems too silent on this until now even it's a serious matter. ", "votes": 2, "replies": [ { "id": 1091939, "postDate": "2020-11-26T11:52:59.730Z", "content": "@juliaelliott Could you please respond to this ongoing discussion? Thanks!
", "rawMarkdown": "@juliaelliott Could you please respond to this ongoing discussion? Thanks!", "votes": 1 }, { "id": 1096875, "postDate": "2020-11-30T20:51:02.307Z", "content": "The competition metric as defined will remain as-is.
", "rawMarkdown": "The competition metric as defined will remain as-is.", "votes": 4 }, { "id": 1096876, "postDate": "2020-11-30T20:54:12.097Z", "content": "@juliaelliott thanks for the clarification!
", "rawMarkdown": "@juliaelliott thanks for the clarification!" } ] }, { "id": 1085927, "postDate": "2020-11-21T09:29:29.340Z", "content": "Mathews Correlation Coefficient or F1-Score can handle the metrics for imbalanced class datasets.
", "rawMarkdown": "Mathews Correlation Coefficient or F1-Score can handle the metrics for imbalanced class datasets.", "votes": 2 }, { "id": 1085753, "postDate": "2020-11-21T07:15:11.210Z", "content": "I would vote for categorical log loss or even micro-f1 score. Accuracy is not a stable indicator, especially when the validation\\testing set is small.
", "rawMarkdown": "I would vote for categorical log loss or even micro-f1 score. Accuracy is not a stable indicator, especially when the validation\\testing set is small.", "votes": 2 }, { "id": 1084807, "postDate": "2020-11-20T12:08:16.370Z", "content": "Yes, as the data is highly imbalanced, only predicting class 3, in this case, would achieve decent accuracy!
\nEdit: Here's the proof: https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/198410
", "rawMarkdown": "Yes, as the data is highly imbalanced, only predicting class 3, in this case, would achieve decent accuracy!\n\nEdit: Here's the proof: https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/198410", "votes": 2 }, { "id": 1084603, "postDate": "2020-11-20T07:48:44.140Z", "content": "For this type of imbalanced data categorical loss is the best metric to evaluate the performance of the model. @ihelon
", "rawMarkdown": "For this type of imbalanced data categorical loss is the best metric to evaluate the performance of the model. @ihelon ", "votes": 2, "replies": [ { "id": 1084660, "postDate": "2020-11-20T09:00:29.887Z", "content": "@akshaychavan123 But to use this metric, they need to change the prediction format - probabilities instead of classes
", "rawMarkdown": "@akshaychavan123 But to use this metric, they need to change the prediction format - probabilities instead of classes", "votes": 1 } ] }, { "id": 1084460, "postDate": "2020-11-20T03:57:16.933Z", "content": "if the host want a category prediction rather than the probability, macro/micro fbeta maybe a good choice
", "rawMarkdown": "if the host want a category prediction rather than the probability, macro/micro fbeta maybe a good choice", "votes": 2, "replies": [ { "id": 1084583, "postDate": "2020-11-20T07:27:22.080Z", "content": "@steamedsheep yes, I think it's a good idea to leave category prediction but change the metric
", "rawMarkdown": "@steamedsheep yes, I think it's a good idea to leave category prediction but change the metric", "votes": 1 } ] }, { "id": 1150967, "postDate": "2021-01-13T02:44:14.533Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1086886, "postDate": "2020-11-22T06:37:32.433Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1084737, "author_name": "Jacopo Repossi", "author_url": "", "post_date": "2020-11-20T10:42:51.350000", "content": "Agree, F1 or (even better) the Matthews Correlation Coefficient might be better alternatives
", "votes": 6, "replies": [] }, { "id": 1087385, "author_name": "DatNT", "author_url": "", "post_date": "2020-11-22T16:55:47.270000", "content": "whatever the metric is, who gets the most accurate predictions wins. so I don't think it matters much
", "votes": 5, "replies": [ { "id": 1087413, "author_name": "Kaushal Shah", "author_url": "", "post_date": "2020-11-22T17:28:05.927000", "content": "@moewie94 That's correct. It doesn't matter to us but assume a scenario that the winner's model is working very well on imbalanced class (which have very high number of data points) but performs average on other classes. This will impact to organizers.
", "votes": 3, "replies": [] }, { "id": 1087694, "author_name": "DatNT", "author_url": "", "post_date": "2020-11-23T02:20:10.700000", "content": "this can even be the way the organizer wants it. they surely aware of this matters when they split the test set (they can just exclude some class 3 samples in the test set to make to test set class balanced) but chose to not do it so it might not be a big problem. furthermore, this can even be the way this problem work in real-life scenario (for example the third class appears most in nature). so just split everything to be balanced is not a good way sometimes.
", "votes": 2, "replies": [] }, { "id": 1088086, "author_name": "vinay", "author_url": "", "post_date": "2020-11-23T10:14:03.263000", "content": "@moewie94 It matters because finally we have to classify all the 4 diseases correctly. As there is a less data for remaining 3 diseases accuracy score will give less weightage to those 3 diseases. Our model wont get enough data to train properly for these 3 diseases compared to the class 3 disease(higher one)
", "votes": 2, "replies": [] }, { "id": 1088338, "author_name": "DatNT", "author_url": "", "post_date": "2020-11-23T14:30:54.187000", "content": "my point is the organizers are fully aware of this matter and they let it happen this way so this imbalance can be their very intention. every organizer has to pay tens of thousands of dollars for kaggle consultant services and every dataset must be analyzed by the kaggle ds team before being accepted for a competition. so i think it's unlikely they are naive enough to not know this class imbalance and how it affects the scoring system.
\nsometimes the test set distribution should be imbalanced like the way they are in real life, not like the way they are in MNIST.
", "votes": 8, "replies": [] }, { "id": 1088344, "author_name": "Yaroslav Isaienkov", "author_url": "", "post_date": "2020-11-23T14:36:12.837000", "content": "@moewie94 I totally agree with you (I don't know why you got so many dislikes)
\nMy goal was to open this discussion to make sure that the metric will not be changed in the middle of the competition.
Agreed! Accuracy is a very bad choice for this dataset. However I've never seen a change of metrics in the middle of a competition.
", "votes": 3, "replies": [ { "id": 1085900, "author_name": "Serigne ", "author_url": "", "post_date": "2020-11-21T09:11:17.543000", "content": "It happened many times (first Jigsaw competition , Ion Switching etc. )
\nFor Jigsaw , the metric was changed about a month after the beginning. But Kaggle extended the deadline thereafter.
", "votes": 5, "replies": [] }, { "id": 1085954, "author_name": "sin", "author_url": "", "post_date": "2020-11-21T09:50:34.197000", "content": "Thanks for the info. I hope this accuracy metrics can change! Otherwise I don't even know which loss function to choose.
", "votes": 2, "replies": [] } ] }, { "id": 1084365, "author_name": "Praveen ", "author_url": "", "post_date": "2020-11-20T01:03:58.410000", "content": "Categorical log loss metric holds good for imbalance classification @ihelon
", "votes": 3, "replies": [] }, { "id": 1085132, "author_name": "jeffalltogether", "author_url": "", "post_date": "2020-11-20T17:35:28.750000", "content": "From my perspective, I am not sure how the competition would improve with a different metric. Accuracy always increases as the number of correct predictions increase, and the competition is won by the submission with the highest number of correct predictions.
", "votes": 2, "replies": [ { "id": 1085135, "author_name": "Yaroslav Isaienkov", "author_url": "", "post_date": "2020-11-20T17:38:26.397000", "content": "@jeffalltogether Yes, your point of view may also be correct. In this case, we need to think over an approach for fitting the imbalance
", "votes": 3, "replies": [] }, { "id": 1088022, "author_name": "vinay", "author_url": "", "post_date": "2020-11-23T08:47:33.197000", "content": "@jeffalltogether Main aim of this competition is to classify these 4 diseases correctly because farmers has to use the fertilizers for that particular disease.
\n As there are more class 3 diseases our model will learn perfectly for class 3 but for remaining class it will fail. As the data for remaining 3 diseases is less, in accuracy score they will have less weightage.
\n So along with accuracy score we also have to look about how accurately we are predicting for each class of disease
I think ROC-AUC score is best fit for imbalance dataset. You can calculate ROC-AUC via sklearn.metrics.roc_auc_score on the test set.
", "votes": 2, "replies": [] }, { "id": 1090333, "author_name": "Tensor Girl", "author_url": "", "post_date": "2020-11-25T09:03:38.980000", "content": "@ihelon Thanks for creating this post on metric Yaroslav . I enjoyed going through the discussion .
", "votes": 1, "replies": [ { "id": 1095307, "author_name": "Yaroslav Isaienkov", "author_url": "", "post_date": "2020-11-29T13:17:25.253000", "content": "@usharengaraju u are welcome)
", "votes": 1, "replies": [] } ] }, { "id": 1087967, "author_name": "tropicbird", "author_url": "", "post_date": "2020-11-23T07:47:03.670000", "content": "Agree. Since at least the public dataset has imbalanced classes (https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/198410 ), other evaluation metrics (e.g. F1) would be suitable for this competition.
", "votes": 1, "replies": [ { "id": 1088041, "author_name": "Yaroslav Isaienkov", "author_url": "", "post_date": "2020-11-23T09:08:38.500000", "content": "Yes you are right
\nI showed this fact first in the notebook
\nhttps://www.kaggle.com/ihelon/cassava-leaf-disease-exploratory-data-analysis
The imbalance in the dataset seems to reflect the proportions one would encounter in a practical scenario, based on domain research, as I discuss in the post linked below, so I think we can safely assume that the private test set is likely to be similar.
\nhttps://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/198584
Good point! I would like to see an answer to this!
", "votes": 2, "replies": [ { "id": 1095306, "author_name": "Yaroslav Isaienkov", "author_url": "", "post_date": "2020-11-29T13:17:00.043000", "content": "@aldi1t I am waiting too)
", "votes": 0, "replies": [] } ] }, { "id": 1088438, "author_name": "Bayartsogt Yadamsuren", "author_url": "", "post_date": "2020-11-23T15:52:52.440000", "content": "I think this is a good point! Seems like competition hosts should reply here.
", "votes": 2, "replies": [ { "id": 1095305, "author_name": "Yaroslav Isaienkov", "author_url": "", "post_date": "2020-11-29T13:16:16.200000", "content": "@bayartsogtya agree
", "votes": 0, "replies": [] } ] }, { "id": 1087066, "author_name": "Innat", "author_url": "", "post_date": "2020-11-22T10:03:21.363000", "content": "The organizer seems too silent on this until now even it's a serious matter.
", "votes": 2, "replies": [ { "id": 1091939, "author_name": "tropicbird", "author_url": "", "post_date": "2020-11-26T11:52:59.730000", "content": "@juliaelliott Could you please respond to this ongoing discussion? Thanks!
", "votes": 1, "replies": [] }, { "id": 1096875, "author_name": "Julia Elliott", "author_url": "", "post_date": "2020-11-30T20:51:02.307000", "content": "The competition metric as defined will remain as-is.
", "votes": 4, "replies": [] }, { "id": 1096876, "author_name": "Yaroslav Isaienkov", "author_url": "", "post_date": "2020-11-30T20:54:12.097000", "content": "@juliaelliott thanks for the clarification!
", "votes": 0, "replies": [] } ] }, { "id": 1085927, "author_name": "Kevin Joseph Scaria", "author_url": "", "post_date": "2020-11-21T09:29:29.340000", "content": "Mathews Correlation Coefficient or F1-Score can handle the metrics for imbalanced class datasets.
", "votes": 2, "replies": [] }, { "id": 1085753, "author_name": "khyeh", "author_url": "", "post_date": "2020-11-21T07:15:11.210000", "content": "I would vote for categorical log loss or even micro-f1 score. Accuracy is not a stable indicator, especially when the validation\\testing set is small.
", "votes": 2, "replies": [] }, { "id": 1084807, "author_name": "Kaushal Shah", "author_url": "", "post_date": "2020-11-20T12:08:16.370000", "content": "Yes, as the data is highly imbalanced, only predicting class 3, in this case, would achieve decent accuracy!
\nEdit: Here's the proof: https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/198410
", "votes": 2, "replies": [] }, { "id": 1084603, "author_name": "Akshay Chavan", "author_url": "", "post_date": "2020-11-20T07:48:44.140000", "content": "For this type of imbalanced data categorical loss is the best metric to evaluate the performance of the model. @ihelon
", "votes": 2, "replies": [ { "id": 1084660, "author_name": "Yaroslav Isaienkov", "author_url": "", "post_date": "2020-11-20T09:00:29.887000", "content": "@akshaychavan123 But to use this metric, they need to change the prediction format - probabilities instead of classes
", "votes": 1, "replies": [] } ] }, { "id": 1084460, "author_name": "sheep", "author_url": "", "post_date": "2020-11-20T03:57:16.933000", "content": "if the host want a category prediction rather than the probability, macro/micro fbeta maybe a good choice
", "votes": 2, "replies": [ { "id": 1084583, "author_name": "Yaroslav Isaienkov", "author_url": "", "post_date": "2020-11-20T07:27:22.080000", "content": "@steamedsheep yes, I think it's a good idea to leave category prediction but change the metric
", "votes": 1, "replies": [] } ] }, { "id": 1150967, "author_name": "", "author_url": "", "post_date": "2021-01-13T02:44:14.533000", "content": "", "votes": 0, "replies": [] }, { "id": 1086886, "author_name": "", "author_url": "", "post_date": "2020-11-22T06:37:32.433000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1084305": "Did you think about using another metric in this competition?\n\nFor example categorical log loss.\n\nBecause I think many players will be able to achieve a very high accuracy score, and for many, this indicator will be equal. And if your dataset has an imbalance, then the accuracy metric is not the best choice for evaluating models.", "1084737": "Agree, F1 or (even better) the Matthews Correlation Coefficient might be better alternatives", "1087385": "whatever the metric is, who gets the most accurate predictions wins. so I don't think it matters much", "1085513": "Agreed! Accuracy is a very bad choice for this dataset. However I've never seen a change of metrics in the middle of a competition. ", "1084365": "Categorical log loss metric holds good for imbalance classification @ihelon ", "1085132": "From my perspective, I am not sure how the competition would improve with a different metric. Accuracy always increases as the number of correct predictions increase, and the competition is won by the submission with the highest number of correct predictions.", "1084459": "I think ROC-AUC score is best fit for imbalance dataset. You can calculate ROC-AUC via sklearn.metrics.roc_auc_score on the test set.", "1090333": "@ihelon Thanks for creating this post on metric Yaroslav . I enjoyed going through the discussion . ", "1087967": "Agree. Since at least the public dataset has imbalanced classes (https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/198410 ), other evaluation metrics (e.g. F1) would be suitable for this competition.", "1089735": "Good point! I would like to see an answer to this!", "1088438": "I think this is a good point! Seems like competition hosts should reply here.", "1087066": "The organizer seems too silent on this until now even it's a serious matter. ", "1085927": "Mathews Correlation Coefficient or F1-Score can handle the metrics for imbalanced class datasets.", "1085753": "I would vote for categorical log loss or even micro-f1 score. Accuracy is not a stable indicator, especially when the validation\\testing set is small.", "1084807": "Yes, as the data is highly imbalanced, only predicting class 3, in this case, would achieve decent accuracy!\n\nEdit: Here's the proof: https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/198410", "1084603": "For this type of imbalanced data categorical loss is the best metric to evaluate the performance of the model. @ihelon ", "1084460": "if the host want a category prediction rather than the probability, macro/micro fbeta maybe a good choice", "1150967": "", "1086886": "" } }