![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F4022661%2F8c10d50c070290bc6f9ee0f978d35340%2FROC_PR.PNG?generation=1596605674284506&alt=media\")
Has anyone tried looking at their model's precision and recall metrics? For my model that is trained with CV 3 folds, it got a good out-of-fold AUROC score but poor average-precision score. Here is a plot:
\n
Given that this competition's training set has a very high class imbalance (way more benign than malignant images), AUROC will tend to give an over-optimistic view of the model performance. This is because:
\n\n\nFor example, a big improvement in the number of false positives only leads to a small change in the false positive rate when using ROC. Precision, on the other hand, by comparing false positives to true positives rather than true negatives, captures the effect of the large number of negative examples on the model’s performance.
\n
Here is another example from a website showing how an imbalanced data set gives a much better ROC curve compared to a balanced set while the PRC is not as much affected:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4022661%2Fcf53fccc821d365fbe2a02da57f69eeb%2FAnnotation%202020-08-16%20103704.png?generation=1597545746676343&alt=media\")
This paper titled The Relationship Between Precision-Recall and ROC Curves provides a great explanation of the relationship.
\nMore resources here: The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets.
So I am wondering why the competition is not using another metric for evaluation such as average precision score. Anyone have any opinions?
\nI would also encourage those who are interested in making their model more robust or 'useful' to use the AUPRC as a supplement to the AUROC to get a more comprehensive picture.
\nIn my quest to get a higher LB AUROC score (this is my first Kaggle competition), which is now ~0.95, I see my AURPC getting worse. Model was trained on EffNet B2 with 5 fold CV:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4022661%2Fd4781c944060a8b63678ebac87f9df51%2FAnnotation%202020-08-16%20104654.png?generation=1597546147873537&alt=media\")
Which makes me wonder if I have done anything useful at all as this latest model would not even do well in future predictions despite having a high AUROC score.
", "messages": [ { "id": "958718", "postDate": "08/05/2020 05:52:49", "content": "Has anyone tried looking at their model's precision and recall metrics? For my model that is trained with CV 3 folds, it got a good out-of-fold AUROC score but poor average-precision score. Here is a plot:
\n
Given that this competition's training set has a very high class imbalance (way more benign than malignant images), AUROC will tend to give an over-optimistic view of the model performance. This is because:
\n\n\nFor example, a big improvement in the number of false positives only leads to a small change in the false positive rate when using ROC. Precision, on the other hand, by comparing false positives to true positives rather than true negatives, captures the effect of the large number of negative examples on the model’s performance.
\n
Here is another example from a website showing how an imbalanced data set gives a much better ROC curve compared to a balanced set while the PRC is not as much affected:
\n
This paper titled The Relationship Between Precision-Recall and ROC Curves provides a great explanation of the relationship.
\nMore resources here: The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets.
So I am wondering why the competition is not using another metric for evaluation such as average precision score. Anyone have any opinions?
\nI would also encourage those who are interested in making their model more robust or 'useful' to use the AUPRC as a supplement to the AUROC to get a more comprehensive picture.
\nIn my quest to get a higher LB AUROC score (this is my first Kaggle competition), which is now ~0.95, I see my AURPC getting worse. Model was trained on EffNet B2 with 5 fold CV:
\n
Which makes me wonder if I have done anything useful at all as this latest model would not even do well in future predictions despite having a high AUROC score.
", "rawMarkdown": "Has anyone tried looking at their model's precision and recall metrics? For my model that is trained with CV 3 folds, it got a good out-of-fold AUROC score but poor average-precision score. Here is a plot: \n\n\n\nGiven that this competition's training set has a very high class imbalance (way more benign than malignant images), AUROC will tend to give an over-optimistic view of the model performance. This is because:\n\n> For example, a big improvement in the number of false positives only leads to a small change in the false positive rate when using ROC. Precision, on the other hand, by comparing false positives to true positives rather than true negatives, captures the effect of the large number of negative examples on the model’s performance.\n\nHere is another example from a website showing how an imbalanced data set gives a much better ROC curve compared to a balanced set while the PRC is not as much affected:\n\n\n\nThis paper titled [The Relationship Between Precision-Recall and ROC Curves](http://pages.cs.wisc.edu/~jdavis/davisgoadrichcamera2.pdf) provides a great explanation of the relationship.\nMore resources here: [The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0118432).\n\nSo I am wondering why the competition is not using another metric for evaluation such as average precision score. Anyone have any opinions?\n\nI would also encourage those who are interested in making their model more robust or 'useful' to use the AUPRC as a supplement to the AUROC to get a more comprehensive picture.\n\n## UPDATE:\n\nIn my quest to get a higher LB AUROC score (this is my first Kaggle competition), which is now ~0.95, I see my AURPC getting worse. Model was trained on EffNet B2 with 5 fold CV:\n\n\nWhich makes me wonder if I have done anything useful at all as this latest model would not even do well in future predictions despite having a high AUROC score.", "votes": null }, { "id": "970939", "postDate": "08/15/2020 02:52:19", "content": "I agree, I'm surprised AUROC is the evaluation metric in this competition considering the primary challenge is the small number of positive observations and the very high class imbalance. I'd rather see an evaluation metric that emphasizes the importance of rare observations and I'd like to hear what other people think about this.
\nThe problem with AUROC:
\nThe ROC space includes true negatives which becomes swamped by the large portion of true negatives in the dataset.
ROC space\nTPR = TP / TP + FN #Aka Recall\nFPR = FP / FP + TN\nAUPRC is more informative:
\nAUPRC is a better metric in this case as it does not include true negatives. The False Positive Rate (FPR) is replaced with Precision, measuring instead the fraction of true positives over positive predictions.
PR Space\nPrecision = TP / TP + FP\nRecall = TP / TP + FN #Aka TPR\nOne challenge with using AUPRC is the interpretation is not as easy as AUROC. Where the random baseline for AUROC is 0.5. The baseline for AUPRC is # positive examples / total # examples, which in this case is ~1.8% malignant images so a AURPC of 0.73 is pretty good.
\nSources:
\nThe Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets
\nMeasuring Performance: AUPRC and Average Precision
Thanks for the insights. Using my above CV AUROC, I got around 0.92 on the LB. However, as I improved on my LB AUROC (~0.95), which everyone else is doing at the end of the competition, my AUPRC is dropping way too low (~0.3) and I am just wondering if there is any point in the model (its pretty much a useless model although it gets a high AUROC).
\nSo I would encourage people to check on both their AUROC and AUPRC to gauge their model's 'usefulness' in the real world.
", "rawMarkdown": "Thanks for the insights. Using my above CV AUROC, I got around 0.92 on the LB. However, as I improved on my LB AUROC (~0.95), which everyone else is doing at the end of the competition, my AUPRC is dropping way too low (~0.3) and I am just wondering if there is any point in the model (its pretty much a useless model although it gets a high AUROC).\n\nSo I would encourage people to check on both their AUROC and AUPRC to gauge their model's 'usefulness' in the real world.", "votes": null }, { "id": "1051218", "postDate": "10/16/2020 09:12:54", "content": "Great post.
\nUnfortunately , not many discussed this issue . All were about get the high LB.
\nI was thinking the same going through all the post . AUPRC makes more in real world than AUROC.
\nIt is very hard to build good model with high AUPRC compared AUROC.
I agree, I'm surprised AUROC is the evaluation metric in this competition considering the primary challenge is the small number of positive observations and the very high class imbalance. I'd rather see an evaluation metric that emphasizes the importance of rare observations and I'd like to hear what other people think about this.
\nThe problem with AUROC:
\nThe ROC space includes true negatives which becomes swamped by the large portion of true negatives in the dataset.
ROC space\nTPR = TP / TP + FN #Aka Recall\nFPR = FP / FP + TN\nAUPRC is more informative:
\nAUPRC is a better metric in this case as it does not include true negatives. The False Positive Rate (FPR) is replaced with Precision, measuring instead the fraction of true positives over positive predictions.
PR Space\nPrecision = TP / TP + FP\nRecall = TP / TP + FN #Aka TPR\nOne challenge with using AUPRC is the interpretation is not as easy as AUROC. Where the random baseline for AUROC is 0.5. The baseline for AUPRC is # positive examples / total # examples, which in this case is ~1.8% malignant images so a AURPC of 0.73 is pretty good.
\nSources:
\nThe Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets
\nMeasuring Performance: AUPRC and Average Precision
Thanks for the insights. Using my above CV AUROC, I got around 0.92 on the LB. However, as I improved on my LB AUROC (~0.95), which everyone else is doing at the end of the competition, my AUPRC is dropping way too low (~0.3) and I am just wondering if there is any point in the model (its pretty much a useless model although it gets a high AUROC).
\nSo I would encourage people to check on both their AUROC and AUPRC to gauge their model's 'usefulness' in the real world.
", "votes": null, "replies": [] }, { "id": 1051218, "author_name": "anthonyleo86", "author_url": "", "post_date": "10/16/2020 09:12:54", "content": "Great post.
\nUnfortunately , not many discussed this issue . All were about get the high LB.
\nI was thinking the same going through all the post . AUPRC makes more in real world than AUROC.
\nIt is very hard to build good model with high AUPRC compared AUROC.