Code for Python users (if you are not sure how exactly this metric is calculated this code is a good place to start): https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/quadratic_weighted_kappa.py
Here is an illustration of how this metric works broken into steps: https://www.kaggle.com/aroraaman/quadratic-kappa-metric-explained-in-5-simple-steps
Quadratic Kappa Metric is the same as cohen kappa metric in Sci-kit learn sklearn.metrics.cohen_kappa_score when weights are set to 'Quadratic'.
Good Wikipedia article on Coheh's kappa with illustrative examples: https://en.wikipedia.org/wiki/Cohen%27s_kappa
Code for Python users (if you are not sure how exactly this metric is calculated this code is a good place to start): https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/quadratic_weighted_kappa.py
Here is an illustration of how this metric works broken into steps: https://www.kaggle.com/aroraaman/quadratic-kappa-metric-explained-in-5-simple-steps
Quadratic Kappa Metric is the same as cohen kappa metric in Sci-kit learn sklearn.metrics.cohen_kappa_score when weights are set to 'Quadratic'.
Good Wikipedia article on Coheh's kappa with illustrative examples: https://en.wikipedia.org/wiki/Cohen%27s_kappa
This metric was used in PetFinder.my Adoption Prediction held until March 2019. There are some discussions for the metric.
", "rawMarkdown": "This metric was used in [PetFinder.my Adoption Prediction](https://www.kaggle.com/c/petfinder-adoption-prediction) held until March 2019. There are some discussions for the metric.", "votes": null }, { "id": "563244", "postDate": "06/28/2019 04:01:46", "content": "For a while ,i was curious how competition submission score are generated, never thought that to be a 6 steps complicated procedure at play to generate my final submission score , Thanks for sharing.
", "rawMarkdown": "For a while ,i was curious how competition submission score are generated, never thought that to be a 6 steps complicated procedure at play to generate my final submission score , Thanks for sharing.", "votes": null }, { "id": "563246", "postDate": "06/28/2019 04:04:26", "content": "It depends on competition. For this one the metric is quite involved -- it make sense to go through the code available at the first link above just to make sure that you understand how it works.
", "rawMarkdown": "It depends on competition. For this one the metric is quite involved -- it make sense to go through the code available at the first link above just to make sure that you understand how it works.", "votes": null }, { "id": "563728", "postDate": "06/28/2019 15:59:19", "content": "fast.ai have in-built support for kappa metric. Choose quadratic weight type.
", "rawMarkdown": "fast.ai have in-built support for kappa metric. Choose quadratic weight type.", "votes": null }, { "id": "565194", "postDate": "06/30/2019 15:04:26", "content": "Very useful, thanks!
", "rawMarkdown": "Very useful, thanks!", "votes": null }, { "id": "565363", "postDate": "06/30/2019 20:20:24", "content": "Thank you very much!
", "rawMarkdown": "Thank you very much!", "votes": null }, { "id": "568849", "postDate": "07/05/2019 14:40:22", "content": "Thank you
", "rawMarkdown": "Thank you", "votes": null }, { "id": "577129", "postDate": "07/16/2019 11:48:34", "content": "I am using the code that is provided here in 1 and I get kappa scores like 0.67 etc on my validation data. However, when I submit my response on the competition, the score is calculated to be 0.0000.\nDoes that make sense? could I be making a rookie mistake?
", "rawMarkdown": "I am using the code that is provided here in 1 and I get kappa scores like 0.67 etc on my validation data. However, when I submit my response on the competition, the score is calculated to be 0.0000.\nDoes that make sense? could I be making a rookie mistake?", "votes": null }, { "id": "579681", "postDate": "07/19/2019 05:32:29", "content": "I use sklearn.metrics.cohen_kappa_score .
But I have a question. When you train the model and compute cohenkappascore, have you ever met the situation of follows:
\n\nRuntimeWarning: invalid value encountered in true_divide k = np.sum(w_mat * confusion) / np.sum(w_mat * expected)
\n\nI used same code as yours:
\n\na = target.data.cpu().numpy() b = output.detach().cpu().numpy() train_ck_score.append(cohen_k_score(a, b))
\n\nDo not understand why there is nan in the denominator. Any suggestions?
", "rawMarkdown": "I use `sklearn.metrics.cohen_kappa_score` .\n\nBut I have a question. When you train the model and compute cohenkappascore, have you ever met the situation of follows:\n\nRuntimeWarning: invalid value encountered in true_divide k = np.sum(w_mat * confusion) / np.sum(w_mat * expected)\n\nI used same code as yours:\n\na = target.data.cpu().numpy() b = output.detach().cpu().numpy() train_ck_score.append(cohen_k_score(a, b))\n\nDo not understand why there is nan in the denominator. Any suggestions?", "votes": null }, { "id": "891386", "postDate": "06/18/2020 06:35:23", "content": "@xiaojiu1414 have you figured out what was wrong, i am also getting same error.
", "rawMarkdown": "xiaojiu1414 have you figured out what was wrong, i am also getting same error.", "votes": null } ], "comments": [ { "id": 563180, "author_name": "sishihara", "author_url": "", "post_date": "06/28/2019 02:03:53", "content": "This metric was used in PetFinder.my Adoption Prediction held until March 2019. There are some discussions for the metric.
", "votes": null, "replies": [] }, { "id": 563244, "author_name": "gunishj", "author_url": "", "post_date": "06/28/2019 04:01:46", "content": "For a while ,i was curious how competition submission score are generated, never thought that to be a 6 steps complicated procedure at play to generate my final submission score , Thanks for sharing.
", "votes": null, "replies": [ { "id": 563246, "author_name": "graf10a", "author_url": "", "post_date": "06/28/2019 04:04:26", "content": "It depends on competition. For this one the metric is quite involved -- it make sense to go through the code available at the first link above just to make sure that you understand how it works.
", "votes": null, "replies": [] } ] }, { "id": 563728, "author_name": "mktripathi456", "author_url": "", "post_date": "06/28/2019 15:59:19", "content": "fast.ai have in-built support for kappa metric. Choose quadratic weight type.
", "votes": null, "replies": [] }, { "id": 565194, "author_name": "dimitreoliveira", "author_url": "", "post_date": "06/30/2019 15:04:26", "content": "Very useful, thanks!
", "votes": null, "replies": [] }, { "id": 565363, "author_name": "andreylitvinchuk", "author_url": "", "post_date": "06/30/2019 20:20:24", "content": "Thank you very much!
", "votes": null, "replies": [] }, { "id": 568849, "author_name": "jyshin", "author_url": "", "post_date": "07/05/2019 14:40:22", "content": "Thank you
", "votes": null, "replies": [] }, { "id": 577129, "author_name": "mittalneha", "author_url": "", "post_date": "07/16/2019 11:48:34", "content": "I am using the code that is provided here in 1 and I get kappa scores like 0.67 etc on my validation data. However, when I submit my response on the competition, the score is calculated to be 0.0000.\nDoes that make sense? could I be making a rookie mistake?
", "votes": null, "replies": [] }, { "id": 579681, "author_name": "xiaojiu1414", "author_url": "", "post_date": "07/19/2019 05:32:29", "content": "I use sklearn.metrics.cohen_kappa_score .
But I have a question. When you train the model and compute cohenkappascore, have you ever met the situation of follows:
\n\nRuntimeWarning: invalid value encountered in true_divide k = np.sum(w_mat * confusion) / np.sum(w_mat * expected)
\n\nI used same code as yours:
\n\na = target.data.cpu().numpy() b = output.detach().cpu().numpy() train_ck_score.append(cohen_k_score(a, b))
\n\nDo not understand why there is nan in the denominator. Any suggestions?
", "votes": null, "replies": [ { "id": 891386, "author_name": "rohitsingh9990", "author_url": "", "post_date": "06/18/2020 06:35:23", "content": "@xiaojiu1414 have you figured out what was wrong, i am also getting same error.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "563173": "1. Code for Python users (if you are not sure how exactly this metric is calculated this code is a good place to start): https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/quadratic_weighted_kappa.py\n\n2. Here is an illustration of how this metric works broken into steps: https://www.kaggle.com/aroraaman/quadratic-kappa-metric-explained-in-5-simple-steps\n\n3. Quadratic Kappa Metric is the same as cohen kappa metric in Sci-kit learn `sklearn.metrics.cohen_kappa_score` when `weights` are set to `'Quadratic'`. \n\n4. Good Wikipedia article on Coheh's kappa with illustrative examples: https://en.wikipedia.org/wiki/Cohen%27s_kappa", "563180": "This metric was used in [PetFinder.my Adoption Prediction](https://www.kaggle.com/c/petfinder-adoption-prediction) held until March 2019. There are some discussions for the metric.", "563244": "For a while ,i was curious how competition submission score are generated, never thought that to be a 6 steps complicated procedure at play to generate my final submission score , Thanks for sharing.", "563246": "It depends on competition. For this one the metric is quite involved -- it make sense to go through the code available at the first link above just to make sure that you understand how it works.", "563728": "fast.ai have in-built support for kappa metric. Choose quadratic weight type.", "565194": "Very useful, thanks!", "565363": "Thank you very much!", "568849": "Thank you", "577129": "I am using the code that is provided here in 1 and I get kappa scores like 0.67 etc on my validation data. However, when I submit my response on the competition, the score is calculated to be 0.0000.\nDoes that make sense? could I be making a rookie mistake?", "579681": "I use `sklearn.metrics.cohen_kappa_score` .\n\nBut I have a question. When you train the model and compute cohenkappascore, have you ever met the situation of follows:\n\nRuntimeWarning: invalid value encountered in true_divide k = np.sum(w_mat * confusion) / np.sum(w_mat * expected)\n\nI used same code as yours:\n\na = target.data.cpu().numpy() b = output.detach().cpu().numpy() train_ck_score.append(cohen_k_score(a, b))\n\nDo not understand why there is nan in the denominator. Any suggestions?", "891386": "xiaojiu1414 have you figured out what was wrong, i am also getting same error." }, "source": "meta" }