{
  "id": 372942,
  "title": "Question about Evaluation Metric",
  "url": "/competitions/nfl-player-contact-detection/discussion/372942",
  "author_name": "",
  "post_date": "2022-12-18T20:33:22.098051500Z",
  "votes": 3,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Hello <a href=\"https://www.kaggle.com/robikscube\" target=\"_blank\">@robikscube</a> and everyone,</p>\n<p>I have created a model for contact detection. However, need clarification regarding the evaluation metric and problem statement with a few suggestions to improve the results thank you.</p>\n<p>Firstly, as we all know our aim is to reduce <strong>Type-1 Error (False Positive)</strong> and <strong>Type-2 Error (False Negative)</strong>. However, I believe in this use case <strong>Precision</strong> matters the most as we need to predict whether there will be contact between the players during ongoing play or before the contact happens right? </p>\n<ul>\n<li><strong>So, do we need more precise predictions or equal recall and precision</strong>?</li>\n</ul>\n<h2>Model Summary</h2>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F84c25f112f1bb94c04dd316c0be7baf8%2Fepoch_nfl_1.png?generation=1671393605648586&amp;alt=media\" alt=\"\"><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F9ddb7a0b9e3e44c966afc180964c6133%2Fepoch_nfl_2.png?generation=1671393616944657&amp;alt=media\" alt=\"\"><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F3a02e8dee97d15b2e3c879e08c545978%2Fepoch_nfl_3.png?generation=1671393635562963&amp;alt=media\" alt=\"\"></p>\n<h4>Model Loss</h4>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2Fd8a521cacd223533342267c286efb431%2Floss.png?generation=1671393997509100&amp;alt=media\" alt=\"\"></p>\n<h4>Model Accuracy</h4>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F829363ce8f9e6ab638023f1d83aa9160%2Facc.png?generation=1671394044734069&amp;alt=media\" alt=\"\"></p>\n<h4>F1-Score</h4>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2Fdea83854b7406eadaa1e52e91175d7c5%2Ff1.png?generation=1671394066474974&amp;alt=media\" alt=\"\"></p>\n<h4>MCC</h4>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F652248aa0e62f33a34932c808c3eca39%2Fmcc_l.png?generation=1671394081049495&amp;alt=media\" alt=\"\"></p>\n<h4>Cohen's Kappa</h4>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2Fd6134b0ae519432bf83d8a7dbcd48d7f%2Fck.png?generation=1671394282184915&amp;alt=media\" alt=\"\"></p>\n<h4>Results</h4>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F77219c34cf1b53edcf5d2c41067e843a%2Fcm_nfl.png?generation=1671394296702685&amp;alt=media\" alt=\"\"><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F08996f67fca928baaee74e05833eac23%2Fcr_nfl.png?generation=1671393939924033&amp;alt=media\" alt=\"\"><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F19cc8b93cfd446ca1ae6802235f3962c%2Fpr_curve_nfl.png?generation=1671394354616170&amp;alt=media\" alt=\"\"><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2Fc535a8c2f12142df5817908a293cae61%2Fmcc_nfl.png?generation=1671394423002403&amp;alt=media\" alt=\"\"></p>",
  "messages": [
    {
      "id": "2069290",
      "postDate": "12/18/2022 20:33:22",
      "content": "<p>Hello <a href=\"https://www.kaggle.com/robikscube\" target=\"_blank\">@robikscube</a> and everyone,</p>\n<p>I have created a model for contact detection. However, need clarification regarding the evaluation metric and problem statement with a few suggestions to improve the results thank you.</p>\n<p>Firstly, as we all know our aim is to reduce <strong>Type-1 Error (False Positive)</strong> and <strong>Type-2 Error (False Negative)</strong>. However, I believe in this use case <strong>Precision</strong> matters the most as we need to predict whether there will be contact between the players during ongoing play or before the contact happens right? </p>\n<ul>\n<li><strong>So, do we need more precise predictions or equal recall and precision</strong>?</li>\n</ul>\n<h2>Model Summary</h2>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F84c25f112f1bb94c04dd316c0be7baf8%2Fepoch_nfl_1.png?generation=1671393605648586&amp;alt=media\" alt=\"\"><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F9ddb7a0b9e3e44c966afc180964c6133%2Fepoch_nfl_2.png?generation=1671393616944657&amp;alt=media\" alt=\"\"><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F3a02e8dee97d15b2e3c879e08c545978%2Fepoch_nfl_3.png?generation=1671393635562963&amp;alt=media\" alt=\"\"></p>\n<h4>Model Loss</h4>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2Fd8a521cacd223533342267c286efb431%2Floss.png?generation=1671393997509100&amp;alt=media\" alt=\"\"></p>\n<h4>Model Accuracy</h4>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F829363ce8f9e6ab638023f1d83aa9160%2Facc.png?generation=1671394044734069&amp;alt=media\" alt=\"\"></p>\n<h4>F1-Score</h4>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2Fdea83854b7406eadaa1e52e91175d7c5%2Ff1.png?generation=1671394066474974&amp;alt=media\" alt=\"\"></p>\n<h4>MCC</h4>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F652248aa0e62f33a34932c808c3eca39%2Fmcc_l.png?generation=1671394081049495&amp;alt=media\" alt=\"\"></p>\n<h4>Cohen's Kappa</h4>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2Fd6134b0ae519432bf83d8a7dbcd48d7f%2Fck.png?generation=1671394282184915&amp;alt=media\" alt=\"\"></p>\n<h4>Results</h4>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F77219c34cf1b53edcf5d2c41067e843a%2Fcm_nfl.png?generation=1671394296702685&amp;alt=media\" alt=\"\"><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F08996f67fca928baaee74e05833eac23%2Fcr_nfl.png?generation=1671393939924033&amp;alt=media\" alt=\"\"><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F19cc8b93cfd446ca1ae6802235f3962c%2Fpr_curve_nfl.png?generation=1671394354616170&amp;alt=media\" alt=\"\"><br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2Fc535a8c2f12142df5817908a293cae61%2Fmcc_nfl.png?generation=1671394423002403&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "Hello @robikscube and everyone,\n\nI have created a model for contact detection. However, need clarification regarding the evaluation metric and problem statement with a few suggestions to improve the results thank you.\n\nFirstly, as we all know our aim is to reduce **Type-1 Error (False Positive)** and **Type-2 Error (False Negative)**. However, I believe in this use case **Precision** matters the most as we need to predict whether there will be contact between the players during ongoing play or before the contact happens right? \n\n- **So, do we need more precise predictions or equal recall and precision**?\n\n## Model Summary\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F84c25f112f1bb94c04dd316c0be7baf8%2Fepoch_nfl_1.png?generation=1671393605648586&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F9ddb7a0b9e3e44c966afc180964c6133%2Fepoch_nfl_2.png?generation=1671393616944657&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F3a02e8dee97d15b2e3c879e08c545978%2Fepoch_nfl_3.png?generation=1671393635562963&alt=media)\n#### Model Loss\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2Fd8a521cacd223533342267c286efb431%2Floss.png?generation=1671393997509100&alt=media)\n#### Model Accuracy\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F829363ce8f9e6ab638023f1d83aa9160%2Facc.png?generation=1671394044734069&alt=media)\n#### F1-Score\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2Fdea83854b7406eadaa1e52e91175d7c5%2Ff1.png?generation=1671394066474974&alt=media)\n#### MCC\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F652248aa0e62f33a34932c808c3eca39%2Fmcc_l.png?generation=1671394081049495&alt=media)\n#### Cohen's Kappa\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2Fd6134b0ae519432bf83d8a7dbcd48d7f%2Fck.png?generation=1671394282184915&alt=media)\n#### Results\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F77219c34cf1b53edcf5d2c41067e843a%2Fcm_nfl.png?generation=1671394296702685&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F08996f67fca928baaee74e05833eac23%2Fcr_nfl.png?generation=1671393939924033&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F19cc8b93cfd446ca1ae6802235f3962c%2Fpr_curve_nfl.png?generation=1671394354616170&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2Fc535a8c2f12142df5817908a293cae61%2Fmcc_nfl.png?generation=1671394423002403&alt=media)",
      "votes": null
    },
    {
      "id": "2069332",
      "postDate": "12/18/2022 21:17:51",
      "content": "<p>Thank you for your question, <a href=\"https://www.kaggle.com/kushtrivedi14728\" target=\"_blank\">@kushtrivedi14728</a>. It seems that you are taking a thorough approach to your work.</p>\n<p>The Matthews correlation coefficient (MCC) is a measure of the quality of binary classification, and the formula for calculating it can be found on Wikipedia. You can also refer to the raw code used by the sklearn library to compute MCC, which should be the same as how the leaderboard calculates it.</p>\n<ul>\n<li>Wiki:  <a href=\"https://en.wikipedia.org/wiki/Phi_coefficient\" target=\"_blank\">https://en.wikipedia.org/wiki/Phi_coefficient</a></li>\n<li>Sklearn: <a href=\"https://github.com/scikit-learn/scikit-learn/blob/dc580a8ef/sklearn/metrics/_classification.py#L848\" target=\"_blank\">https://github.com/scikit-learn/scikit-learn/blob/dc580a8ef/sklearn/metrics/_classification.py#L848</a></li>\n</ul>\n<p>As for optimizing for this metric, I can’t provide much insight as that’s really up to you. Looks like others have posted some references on the metric that you might find helpful. Good luck!</p>",
      "rawMarkdown": "Thank you for your question, @kushtrivedi14728. It seems that you are taking a thorough approach to your work.\n\nThe Matthews correlation coefficient (MCC) is a measure of the quality of binary classification, and the formula for calculating it can be found on Wikipedia. You can also refer to the raw code used by the sklearn library to compute MCC, which should be the same as how the leaderboard calculates it.\n- Wiki:  https://en.wikipedia.org/wiki/Phi_coefficient\n- Sklearn: https://github.com/scikit-learn/scikit-learn/blob/dc580a8ef/sklearn/metrics/_classification.py#L848\n\nAs for optimizing for this metric, I can’t provide much insight as that’s really up to you. Looks like others have posted some references on the metric that you might find helpful. Good luck!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2069332,
      "author_name": "robikscube",
      "author_url": "",
      "post_date": "12/18/2022 21:17:51",
      "content": "<p>Thank you for your question, <a href=\"https://www.kaggle.com/kushtrivedi14728\" target=\"_blank\">@kushtrivedi14728</a>. It seems that you are taking a thorough approach to your work.</p>\n<p>The Matthews correlation coefficient (MCC) is a measure of the quality of binary classification, and the formula for calculating it can be found on Wikipedia. You can also refer to the raw code used by the sklearn library to compute MCC, which should be the same as how the leaderboard calculates it.</p>\n<ul>\n<li>Wiki:  <a href=\"https://en.wikipedia.org/wiki/Phi_coefficient\" target=\"_blank\">https://en.wikipedia.org/wiki/Phi_coefficient</a></li>\n<li>Sklearn: <a href=\"https://github.com/scikit-learn/scikit-learn/blob/dc580a8ef/sklearn/metrics/_classification.py#L848\" target=\"_blank\">https://github.com/scikit-learn/scikit-learn/blob/dc580a8ef/sklearn/metrics/_classification.py#L848</a></li>\n</ul>\n<p>As for optimizing for this metric, I can’t provide much insight as that’s really up to you. Looks like others have posted some references on the metric that you might find helpful. Good luck!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2069290": "Hello @robikscube and everyone,\n\nI have created a model for contact detection. However, need clarification regarding the evaluation metric and problem statement with a few suggestions to improve the results thank you.\n\nFirstly, as we all know our aim is to reduce **Type-1 Error (False Positive)** and **Type-2 Error (False Negative)**. However, I believe in this use case **Precision** matters the most as we need to predict whether there will be contact between the players during ongoing play or before the contact happens right? \n\n- **So, do we need more precise predictions or equal recall and precision**?\n\n## Model Summary\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F84c25f112f1bb94c04dd316c0be7baf8%2Fepoch_nfl_1.png?generation=1671393605648586&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F9ddb7a0b9e3e44c966afc180964c6133%2Fepoch_nfl_2.png?generation=1671393616944657&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F3a02e8dee97d15b2e3c879e08c545978%2Fepoch_nfl_3.png?generation=1671393635562963&alt=media)\n#### Model Loss\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2Fd8a521cacd223533342267c286efb431%2Floss.png?generation=1671393997509100&alt=media)\n#### Model Accuracy\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F829363ce8f9e6ab638023f1d83aa9160%2Facc.png?generation=1671394044734069&alt=media)\n#### F1-Score\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2Fdea83854b7406eadaa1e52e91175d7c5%2Ff1.png?generation=1671394066474974&alt=media)\n#### MCC\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F652248aa0e62f33a34932c808c3eca39%2Fmcc_l.png?generation=1671394081049495&alt=media)\n#### Cohen's Kappa\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2Fd6134b0ae519432bf83d8a7dbcd48d7f%2Fck.png?generation=1671394282184915&alt=media)\n#### Results\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F77219c34cf1b53edcf5d2c41067e843a%2Fcm_nfl.png?generation=1671394296702685&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F08996f67fca928baaee74e05833eac23%2Fcr_nfl.png?generation=1671393939924033&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2F19cc8b93cfd446ca1ae6802235f3962c%2Fpr_curve_nfl.png?generation=1671394354616170&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8412510%2Fc535a8c2f12142df5817908a293cae61%2Fmcc_nfl.png?generation=1671394423002403&alt=media)",
    "2069332": "Thank you for your question, @kushtrivedi14728. It seems that you are taking a thorough approach to your work.\n\nThe Matthews correlation coefficient (MCC) is a measure of the quality of binary classification, and the formula for calculating it can be found on Wikipedia. You can also refer to the raw code used by the sklearn library to compute MCC, which should be the same as how the leaderboard calculates it.\n- Wiki:  https://en.wikipedia.org/wiki/Phi_coefficient\n- Sklearn: https://github.com/scikit-learn/scikit-learn/blob/dc580a8ef/sklearn/metrics/_classification.py#L848\n\nAs for optimizing for this metric, I can’t provide much insight as that’s really up to you. Looks like others have posted some references on the metric that you might find helpful. Good luck!"
  },
  "source": "meta"
}