{
  "id": 475878,
  "title": "Understanding the gini stability metric",
  "url": "/competitions/home-credit-credit-risk-model-stability/discussion/475878",
  "author_name": "Anthony Chiu",
  "post_date": "2024-02-10T10:12:36.098000",
  "votes": 57,
  "comment_count": 0,
  "views": 0,
  "content": "<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F690886%2F6559e36c2ad0f3036505f549d8c54adc%2Fmetric.jpg?generation=1707559036686140&amp;alt=media\"></p>\n<p>The metric is an (Avg) Gini score that is penalized by<br>\n1) Falling Rate of weekly Gini score<br>\n2) variability of the predictions</p>\n<p>The first penalized term is the slope of the blue line in the diagram above. min(0, a) means we don't care about the positive slope case, which might be too good to be true.</p>\n<p>The second term concerns residuals between the blue line and the weekly score data points. There are 2 ways to calculate it</p>\n<ol>\n<li>The mean of the residuals</li>\n<li>The std of the residuals.</li>\n</ol>\n<p>The second option is chosen here. Also, the metric is using std instead of downside std. This means we also don't want the model performance jumping up again. (upside std: surprisingly improved, downside std: surprisingly worsen.)</p>\n<p>Therefore, we seek a method to reduce the model performance degradation but keep the whole situation predictable (change linearly over time).</p>\n<p>. This post is trying to guess the purpose of these constants<br>\n<a href=\"https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/475118\" target=\"_blank\">https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/475118</a></p>\n<p><strong>Update:</strong></p>\n<p>After plotting the diagram below, I think the metric can be also considered as a projection of the Gini score after 88 more weeks. </p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F690886%2F6e01a070a3b300c57d95b468b0dc672d%2Fmetric%20projection.jpg?generation=1707562905296972&amp;alt=media\"></p>\n<ul>\n<li>Point A above is the first term of the metric</li>\n<li>Point B is the first + second (88 means 88 weeks later)</li>\n<li>Point C is the final (0.5 is cutting the std to half to guess the extra downside)</li>\n</ul>",
  "messages": [
    {
      "id": 2645540,
      "postDate": "2024-02-10T10:12:36.100Z",
      "content": "<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F690886%2F6559e36c2ad0f3036505f549d8c54adc%2Fmetric.jpg?generation=1707559036686140&amp;alt=media\"></p>\n<p>The metric is an (Avg) Gini score that is penalized by<br>\n1) Falling Rate of weekly Gini score<br>\n2) variability of the predictions</p>\n<p>The first penalized term is the slope of the blue line in the diagram above. min(0, a) means we don't care about the positive slope case, which might be too good to be true.</p>\n<p>The second term concerns residuals between the blue line and the weekly score data points. There are 2 ways to calculate it</p>\n<ol>\n<li>The mean of the residuals</li>\n<li>The std of the residuals.</li>\n</ol>\n<p>The second option is chosen here. Also, the metric is using std instead of downside std. This means we also don't want the model performance jumping up again. (upside std: surprisingly improved, downside std: surprisingly worsen.)</p>\n<p>Therefore, we seek a method to reduce the model performance degradation but keep the whole situation predictable (change linearly over time).</p>\n<p>. This post is trying to guess the purpose of these constants<br>\n<a href=\"https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/475118\" target=\"_blank\">https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/475118</a></p>\n<p><strong>Update:</strong></p>\n<p>After plotting the diagram below, I think the metric can be also considered as a projection of the Gini score after 88 more weeks. </p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F690886%2F6e01a070a3b300c57d95b468b0dc672d%2Fmetric%20projection.jpg?generation=1707562905296972&amp;alt=media\"></p>\n<ul>\n<li>Point A above is the first term of the metric</li>\n<li>Point B is the first + second (88 means 88 weeks later)</li>\n<li>Point C is the final (0.5 is cutting the std to half to guess the extra downside)</li>\n</ul>",
      "rawMarkdown": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F690886%2F6559e36c2ad0f3036505f549d8c54adc%2Fmetric.jpg?generation=1707559036686140&alt=media)\n\n\nThe metric is an (Avg) Gini score that is penalized by\n1) Falling Rate of weekly Gini score\n2) variability of the predictions\n\nThe first penalized term is the slope of the blue line in the diagram above. min(0, a) means we don't care about the positive slope case, which might be too good to be true.\n\nThe second term concerns residuals between the blue line and the weekly score data points. There are 2 ways to calculate it\n1. The mean of the residuals\n2. The std of the residuals.\n\nThe second option is chosen here. Also, the metric is using std instead of downside std. This means we also don't want the model performance jumping up again. (upside std: surprisingly improved, downside std: surprisingly worsen.)\n\nTherefore, we seek a method to reduce the model performance degradation but keep the whole situation predictable (change linearly over time).\n\n~~I am unsure why 88 and 0.5 were chosen in the formula~~. This post is trying to guess the purpose of these constants\nhttps://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/475118\n\n**Update:**\n\nAfter plotting the diagram below, I think the metric can be also considered as a projection of the Gini score after 88 more weeks. \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F690886%2F6e01a070a3b300c57d95b468b0dc672d%2Fmetric%20projection.jpg?generation=1707562905296972&alt=media)\n\n- Point A above is the first term of the metric\n- Point B is the first + second (88 means 88 weeks later)\n- Point C is the final (0.5 is cutting the std to half to guess the extra downside)",
      "votes": 56
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "2645540": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F690886%2F6559e36c2ad0f3036505f549d8c54adc%2Fmetric.jpg?generation=1707559036686140&alt=media)\n\n\nThe metric is an (Avg) Gini score that is penalized by\n1) Falling Rate of weekly Gini score\n2) variability of the predictions\n\nThe first penalized term is the slope of the blue line in the diagram above. min(0, a) means we don't care about the positive slope case, which might be too good to be true.\n\nThe second term concerns residuals between the blue line and the weekly score data points. There are 2 ways to calculate it\n1. The mean of the residuals\n2. The std of the residuals.\n\nThe second option is chosen here. Also, the metric is using std instead of downside std. This means we also don't want the model performance jumping up again. (upside std: surprisingly improved, downside std: surprisingly worsen.)\n\nTherefore, we seek a method to reduce the model performance degradation but keep the whole situation predictable (change linearly over time).\n\n~~I am unsure why 88 and 0.5 were chosen in the formula~~. This post is trying to guess the purpose of these constants\nhttps://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/475118\n\n**Update:**\n\nAfter plotting the diagram below, I think the metric can be also considered as a projection of the Gini score after 88 more weeks. \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F690886%2F6e01a070a3b300c57d95b468b0dc672d%2Fmetric%20projection.jpg?generation=1707562905296972&alt=media)\n\n- Point A above is the first term of the metric\n- Point B is the first + second (88 means 88 weeks later)\n- Point C is the final (0.5 is cutting the std to half to guess the extra downside)"
  }
}