{
  "id": 491242,
  "title": "Why is the score on the leaderboard so low?",
  "url": "/competitions/home-credit-credit-risk-model-stability/discussion/491242",
  "author_name": "",
  "post_date": "2024-04-05T05:49:58.531209900Z",
  "votes": 1,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I have 1 million applications for the training sample and 500,000 for the test sample. I received many different models with an AUC on a test sample of 83-86%. When I upload it to submission, I get 15-25%. I don't understand the connection between the AUC on my 500,000 clients and what the submission gives out. Why do I get such a bad result? Help me please</p>",
  "messages": [
    {
      "id": "2736280",
      "postDate": "04/05/2024 05:49:58",
      "content": "<p>I have 1 million applications for the training sample and 500,000 for the test sample. I received many different models with an AUC on a test sample of 83-86%. When I upload it to submission, I get 15-25%. I don't understand the connection between the AUC on my 500,000 clients and what the submission gives out. Why do I get such a bad result? Help me please</p>",
      "rawMarkdown": "I have 1 million applications for the training sample and 500,000 for the test sample. I received many different models with an AUC on a test sample of 83-86%. When I upload it to submission, I get 15-25%. I don't understand the connection between the AUC on my 500,000 clients and what the submission gives out. Why do I get such a bad result? Help me please",
      "votes": null
    },
    {
      "id": "2736325",
      "postDate": "04/05/2024 06:18:27",
      "content": "<p>The metric is the stability index, your model may have a good GINi score at the start but it may fall off later causing the fall in the metric. This metric is perhaps quite difficult to optimize in my opinion. I suggest you could incorporate it in the training process to perhaps assess if your base model is stable and then submit to the leaderboard.<br>\nThis is a far more difficult challenge than it's initial appearance, wishing you the best <a href=\"https://www.kaggle.com/rinatbayanov\" target=\"_blank\">@rinatbayanov</a> </p>",
      "rawMarkdown": "The metric is the stability index, your model may have a good GINi score at the start but it may fall off later causing the fall in the metric. This metric is perhaps quite difficult to optimize in my opinion. I suggest you could incorporate it in the training process to perhaps assess if your base model is stable and then submit to the leaderboard.\nThis is a far more difficult challenge than it's initial appearance, wishing you the best @rinatbayanov",
      "votes": null
    },
    {
      "id": "2736930",
      "postDate": "04/05/2024 13:31:59",
      "content": "<p>Why did you upset me? I want to cry</p>\n<p>I'm kidding, thanks!</p>",
      "rawMarkdown": "Why did you upset me? I want to cry\n\nI'm kidding, thanks!",
      "votes": null
    },
    {
      "id": "2768973",
      "postDate": "04/23/2024 05:27:18",
      "content": "<p>dude, its okay. Keep reading good solutions and discussions, you'll get there</p>",
      "rawMarkdown": "dude, its okay. Keep reading good solutions and discussions, you'll get there",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2736325,
      "author_name": "ravi20076",
      "author_url": "",
      "post_date": "04/05/2024 06:18:27",
      "content": "<p>The metric is the stability index, your model may have a good GINi score at the start but it may fall off later causing the fall in the metric. This metric is perhaps quite difficult to optimize in my opinion. I suggest you could incorporate it in the training process to perhaps assess if your base model is stable and then submit to the leaderboard.<br>\nThis is a far more difficult challenge than it's initial appearance, wishing you the best <a href=\"https://www.kaggle.com/rinatbayanov\" target=\"_blank\">@rinatbayanov</a> </p>",
      "votes": null,
      "replies": [
        {
          "id": 2736930,
          "author_name": "rinatbayanov",
          "author_url": "",
          "post_date": "04/05/2024 13:31:59",
          "content": "<p>Why did you upset me? I want to cry</p>\n<p>I'm kidding, thanks!</p>",
          "votes": null,
          "replies": [
            {
              "id": 2768973,
              "author_name": "luciferisback",
              "author_url": "",
              "post_date": "04/23/2024 05:27:18",
              "content": "<p>dude, its okay. Keep reading good solutions and discussions, you'll get there</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2736280": "I have 1 million applications for the training sample and 500,000 for the test sample. I received many different models with an AUC on a test sample of 83-86%. When I upload it to submission, I get 15-25%. I don't understand the connection between the AUC on my 500,000 clients and what the submission gives out. Why do I get such a bad result? Help me please",
    "2736325": "The metric is the stability index, your model may have a good GINi score at the start but it may fall off later causing the fall in the metric. This metric is perhaps quite difficult to optimize in my opinion. I suggest you could incorporate it in the training process to perhaps assess if your base model is stable and then submit to the leaderboard.\nThis is a far more difficult challenge than it's initial appearance, wishing you the best @rinatbayanov",
    "2736930": "Why did you upset me? I want to cry\n\nI'm kidding, thanks!",
    "2768973": "dude, its okay. Keep reading good solutions and discussions, you'll get there"
  },
  "source": "meta"
}