{
  "id": 493417,
  "title": "What about the gini stability metric of your local CV?",
  "url": "/competitions/home-credit-credit-risk-model-stability/discussion/493417",
  "author_name": "",
  "post_date": "2024-04-13T10:31:44.564252500Z",
  "votes": 4,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Many notebooks evaluate offline performance using the AUC metric. However, the official evaluation metric is the gini stability metric.</p>\n<p>I referred to a <a href=\"https://www.kaggle.com/code/hideyukizushi/home-aftersubmissionsopen-3-11-2024-lb-567\" target=\"_blank\">notebook</a> to calculate the offline performance of <a href=\"https://www.kaggle.com/code/bruceqdu/simple-ensemble-with-lgb-and-catboost?scriptVersionId=171799905\" target=\"_blank\">my solution</a>. To my surprise, the Gini stability metric's offline performance was only 0.43 while the online score was 0.58. </p>\n<p>The offline and online scores (0.574 vs 0.567) of the notebook I referred to were quite consistent.  What about your local cv? Additionally, if you find any errors in my solution, please leave a comment.</p>",
  "messages": [
    {
      "id": "2749882",
      "postDate": "04/13/2024 10:31:44",
      "content": "<p>Many notebooks evaluate offline performance using the AUC metric. However, the official evaluation metric is the gini stability metric.</p>\n<p>I referred to a <a href=\"https://www.kaggle.com/code/hideyukizushi/home-aftersubmissionsopen-3-11-2024-lb-567\" target=\"_blank\">notebook</a> to calculate the offline performance of <a href=\"https://www.kaggle.com/code/bruceqdu/simple-ensemble-with-lgb-and-catboost?scriptVersionId=171799905\" target=\"_blank\">my solution</a>. To my surprise, the Gini stability metric's offline performance was only 0.43 while the online score was 0.58. </p>\n<p>The offline and online scores (0.574 vs 0.567) of the notebook I referred to were quite consistent.  What about your local cv? Additionally, if you find any errors in my solution, please leave a comment.</p>",
      "rawMarkdown": "Many notebooks evaluate offline performance using the AUC metric. However, the official evaluation metric is the gini stability metric.\n\nI referred to a [notebook](https://www.kaggle.com/code/hideyukizushi/home-aftersubmissionsopen-3-11-2024-lb-567) to calculate the offline performance of [my solution](https://www.kaggle.com/code/bruceqdu/simple-ensemble-with-lgb-and-catboost?scriptVersionId=171799905). To my surprise, the Gini stability metric's offline performance was only 0.43 while the online score was 0.58. \n\nThe offline and online scores (0.574 vs 0.567) of the notebook I referred to were quite consistent.  What about your local cv? Additionally, if you find any errors in my solution, please leave a comment.",
      "votes": null
    },
    {
      "id": "2750154",
      "postDate": "04/13/2024 13:36:02",
      "content": "<p>I just found the problem. It turns out that only 50,000 samples were used for training before submission.</p>\n<pre><code>sample = pd.read_csv()\ndevice=\n\nn_est=\nDRY_RUN =   sample.shape[] ==      \n DRY_RUN:\n    device=\n    df_train = df_train.iloc[:]\n    \n    n_est=\n(device)\n</code></pre>",
      "rawMarkdown": "I just found the problem. It turns out that only 50,000 samples were used for training before submission.\n\n```python\nsample = pd.read_csv(\"/kaggle/input/home-credit-credit-risk-model-stability/sample_submission.csv\")\ndevice='gpu'\n\nn_est=6000\nDRY_RUN = True if sample.shape[0] == 10 else False   \nif DRY_RUN:\n    device='cpu'\n    df_train = df_train.iloc[:50000]\n    #n_samples=10000\n    n_est=600\nprint(device)\n```",
      "votes": null
    },
    {
      "id": "2750175",
      "postDate": "04/13/2024 13:50:06",
      "content": "<p>Please watch out for the keyword - <strong>DRY_RUN</strong>, this indicates that the developer id checking the code and is not running on the entire data <a href=\"https://www.kaggle.com/bruceqdu\" target=\"_blank\">@bruceqdu</a> </p>",
      "rawMarkdown": "Please watch out for the keyword - **DRY_RUN**, this indicates that the developer id checking the code and is not running on the entire data @bruceqdu",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2750154,
      "author_name": "bruceqdu",
      "author_url": "",
      "post_date": "04/13/2024 13:36:02",
      "content": "<p>I just found the problem. It turns out that only 50,000 samples were used for training before submission.</p>\n<pre><code>sample = pd.read_csv()\ndevice=\n\nn_est=\nDRY_RUN =   sample.shape[] ==      \n DRY_RUN:\n    device=\n    df_train = df_train.iloc[:]\n    \n    n_est=\n(device)\n</code></pre>",
      "votes": null,
      "replies": [
        {
          "id": 2750175,
          "author_name": "ravi20076",
          "author_url": "",
          "post_date": "04/13/2024 13:50:06",
          "content": "<p>Please watch out for the keyword - <strong>DRY_RUN</strong>, this indicates that the developer id checking the code and is not running on the entire data <a href=\"https://www.kaggle.com/bruceqdu\" target=\"_blank\">@bruceqdu</a> </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2749882": "Many notebooks evaluate offline performance using the AUC metric. However, the official evaluation metric is the gini stability metric.\n\nI referred to a [notebook](https://www.kaggle.com/code/hideyukizushi/home-aftersubmissionsopen-3-11-2024-lb-567) to calculate the offline performance of [my solution](https://www.kaggle.com/code/bruceqdu/simple-ensemble-with-lgb-and-catboost?scriptVersionId=171799905). To my surprise, the Gini stability metric's offline performance was only 0.43 while the online score was 0.58. \n\nThe offline and online scores (0.574 vs 0.567) of the notebook I referred to were quite consistent.  What about your local cv? Additionally, if you find any errors in my solution, please leave a comment.",
    "2750154": "I just found the problem. It turns out that only 50,000 samples were used for training before submission.\n\n```python\nsample = pd.read_csv(\"/kaggle/input/home-credit-credit-risk-model-stability/sample_submission.csv\")\ndevice='gpu'\n\nn_est=6000\nDRY_RUN = True if sample.shape[0] == 10 else False   \nif DRY_RUN:\n    device='cpu'\n    df_train = df_train.iloc[:50000]\n    #n_samples=10000\n    n_est=600\nprint(device)\n```",
    "2750175": "Please watch out for the keyword - **DRY_RUN**, this indicates that the developer id checking the code and is not running on the entire data @bruceqdu"
  },
  "source": "meta"
}