I trained cat and lgb and got a cv score of 0.78, then I used stacking and got a cv score of 0.93 (I was surprised why it was so high), but the lb score was negative.
", "messages": [ { "id": 2796853, "postDate": "2024-05-06T12:36:26.860Z", "content": "It could be that the even though the AUC score is high, but the stability of the model is bad. Note that we have the term 88.0⋅𝑚𝑖𝑛(0,𝑎) (negative a would have large negative impact for the final score) for the stability metrics, which is used for the LB score after all.
During the validation stage, you can not only calculate the cv score, but also the stability score according to the evaluation, to inspect the stability of your model.
\ne.g.,
\n numpy np\n sklearn.linear_model LinearRegression\n sklearn.metrics roc_auc_score\n\n ():\n weeks = ((gini_scores))\n weeks = np.array(weeks).reshape(-, )\n gini_scores = np.array(gini_scores)\n\n \n model = LinearRegression()\n model.fit(weeks, gini_scores)\n\n \n falling_rate = (, model.coef_[])\n\n \n residuals = gini_scores - model.predict(weeks)\n std_residuals = np.std(residuals)\n\n \n stability_metric = np.mean(gini_scores) + * falling_rate - * std_residuals\n\n stability_metric\nI trained cat and lgb and got a cv score of 0.78, then I used stacking and got a cv score of 0.93 (I was surprised why it was so high), but the lb score was negative.
", "rawMarkdown": "I trained cat and lgb and got a cv score of 0.78, then I used stacking and got a cv score of 0.93 (I was surprised why it was so high), but the lb score was negative.", "votes": 1 }, { "id": 2803499, "postDate": "2024-05-09T14:32:37.373Z", "content": "If use stacking, make sure there is no leakage in training process.
", "rawMarkdown": "If use stacking, make sure there is no leakage in training process.", "votes": 2 }, { "id": 2795812, "postDate": "2024-05-06T02:11:38.937Z", "content": "Perhaps your model's performance dropped significantly and a negative “a” will cause cliff fall on lb
", "rawMarkdown": "Perhaps your model's performance dropped significantly and a negative “a” will cause cliff fall on lb", "votes": 2, "isDeleted": true } ], "comments": [ { "id": 2796853, "author_name": "faithk7u", "author_url": "", "post_date": "2024-05-06T12:36:26.860000", "content": "It could be that the even though the AUC score is high, but the stability of the model is bad. Note that we have the term 88.0⋅𝑚𝑖𝑛(0,𝑎) (negative a would have large negative impact for the final score) for the stability metrics, which is used for the LB score after all.
During the validation stage, you can not only calculate the cv score, but also the stability score according to the evaluation, to inspect the stability of your model.
\ne.g.,
\n numpy np\n sklearn.linear_model LinearRegression\n sklearn.metrics roc_auc_score\n\n ():\n weeks = ((gini_scores))\n weeks = np.array(weeks).reshape(-, )\n gini_scores = np.array(gini_scores)\n\n \n model = LinearRegression()\n model.fit(weeks, gini_scores)\n\n \n falling_rate = (, model.coef_[])\n\n \n residuals = gini_scores - model.predict(weeks)\n std_residuals = np.std(residuals)\n\n \n stability_metric = np.mean(gini_scores) + * falling_rate - * std_residuals\n\n stability_metric\nIf use stacking, make sure there is no leakage in training process.
", "votes": 2, "replies": [] }, { "id": 2795812, "author_name": "", "author_url": "", "post_date": "2024-05-06T02:11:38.937000", "content": "Perhaps your model's performance dropped significantly and a negative “a” will cause cliff fall on lb
", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "2796853": "It could be that the even though the AUC score is high, but the stability of the model is bad. Note that we have the term `88.0⋅𝑚𝑖𝑛(0,𝑎)` (negative `a` would have large negative impact for the final score) for the stability metrics, which is used for the LB score after all.\n\nDuring the validation stage, you can not only calculate the cv score, but also the stability score according to the evaluation, to inspect the stability of your model.\n\ne.g.,\n```\nimport numpy as np\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.metrics import roc_auc_score\n\ndef calculate_stability_score(gini_scores):\n weeks = range(len(gini_scores))\n weeks = np.array(weeks).reshape(-1, 1)\n gini_scores = np.array(gini_scores)\n\n # Fit a linear regression to the gini scores\n model = LinearRegression()\n model.fit(weeks, gini_scores)\n\n # Calculate the falling_rate\n falling_rate = min(0, model.coef_[0])\n\n # Calculate the residuals and their standard deviation\n residuals = gini_scores - model.predict(weeks)\n std_residuals = np.std(residuals)\n\n # Calculate the stability metric\n stability_metric = np.mean(gini_scores) + 88.0 * falling_rate - 0.5 * std_residuals\n\n return stability_metric\n```", "2795778": "I trained cat and lgb and got a cv score of 0.78, then I used stacking and got a cv score of 0.93 (I was surprised why it was so high), but the lb score was negative.", "2803499": "If use stacking, make sure there is no leakage in training process.", "2795812": "Perhaps your model's performance dropped significantly and a negative “a” will cause cliff fall on lb" } }