Hi everyone,
\nI'm facing an overfitting issue while training a CatBoost model. Here's what happened:
\nš¹ First Notebook (Baseline):
\nhttps://www.kaggle.com/code/qamarmath/catboostregressor-baseline
\nTrain Pearson Correlation: 0.3946
Validation Pearson Correlation: 0.1142
\nSubmission Score: 0.07
\nThe model clearly overfit the training data.
\nš¹ Second Notebook (After Hyperparameter Tuning):
\nhttps://www.kaggle.com/code/qamarmath/hyperparameter-tuning-of-catboost
\nTrain Pearson Correlation: 0.2250 ā
Validation Pearson Correlation: 0.1251 ā
\nSubmission Score: 0.04 ā
\nThis model shows less overfitting and performs slightly better on validation ā but the submission score dropped even more.
\nI'm trying to understand why the second model, which looks better on both train and validation, performs worse on the test data.
", "messages": [ { "id": "3248188", "postDate": "07/14/2025 06:09:12", "content": "Hi everyone,
\nI'm facing an overfitting issue while training a CatBoost model. Here's what happened:
\nš¹ First Notebook (Baseline):
\nhttps://www.kaggle.com/code/qamarmath/catboostregressor-baseline
\nTrain Pearson Correlation: 0.3946
Validation Pearson Correlation: 0.1142
\nSubmission Score: 0.07
\nThe model clearly overfit the training data.
\nš¹ Second Notebook (After Hyperparameter Tuning):
\nhttps://www.kaggle.com/code/qamarmath/hyperparameter-tuning-of-catboost
\nTrain Pearson Correlation: 0.2250 ā
Validation Pearson Correlation: 0.1251 ā
\nSubmission Score: 0.04 ā
\nThis model shows less overfitting and performs slightly better on validation ā but the submission score dropped even more.
\nI'm trying to understand why the second model, which looks better on both train and validation, performs worse on the test data.
", "rawMarkdown": "Hi everyone,\n\nI'm facing an overfitting issue while training a CatBoost model. Here's what happened:\n\nš¹ First Notebook (Baseline):\nhttps://www.kaggle.com/code/qamarmath/catboostregressor-baseline\nTrain Pearson Correlation: 0.3946\n\nValidation Pearson Correlation: 0.1142\n\nSubmission Score: 0.07\n\nThe model clearly overfit the training data.\nš¹ Second Notebook (After Hyperparameter Tuning):\nhttps://www.kaggle.com/code/qamarmath/hyperparameter-tuning-of-catboost\nTrain Pearson Correlation: 0.2250 ā \n\nValidation Pearson Correlation: 0.1251 ā \n\nSubmission Score: 0.04 ā\n\nThis model shows less overfitting and performs slightly better on validation ā but the submission score dropped even more.\n\nI'm trying to understand why the second model, which looks better on both train and validation, performs worse on the test data.", "votes": null }, { "id": "3248249", "postDate": "07/14/2025 07:57:10", "content": "The features are non stationnary, meaning that the relation you have modeled in train and validated in val may have completely vanished in the test set. You can try simpler models (capturing real structural patterns and not just temporary statistical behaviors) or ensemble many more models to decrease your variance or even model the different regimes of the market.
", "rawMarkdown": "The features are non stationnary, meaning that the relation you have modeled in train and validated in val may have completely vanished in the test set. You can try simpler models (capturing real structural patterns and not just temporary statistical behaviors) or ensemble many more models to decrease your variance or even model the different regimes of the market.", "votes": null }, { "id": "3248299", "postDate": "07/14/2025 10:15:32", "content": "For a start you need to 1) close the cv gap. There are reasons you are overfitting (feature engineering ?, lack of regularisation ?). and 2) add robust evaluation. For exemple your models can be both 0.05 +/- 0.02 and you current evaluation is just not enough to conclude.
", "rawMarkdown": "For a start you need to 1) close the cv gap. There are reasons you are overfitting (feature engineering ?, lack of regularisation ?). and 2) add robust evaluation. For exemple your models can be both 0.05 +/- 0.02 and you current evaluation is just not enough to conclude.", "votes": null }, { "id": "3248365", "postDate": "07/14/2025 12:42:06", "content": "Agree with the comments from @lucasmorin and @yannfb .
\nThe features that show importance in the train dataset may drift/have completely different behavior in the test dataset.
\nIf you look at the charts from these notebook, you can see that from a stability point of view, there are very few features that have good stability in the test and train dataset.
\nhttps://www.kaggle.com/code/taylorsamarel/feature-select-using-time-series-pseudo-labels-2-0
\nhttps://www.kaggle.com/code/taylorsamarel/feature-select-pseudo-labels-x-lightgbm
\nhttps://www.kaggle.com/code/taylorsamarel/feature-select-pseudo-labels-x-xgb
I have found that in addition to regularization, noise injection, and drop out, using an overfitting discriminator also worked well.
", "rawMarkdown": "Agree with the comments from @lucasmorin and @yannfb .\n\nThe features that show importance in the train dataset may drift/have completely different behavior in the test dataset.\n\nIf you look at the charts from these notebook, you can see that from a stability point of view, there are very few features that have good stability in the test and train dataset.\n\nhttps://www.kaggle.com/code/taylorsamarel/feature-select-using-time-series-pseudo-labels-2-0\nhttps://www.kaggle.com/code/taylorsamarel/feature-select-pseudo-labels-x-lightgbm\nhttps://www.kaggle.com/code/taylorsamarel/feature-select-pseudo-labels-x-xgb\n\nI have found that in addition to regularization, noise injection, and drop out, using an overfitting discriminator also worked well.", "votes": null }, { "id": "3248400", "postDate": "07/14/2025 13:43:16", "content": "To add to this, look into techniques for detecting distributional shift, e.g. adversarial validation. Especially in crypto distributional shifts can be strong.
", "rawMarkdown": "To add to this, look into techniques for detecting distributional shift, e.g. adversarial validation. Especially in crypto distributional shifts can be strong.", "votes": null }, { "id": "3248404", "postDate": "07/14/2025 13:58:15", "content": "Good point, adversarial validation is a great tool.
", "rawMarkdown": "Good point, adversarial validation is a great tool.", "votes": null } ], "comments": [ { "id": 3248249, "author_name": "yannfb", "author_url": "", "post_date": "07/14/2025 07:57:10", "content": "The features are non stationnary, meaning that the relation you have modeled in train and validated in val may have completely vanished in the test set. You can try simpler models (capturing real structural patterns and not just temporary statistical behaviors) or ensemble many more models to decrease your variance or even model the different regimes of the market.
", "votes": null, "replies": [] }, { "id": 3248299, "author_name": "lucasmorin", "author_url": "", "post_date": "07/14/2025 10:15:32", "content": "For a start you need to 1) close the cv gap. There are reasons you are overfitting (feature engineering ?, lack of regularisation ?). and 2) add robust evaluation. For exemple your models can be both 0.05 +/- 0.02 and you current evaluation is just not enough to conclude.
", "votes": null, "replies": [] }, { "id": 3248365, "author_name": "taylorsamarel", "author_url": "", "post_date": "07/14/2025 12:42:06", "content": "Agree with the comments from @lucasmorin and @yannfb .
\nThe features that show importance in the train dataset may drift/have completely different behavior in the test dataset.
\nIf you look at the charts from these notebook, you can see that from a stability point of view, there are very few features that have good stability in the test and train dataset.
\nhttps://www.kaggle.com/code/taylorsamarel/feature-select-using-time-series-pseudo-labels-2-0
\nhttps://www.kaggle.com/code/taylorsamarel/feature-select-pseudo-labels-x-lightgbm
\nhttps://www.kaggle.com/code/taylorsamarel/feature-select-pseudo-labels-x-xgb
I have found that in addition to regularization, noise injection, and drop out, using an overfitting discriminator also worked well.
", "votes": null, "replies": [ { "id": 3248400, "author_name": "drshredz", "author_url": "", "post_date": "07/14/2025 13:43:16", "content": "To add to this, look into techniques for detecting distributional shift, e.g. adversarial validation. Especially in crypto distributional shifts can be strong.
", "votes": null, "replies": [ { "id": 3248404, "author_name": "taylorsamarel", "author_url": "", "post_date": "07/14/2025 13:58:15", "content": "Good point, adversarial validation is a great tool.
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3248188": "Hi everyone,\n\nI'm facing an overfitting issue while training a CatBoost model. Here's what happened:\n\nš¹ First Notebook (Baseline):\nhttps://www.kaggle.com/code/qamarmath/catboostregressor-baseline\nTrain Pearson Correlation: 0.3946\n\nValidation Pearson Correlation: 0.1142\n\nSubmission Score: 0.07\n\nThe model clearly overfit the training data.\nš¹ Second Notebook (After Hyperparameter Tuning):\nhttps://www.kaggle.com/code/qamarmath/hyperparameter-tuning-of-catboost\nTrain Pearson Correlation: 0.2250 ā \n\nValidation Pearson Correlation: 0.1251 ā \n\nSubmission Score: 0.04 ā\n\nThis model shows less overfitting and performs slightly better on validation ā but the submission score dropped even more.\n\nI'm trying to understand why the second model, which looks better on both train and validation, performs worse on the test data.", "3248249": "The features are non stationnary, meaning that the relation you have modeled in train and validated in val may have completely vanished in the test set. You can try simpler models (capturing real structural patterns and not just temporary statistical behaviors) or ensemble many more models to decrease your variance or even model the different regimes of the market.", "3248299": "For a start you need to 1) close the cv gap. There are reasons you are overfitting (feature engineering ?, lack of regularisation ?). and 2) add robust evaluation. For exemple your models can be both 0.05 +/- 0.02 and you current evaluation is just not enough to conclude.", "3248365": "Agree with the comments from @lucasmorin and @yannfb .\n\nThe features that show importance in the train dataset may drift/have completely different behavior in the test dataset.\n\nIf you look at the charts from these notebook, you can see that from a stability point of view, there are very few features that have good stability in the test and train dataset.\n\nhttps://www.kaggle.com/code/taylorsamarel/feature-select-using-time-series-pseudo-labels-2-0\nhttps://www.kaggle.com/code/taylorsamarel/feature-select-pseudo-labels-x-lightgbm\nhttps://www.kaggle.com/code/taylorsamarel/feature-select-pseudo-labels-x-xgb\n\nI have found that in addition to regularization, noise injection, and drop out, using an overfitting discriminator also worked well.", "3248400": "To add to this, look into techniques for detecting distributional shift, e.g. adversarial validation. Especially in crypto distributional shifts can be strong.", "3248404": "Good point, adversarial validation is a great tool." }, "source": "meta" }