Hey everyone ,
\nI’ve been working on the DRW Crypto Market Prediction competition and wanted to share some unexpected findings and hard-won lessons from trying to model short-term price movements using an MLPRegressor on the full training set.
\nThe Setup
\nDataset: ~525,000 rows with ~890 features
\nTarget: label (appears to represent short-term price movement)
\nMetric: Pearson correlation
\nModel: MLPRegressor (sklearn), incremental feature selection loop from top-k F-scores
\nGoal: Find the optimal number of features (in steps of 5) that yield the best Pearson correlation
\nKey Observations
Has anyone seen consistent gains using MLPs or Keras models?
\nHas feature engineering helped over brute force top-k methods?
\nAnyone tried building hybrid models using both proprietary and public volume-based features?
\nLet’s discuss below — I’d love to hear your experiments, failures, and wins 🙌
Hey everyone ,
\nI’ve been working on the DRW Crypto Market Prediction competition and wanted to share some unexpected findings and hard-won lessons from trying to model short-term price movements using an MLPRegressor on the full training set.
\nThe Setup
\nDataset: ~525,000 rows with ~890 features
\nTarget: label (appears to represent short-term price movement)
\nMetric: Pearson correlation
\nModel: MLPRegressor (sklearn), incremental feature selection loop from top-k F-scores
\nGoal: Find the optimal number of features (in steps of 5) that yield the best Pearson correlation
\nKey Observations
Has anyone seen consistent gains using MLPs or Keras models?
\nHas feature engineering helped over brute force top-k methods?
\nAnyone tried building hybrid models using both proprietary and public volume-based features?
\nLet’s discuss below — I’d love to hear your experiments, failures, and wins 🙌
Top 5 features: Pearson = 0.0296
\nTop 25 features: Pearson = 0.0006
\nTop 30 features: Pearson = 0.0565
\nTop 40 features: Pearson = 0.0146
Are these CV scores or LB scores? What CV are you using?
", "rawMarkdown": "Top 5 features: Pearson = 0.0296\nTop 25 features: Pearson = 0.0006\nTop 30 features: Pearson = 0.0565\nTop 40 features: Pearson = 0.0146\n\nAre these CV scores or LB scores? What CV are you using?", "votes": null }, { "id": "3231489", "postDate": "06/24/2025 13:00:15", "content": "Hi byunjins,
\nThanks for your question! Those Pearson scores are from the validation set (CV) during incremental feature selection — specifically a time-based split where the last 20% of the training data is used as validation to avoid future data leakage.
\nThey are not leaderboard (LB) scores — I haven’t yet submitted those predictions.
\nI’m using a simple train_test_split with shuffle=False to preserve the temporal order, mimicking a realistic scenario for the crypto price prediction task.
\nLet me know if you want me to share the exact CV code or results on the LB!
How are you getting such low scores(assuming they reflect actual LB like scores) on CV on pearson? Am I missing something? Because when I try pearson I get high scores. So I thought that my test data is from different distribution. I am not even able to debug if there is something else causing a dataleak.
", "rawMarkdown": "How are you getting such low scores(assuming they reflect actual LB like scores) on CV on pearson? Am I missing something? Because when I try pearson I get high scores. So I thought that my test data is from different distribution. I am not even able to debug if there is something else causing a dataleak.", "votes": null }, { "id": "3232326", "postDate": "06/25/2025 15:22:11", "content": "I used a BP neural network model with 2 hidden layers, AdamW optimizer, and CosineAnnealingLR scheduler. I also used 5 - fold cross - validation. The highest LB score is 0.10459, but the model may be unstable.
", "rawMarkdown": "I used a BP neural network model with 2 hidden layers, AdamW optimizer, and CosineAnnealingLR scheduler. I also used 5 - fold cross - validation. The highest LB score is 0.10459, but the model may be unstable.", "votes": null }, { "id": "3232470", "postDate": "06/25/2025 18:59:09", "content": "Really insightful post — thanks for sharing your findings so transparently!
\nYour experience with MLPRegressor resonates a lot. In noisy financial data like crypto, I’ve noticed that increasing feature count often adds more noise than signal, especially with models like MLPs that don’t inherently handle feature importance or sparsity well. Regularization can help a bit, but it’s often not enough.
\nA few quick thoughts and follow-up questions:
\nHave you tried dimensionality reduction methods like PCA or autoencoders before feeding into the MLP? Sometimes they help stabilize convergence.
\nFor the label distribution, did you experiment with scaling or clipping extreme values? That can reduce the influence of outliers during training.
\nI’ve seen some gains using hybrid models — combining MLPs with tree-based models or feature selectors from LGBM/XGBoost. The idea is to let the trees handle the feature noise and then pass refined features to a neural net.
\nAlso curious if anyone has tried contrastive learning or any self-supervised approaches on crypto time series — might be a better way to learn useful representations in such a volatile domain.
", "rawMarkdown": "Really insightful post — thanks for sharing your findings so transparently!\n\nYour experience with MLPRegressor resonates a lot. In noisy financial data like crypto, I’ve noticed that increasing feature count often adds more noise than signal, especially with models like MLPs that don’t inherently handle feature importance or sparsity well. Regularization can help a bit, but it’s often not enough.\n\nA few quick thoughts and follow-up questions:\n\nHave you tried dimensionality reduction methods like PCA or autoencoders before feeding into the MLP? Sometimes they help stabilize convergence.\n\nFor the label distribution, did you experiment with scaling or clipping extreme values? That can reduce the influence of outliers during training.\n\nI’ve seen some gains using hybrid models — combining MLPs with tree-based models or feature selectors from LGBM/XGBoost. The idea is to let the trees handle the feature noise and then pass refined features to a neural net.\n\n\nAlso curious if anyone has tried contrastive learning or any self-supervised approaches on crypto time series — might be a better way to learn useful representations in such a volatile domain.", "votes": null }, { "id": "3232765", "postDate": "06/26/2025 07:08:59", "content": "Spot on. That \"more features = worse performance\" issue is a classic.
\nI've found it often comes down to two things:
\nData Leakage: Some features might have look-ahead bias baked into them.
\nFactor Decay: Some features look great on the training set but their predictive power just dies off on new data.
\nI had the same experience—I removed a single feature, \"X612\", and my model's performance got a noticeable boost. It really is quality over quantity.
Hi @visterbai ,
\nThanks for your thought-provoking insight! I am curious how you detect data leakage in a systematic way? I did find some features seem to be filled nan with future info (if I remember correctly
", "rawMarkdown": "Hi @visterbai ,\n\nThanks for your thought-provoking insight! I am curious how you detect data leakage in a systematic way? I did find some features seem to be filled nan with future info (if I remember correctly", "votes": null }, { "id": "3233398", "postDate": "06/26/2025 18:29:55", "content": "I ran a few simple tests and extracted some features that seemed important based on SHAP values, then submitted them individually as submissions. I noticed that some features have a much higher IC (information coefficient) on the test set than on the training set, while others show the opposite. If a feature performs similarly on both the training and test sets with an IC above 0.02🌟, it likely has good generalization and tends to yield solid results.
", "rawMarkdown": "I ran a few simple tests and extracted some features that seemed important based on SHAP values, then submitted them individually as submissions. I noticed that some features have a much higher IC (information coefficient) on the test set than on the training set, while others show the opposite. If a feature performs similarly on both the training and test sets with an IC above 0.02🌟, it likely has good generalization and tends to yield solid results.", "votes": null }, { "id": "3233499", "postDate": "06/26/2025 21:43:40", "content": "Absolutely, totally agree.
\nI’ve run into the same issues — especially factor decay — where certain features shine during training but completely fall apart in live or test data.
\nAnd yeah, data leakage is sneaky. One seemingly harmless feature can wreck your entire validation if you're not careful.
\nRemoving \"X612\" and seeing a boost is a perfect example — sometimes less is more. I’ve started adopting a \"trust but verify\" approach for every feature now.
I also use a neural network, but my model is deeper, I use 4 to 8 layers of hidden layers. My best score is only around 0.03
", "rawMarkdown": "I also use a neural network, but my model is deeper, I use 4 to 8 layers of hidden layers. My best score is only around 0.03", "votes": null }, { "id": "3237397", "postDate": "07/01/2025 04:34:17", "content": "Try injecting noise and using high drop out levels.
", "rawMarkdown": "Try injecting noise and using high drop out levels.", "votes": null }, { "id": "3243188", "postDate": "07/06/2025 21:09:21", "content": "Very interesting post and questions, thank you for sharing!!
\nLooking forword to see other people's responses and experiences about this.
\nThank you all !!
I've been able to get MLP models and GANDALF models to train with significant stability to around 105-150 features. However, adding more features is becoming difficult again!
", "rawMarkdown": "I've been able to get MLP models and GANDALF models to train with significant stability to around 105-150 features. However, adding more features is becoming difficult again!", "votes": null }, { "id": "3243438", "postDate": "07/07/2025 06:28:12", "content": "Have you tried ensemble of models by training them on few set of features on each in combination? If so could you share was it useful doing it or not?
", "rawMarkdown": "Have you tried ensemble of models by training them on few set of features on each in combination? If so could you share was it useful doing it or not?", "votes": null }, { "id": "3243647", "postDate": "07/07/2025 11:43:29", "content": "This is possible, and is something I am trying at the moment - with marginal success. However, the task of adding new features becomes slower and slower and I find myself only getting useful results using a greedy approach of manually trying different ensemble combinations and feature sets. Everytime I try to do an advanced algorithm to optimize everything at once I either get extreme overfitting, instability, or run out of memory/compute time.
", "rawMarkdown": "This is possible, and is something I am trying at the moment - with marginal success. However, the task of adding new features becomes slower and slower and I find myself only getting useful results using a greedy approach of manually trying different ensemble combinations and feature sets. Everytime I try to do an advanced algorithm to optimize everything at once I either get extreme overfitting, instability, or run out of memory/compute time.", "votes": null } ], "comments": [ { "id": 3231479, "author_name": "byunjins", "author_url": "", "post_date": "06/24/2025 12:39:10", "content": "Top 5 features: Pearson = 0.0296
\nTop 25 features: Pearson = 0.0006
\nTop 30 features: Pearson = 0.0565
\nTop 40 features: Pearson = 0.0146
Are these CV scores or LB scores? What CV are you using?
", "votes": null, "replies": [ { "id": 3231489, "author_name": "sanketpai", "author_url": "", "post_date": "06/24/2025 13:00:15", "content": "Hi byunjins,
\nThanks for your question! Those Pearson scores are from the validation set (CV) during incremental feature selection — specifically a time-based split where the last 20% of the training data is used as validation to avoid future data leakage.
\nThey are not leaderboard (LB) scores — I haven’t yet submitted those predictions.
\nI’m using a simple train_test_split with shuffle=False to preserve the temporal order, mimicking a realistic scenario for the crypto price prediction task.
\nLet me know if you want me to share the exact CV code or results on the LB!
How are you getting such low scores(assuming they reflect actual LB like scores) on CV on pearson? Am I missing something? Because when I try pearson I get high scores. So I thought that my test data is from different distribution. I am not even able to debug if there is something else causing a dataleak.
", "votes": null, "replies": [] } ] } ] }, { "id": 3232326, "author_name": "jiaoyouzhang", "author_url": "", "post_date": "06/25/2025 15:22:11", "content": "I used a BP neural network model with 2 hidden layers, AdamW optimizer, and CosineAnnealingLR scheduler. I also used 5 - fold cross - validation. The highest LB score is 0.10459, but the model may be unstable.
", "votes": null, "replies": [ { "id": 3235441, "author_name": "jinghongliu", "author_url": "", "post_date": "06/29/2025 08:52:13", "content": "I also use a neural network, but my model is deeper, I use 4 to 8 layers of hidden layers. My best score is only around 0.03
", "votes": null, "replies": [] } ] }, { "id": 3232470, "author_name": "mohamedmahmoud111", "author_url": "", "post_date": "06/25/2025 18:59:09", "content": "Really insightful post — thanks for sharing your findings so transparently!
\nYour experience with MLPRegressor resonates a lot. In noisy financial data like crypto, I’ve noticed that increasing feature count often adds more noise than signal, especially with models like MLPs that don’t inherently handle feature importance or sparsity well. Regularization can help a bit, but it’s often not enough.
\nA few quick thoughts and follow-up questions:
\nHave you tried dimensionality reduction methods like PCA or autoencoders before feeding into the MLP? Sometimes they help stabilize convergence.
\nFor the label distribution, did you experiment with scaling or clipping extreme values? That can reduce the influence of outliers during training.
\nI’ve seen some gains using hybrid models — combining MLPs with tree-based models or feature selectors from LGBM/XGBoost. The idea is to let the trees handle the feature noise and then pass refined features to a neural net.
\nAlso curious if anyone has tried contrastive learning or any self-supervised approaches on crypto time series — might be a better way to learn useful representations in such a volatile domain.
", "votes": null, "replies": [] }, { "id": 3232765, "author_name": "visterbai", "author_url": "", "post_date": "06/26/2025 07:08:59", "content": "Spot on. That \"more features = worse performance\" issue is a classic.
\nI've found it often comes down to two things:
\nData Leakage: Some features might have look-ahead bias baked into them.
\nFactor Decay: Some features look great on the training set but their predictive power just dies off on new data.
\nI had the same experience—I removed a single feature, \"X612\", and my model's performance got a noticeable boost. It really is quality over quantity.
Hi @visterbai ,
\nThanks for your thought-provoking insight! I am curious how you detect data leakage in a systematic way? I did find some features seem to be filled nan with future info (if I remember correctly
", "votes": null, "replies": [ { "id": 3233398, "author_name": "visterbai", "author_url": "", "post_date": "06/26/2025 18:29:55", "content": "I ran a few simple tests and extracted some features that seemed important based on SHAP values, then submitted them individually as submissions. I noticed that some features have a much higher IC (information coefficient) on the test set than on the training set, while others show the opposite. If a feature performs similarly on both the training and test sets with an IC above 0.02🌟, it likely has good generalization and tends to yield solid results.
", "votes": null, "replies": [] } ] }, { "id": 3233499, "author_name": "mohamedmahmoud111", "author_url": "", "post_date": "06/26/2025 21:43:40", "content": "Absolutely, totally agree.
\nI’ve run into the same issues — especially factor decay — where certain features shine during training but completely fall apart in live or test data.
\nAnd yeah, data leakage is sneaky. One seemingly harmless feature can wreck your entire validation if you're not careful.
\nRemoving \"X612\" and seeing a boost is a perfect example — sometimes less is more. I’ve started adopting a \"trust but verify\" approach for every feature now.
Try injecting noise and using high drop out levels.
", "votes": null, "replies": [] }, { "id": 3243188, "author_name": "abdelhakouanzougui2", "author_url": "", "post_date": "07/06/2025 21:09:21", "content": "Very interesting post and questions, thank you for sharing!!
\nLooking forword to see other people's responses and experiences about this.
\nThank you all !!
I've been able to get MLP models and GANDALF models to train with significant stability to around 105-150 features. However, adding more features is becoming difficult again!
", "votes": null, "replies": [ { "id": 3243438, "author_name": "electroknight", "author_url": "", "post_date": "07/07/2025 06:28:12", "content": "Have you tried ensemble of models by training them on few set of features on each in combination? If so could you share was it useful doing it or not?
", "votes": null, "replies": [ { "id": 3243647, "author_name": "taylorsamarel", "author_url": "", "post_date": "07/07/2025 11:43:29", "content": "This is possible, and is something I am trying at the moment - with marginal success. However, the task of adding new features becomes slower and slower and I find myself only getting useful results using a greedy approach of manually trying different ensemble combinations and feature sets. Everytime I try to do an advanced algorithm to optimize everything at once I either get extreme overfitting, instability, or run out of memory/compute time.
", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "3231474": "Hey everyone ,\n\nI’ve been working on the DRW Crypto Market Prediction competition and wanted to share some unexpected findings and hard-won lessons from trying to model short-term price movements using an MLPRegressor on the full training set.\nThe Setup\nDataset: ~525,000 rows with ~890 features\nTarget: label (appears to represent short-term price movement)\nMetric: Pearson correlation\nModel: MLPRegressor (sklearn), incremental feature selection loop from top-k F-scores\nGoal: Find the optimal number of features (in steps of 5) that yield the best Pearson correlation\nKey Observations\n1. More features ≠ better performance\nI naively assumed adding more features would boost performance, but RMSE and Pearson correlation actually worsened as I increased feature count in some cases. The model became less stable, and generalization dropped — even after F-test feature filtering.\nTop 5 features: Pearson = 0.0296\nTop 25 features: Pearson = 0.0006\nTop 30 features: Pearson = 0.0565\nTop 40 features: Pearson = 0.0146\n2. MLPRegressor struggled to converge\nEven with max_iter=300, I received convergence warnings at nearly every step. As feature count increased, training time ballooned (some steps took over 20 minutes), but results didn’t improve proportionally.\n3. Label distribution is wide-tailed but centered\nThe label column has:\nMean ~0.03\nStd ~1.01\nRange: –24 to +20\nThe label isn’t flat, which is great for modeling, but the outliers massively skew RMSE and make convergence harder for neural nets.\n\n\n\n\nHas anyone seen consistent gains using MLPs or Keras models?\nHas feature engineering helped over brute force top-k methods?\nAnyone tried building hybrid models using both proprietary and public volume-based features?\nLet’s discuss below — I’d love to hear your experiments, failures, and wins 🙌", "3231479": "Top 5 features: Pearson = 0.0296\nTop 25 features: Pearson = 0.0006\nTop 30 features: Pearson = 0.0565\nTop 40 features: Pearson = 0.0146\n\nAre these CV scores or LB scores? What CV are you using?", "3231489": "Hi byunjins,\nThanks for your question! Those Pearson scores are from the validation set (CV) during incremental feature selection — specifically a time-based split where the last 20% of the training data is used as validation to avoid future data leakage.\nThey are not leaderboard (LB) scores — I haven’t yet submitted those predictions.\nI’m using a simple train_test_split with shuffle=False to preserve the temporal order, mimicking a realistic scenario for the crypto price prediction task.\nLet me know if you want me to share the exact CV code or results on the LB!", "3232082": "How are you getting such low scores(assuming they reflect actual LB like scores) on CV on pearson? Am I missing something? Because when I try pearson I get high scores. So I thought that my test data is from different distribution. I am not even able to debug if there is something else causing a dataleak.", "3232326": "I used a BP neural network model with 2 hidden layers, AdamW optimizer, and CosineAnnealingLR scheduler. I also used 5 - fold cross - validation. The highest LB score is 0.10459, but the model may be unstable.", "3232470": "Really insightful post — thanks for sharing your findings so transparently!\n\nYour experience with MLPRegressor resonates a lot. In noisy financial data like crypto, I’ve noticed that increasing feature count often adds more noise than signal, especially with models like MLPs that don’t inherently handle feature importance or sparsity well. Regularization can help a bit, but it’s often not enough.\n\nA few quick thoughts and follow-up questions:\n\nHave you tried dimensionality reduction methods like PCA or autoencoders before feeding into the MLP? Sometimes they help stabilize convergence.\n\nFor the label distribution, did you experiment with scaling or clipping extreme values? That can reduce the influence of outliers during training.\n\nI’ve seen some gains using hybrid models — combining MLPs with tree-based models or feature selectors from LGBM/XGBoost. The idea is to let the trees handle the feature noise and then pass refined features to a neural net.\n\n\nAlso curious if anyone has tried contrastive learning or any self-supervised approaches on crypto time series — might be a better way to learn useful representations in such a volatile domain.", "3232765": "Spot on. That \"more features = worse performance\" issue is a classic.\n\nI've found it often comes down to two things:\n\nData Leakage: Some features might have look-ahead bias baked into them.\nFactor Decay: Some features look great on the training set but their predictive power just dies off on new data.\nI had the same experience—I removed a single feature, \"X612\", and my model's performance got a noticeable boost. It really is quality over quantity.", "3233317": "Hi @visterbai ,\n\nThanks for your thought-provoking insight! I am curious how you detect data leakage in a systematic way? I did find some features seem to be filled nan with future info (if I remember correctly", "3233398": "I ran a few simple tests and extracted some features that seemed important based on SHAP values, then submitted them individually as submissions. I noticed that some features have a much higher IC (information coefficient) on the test set than on the training set, while others show the opposite. If a feature performs similarly on both the training and test sets with an IC above 0.02🌟, it likely has good generalization and tends to yield solid results.", "3233499": "Absolutely, totally agree.\nI’ve run into the same issues — especially factor decay — where certain features shine during training but completely fall apart in live or test data.\nAnd yeah, data leakage is sneaky. One seemingly harmless feature can wreck your entire validation if you're not careful.\n\nRemoving \"X612\" and seeing a boost is a perfect example — sometimes less is more. I’ve started adopting a \"trust but verify\" approach for every feature now.", "3235441": "I also use a neural network, but my model is deeper, I use 4 to 8 layers of hidden layers. My best score is only around 0.03", "3237397": "Try injecting noise and using high drop out levels.", "3243188": "Very interesting post and questions, thank you for sharing!!\nLooking forword to see other people's responses and experiences about this. \nThank you all !!", "3243242": "I've been able to get MLP models and GANDALF models to train with significant stability to around 105-150 features. However, adding more features is becoming difficult again!", "3243438": "Have you tried ensemble of models by training them on few set of features on each in combination? If so could you share was it useful doing it or not?", "3243647": "This is possible, and is something I am trying at the moment - with marginal success. However, the task of adding new features becomes slower and slower and I find myself only getting useful results using a greedy approach of manually trying different ensemble combinations and feature sets. Everytime I try to do an advanced algorithm to optimize everything at once I either get extreme overfitting, instability, or run out of memory/compute time." }, "source": "meta" }