Validation Strategy: Nested Cross-Validation
\nThe most critical element of our success was the nested cross-validation (Nested CV) architecture we designed. For the outer loop, we used a Time Series Split because it showed a high correlation with the leaderboard scores and respected the temporal nature of the dataset. This approach prevented data leakage while providing us with a reliable local validation score. For the inner loop, we chose K-Fold cross-validation for hyperparameter tuning. The reason for this was that using K-Fold to search for parameters yielded more stable and higher scores on the leaderboard. This dual structure allowed us to perform a general validation that was true to the time-series nature of the data, while also finding the most stable parameters within each time slice.
\nHyperparameter Optimization
\nTo automate and efficiently tune our hyperparameters, we utilized the Optuna library. Our strategy was to focus Optuna solely on improving the score of our outer loop's Time Series validation. We intentionally did not touch the inner K-Fold structure with Optuna, as its purpose was to identify stable parameter candidates. By having Optuna focus only on the outer loop, the optimization process became more targeted and geared towards enhancing overall performance, which enabled us to find the best set of hyperparameters.
\nFeature Selection and Engineering
\nTo identify the most effective features, we employed a combination of three distinct and powerful algorithms: LOFO (Leave One Feature Out), SHAP (SHapley Additive exPlanations), and mRMR (Minimum Redundancy Maximum Relevance). The combination of these three methods gave us the opportunity to understand both the individual importance of features and their collective impact on the model. Based on the outputs from these algorithms, we ranked all features by importance. Next, we carefully analyzed the distributions of the top-ranked features in both the training and test sets to ensure our model's ability to generalize to new data. Finally, we generated new interaction features from this set of best features and trained our model on this enriched and carefully selected dataset.
\nFinal Model Training
\nFor training our final model, we used the optimal set of hyperparameters discovered through the K-Fold process within our nested validation structure. With these parameters, we trained our model one last time on the entire training dataset to generate our final predictions for submission. This final training step ensured that the most robust parameters, proven during validation, were combined with the knowledge from the entire dataset.
Validation Strategy: Nested Cross-Validation
\nThe most critical element of our success was the nested cross-validation (Nested CV) architecture we designed. For the outer loop, we used a Time Series Split because it showed a high correlation with the leaderboard scores and respected the temporal nature of the dataset. This approach prevented data leakage while providing us with a reliable local validation score. For the inner loop, we chose K-Fold cross-validation for hyperparameter tuning. The reason for this was that using K-Fold to search for parameters yielded more stable and higher scores on the leaderboard. This dual structure allowed us to perform a general validation that was true to the time-series nature of the data, while also finding the most stable parameters within each time slice.
\nHyperparameter Optimization
\nTo automate and efficiently tune our hyperparameters, we utilized the Optuna library. Our strategy was to focus Optuna solely on improving the score of our outer loop's Time Series validation. We intentionally did not touch the inner K-Fold structure with Optuna, as its purpose was to identify stable parameter candidates. By having Optuna focus only on the outer loop, the optimization process became more targeted and geared towards enhancing overall performance, which enabled us to find the best set of hyperparameters.
\nFeature Selection and Engineering
\nTo identify the most effective features, we employed a combination of three distinct and powerful algorithms: LOFO (Leave One Feature Out), SHAP (SHapley Additive exPlanations), and mRMR (Minimum Redundancy Maximum Relevance). The combination of these three methods gave us the opportunity to understand both the individual importance of features and their collective impact on the model. Based on the outputs from these algorithms, we ranked all features by importance. Next, we carefully analyzed the distributions of the top-ranked features in both the training and test sets to ensure our model's ability to generalize to new data. Finally, we generated new interaction features from this set of best features and trained our model on this enriched and carefully selected dataset.
\nFinal Model Training
\nFor training our final model, we used the optimal set of hyperparameters discovered through the K-Fold process within our nested validation structure. With these parameters, we trained our model one last time on the entire training dataset to generate our final predictions for submission. This final training step ensured that the most robust parameters, proven during validation, were combined with the knowledge from the entire dataset.