We are thrilled to achieve a gold medal in our first Kaggle competition and deeply grateful to everyone at Kaggle for all that you’ve shared, which greatly contributed to our learning journey. In this write-up, we’re excited to share our solution and key takeaways.
\nOur model builds on this notebook: https://www.kaggle.com/code/laiyunghwei/lb-0-493. We used three gradient boosting models-LightGBM, XGBoost, and CatBoost-and combined them using a voting regressor with equal weights.
\nMissing data was a significant challenge. Nearly all features, except demographic ones, contained missing values, with about 10 features having missing rates exceeding 70%. The target variable also had approximately 30% missing data.
\nAdditionally, features from the same instrument had similar missing rates, and the missing values were randomly distributed, allowing us to handle them through either direct removal or imputation.
\nWe identified multiple outliers after a detailed examination of each feature:
\n| Feature | \nOutlier Standard | \nNumber of Outliers | \n
|---|---|---|
CGAS-CGAS_Score | \nExtremely high value (e.g., 999) | \n1 | \n
Physical-Weight | \nInvalid weight value (e.g., 0) | \n61 | \n
BIA related features (such as BIA-BIA_TBW, BIA-BIA_ICW) | \nValues significantly higher or lower compared to the normal range | \n2 | \n
Physical-HeartRate | \nAbnormally low heart rate (<30 bpm) | \n1 | \n
Total Body Water percentage (BIA-BIA_TBW/Physical-Weight) | \n<20% or >100% | \n10 | \n
A correlation heatmap revealed that features from the same instrument were often highly correlated (e.g., Bio-electric Impedance Analysis (BIA) features had correlations > 0.9).
\nAdditionally, some features, like BMI, could be derived from others (BMI = FFMI + FMI). Reducing such redundancy was crucial for simplifying gradient boosting models.
\nPhysical features like height and weight exhibited strong correlations with age. By normalizing these features (e.g., height/age, weight/age), we can extract more meaningful information. And age-based regression seems a reasonable method for imputing missing values.
\nTime series data had high missing rates and minimal impact on prediction accuracy, so we ultimately excluded it from our solution.
\nWe removed entries with outlier values (explained in the EDA part) to avoid skewing the model.
\nWe used feature engineering to extract more valuable insights and remove duplicate information, and dropped features with too many missing values, redundancy, or strong correlations. The final set of selected features included:
\n| No. | \nFeature | \nExplanation | \n
|---|---|---|
| 1 | \nBasic_Demos-Age | \nParticipant's age | \n
| 2 | \nBasic_Demos-Sex | \nParticipant's sex | \n
| 3 | \nCGAS-CGAS_Score | \nChildren's Global Assessment Scale (CGAS) score | \n
| 4 | \nPhysical-Height_per_Age | \nPhysical-Height/Basic_Demos-Age | \n
| 5 | \nPhysical-Weight_per_Age | \nPhysical-Weight/Basic_Demos-Age | \n
| 6 | \nFGC_Zone_Total | \nFGC_CU_Zone+FGC_SRL_Zone+FGC_SRR_Zone+FGC_TL_Zone | \n
| 7 | \nBIA-BIA_Activity_Level_num | \nActivity level | \n
| 8 | \nBIA-BIA_BMR | \nBasal Metabolic Rate | \n
| 9 | \nBIA-BIA_DEE | \nDaily Energy Expenditure | \n
| 10 | \nBIA-BIA_FFMI | \nFat-Free Mass Index | \n
| 11 | \nBIA-BIA_FMI | \nFat Mass Index | \n
| 12 | \nBIA-BIA_Frame_num | \nFrame size | \n
| 13 | \nBIA-BIA_SMM | \nSkeletal Muscle Mass | \n
| 14 | \nBIA-BIA_TBW | \nTotal Body Water volume | \n
| 15 | \nSDS-SDS_Total_T | \nSleep Disturbance Scale total score | \n
| 16 | \nPreInt_EduHx-computerinternet_hoursday | \nAverage hours per day spent on computer/internet usage | \n
| 17 | \nPAQ_Total | \nPhysical Activity Questionnaire score, combined by PAQ_A-PAQ_A_Total and PAQ_C-PAQ_C_Total | \n
We evaluated two imputation methods: age-based regression and a hybrid approach combining K-Nearest Neighbors (KNN) for continuous variables and Random Forest for categorical variables. Both methods are reasonable depending on the data characteristics.
\nIn this project, most features are closely related to physical development, making linear regression with age a suitable choice for imputing missing values. This approach ensures that the imputed values align with the observed data distribution.
\nThe hybrid method leverages the assumption that continuous features follow local similarity patterns, which KNN can effectively capture, and categorical features are sufficiently correlated with other features, allowing Random Forest to provide accurate predictions.
\nAfter testing both methods, age-based regression was selected as it demonstrated better performance for this dataset.
\nWe tested three methods: 1) Ridge Regression; 2) Multilayer Perceptron (MLP); 3) Gradient Boosting. Gradient boosting consistently outperformed the other methods with a stronger Quadratic Weighted Kappa (QWK) score, so we stick to this method. We optimized hyperparameters using Optuna and validated results with 5-fold cross-validation.
\nThis competition was not only a test of technical skills but also an exercise in handling real-world data challenges. While proud of our results, we see room for improvement and valuable lessons for future projects.
\nPreInt_EduHx_computerinternet_hoursday, which directly tracks computer usage time, seems like a more reliable indicator.We are thrilled to achieve a gold medal in our first Kaggle competition and deeply grateful to everyone at Kaggle for all that you’ve shared, which greatly contributed to our learning journey. In this write-up, we’re excited to share our solution and key takeaways.
\nOur model builds on this notebook: https://www.kaggle.com/code/laiyunghwei/lb-0-493. We used three gradient boosting models-LightGBM, XGBoost, and CatBoost-and combined them using a voting regressor with equal weights.
\nMissing data was a significant challenge. Nearly all features, except demographic ones, contained missing values, with about 10 features having missing rates exceeding 70%. The target variable also had approximately 30% missing data.
\nAdditionally, features from the same instrument had similar missing rates, and the missing values were randomly distributed, allowing us to handle them through either direct removal or imputation.
\nWe identified multiple outliers after a detailed examination of each feature:
\n| Feature | \nOutlier Standard | \nNumber of Outliers | \n
|---|---|---|
CGAS-CGAS_Score | \nExtremely high value (e.g., 999) | \n1 | \n
Physical-Weight | \nInvalid weight value (e.g., 0) | \n61 | \n
BIA related features (such as BIA-BIA_TBW, BIA-BIA_ICW) | \nValues significantly higher or lower compared to the normal range | \n2 | \n
Physical-HeartRate | \nAbnormally low heart rate (<30 bpm) | \n1 | \n
Total Body Water percentage (BIA-BIA_TBW/Physical-Weight) | \n<20% or >100% | \n10 | \n
A correlation heatmap revealed that features from the same instrument were often highly correlated (e.g., Bio-electric Impedance Analysis (BIA) features had correlations > 0.9).
\nAdditionally, some features, like BMI, could be derived from others (BMI = FFMI + FMI). Reducing such redundancy was crucial for simplifying gradient boosting models.
\nPhysical features like height and weight exhibited strong correlations with age. By normalizing these features (e.g., height/age, weight/age), we can extract more meaningful information. And age-based regression seems a reasonable method for imputing missing values.
\nTime series data had high missing rates and minimal impact on prediction accuracy, so we ultimately excluded it from our solution.
\nWe removed entries with outlier values (explained in the EDA part) to avoid skewing the model.
\nWe used feature engineering to extract more valuable insights and remove duplicate information, and dropped features with too many missing values, redundancy, or strong correlations. The final set of selected features included:
\n| No. | \nFeature | \nExplanation | \n
|---|---|---|
| 1 | \nBasic_Demos-Age | \nParticipant's age | \n
| 2 | \nBasic_Demos-Sex | \nParticipant's sex | \n
| 3 | \nCGAS-CGAS_Score | \nChildren's Global Assessment Scale (CGAS) score | \n
| 4 | \nPhysical-Height_per_Age | \nPhysical-Height/Basic_Demos-Age | \n
| 5 | \nPhysical-Weight_per_Age | \nPhysical-Weight/Basic_Demos-Age | \n
| 6 | \nFGC_Zone_Total | \nFGC_CU_Zone+FGC_SRL_Zone+FGC_SRR_Zone+FGC_TL_Zone | \n
| 7 | \nBIA-BIA_Activity_Level_num | \nActivity level | \n
| 8 | \nBIA-BIA_BMR | \nBasal Metabolic Rate | \n
| 9 | \nBIA-BIA_DEE | \nDaily Energy Expenditure | \n
| 10 | \nBIA-BIA_FFMI | \nFat-Free Mass Index | \n
| 11 | \nBIA-BIA_FMI | \nFat Mass Index | \n
| 12 | \nBIA-BIA_Frame_num | \nFrame size | \n
| 13 | \nBIA-BIA_SMM | \nSkeletal Muscle Mass | \n
| 14 | \nBIA-BIA_TBW | \nTotal Body Water volume | \n
| 15 | \nSDS-SDS_Total_T | \nSleep Disturbance Scale total score | \n
| 16 | \nPreInt_EduHx-computerinternet_hoursday | \nAverage hours per day spent on computer/internet usage | \n
| 17 | \nPAQ_Total | \nPhysical Activity Questionnaire score, combined by PAQ_A-PAQ_A_Total and PAQ_C-PAQ_C_Total | \n
We evaluated two imputation methods: age-based regression and a hybrid approach combining K-Nearest Neighbors (KNN) for continuous variables and Random Forest for categorical variables. Both methods are reasonable depending on the data characteristics.
\nIn this project, most features are closely related to physical development, making linear regression with age a suitable choice for imputing missing values. This approach ensures that the imputed values align with the observed data distribution.
\nThe hybrid method leverages the assumption that continuous features follow local similarity patterns, which KNN can effectively capture, and categorical features are sufficiently correlated with other features, allowing Random Forest to provide accurate predictions.
\nAfter testing both methods, age-based regression was selected as it demonstrated better performance for this dataset.
\nWe tested three methods: 1) Ridge Regression; 2) Multilayer Perceptron (MLP); 3) Gradient Boosting. Gradient boosting consistently outperformed the other methods with a stronger Quadratic Weighted Kappa (QWK) score, so we stick to this method. We optimized hyperparameters using Optuna and validated results with 5-fold cross-validation.
\nThis competition was not only a test of technical skills but also an exercise in handling real-world data challenges. While proud of our results, we see room for improvement and valuable lessons for future projects.
\nPreInt_EduHx_computerinternet_hoursday, which directly tracks computer usage time, seems like a more reliable indicator.