{ "id": 585124, "title": "🚀 LightGBM + Feature Selection: Performance Breakthrough.", "url": "/competitions/drw-crypto-market-prediction/discussion/585124", "author_name": "Sanket Pai", "post_date": "2025-06-18T05:17:32.394000", "votes": 0, "comment_count": 0, "views": 0, "content": "

Hi everyone,
\nI recently ran an experiment on a regression problem (~525k rows) using LightGBM with GPU acceleration (Tesla T4) to evaluate how performance scales with incremental feature selection. Thought I’d share my results here for anyone exploring similar approaches.

\n

🧪 Setup
\nTarget: Continuous variable (non-categorical)
\nFeatures: ~200 after correlation filtering
\nFeature Selection: SelectKBest with F-test (f_regression)
\nScaler: StandardScaler
\nModel: LightGBM with GPU (OpenCL)
\nTrain Size: 420,709 rows
\nMetrics: R², RMSE, MAE (on validation split)

\n

below I have posted my results with the corresponding metrics for the k top features

\n

📊 Results
\nK R² RMSE MAE
\n5 0.3723 0.8053 0.5462
\n10 0.4516 0.7527 0.5216
\n15 0.4950 0.7222 0.5046
\n20 0.5181 0.7055 0.4960
\n25 0.5371 0.6915 0.4871
\n30 0.5528 0.6797 0.4817
\n35 0.5642 0.6710 0.4764
\n40 0.5670 0.6688 0.4750
\n45 0.5739 0.6634 0.4718
\n50 0.5801 0.6586 0.4686
\n55 0.5800 0.6586 0.4668
\n60 0.5819 0.6572 0.4661
\n65 0.5840 0.6555 0.4635
\n📌 Early stopped at 65 features due to R² plateau.

\n

🔍 Observations
\nLargest gains were seen in the first 30–40 features.
\nMarginal returns diminish after 50.
\nGPU training times were super fast (~10–20MB GPU memory per batch).
\nEven sparse features didn’t drastically improve performance after a point.

", "messages": [ { "id": 3226795, "postDate": "2025-06-18T05:17:32.393Z", "content": "

Hi everyone,
\nI recently ran an experiment on a regression problem (~525k rows) using LightGBM with GPU acceleration (Tesla T4) to evaluate how performance scales with incremental feature selection. Thought I’d share my results here for anyone exploring similar approaches.

\n

🧪 Setup
\nTarget: Continuous variable (non-categorical)
\nFeatures: ~200 after correlation filtering
\nFeature Selection: SelectKBest with F-test (f_regression)
\nScaler: StandardScaler
\nModel: LightGBM with GPU (OpenCL)
\nTrain Size: 420,709 rows
\nMetrics: R², RMSE, MAE (on validation split)

\n

below I have posted my results with the corresponding metrics for the k top features

\n

📊 Results
\nK R² RMSE MAE
\n5 0.3723 0.8053 0.5462
\n10 0.4516 0.7527 0.5216
\n15 0.4950 0.7222 0.5046
\n20 0.5181 0.7055 0.4960
\n25 0.5371 0.6915 0.4871
\n30 0.5528 0.6797 0.4817
\n35 0.5642 0.6710 0.4764
\n40 0.5670 0.6688 0.4750
\n45 0.5739 0.6634 0.4718
\n50 0.5801 0.6586 0.4686
\n55 0.5800 0.6586 0.4668
\n60 0.5819 0.6572 0.4661
\n65 0.5840 0.6555 0.4635
\n📌 Early stopped at 65 features due to R² plateau.

\n

🔍 Observations
\nLargest gains were seen in the first 30–40 features.
\nMarginal returns diminish after 50.
\nGPU training times were super fast (~10–20MB GPU memory per batch).
\nEven sparse features didn’t drastically improve performance after a point.

", "rawMarkdown": "Hi everyone,\nI recently ran an experiment on a regression problem (~525k rows) using LightGBM with GPU acceleration (Tesla T4) to evaluate how performance scales with incremental feature selection. Thought I’d share my results here for anyone exploring similar approaches.\n\n\n\n🧪 Setup\nTarget: Continuous variable (non-categorical)\nFeatures: ~200 after correlation filtering\nFeature Selection: SelectKBest with F-test (f_regression)\nScaler: StandardScaler\nModel: LightGBM with GPU (OpenCL)\nTrain Size: 420,709 rows\nMetrics: R², RMSE, MAE (on validation split)\n\nbelow I have posted my results with the corresponding metrics for the k top features\n\n📊 Results\nK \t R²\tRMSE\tMAE\n5\t 0.3723\t0.8053\t0.5462\n10\t 0.4516\t0.7527\t0.5216\n15 \t 0.4950\t0.7222\t0.5046\n20\t 0.5181\t0.7055\t0.4960\n25\t 0.5371\t0.6915\t0.4871\n30\t 0.5528\t0.6797\t0.4817\n35\t 0.5642\t0.6710\t0.4764\n40\t 0.5670\t0.6688\t0.4750\n45\t 0.5739\t0.6634\t0.4718\n50\t 0.5801\t0.6586\t0.4686\n55\t 0.5800\t0.6586\t0.4668\n60\t 0.5819\t0.6572\t0.4661\n65\t 0.5840\t0.6555\t0.4635\n📌 Early stopped at 65 features due to R² plateau.\n\n🔍 Observations\nLargest gains were seen in the first 30–40 features.\nMarginal returns diminish after 50.\nGPU training times were super fast (~10–20MB GPU memory per batch).\nEven sparse features didn’t drastically improve performance after a point.\n\n" } ], "comments": [], "raw_markdown_by_id": { "3226795": "Hi everyone,\nI recently ran an experiment on a regression problem (~525k rows) using LightGBM with GPU acceleration (Tesla T4) to evaluate how performance scales with incremental feature selection. Thought I’d share my results here for anyone exploring similar approaches.\n\n\n\n🧪 Setup\nTarget: Continuous variable (non-categorical)\nFeatures: ~200 after correlation filtering\nFeature Selection: SelectKBest with F-test (f_regression)\nScaler: StandardScaler\nModel: LightGBM with GPU (OpenCL)\nTrain Size: 420,709 rows\nMetrics: R², RMSE, MAE (on validation split)\n\nbelow I have posted my results with the corresponding metrics for the k top features\n\n📊 Results\nK \t R²\tRMSE\tMAE\n5\t 0.3723\t0.8053\t0.5462\n10\t 0.4516\t0.7527\t0.5216\n15 \t 0.4950\t0.7222\t0.5046\n20\t 0.5181\t0.7055\t0.4960\n25\t 0.5371\t0.6915\t0.4871\n30\t 0.5528\t0.6797\t0.4817\n35\t 0.5642\t0.6710\t0.4764\n40\t 0.5670\t0.6688\t0.4750\n45\t 0.5739\t0.6634\t0.4718\n50\t 0.5801\t0.6586\t0.4686\n55\t 0.5800\t0.6586\t0.4668\n60\t 0.5819\t0.6572\t0.4661\n65\t 0.5840\t0.6555\t0.4635\n📌 Early stopped at 65 features due to R² plateau.\n\n🔍 Observations\nLargest gains were seen in the first 30–40 features.\nMarginal returns diminish after 50.\nGPU training times were super fast (~10–20MB GPU memory per batch).\nEven sparse features didn’t drastically improve performance after a point.\n\n" } }