{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.11.11","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":96164,"databundleVersionId":11418275,"sourceType":"competition"}],"dockerImageVersionId":31040,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Feature Discovery Analysis - Building on 0.12 Success\n\n## Overview\nThis notebook analyzes the successful approach from [Lion-li-li](https://www.kaggle.com/nina2025https://www.kaggle.com/nina2025) on [DRW - Crypto Market Prediction | TidaDRW - Crypto Market Prediction | Tida](https://www.kaggle.com/code/nina2025/drw-crypto-market-prediction-tida) to identify more predictive X features and optimize feature engineering.\n\n### Successful Features from Lion-li-li approach:\n```\nX863, X856, X598, X862, X385, X852, X603, X860, X674,\nX415, X345, X855, X174, X302, X178, X168, X612,\nX888, X421, X333\n```\n\n### Goals:\n1. **Analyze successful features** - What makes them predictive?\n2. **Find similar features** - Discover more X features with strong correlations\n3. **Advanced feature engineering** - Build on successful patterns\n4. **Feature selection optimization** - Remove noise, keep signal","metadata":{}},{"cell_type":"markdown","source":"## 1. Environment Setup","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom scipy import stats\nfrom scipy.stats import pearsonr\nfrom sklearn.model_selection import KFold\nfrom sklearn.preprocessing import RobustScaler\nfrom sklearn.feature_selection import SelectKBest, f_regression, mutual_info_regression\nfrom sklearn.ensemble import RandomForestRegressor\nimport lightgbm as lgb\nimport xgboost as xgb\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Set display options\npd.set_option('display.max_columns', 100)\npd.set_option('display.max_rows', 100)\nplt.style.use('seaborn-v0_8-darkgrid')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T17:25:47.980699Z","iopub.execute_input":"2025-06-25T17:25:47.980978Z","iopub.status.idle":"2025-06-25T17:26:00.489410Z","shell.execute_reply.started":"2025-06-25T17:25:47.980946Z","shell.execute_reply":"2025-06-25T17:26:00.488464Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2. Data Loading and Memory Optimization","metadata":{}},{"cell_type":"code","source":"class CFG:\n    train_path = \"/kaggle/input/drw-crypto-market-prediction/train.parquet\"\n    test_path = \"/kaggle/input/drw-crypto-market-prediction/test.parquet\"\n    \n    SUCCESSFUL_X_FEATURES = [\n        \"X863\", \"X856\", \"X598\", \"X862\", \"X385\", \"X852\", \"X603\", \"X860\", \"X674\",\n        \"X415\", \"X345\", \"X855\", \"X174\", \"X302\", \"X178\", \"X168\", \"X612\",\n        \"X888\", \"X421\", \"X333\"\n    ]\n    \n    MARKET_FEATURES = [\"bid_qty\", \"ask_qty\", \"buy_qty\", \"sell_qty\", \"volume\"]\n\ndef reduce_mem_usage(dataframe, dataset):    \n    \"\"\"Reduce memory usage by optimizing data types\"\"\"\n    print(f'Reducing memory usage for: {dataset}')\n    initial_mem_usage = dataframe.memory_usage().sum() / 1024**2\n    \n    for col in dataframe.columns:\n        col_type = dataframe[col].dtype\n        c_min = dataframe[col].min()\n        c_max = dataframe[col].max()\n        \n        if str(col_type)[:3] == 'int':\n            if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                dataframe[col] = dataframe[col].astype(np.int8)\n            elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                dataframe[col] = dataframe[col].astype(np.int16)\n            elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                dataframe[col] = dataframe[col].astype(np.int32)\n        else:\n            if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                dataframe[col] = dataframe[col].astype(np.float16)\n            elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                dataframe[col] = dataframe[col].astype(np.float32)\n\n    final_mem_usage = dataframe.memory_usage().sum() / 1024**2\n    print(f'--- Memory usage before: {initial_mem_usage:.2f} MB')\n    print(f'--- Memory usage after: {final_mem_usage:.2f} MB')\n    print(f'--- Decreased by: {100 * (initial_mem_usage - final_mem_usage) / initial_mem_usage:.1f}%\\n')\n    return dataframe\n\ntrain_df = pd.read_parquet(CFG.train_path).reset_index(drop=True)\ntrain_df = reduce_mem_usage(train_df, 'train')\n\ntest_df = pd.read_parquet(CFG.test_path).reset_index(drop=True)\ntest_df = reduce_mem_usage(test_df, 'test')\n\nprint(f\"Data shapes - Train: {train_df.shape}, Test: {test_df.shape}\")\nprint(f\"Target stats: Mean={train_df['label'].mean():.6f}, Std={train_df['label'].std():.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T17:29:08.425038Z","iopub.execute_input":"2025-06-25T17:29:08.425399Z","iopub.status.idle":"2025-06-25T17:30:17.867835Z","shell.execute_reply.started":"2025-06-25T17:29:08.425374Z","shell.execute_reply":"2025-06-25T17:30:17.866724Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 3. Analyze Successful Features\n\nFirst, let's understand what makes the successful features from the 0.12 model special.","metadata":{}},{"cell_type":"code","source":"def analyze_successful_features():\n    \"\"\"Analyze the characteristics of successful features\"\"\"\n    \n    print(\"=== ANALYSIS OF SUCCESSFUL FEATURES ===\")\n    \n    successful_correlations = {}\n    successful_stats = {}\n    \n    for feature in CFG.SUCCESSFUL_X_FEATURES:\n        if feature in train_df.columns:\n            # Correlation with target\n            corr = train_df[feature].corr(train_df['label'])\n            successful_correlations[feature] = corr\n            \n            # Statistical properties\n            stats_dict = {\n                'min': train_df[feature].min(),\n                'max': train_df[feature].max(),\n                'mean': train_df[feature].mean(),\n                'std': train_df[feature].std(),\n                'skew': train_df[feature].skew(),\n                'unique_values': train_df[feature].nunique(),\n                'correlation': corr\n            }\n            successful_stats[feature] = stats_dict\n    \n    stats_df = pd.DataFrame(successful_stats).T\n    stats_df = stats_df.sort_values('correlation', key=abs, ascending=False)\n    \n    print(\"\\nTop 10 successful features by correlation:\")\n    print(stats_df[['correlation', 'mean', 'std', 'unique_values']].head(10))\n    \n    plt.figure(figsize=(12, 8))\n    correlations = stats_df['correlation'].values\n    features = stats_df.index.tolist()\n    \n    colors = ['red' if abs(x) > 0.02 else 'orange' if abs(x) > 0.01 else 'blue' for x in correlations]\n    plt.barh(range(len(features)), correlations, color=colors)\n    plt.yticks(range(len(features)), features)\n    plt.xlabel('Correlation with Target')\n    plt.title('Successful Features: Target Correlations')\n    plt.axvline(x=0, color='black', linestyle='-', linewidth=0.5)\n    plt.axvline(x=0.02, color='red', linestyle='--', alpha=0.5, label='|corr| > 0.02')\n    plt.axvline(x=-0.02, color='red', linestyle='--', alpha=0.5)\n    plt.axvline(x=0.01, color='orange', linestyle='--', alpha=0.5, label='|corr| > 0.01')\n    plt.axvline(x=-0.01, color='orange', linestyle='--', alpha=0.5)\n    plt.legend()\n    plt.gca().invert_yaxis()\n    plt.tight_layout()\n    plt.show()\n    \n    print(f\"\\n=== SUCCESSFUL FEATURES SUMMARY ===\")\n    print(f\"Total successful features: {len(successful_correlations)}\")\n    print(f\"Mean absolute correlation: {np.mean([abs(c) for c in successful_correlations.values()]):.4f}\")\n    print(f\"Features with |corr| > 0.02: {sum(1 for c in successful_correlations.values() if abs(c) > 0.02)}\")\n    print(f\"Features with |corr| > 0.01: {sum(1 for c in successful_correlations.values() if abs(c) > 0.01)}\")\n    \n    return stats_df, successful_correlations\n\nsuccessful_stats_df, successful_corrs = analyze_successful_features()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T17:35:56.257682Z","iopub.execute_input":"2025-06-25T17:35:56.259299Z","iopub.status.idle":"2025-06-25T17:35:59.461110Z","shell.execute_reply.started":"2025-06-25T17:35:56.259265Z","shell.execute_reply":"2025-06-25T17:35:59.459944Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 4. Comprehensive X Feature Discovery\n\nAnalyze ALL X features to find more with similar predictive power.","metadata":{}},{"cell_type":"code","source":"def comprehensive_x_feature_analysis():\n    \"\"\"Analyze all X features to find more predictive ones\"\"\"\n    \n    print(\"=== COMPREHENSIVE X FEATURE ANALYSIS ===\")\n    \n    all_x_features = [col for col in train_df.columns if col.startswith('X')]\n    print(f\"Total X features available: {len(all_x_features)}\")\n    \n    all_correlations = {}\n    all_stats = {}\n    \n    print(\"Calculating correlations for all X features...\")\n    for i, feature in enumerate(all_x_features):\n        if i % 100 == 0:\n            print(f\"Progress: {i}/{len(all_x_features)}\")\n            \n        corr = train_df[feature].corr(train_df['label'])\n        if not np.isnan(corr):\n            all_correlations[feature] = corr\n            \n            if abs(corr) > 0.005:  # Only calculate detailed stats for promising features\n                all_stats[feature] = {\n                    'correlation': corr,\n                    'mean': train_df[feature].mean(),\n                    'std': train_df[feature].std(),\n                    'unique_values': train_df[feature].nunique(),\n                    'is_successful': feature in CFG.SUCCESSFUL_X_FEATURES\n                }\n    \n    # Sort by absolute correlation\n    sorted_correlations = sorted(all_correlations.items(), key=lambda x: abs(x[1]), reverse=True)\n    \n    print(f\"\\nTop 50 X features by absolute correlation:\")\n    for i, (feature, corr) in enumerate(sorted_correlations[:50]):\n        is_successful = \"✓\" if feature in CFG.SUCCESSFUL_X_FEATURES else \" \"\n        print(f\"{i+1:2d}. {feature}: {corr:7.4f} {is_successful}\")\n    \n\n    strong_threshold = 0.015  # Threshold for strong correlation\n    strong_features = [f for f, c in sorted_correlations if abs(c) > strong_threshold]\n    new_strong_features = [f for f in strong_features if f not in CFG.SUCCESSFUL_X_FEATURES]\n    \n    print(f\"\\n=== DISCOVERY RESULTS ===\")\n    print(f\"Features with |correlation| > {strong_threshold}: {len(strong_features)}\")\n    print(f\"New strong features (not in successful set): {len(new_strong_features)}\")\n    \n    if new_strong_features:\n        print(f\"\\nNEW STRONG FEATURES TO TEST:\")\n        for feature in new_strong_features[:15]:  # Show top 15 new ones\n            corr = all_correlations[feature]\n            print(f\"  {feature}: {corr:.4f}\")\n    \n    stats_df = pd.DataFrame(all_stats).T\n    \n    return sorted_correlations, new_strong_features, stats_df\n\nall_x_correlations, new_strong_features, comprehensive_stats = comprehensive_x_feature_analysis()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T17:36:07.297079Z","iopub.execute_input":"2025-06-25T17:36:07.297514Z","iopub.status.idle":"2025-06-25T17:37:09.525597Z","shell.execute_reply.started":"2025-06-25T17:36:07.297485Z","shell.execute_reply":"2025-06-25T17:37:09.524567Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 5. Feature Selection and Validation\n\nUse multiple feature selection methods to validate our findings.","metadata":{}},{"cell_type":"code","source":"def advanced_feature_selection():\n  \"\"\"Use multiple feature selection methods - FIXED VERSION\"\"\"\n\n  print(\"=== ADVANCED FEATURE SELECTION ===\")\n\n  all_x_features = [col for col in train_df.columns if col.startswith('X')]\n  X = train_df[all_x_features].copy()\n  y = train_df['label'].copy()\n\n  print(f\"Original shape: {X.shape}\")\n\n  X = X.replace([np.inf, -np.inf], np.nan)\n\n  for col in X.columns:\n      q99 = X[col].quantile(0.99)\n      q01 = X[col].quantile(0.01)\n      if not pd.isna(q99) and not pd.isna(q01):\n          X[col] = X[col].clip(lower=q01, upper=q99)\n\n  for col in X.columns:\n      if X[col].isna().any():\n          median_val = X[col].median()\n          X[col] = X[col].fillna(median_val if not pd.isna(median_val) else 0)\n\n  print(f\"Cleaned shape: {X.shape}\")\n\n  correlation_based = [f for f, c in all_x_correlations[:100] if abs(c) > 0.005]\n\n  print(f\"Selected {len(correlation_based)} features with |correlation| > 0.005\")\n\n  consensus_features = [(f, dict(all_x_correlations)[f]) for f in correlation_based]\n  consensus_features = sorted(consensus_features, key=lambda x: abs(x[1]), reverse=True)\n\n  return consensus_features, {'correlation_based': correlation_based}\n\nconsensus_features, selection_methods = advanced_feature_selection()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T17:41:11.421326Z","iopub.execute_input":"2025-06-25T17:41:11.422079Z","iopub.status.idle":"2025-06-25T17:42:06.397778Z","shell.execute_reply.started":"2025-06-25T17:41:11.422053Z","shell.execute_reply":"2025-06-25T17:42:06.396406Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 6. Create Enhanced Feature Set\n\nCombine successful features with newly discovered ones.","metadata":{}},{"cell_type":"code","source":"def create_enhanced_feature_set():\n    \"\"\"Create an enhanced feature set combining successful and new features\"\"\"\n    \n    print(\"=== CREATING ENHANCED FEATURE SET ===\")\n    \n    # Start with successful features as base\n    enhanced_features = CFG.SUCCESSFUL_X_FEATURES.copy()\n    print(f\"Base successful features: {len(enhanced_features)}\")\n    \n    # Add market features\n    enhanced_features.extend(CFG.MARKET_FEATURES)\n    print(f\"After adding market features: {len(enhanced_features)}\")\n    \n    # Add top new features from correlation analysis\n    new_from_correlation = new_strong_features[:10]  # Top 10 new strong features\n    enhanced_features.extend(new_from_correlation)\n    print(f\"After adding new correlation features: {len(enhanced_features)}\")\n    \n    # Add top consensus features not already included\n    consensus_to_add = []\n    for feature, score in consensus_features[:30]:  # Check top 30 consensus\n        if feature not in enhanced_features and score > 2:  # High consensus score\n            consensus_to_add.append(feature)\n        if len(consensus_to_add) >= 15:  # Limit to 15 additional\n            break\n    \n    enhanced_features.extend(consensus_to_add)\n    print(f\"After adding consensus features: {len(enhanced_features)}\")\n    \n    # Remove duplicates and sort\n    enhanced_features = list(set(enhanced_features))\n    enhanced_features.sort()\n    \n    print(f\"\\nFinal enhanced feature set: {len(enhanced_features)} features\")\n    \n    # Calculate correlations for enhanced set\n    enhanced_correlations = {}\n    for feature in enhanced_features:\n        if feature in train_df.columns and feature != 'label':\n            corr = train_df[feature].corr(train_df['label'])\n            if not np.isnan(corr):\n                enhanced_correlations[feature] = corr\n    \n    # Sort by correlation strength\n    sorted_enhanced = sorted(enhanced_correlations.items(), key=lambda x: abs(x[1]), reverse=True)\n    \n    print(f\"\\nTop 20 features in enhanced set:\")\n    for i, (feature, corr) in enumerate(sorted_enhanced[:20]):\n        feature_type = \"Market\" if feature in CFG.MARKET_FEATURES else \"Successful\" if feature in CFG.SUCCESSFUL_X_FEATURES else \"New\"\n        print(f\"{i+1:2d}. {feature}: {corr:7.4f} ({feature_type})\")\n    \n    return enhanced_features, enhanced_correlations\n\nenhanced_feature_set, enhanced_correlations = create_enhanced_feature_set()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T17:42:19.931321Z","iopub.execute_input":"2025-06-25T17:42:19.931677Z","iopub.status.idle":"2025-06-25T17:42:20.528062Z","shell.execute_reply.started":"2025-06-25T17:42:19.931653Z","shell.execute_reply":"2025-06-25T17:42:20.526928Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 7. Advanced Feature Engineering\n\nCreate sophisticated features based on successful patterns.","metadata":{}},{"cell_type":"code","source":"def advanced_feature_engineering(df, feature_set):\n    \"\"\"Create advanced features based on successful patterns\"\"\"\n    \n    print(\"=== ADVANCED FEATURE ENGINEERING ===\")\n    \n    result_df = df.copy()\n    \n    # 1. Successful microstructure features from 0.12 model\n    if all(col in df.columns for col in CFG.MARKET_FEATURES):\n        print(\"Creating microstructure features...\")\n        \n        # Original successful features\n        result_df['log_volume'] = np.log1p(df['volume'])\n        result_df['bid_ask_imbalance'] = (df['bid_qty'] - df['ask_qty']) / (df['bid_qty'] + df['ask_qty'] + 1e-8)\n        result_df['order_flow_imbalance'] = (df['buy_qty'] - df['sell_qty']) / (df['buy_qty'] + df['sell_qty'] + 1e-8)\n        result_df['liquidity_ratio'] = (df['bid_qty'] + df['ask_qty']) / (df['volume'] + 1e-8)\n        \n        # Additional microstructure features\n        result_df['volume_weighted_sell'] = df['sell_qty'] * df['volume']\n        result_df['buy_sell_ratio'] = df['buy_qty'] / (df['sell_qty'] + 1e-8)\n        result_df['selling_pressure'] = df['sell_qty'] / (df['volume'] + 1e-8)\n        result_df['effective_spread_proxy'] = np.abs(df['buy_qty'] - df['sell_qty']) / (df['volume'] + 1e-8)\n        \n        # Enhanced ratios\n        result_df['volume_concentration'] = df['volume'] / (df['bid_qty'] + df['ask_qty'] + df['volume'] + 1e-8)\n        result_df['aggressive_ratio'] = (df['buy_qty'] + df['sell_qty']) / (df['volume'] + 1e-8)\n        result_df['market_efficiency'] = np.abs(df['buy_qty'] - df['sell_qty']) / (df['volume'] + 1e-8)\n        \n        # Log transformations\n        for col in CFG.MARKET_FEATURES:\n            result_df[f'log_{col}'] = np.log1p(df[col])\n    \n    # 2. Top X feature interactions\n    top_x_features = [f for f in enhanced_feature_set if f.startswith('X')][:10]  # Top 10 X features\n    \n    if len(top_x_features) >= 2:\n        print(f\"Creating interactions for top {len(top_x_features)} X features...\")\n        \n        # Pairwise interactions for top features\n        for i, feat1 in enumerate(top_x_features[:5]):  # Limit to avoid explosion\n            if feat1 in df.columns:\n                for feat2 in top_x_features[i+1:6]:  # Max 5 interactions per feature\n                    if feat2 in df.columns:\n                        # Multiplicative interaction\n                        result_df[f'{feat1}_x_{feat2}'] = df[feat1] * df[feat2]\n                        \n                        # Ratio interaction\n                        result_df[f'{feat1}_div_{feat2}'] = df[feat1] / (df[feat2] + 1e-8)\n    \n    # 3. X features with market features interactions\n    print(\"Creating X-feature and market feature interactions...\")\n    for x_feat in top_x_features[:5]:  # Top 5 X features\n        if x_feat in df.columns:\n            for market_feat in CFG.MARKET_FEATURES:\n                if market_feat in df.columns:\n                    # Key interactions\n                    result_df[f'{x_feat}_x_{market_feat}'] = df[x_feat] * df[market_feat]\n                    result_df[f'{x_feat}_div_{market_feat}'] = df[x_feat] / (df[market_feat] + 1e-8)\n    \n    # 4. Polynomial features for strongest predictors\n    strongest_features = [f for f, c in sorted(enhanced_correlations.items(), key=lambda x: abs(x[1]), reverse=True)[:5]]\n    \n    print(f\"Creating polynomial features for strongest predictors...\")\n    for feature in strongest_features:\n        if feature in df.columns:\n            result_df[f'{feature}_squared'] = df[feature] ** 2\n            result_df[f'{feature}_sqrt'] = np.sign(df[feature]) * np.sqrt(np.abs(df[feature]))\n    \n    # 5. Rolling statistics (simplified for static data)\n    print(\"Creating statistical transformations...\")\n    for feature in top_x_features[:3] + CFG.MARKET_FEATURES:\n        if feature in df.columns:\n            # Rank normalization\n            result_df[f'{feature}_rank'] = df[feature].rank(pct=True)\n            \n            # Z-score normalization\n            mean_val = df[feature].mean()\n            std_val = df[feature].std()\n            if std_val > 0:\n                result_df[f'{feature}_zscore'] = (df[feature] - mean_val) / std_val\n    \n    # Clean up inf/nan values\n    result_df = result_df.replace([np.inf, -np.inf], np.nan)\n    \n    # Fill NaN with median for robustness\n    for col in result_df.columns:\n        if result_df[col].isna().any():\n            median_val = result_df[col].median()\n            result_df[col] = result_df[col].fillna(median_val if not pd.isna(median_val) else 0)\n    \n    new_features = [col for col in result_df.columns if col not in df.columns]\n    print(f\"Created {len(new_features)} new engineered features\")\n    \n    return result_df, new_features\n\n# Apply feature engineering\ntrain_engineered, new_feature_names = advanced_feature_engineering(train_df, enhanced_feature_set)\ntest_engineered, _ = advanced_feature_engineering(test_df, enhanced_feature_set)\n\nprint(f\"\\nEngineered data shapes - Train: {train_engineered.shape}, Test: {test_engineered.shape}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T17:42:27.061069Z","iopub.execute_input":"2025-06-25T17:42:27.061436Z","iopub.status.idle":"2025-06-25T17:43:00.453244Z","shell.execute_reply.started":"2025-06-25T17:42:27.061412Z","shell.execute_reply":"2025-06-25T17:43:00.452119Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 8. Feature Selection Optimization\n\nSelect the best features for modeling from our enhanced set.","metadata":{}},{"cell_type":"code","source":"def optimize_feature_selection():\n    \"\"\"Select optimal features from enhanced set\"\"\"\n    \n    print(\"=== OPTIMIZING FEATURE SELECTION ===\")\n    \n    # Calculate correlations for all features\n    all_feature_correlations = {}\n    \n    for col in train_engineered.columns:\n        if col != 'label':\n            corr = train_engineered[col].corr(train_engineered['label'])\n            if not np.isnan(corr):\n                all_feature_correlations[col] = corr\n    \n    # Sort by absolute correlation\n    sorted_all_features = sorted(all_feature_correlations.items(), key=lambda x: abs(x[1]), reverse=True)\n    \n    print(f\"Total features available: {len(all_feature_correlations)}\")\n    \n    # Feature selection criteria\n    correlation_threshold = 0.008  # Minimum correlation threshold\n    max_features = 60  # Maximum number of features\n    \n    # Select features above threshold\n    selected_features = []\n    selected_correlations = {}\n    \n    for feature, corr in sorted_all_features:\n        if abs(corr) >= correlation_threshold and len(selected_features) < max_features:\n            selected_features.append(feature)\n            selected_correlations[feature] = corr\n    \n    print(f\"Selected {len(selected_features)} features with |correlation| >= {correlation_threshold}\")\n    \n    # Analyze feature types\n    feature_types = {\n        'original_x': [f for f in selected_features if f.startswith('X') and f in CFG.SUCCESSFUL_X_FEATURES],\n        'new_x': [f for f in selected_features if f.startswith('X') and f not in CFG.SUCCESSFUL_X_FEATURES],\n        'market': [f for f in selected_features if f in CFG.MARKET_FEATURES],\n        'engineered': [f for f in selected_features if f in new_feature_names]\n    }\n    \n    print(f\"\\n=== FEATURE BREAKDOWN ===\")\n    for ftype, features in feature_types.items():\n        print(f\"{ftype}: {len(features)} features\")\n        if features:\n            avg_corr = np.mean([abs(selected_correlations[f]) for f in features])\n            print(f\"  Average |correlation|: {avg_corr:.4f}\")\n    \n    print(f\"\\nTop 20 selected features:\")\n    for i, feature in enumerate(selected_features[:20]):\n        corr = selected_correlations[feature]\n        ftype = 'Market' if feature in CFG.MARKET_FEATURES else 'Orig-X' if feature in CFG.SUCCESSFUL_X_FEATURES else 'New-X' if feature.startswith('X') else 'Eng'\n        print(f\"{i+1:2d}. {feature}: {corr:7.4f} ({ftype})\")\n    \n    return selected_features, selected_correlations, feature_types\n\noptimal_features, optimal_correlations, feature_breakdown = optimize_feature_selection()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T17:44:28.936528Z","iopub.execute_input":"2025-06-25T17:44:28.936921Z","iopub.status.idle":"2025-06-25T17:44:45.166235Z","shell.execute_reply.started":"2025-06-25T17:44:28.936895Z","shell.execute_reply":"2025-06-25T17:44:45.165252Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 9. Quick Model Validation\n\nTest our enhanced feature set with a simple model to validate improvement.","metadata":{}},{"cell_type":"code","source":"def quick_model_validation():\n    \"\"\"Quick validation of enhanced feature set\"\"\"\n    \n    print(\"=== QUICK MODEL VALIDATION ===\")\n    \n    # Prepare data\n    X_train = train_engineered[optimal_features].fillna(0)\n    y_train = train_engineered['label']\n    X_test = test_engineered[optimal_features].fillna(0)\n    \n    print(f\"Training data shape: {X_train.shape}\")\n    print(f\"Test data shape: {X_test.shape}\")\n    \n    # Simple time series split for validation\n    split_idx = int(0.8 * len(X_train))\n    \n    X_train_split = X_train.iloc[:split_idx]\n    X_val_split = X_train.iloc[split_idx:]\n    y_train_split = y_train.iloc[:split_idx]\n    y_val_split = y_train.iloc[split_idx:]\n    \n    print(f\"Training split: {X_train_split.shape}\")\n    print(f\"Validation split: {X_val_split.shape}\")\n    \n    results = {}\n    \n    # Model 1: LightGBM\n    print(\"\\n1. Testing LightGBM...\")\n    lgb_model = lgb.LGBMRegressor(\n        n_estimators=300,\n        learning_rate=0.05,\n        num_leaves=31,\n        min_child_samples=50,\n        subsample=0.8,\n        colsample_bytree=0.8,\n        reg_alpha=10,\n        reg_lambda=10,\n        random_state=42,\n        verbose=-1\n    )\n    \n    lgb_model.fit(X_train_split, y_train_split, \n                  eval_set=[(X_val_split, y_val_split)],\n                  callbacks=[])\n    \n    lgb_train_pred = lgb_model.predict(X_train)\n    lgb_val_pred = lgb_model.predict(X_val_split)\n    lgb_test_pred = lgb_model.predict(X_test)\n    \n    lgb_train_corr = pearsonr(y_train, lgb_train_pred)[0]\n    lgb_val_corr = pearsonr(y_val_split, lgb_val_pred)[0]\n    \n    results['LightGBM'] = {\n        'train_corr': lgb_train_corr,\n        'val_corr': lgb_val_corr,\n        'test_pred': lgb_test_pred\n    }\n    \n    print(f\"LightGBM - Train: {lgb_train_corr:.4f}, Val: {lgb_val_corr:.4f}\")\n    \n    # Model 2: XGBoost (simplified)\n    print(\"\\n2. Testing XGBoost...\")\n    xgb_model = xgb.XGBRegressor(\n        n_estimators=300,\n        learning_rate=0.05,\n        max_depth=10,\n        min_child_weight=10,\n        subsample=0.8,\n        colsample_bytree=0.8,\n        reg_alpha=20,\n        reg_lambda=40,\n        random_state=42,\n        verbosity=0\n    )\n    \n    xgb_model.fit(X_train_split, y_train_split,\n                  eval_set=[(X_val_split, y_val_split)],\n                  verbose=False)\n    \n    xgb_train_pred = xgb_model.predict(X_train)\n    xgb_val_pred = xgb_model.predict(X_val_split)\n    xgb_test_pred = xgb_model.predict(X_test)\n    \n    xgb_train_corr = pearsonr(y_train, xgb_train_pred)[0]\n    xgb_val_corr = pearsonr(y_val_split, xgb_val_pred)[0]\n    \n    results['XGBoost'] = {\n        'train_corr': xgb_train_corr,\n        'val_corr': xgb_val_corr,\n        'test_pred': xgb_test_pred\n    }\n    \n    print(f\"XGBoost - Train: {xgb_train_corr:.4f}, Val: {xgb_val_corr:.4f}\")\n    \n    # Simple ensemble\n    ensemble_test_pred = 0.5 * lgb_test_pred + 0.5 * xgb_test_pred\n    ensemble_train_pred = 0.5 * lgb_train_pred + 0.5 * xgb_train_pred\n    ensemble_train_corr = pearsonr(y_train, ensemble_train_pred)[0]\n    \n    results['Ensemble'] = {\n        'train_corr': ensemble_train_corr,\n        'val_corr': (lgb_val_corr + xgb_val_corr) / 2,  # Approximate\n        'test_pred': ensemble_test_pred\n    }\n    \n    print(f\"Ensemble - Train: {ensemble_train_corr:.4f}\")\n    \n    # Feature importance analysis\n    print(f\"\\n=== FEATURE IMPORTANCE (LightGBM) ===\")\n    feature_importance = pd.DataFrame({\n        'feature': optimal_features,\n        'importance': lgb_model.feature_importances_\n    }).sort_values('importance', ascending=False)\n    \n    print(\"Top 15 most important features:\")\n    for i, row in feature_importance.head(15).iterrows():\n        ftype = 'Market' if row['feature'] in CFG.MARKET_FEATURES else 'Orig-X' if row['feature'] in CFG.SUCCESSFUL_X_FEATURES else 'New-X' if row['feature'].startswith('X') else 'Eng'\n        print(f\"{row.name+1:2d}. {row['feature']}: {row['importance']:8.1f} ({ftype})\")\n    \n    return results, feature_importance\n\nvalidation_results, feature_importance_df = quick_model_validation()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T17:46:48.717337Z","iopub.execute_input":"2025-06-25T17:46:48.718095Z","iopub.status.idle":"2025-06-25T17:48:12.971015Z","shell.execute_reply.started":"2025-06-25T17:46:48.718065Z","shell.execute_reply":"2025-06-25T17:48:12.969926Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 10. Generate Enhanced Submissions","metadata":{}},{"cell_type":"code","source":"def generate_enhanced_submissions():\n    \"\"\"Generate submission files with enhanced features\"\"\"\n    \n    print(\"=== GENERATING ENHANCED SUBMISSIONS ===\")\n    \n    # Load sample submission format\n    sample_submission = pd.read_csv(\"/kaggle/input/drw-crypto-market-prediction/sample_submission.csv\")\n    \n    # Create submissions for each model\n    submissions = {}\n    \n    for model_name, results in validation_results.items():\n        submission = sample_submission.copy()\n        submission['prediction'] = results['test_pred']\n        \n        filename = f\"enhanced_{model_name.lower()}_submission.csv\"\n        submission.to_csv(filename, index=False)\n        \n        submissions[model_name] = {\n            'filename': filename,\n            'train_corr': results['train_corr'],\n            'val_corr': results['val_corr'],\n            'pred_stats': {\n                'min': results['test_pred'].min(),\n                'max': results['test_pred'].max(),\n                'mean': results['test_pred'].mean(),\n                'std': results['test_pred'].std()\n            }\n        }\n        \n        print(f\"✅ Saved {filename}\")\n        print(f\"   Train correlation: {results['train_corr']:.4f}\")\n        print(f\"   Validation correlation: {results['val_corr']:.4f}\")\n        print(f\"   Prediction range: [{results['test_pred'].min():.3f}, {results['test_pred'].max():.3f}]\")\n        print()\n    \n    return submissions\n\nsubmission_summary = generate_enhanced_submissions()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T17:49:03.008793Z","iopub.execute_input":"2025-06-25T17:49:03.009844Z","iopub.status.idle":"2025-06-25T17:49:06.830239Z","shell.execute_reply.started":"2025-06-25T17:49:03.009807Z","shell.execute_reply":"2025-06-25T17:49:06.829170Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 11. Summary and Recommendations","metadata":{}},{"cell_type":"code","source":"def final_summary_and_recommendations():\n    \"\"\"Provide final summary and next steps\"\"\"\n    \n    print(\"=\" * 80)\n    print(\"ENHANCED FEATURE DISCOVERY - FINAL SUMMARY\")\n    print(\"=\" * 80)\n    \n    print(f\"\\n🎯 FEATURE DISCOVERY RESULTS:\")\n    print(f\"• Analyzed {len([col for col in train_df.columns if col.startswith('X')])} total X features\")\n    print(f\"• Found {len(new_strong_features)} new strong features (|corr| > 0.015)\")\n    print(f\"• Created {len(new_feature_names)} engineered features\")\n    print(f\"• Final optimal set: {len(optimal_features)} features\")\n    \n    print(f\"\\n📊 MODEL PERFORMANCE:\")\n    for model_name, results in validation_results.items():\n        print(f\"• {model_name}: Train {results['train_corr']:.4f}, Val {results['val_corr']:.4f}\")\n    \n    print(f\"\\n🏆 BEST PERFORMERS:\")\n    best_val = max(validation_results.items(), key=lambda x: abs(x[1]['val_corr']))\n    print(f\"• Best validation: {best_val[0]} ({best_val[1]['val_corr']:.4f})\")\n    \n    print(f\"\\n🔧 KEY IMPROVEMENTS IDENTIFIED:\")\n    print(f\"• New predictive X features: {', '.join(new_strong_features[:5])}\")\n    print(f\"• Most important engineered features from top 5:\")\n    \n    top_engineered = feature_importance_df[feature_importance_df['feature'].isin(new_feature_names)].head(5)\n    for _, row in top_engineered.iterrows():\n        print(f\"  - {row['feature']} (importance: {row['importance']:.1f})\")\n    \n    \n    print(f\"\\n📝 FILES CREATED:\")\n    for model_name, details in submission_summary.items():\n        print(f\"• {details['filename']} - {model_name} model\")\n    \n    \n    print(\"\\n\" + \"=\" * 80)\n\nfinal_summary_and_recommendations()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-25T17:49:22.801738Z","iopub.execute_input":"2025-06-25T17:49:22.802629Z","iopub.status.idle":"2025-06-25T17:49:22.825385Z","shell.execute_reply.started":"2025-06-25T17:49:22.802597Z","shell.execute_reply":"2025-06-25T17:49:22.824254Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Summary\n\nThis notebook has systematically analyzed and enhanced the successful initial approach by:\n\n### 🔍 **Discovery Process:**\n1. **Analyzed successful features** from the initial model\n2. **Discovered new predictive X features** through comprehensive correlation analysis\n3. **Applied multiple feature selection methods** for validation\n4. **Created advanced engineered features** based on successful patterns\n5. **Optimized feature selection** for maximum predictive power","metadata":{}}]}