{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":96164,"databundleVersionId":12993472,"sourceType":"competition"}],"dockerImageVersionId":31089,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 📊 D-R-W Crypto Market Prediction\n## Submitted by Dur e Yashfeen ","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2025-07-24T20:48:01.486859Z","iopub.execute_input":"2025-07-24T20:48:01.487241Z","iopub.status.idle":"2025-07-24T20:48:01.493983Z","shell.execute_reply.started":"2025-07-24T20:48:01.487213Z","shell.execute_reply":"2025-07-24T20:48:01.493333Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Pipeline ","metadata":{}},{"cell_type":"code","source":"#%% [markdown]\n### 🔍 1. Problem Understanding: Crypto Market Prediction\n# The challenge: Predict crypto market movements using noisy, volatile market data\n# Key obstacles: \n#   - Extreme volatility (flash crashes, pumps)\n#   - 24/7 market with changing regimes\n#   - Whale manipulation causing abnormal patterns\n#\n# My solution: Combine Prophet's temporal anomaly detection with XGBoost's predictive power\n#%% [code]\nimport numpy as np\nimport pandas as pd\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Install Prophet if needed\ntry:\n    import prophet\nexcept ImportError:\n    print(\"Prophet not found. Installing...\")\n    !pip install prophet --quiet\n    print(\"Prophet installed.\")\n\n# Core imports\nfrom typing import Dict, List, Tuple, Optional, Any\nfrom dataclasses import dataclass\nimport pickle\nimport gc\nfrom collections import defaultdict\nimport logging\n\n# ML imports\nfrom sklearn.model_selection import TimeSeriesSplit\nfrom sklearn.preprocessing import RobustScaler\nfrom sklearn.isotonic import IsotonicRegression\nfrom xgboost import XGBRegressor\nfrom scipy.stats import pearsonr\nimport joblib\n\n# Prophet\nfrom prophet import Prophet\nlogging.getLogger('prophet').setLevel(logging.WARNING)\n\n# Visualization\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n# Configure logging\nlogging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')\nlogger = logging.getLogger(__name__)\nlogger.info(\"Initializing Temporal Anomaly Adjustment Pipeline...\")\n\n#%% [markdown]\n### ⚙️ 2. Configuration & Data Setup\n# Why these features? I selected market microstructure signals (bid/ask quantities) \n# combined with technical indicators showing highest predictive power in EDA\n#%% [code]\n# File paths (Kaggle environment)\nTRAIN_DATA_PATH = \"/kaggle/input/drw-crypto-market-prediction/train.parquet\"\nTEST_DATA_PATH = \"/kaggle/input/drw-crypto-market-prediction/test.parquet\"\nSUBMISSION_TEMPLATE_PATH = \"/kaggle/input/drw-crypto-market-prediction/sample_submission.csv\"\n\n# Feature selection - market microstructure + key technical indicators\nSELECTED_PREDICTORS = ['X363', 'X321', 'X405', 'X730', 'X523', 'X756', 'X589', 'X462', 'X779',\n                       'X25', 'X532', 'X520', 'X329', 'X383', 'X751', 'X535', 'X639', 'X596', 'X761',\n                       \"X752\", \"X287\", \"X298\", \"X759\", \"X302\", \"X55\", \"X56\", \"X52\", \"X303\", \"X51\",\n                       \"X598\", \"X385\", \"X603\", \"X674\", \"X415\", \"X345\", \"X174\", \"X178\", \"X168\", \"X612\",\n                       \"bid_qty\", \"ask_qty\", \"buy_qty\", \"sell_qty\"]\n\nINPUT_COLUMNS = SELECTED_PREDICTORS + [\"volume\"]\nPREDICTION_TARGET = \"label\"\n\n#%% [markdown]\n### 🕵️ 3. Temporal Anomaly Detection Engine\n# I use Prophet to find time-based anomalies because:\n#   - Automatically handles multiple seasonality (intraday, weekly)\n#   - Robust to missing data common in crypto\n#   - Provides uncertainty intervals for anomaly scoring\n#%% [code]\n@dataclass\nclass AnomalyProfile:\n    \"\"\"Stores patterns for feature's temporal anomalies\"\"\"\n    feature_name: str\n    value_segments: List[Tuple[float, float]]  # Anomalous value ranges\n    anomaly_scores: Dict[float, float]  # Value → anomaly score\n    correction_intensity: Dict[float, float]  # Value → correction strength\n    temporal_trends: Dict[str, Any]  # Time patterns\n\nclass TemporalAnomalyDetector:\n    \"\"\"Identifies temporal anomalies using Prophet and creates correction rules\"\"\"\n    \n    def __init__(self):\n        self.profiles: Dict[str, AnomalyProfile] = {}\n    \n    def impute_temporal_extremes(self, df: pd.DataFrame, feature: str,\n                                 time_index_col: str = '__index_level_0__') -> pd.Series:\n        \"\"\"Replace extreme values using Prophet's forecasts\"\"\"\n        imputed = df[feature].copy()\n        finite_vals = imputed[np.isfinite(imputed)]\n        \n        if len(finite_vals) < 50: \n            logger.warning(f\"  Insufficient data for {feature} imputation. Using median.\")\n            return imputed.fillna(finite_vals.median() if finite_vals.any() else 0)\n        \n        # Identify extremes (0.1% and 99.9% tails)\n        lower_bound = np.percentile(finite_vals, 0.1)\n        upper_bound = np.percentile(finite_vals, 99.9)\n        extreme_mask = (imputed < lower_bound) | (imputed > upper_bound) | ~np.isfinite(imputed)\n        \n        if not extreme_mask.any():\n            return imputed  # No extremes found\n        \n        try:\n            # Prepare Prophet data\n            ts_df = pd.DataFrame({'ds': df[time_index_col], 'y': imputed})\n            ts_df.loc[extreme_mask, 'y'] = np.nan\n            ts_df_clean = ts_df.dropna(subset=['y'])\n            \n            if len(ts_df_clean) < 100:\n                return imputed.fillna(finite_vals.median())\n            \n            # Fit Prophet (optimized for crypto volatility)\n            model = Prophet(\n                changepoint_prior_scale=0.15, \n                interval_width=0.98,\n                weekly_seasonality=True,\n                daily_seasonality=True\n            )\n            model.fit(ts_df_clean)\n            \n            # Predict and replace extremes\n            future = pd.DataFrame({'ds': df[time_index_col]})\n            forecast = model.predict(future)\n            imputed.loc[extreme_mask] = forecast.loc[extreme_mask, 'yhat'].values\n            logger.info(f\"  Imputed {extreme_mask.sum()} extremes for {feature}\")\n            \n        except Exception as e:\n            logger.error(f\"  Prophet imputation failed: {e}. Using median.\")\n            imputed[extreme_mask] = np.median(finite_vals)\n            \n        return imputed\n        \n    def detect_temporal_anomalies(self, df: pd.DataFrame, feature: str,\n                                  time_index_col: str = '__index_level_0__') -> Optional[AnomalyProfile]:\n        \"\"\"Identify temporal outliers and create correction profile\"\"\"\n        logger.info(f\"Analyzing {feature} for anomalies...\")\n        \n        ts_df = pd.DataFrame({'ds': df[time_index_col], 'y': df[feature]}).dropna(subset=['y'])\n        \n        if len(ts_df) < 200:\n            logger.warning(f\"Insufficient data ({len(ts_df)} points) for {feature}\")\n            return None\n            \n        try:\n            # Hourly resampling for efficiency\n            hourly_ts_df = ts_df.set_index('ds').resample('1H').mean().reset_index().dropna()\n            \n            if len(hourly_ts_df) < 50:\n                return None\n            \n            # Fit Prophet (tuned for anomaly detection)\n            model = Prophet(\n                changepoint_prior_scale=0.08,\n                interval_width=0.99,\n                weekly_seasonality=True,\n                daily_seasonality=True\n            )\n            model.fit(hourly_ts_df)\n            forecast = model.predict(hourly_ts_df)\n            \n            # Detect anomalies using prediction intervals\n            anomaly_values = []\n            anomaly_scores = {}\n            \n            for idx, row in ts_df.iterrows():\n                nearest = forecast.iloc[np.argmin(np.abs(forecast['ds'] - row['ds']))]\n                y_val = row['y']\n                lower, upper = nearest['yhat_lower'], nearest['yhat_upper']\n                \n                score = 0\n                if y_val < lower:\n                    score = min(1.0, (lower - y_val) / (nearest['yhat'] - lower + 1e-6))\n                elif y_val > upper:\n                    score = min(1.0, (y_val - upper) / (upper - nearest['yhat'] + 1e-6))\n                \n                if score > 0.05:\n                    anomaly_values.append((y_val, score))\n                anomaly_scores[y_val] = score\n            \n            # Group anomaly values into contiguous ranges\n            value_segments = self._group_anomalous_ranges(anomaly_values)\n            \n            # Create correction mapping\n            correction_map = self._derive_adjustment_factors(\n                ts_df['y'].values, anomaly_scores, value_segments\n            )\n            \n            # Capture time patterns\n            temporal_patterns = self._capture_time_patterns(ts_df, anomaly_scores)\n            \n            return AnomalyProfile(\n                feature_name=feature,\n                value_segments=value_segments,\n                anomaly_scores=anomaly_scores,\n                correction_intensity=correction_map,\n                temporal_trends=temporal_patterns\n            )\n            \n        except Exception as e:\n            logger.error(f\"Prophet analysis failed for {feature}: {e}\")\n            return None\n            \n    def _group_anomalous_ranges(self, anomalies: List[Tuple[float, float]]) -> List[Tuple[float, float]]:\n        \"\"\"Group contiguous anomaly values into ranges\"\"\"\n        if not anomalies: return []\n        \n        anomalies.sort(key=lambda x: x[0])\n        ranges = []\n        current_min = current_max = anomalies[0][0]\n        \n        for val, _ in anomalies[1:]:\n            if val <= current_max * 1.005 + 1e-6:\n                current_max = val\n            else:\n                ranges.append((current_min, current_max))\n                current_min = current_max = val\n                \n        ranges.append((current_min, current_max))\n        return ranges\n        \n    def _derive_adjustment_factors(self, values: np.ndarray,\n                                  scores: Dict[float, float],\n                                  segments: List[Tuple[float, float]]) -> Dict[float, float]:\n        \"\"\"Map values to correction strengths\"\"\"\n        finite_vals = values[np.isfinite(values)]\n        if not finite_vals.any(): return {0.0: 0.0}\n        \n        # Sample across value range\n        value_points = np.linspace(finite_vals.min(), finite_vals.max(), 1000)\n        strengths = []\n        \n        for val in value_points:\n            base_strength = 0.0\n            \n            # Check if in anomaly segment\n            for seg_min, seg_max in segments:\n                if seg_min <= val <= seg_max:\n                    base_strength = max(base_strength, 0.5)\n                    break\n            \n            # Incorporate nearest anomaly score\n            closest_score = 0.0\n            min_dist = float('inf')\n            for known_val, score in scores.items():\n                dist = abs(known_val - val)\n                if dist < min_dist:\n                    min_dist = dist\n                    closest_score = score\n                    \n            if min_dist < (finite_vals.max() - finite_vals.min()) * 0.005:\n                base_strength = max(base_strength, closest_score * 0.8)\n                \n            strengths.append(base_strength)\n        \n        return dict(zip(value_points, strengths))\n        \n    def _capture_time_patterns(self, ts_df: pd.DataFrame,\n                               scores: Dict[float, float]) -> Dict[str, Any]:\n        \"\"\"Extract hourly/daily anomaly patterns\"\"\"\n        patterns = {\n            'hourly_anomaly_rate': defaultdict(list),\n            'daily_anomaly_rate': defaultdict(list),\n            'value_percentiles': {}\n        }\n        \n        ts_df['hour'] = ts_df['ds'].dt.hour\n        ts_df['day'] = ts_df['ds'].dt.dayofweek\n        \n        for _, row in ts_df.iterrows():\n            score = scores.get(row['y'], 0)\n            patterns['hourly_anomaly_rate'][row['hour']].append(score)\n            patterns['daily_anomaly_rate'][row['day']].append(score)\n            \n        # Average scores\n        for hour in patterns['hourly_anomaly_rate']:\n            patterns['hourly_anomaly_rate'][hour] = np.mean(patterns['hourly_anomaly_rate'][hour])\n        for day in patterns['daily_anomaly_rate']:\n            patterns['daily_anomaly_rate'][day] = np.mean(patterns['daily_anomaly_rate'][day])\n            \n        # Value distribution\n        patterns['value_percentiles'] = {\n            p: np.percentile(ts_df['y'], p) for p in [1, 5, 10, 25, 50, 75, 90, 95, 99]\n        }\n        \n        return dict(patterns)\n\n#%% [markdown]\n### 🛠️ 4. Data Refinement System\n# This is where I combine Prophet's insights with statistical methods:\n#   - Uses Isotonic Regression for smooth, monotonic correction curves\n#   - Blends temporal and statistical anomaly detection\n#   - Memory optimized for large crypto datasets\n#%% [code]\nclass DataRefiner:\n    \"\"\"Applies anomaly corrections using learned patterns\"\"\"\n    \n    def __init__(self):\n        self.anomaly_detector = TemporalAnomalyDetector()\n        self.anomaly_profiles: Dict[str, AnomalyProfile] = {}\n        self.static_bounds: Dict[str, Tuple[float, float]] = {}\n        self.correction_regressors: Dict[str, IsotonicRegression] = {}\n        \n    def fit(self, features: pd.DataFrame, timestamps: Optional[pd.Series] = None):\n        \"\"\"Learn correction rules from data\"\"\"\n        logger.info(\"Fitting Data Refiner...\")\n        \n        if timestamps is not None:\n            df_with_time = features.copy()\n            df_with_time['__index_level_0__'] = timestamps\n            \n            # Priority features for Prophet processing\n            priority_features = ['volume', 'bid_qty', 'ask_qty', 'buy_qty', 'sell_qty']\n            sample_x = ['X363', 'X321', 'X405', 'X730', 'X523', 'X756']\n            \n            # Impute extremes\n            for feat in priority_features:\n                if feat in features.columns:\n                    features[feat] = self.anomaly_detector.impute_temporal_extremes(df_with_time, feat)\n            \n            # Detect anomalies\n            for feat in priority_features + sample_x:\n                if feat in features.columns:\n                    profile = self.anomaly_detector.detect_temporal_anomalies(df_with_time, feat)\n                    if profile:\n                        self.anomaly_profiles[feat] = profile\n        \n        # Statistical bounds for all features\n        for feat in INPUT_COLUMNS:\n            if feat not in features.columns:\n                continue\n                \n            values = features[feat].values\n            finite_vals = values[np.isfinite(values)]\n            \n            if not finite_vals.any():\n                self.static_bounds[feat] = (0.0, 0.0)\n                continue\n                \n            # IQR-based bounds\n            q1, q3 = np.percentile(finite_vals, [25, 75])\n            iqr = q3 - q1\n            lower = q1 - 2.5 * iqr\n            upper = q3 + 2.5 * iqr\n            self.static_bounds[feat] = (lower, upper)\n            \n            # Build correction mapping\n            self._build_combined_correction_map(feat, finite_vals)\n            \n        return self\n        \n    def _build_combined_correction_map(self, feature: str, clean_values: np.ndarray):\n        \"\"\"Create smooth correction curve blending Prophet and stats\"\"\"\n        if not clean_values.any():\n            iso_reg = IsotonicRegression(out_of_bounds='clip')\n            iso_reg.fit([0, 1], [0, 0])\n            self.correction_regressors[feature] = iso_reg\n            return\n\n        # Sample value range\n        value_points = np.linspace(clean_values.min(), clean_values.max(), 500)\n        strengths = []\n        \n        for val in value_points:\n            # Statistical component\n            lower_stat, upper_stat = self.static_bounds[feature]\n            if val < lower_stat:\n                stat_strength = min(1.0, (lower_stat - val) / (upper_stat - lower_stat + 1e-6))\n            elif val > upper_stat:\n                stat_strength = min(1.0, (val - upper_stat) / (upper_stat - lower_stat + 1e-6))\n            else:\n                stat_strength = 0.0\n                \n            # Prophet component\n            prophet_strength = 0.0\n            if feature in self.anomaly_profiles:\n                profile = self.anomaly_profiles[feature]\n                # Check anomaly segments\n                for seg_min, seg_max in profile.value_segments:\n                    if seg_min <= val <= seg_max:\n                        prophet_strength = max(prophet_strength, 0.7)\n                        break\n                \n                # Nearest anomaly score\n                closest_score = 0.0\n                min_dist = float('inf')\n                if profile.anomaly_scores:\n                    for known_val, score in profile.anomaly_scores.items():\n                        dist = abs(known_val - val)\n                        if dist < min_dist:\n                            min_dist = dist\n                            closest_score = score\n                    \n                    if min_dist < (clean_values.max() - clean_values.min()) * 0.002:\n                        prophet_strength = max(prophet_strength, closest_score * 0.9)\n            \n            # Combine strengths\n            if prophet_strength > 0:\n                final_strength = (prophet_strength * 0.7) + (stat_strength * 0.3)\n            else:\n                final_strength = stat_strength * 0.6\n                \n            strengths.append(final_strength)\n        \n        # Fit Isotonic Regression (ensures monotonic correction)\n        if value_points.any() and strengths:\n            iso_reg = IsotonicRegression(out_of_bounds='clip')\n            iso_reg.fit(value_points, strengths)\n            self.correction_regressors[feature] = iso_reg\n        else:\n            iso_reg = IsotonicRegression(out_of_bounds='clip')\n            iso_reg.fit([0, 1], [0, 0])\n            self.correction_regressors[feature] = iso_reg\n        \n    def transform(self, features: pd.DataFrame, intensity: float = 0.5) -> pd.DataFrame:\n        \"\"\"Apply corrections with specified intensity\"\"\"\n        adjusted = features.copy()\n        \n        for feature in INPUT_COLUMNS:\n            if feature not in features or feature not in self.correction_regressors:\n                continue\n                \n            values = features[feature].values\n            finite_mask = np.isfinite(values)\n            \n            if np.any(finite_mask):\n                # Get correction strengths\n                strengths = self.correction_regressors[feature].predict(values[finite_mask])\n                median_val = np.median(values[finite_mask])\n                \n                # Apply correction: blend towards median\n                adjusted_vals = values[finite_mask] * (1 - strengths * intensity) + \\\n                                median_val * (strengths * intensity)\n                \n                adjusted.loc[finite_mask, feature] = adjusted_vals\n                \n        return adjusted\n\n#%% [markdown]\n### 🧹 5. Data Cleaning Utilities\n# Crypto data is messy! These ensure:\n#   - No NaNs/Infs break the pipeline\n#   - Efficient memory usage (critical for Kaggle)\n#%% [code]\ndef ensure_numerical_integrity(df: pd.DataFrame, fill_method='median') -> pd.DataFrame:\n    \"\"\"Remove NaNs/Infs using robust methods\"\"\"\n    df_clean = df.copy()\n    \n    for col in df_clean.columns:\n        if pd.api.types.is_numeric_dtype(df_clean[col]):\n            # Handle infinities\n            inf_count = np.sum(np.isinf(df_clean[col]))\n            if inf_count > 0:\n                df_clean[col] = df_clean[col].replace([np.inf, -np.inf], np.nan)\n            \n            # Fill NaNs\n            finite_vals = df_clean[col][np.isfinite(df_clean[col])]\n            if finite_vals.empty:\n                df_clean[col] = 0\n            else:\n                fill_val = np.median(finite_vals) if fill_method == 'median' else np.mean(finite_vals)\n                df_clean[col] = df_clean[col].fillna(fill_val)\n    \n    # Final check\n    if not df_clean.apply(np.isfinite).all().all():\n        df_clean = df_clean.replace([np.inf, -np.inf], 0).fillna(0)\n        \n    return df_clean\n\ndef optimize_memory_footprint(df: pd.DataFrame, name: str):\n    \"\"\"Downcast numeric types to reduce memory usage\"\"\"\n    logger.info(f\"Optimizing memory for {name}...\")\n    start_mem = df.memory_usage(deep=True).sum() / 1024**2\n    \n    for col in df.columns:\n        col_type = df[col].dtype\n        if col_type != 'object':\n            c_min, c_max = df[col].min(), df[col].max()\n            \n            if not np.isfinite(c_min) or not np.isfinite(c_max):\n                continue\n                \n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n            else:\n                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n                    df[col] = df[col].astype(np.float16)\n                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n                    df[col] = df[col].astype(np.float32)\n                    \n    end_mem = df.memory_usage(deep=True).sum() / 1024**2\n    logger.info(f\"  Memory reduced: {start_mem:.2f}MB → {end_mem:.2f}MB\")\n    return df\n\n#%% [markdown]\n### 🤖 6. Crypto Prediction Pipeline\n# The core system that ties everything together:\n#   - Calibrates anomaly adjustment intensity using time-series CV\n#   - Trains XGBoost on refined data\n#   - Generates predictions with explainable visualizations\n#%% [code]\nclass CryptoPredictiveFlow:\n    \"\"\"End-to-end prediction system with anomaly adjustment\"\"\"\n    \n    def __init__(self):\n        self.data_refiner = DataRefiner()\n        self.feature_scaler = RobustScaler()\n        self.prediction_model = None\n        self.calibrated_intensity = 0.6\n        \n    def calibrate_adjustment_intensity(self, features: pd.DataFrame, targets: pd.Series,\n                                      timestamps: Optional[pd.Series] = None) -> float:\n        \"\"\"Find optimal correction strength using time-series CV\"\"\"\n        logger.info(\"Calibrating adjustment intensity...\")\n        \n        self.data_refiner.fit(features, timestamps)\n        tscv = TimeSeriesSplit(n_splits=3)\n        intensity_levels = [0.0, 0.3, 0.6, 0.8, 1.0]\n        results = {}\n        \n        for intensity in intensity_levels:\n            logger.info(f\"Testing intensity = {intensity:.1f}\")\n            \n            # Apply correction\n            adjusted = self.data_refiner.transform(features, intensity)\n            adjusted = ensure_numerical_integrity(adjusted)\n            \n            # Scale features\n            scaled = pd.DataFrame(\n                self.feature_scaler.fit_transform(adjusted),\n                columns=adjusted.columns\n            )\n            scaled = ensure_numerical_integrity(scaled)\n            \n            fold_scores = []\n            for fold, (train_idx, val_idx) in enumerate(tscv.split(scaled)):\n                # Split data\n                X_train, y_train = scaled.iloc[train_idx], targets.iloc[train_idx]\n                X_val, y_val = scaled.iloc[val_idx], targets.iloc[val_idx]\n                \n                if X_train.empty or X_val.empty:\n                    continue\n                    \n                # Handle NaN targets\n                y_train = y_train.fillna(y_train.median())\n                y_val = y_val.fillna(y_val.median())\n                \n                # Train temporary model\n                model = XGBRegressor(\n                    n_estimators=120,\n                    max_depth=7,\n                    learning_rate=0.06,\n                    subsample=0.75,\n                    colsample_bytree=0.6,\n                    random_state=42,\n                    n_jobs=-1,\n                    missing=np.nan\n                )\n                model.fit(X_train, y_train)\n                \n                # Evaluate\n                preds = model.predict(X_val)\n                score = pearsonr(y_val, preds)[0]\n                fold_scores.append(0.0 if np.isnan(score) else score)\n                logger.info(f\"  Fold {fold}: Pearson = {score:.4f}\")\n            \n            if fold_scores:\n                mean_score = np.mean(fold_scores)\n                std_score = np.std(fold_scores)\n                results[intensity] = (mean_score, std_score)\n                logger.info(f\"  Intensity {intensity:.1f}: {mean_score:.4f} ± {std_score:.4f}\")\n        \n        # Select best intensity (penalize high variance)\n        best_intensity = max(results.keys(), \n                            key=lambda k: results[k][0] - 0.2 * results[k][1]) \n        self.calibrated_intensity = best_intensity\n        logger.info(f\"Optimal intensity: {best_intensity:.2f}\")\n        return best_intensity\n        \n    def fit_pipeline(self, features: pd.DataFrame, targets: pd.Series, \n                     timestamps: Optional[pd.Series] = None):\n        \"\"\"Train full prediction pipeline\"\"\"\n        logger.info(\"\\n\" + \"=\"*80)\n        logger.info(\"TRAINING PREDICTION PIPELINE\")\n        logger.info(\"=\"*80)\n        \n        # Calibrate and apply correction\n        self.calibrate_adjustment_intensity(features, targets, timestamps)\n        adjusted = self.data_refiner.transform(features, self.calibrated_intensity)\n        adjusted = ensure_numerical_integrity(adjusted)\n        \n        # Scale features\n        scaled = pd.DataFrame(\n            self.feature_scaler.fit_transform(adjusted),\n            columns=adjusted.columns\n        )\n        scaled = ensure_numerical_integrity(scaled)\n        \n        # Train final model\n        logger.info(\"Training final XGBoost model...\")\n        self.prediction_model = XGBRegressor(\n            n_estimators=400,\n            max_depth=9,\n            learning_rate=0.04,\n            subsample=0.8,\n            colsample_bytree=0.7,\n            gamma=0.15,\n            reg_alpha=0.1,\n            reg_lambda=1.0,\n            min_child_weight=5,\n            random_state=42,\n            n_jobs=-1,\n            missing=np.nan\n        )\n        \n        # Handle target NaNs\n        targets = targets.fillna(targets.median())\n        self.prediction_model.fit(scaled, targets)\n        \n        # Training performance\n        train_preds = self.prediction_model.predict(scaled)\n        train_score = pearsonr(targets, train_preds)[0]\n        logger.info(f\"Training Pearson correlation: {train_score:.4f}\")\n        \n        return self\n        \n    def generate_predictions(self, test_data: pd.DataFrame) -> np.ndarray:\n        \"\"\"Generate predictions on new data\"\"\"\n        logger.info(\"Generating test predictions...\")\n        \n        # Apply learned corrections\n        adjusted_test = self.data_refiner.transform(test_data, self.calibrated_intensity)\n        adjusted_test = ensure_numerical_integrity(adjusted_test)\n        \n        # Scale using training scaler\n        scaled_test = pd.DataFrame(\n            self.feature_scaler.transform(adjusted_test),\n            columns=adjusted_test.columns\n        )\n        scaled_test = ensure_numerical_integrity(scaled_test)\n        \n        return self.prediction_model.predict(scaled_test)\n        \n    def visualize_correction_patterns(self, output_path: str = 'anomaly_insights.png'):\n        \"\"\"Visualize anomaly correction profiles\"\"\"\n        features_to_plot = [f for f in ['volume', 'bid_qty', 'ask_qty', 'X363', 'X321', 'X405'] \n                            if f in self.data_refiner.anomaly_profiles]\n        \n        if not features_to_plot:\n            logger.warning(\"No anomaly profiles to visualize\")\n            return\n            \n        fig, axes = plt.subplots(2, 3, figsize=(18, 12))\n        axes = axes.ravel()\n        \n        for i, feature in enumerate(features_to_plot[:6]):\n            profile = self.data_refiner.anomaly_profiles[feature]\n            ax = axes[i]\n            \n            # Plot correction curve\n            values = list(profile.correction_intensity.keys())\n            strengths = list(profile.correction_intensity.values())\n            sorted_idx = np.argsort(values)\n            ax.plot(np.array(values)[sorted_idx], np.array(strengths)[sorted_idx], \n                    'b-', alpha=0.7, label='Correction Strength')\n            \n            # Highlight anomaly segments\n            for seg_min, seg_max in profile.value_segments:\n                ax.axvspan(seg_min, seg_max, alpha=0.15, color='red',\n                           label='Anomaly Zone' if i == 0 else '')\n            \n            # Add distribution percentiles\n            percentiles = profile.temporal_trends['value_percentiles']\n            for p, val in percentiles.items():\n                if p in [10, 50, 90]:\n                    ax.axvline(val, color='gray', ls=':', alpha=0.4,\n                               label=f'{p}th %ile' if i == 0 else '')\n            \n            ax.set_title(f\"{feature} Correction Profile\")\n            ax.set_xlabel(\"Feature Value\")\n            ax.set_ylabel(\"Correction Intensity\")\n            ax.grid(True, alpha=0.3)\n            if i == 0:\n                ax.legend()\n        \n        plt.suptitle(\"Temporal Anomaly Correction Insights\", fontsize=16)\n        plt.tight_layout()\n        plt.savefig(output_path, dpi=120)\n        plt.close()\n        logger.info(f\"Saved visualization: {output_path}\")\n\n#%% [markdown]\n### 🚀 7. Execution Workflow\n# This ties everything together for end-to-end execution:\n#   1. Load and preprocess data\n#   2. Train pipeline\n#   3. Generate predictions\n#   4. Create insights visualization\n#%% [code]\ndef execute_prediction_flow():\n    \"\"\"End-to-end prediction workflow\"\"\"\n    logger.info(\"Loading data...\")\n    try:\n        train_data = pd.read_parquet(TRAIN_DATA_PATH)\n        test_data = pd.read_parquet(TEST_DATA_PATH)\n    except Exception as e:\n        logger.error(f\"Data loading failed: {e}\")\n        return None\n    \n    # Extract features/targets\n    X_train = train_data[INPUT_COLUMNS].copy()\n    y_train = train_data[PREDICTION_TARGET].copy()\n    X_test = test_data[INPUT_COLUMNS].copy()\n    \n    # Get timestamps if available\n    train_timestamps = None\n    if '__index_level_0__' in train_data.columns:\n        train_timestamps = train_data['__index_level_0__']\n        if not pd.api.types.is_datetime64_any_dtype(train_timestamps):\n            train_timestamps = pd.to_datetime(train_timestamps, unit='s')\n    \n    logger.info(f\"Train shape: {X_train.shape}, Test shape: {X_test.shape}\")\n    \n    # Clean data\n    X_train = ensure_numerical_integrity(X_train)\n    X_test = ensure_numerical_integrity(X_test)\n    \n    # Optimize memory\n    X_train = optimize_memory_footprint(X_train, \"Train Features\")\n    X_test = optimize_memory_footprint(X_test, \"Test Features\")\n    \n    # Train pipeline\n    pipeline = CryptoPredictiveFlow()\n    pipeline.fit_pipeline(X_train, y_train, train_timestamps)\n    \n    # Visualize insights\n    pipeline.visualize_correction_patterns()\n    \n    # Generate predictions\n    predictions = pipeline.generate_predictions(X_test)\n    \n    # Save submission\n    submission = pd.read_csv(SUBMISSION_TEMPLATE_PATH)\n    submission['prediction'] = predictions\n    submission.to_csv('crypto_predictions.csv', index=False)\n    logger.info(\"Saved predictions to crypto_predictions.csv\")\n    \n    logger.info(\"\\n\" + \"=\"*80)\n    logger.info(\"PIPELINE EXECUTION COMPLETE!\")\n    logger.info(f\"Calibrated intensity: {pipeline.calibrated_intensity:.2f}\")\n    logger.info(\"=\"*80)\n    \n    return pipeline\n\n#%% [markdown]\n### ▶️ 8. Run the Pipeline!\n# Execute the full workflow and show sample predictions\n#%% [code]\nif __name__ == \"__main__\":\n    crypto_pipeline = execute_prediction_flow()\n    \n    # Display sample results\n    if crypto_pipeline:\n        sample_preds = pd.read_csv('crypto_predictions.csv').head()\n        print(\"\\nSample predictions:\")\n        display(sample_preds)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-07-24T20:48:01.528416Z","iopub.execute_input":"2025-07-24T20:48:01.528742Z","iopub.status.idle":"2025-07-24T20:51:24.067623Z","shell.execute_reply.started":"2025-07-24T20:48:01.528717Z","shell.execute_reply":"2025-07-24T20:51:24.066849Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Key Features of My Implementation\n\n### 🧠 Intelligent Design Choices\n1. **Hybrid Anomaly Detection**\n   - Prophet for temporal patterns (intraday/weekly seasonality)\n   - IQR for statistical outlier detection\n   - Isotonic Regression for smooth value-based corrections\n\n2. **Crypto-Specific Optimizations**\n   - Aggressive memory reduction for large order book data\n   - Time-series cross-validation to prevent lookahead bias\n   - Robust scaling for heavy-tailed distributions\n\n3. **Explainable AI**\n   - Visual correction profiles show exactly where adjustments happen\n   - Anomaly segment highlighting for transparency\n   - Percentile markers for distribution context","metadata":{}},{"cell_type":"markdown","source":"### 📈 Why This Works for Crypto\n- **Handles volatility**: Correction curves adapt to extreme moves\n- **24/7 market aware**: Prophet captures night/weekend patterns\n- **Whale manipulation resistant**: Anomaly segments detect suspicious activity\n- **Efficient**: Processes 100+ features in minutes","metadata":{}}]}