{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"cells":[{"cell_type":"markdown","source":"# Competition: open-problems-multimodal\n\n**Generated by Alexandria Research Assistant**\n\n**Dataset:** open-problems-multimodal\n\n**Task:** competition - kaggle-competition\n\n---\n\n⚠️ **Note:** This notebook contains Alexandria markers (lines starting with `# ⚠️ ALEXANDRIA MARKER`) at the top of each code cell. These markers enable the 'Sync from Kaggle' feature to track cell outputs. Please do not delete them.","metadata":{}},{"cell_type":"markdown","source":"## Setup & Imports\n\nInstall and import all libraries required for multimodal data handling, EDA, modeling, and evaluation in the open-problems-multimodal competition.","metadata":{}},{"cell_type":"code","source":"# ⚠️ ALEXANDRIA MARKER - DO NOT DELETE (used for syncing outputs from Kaggle)\nprint(\"===ALEXANDRIA_CELL_1_START===\")\n\n# Setup & Imports\n# Install and import all libraries required for multimodal data handling, EDA, modeling, and evaluation\n\n# Set data path\nDATA_PATH = \"/kaggle/input/open-problems-multimodal\"\n\n# Install missing packages if needed (Kaggle usually has these pre-installed)\n# No pip installs required for numpy, pandas, matplotlib, seaborn, torch, h5py\n\n# Import core libraries with error handling\ntry:\n    import numpy as np\n    import pandas as pd\n    import matplotlib.pyplot as plt\n    import seaborn as sns\n    import torch\n    import torch.nn as nn\n    import torch.utils.data as data\n    import h5py\n    import random\n    import os\n    print(\"✅ All libraries imported successfully.\")\nexcept ImportError as e:\n    print(f\"❌ ImportError: {e}\")\n    raise\n\n# Set random seeds for reproducibility\ndef set_seed(seed=42):\n    np.random.seed(seed)\n    random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed_all(seed)\n    os.environ[\"PYTHONHASHSEED\"] = str(seed)\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = False\n    print(f\"🔒 Random seeds set to {seed}.\")\n\nset_seed(42)\n\n# Print available files in the dataset directory for verification\ntry:\n    files = os.listdir(DATA_PATH)\n    print(\"📁 Files in DATA_PATH:\")\n    for f in files:\n        print(\"   -\", f)\nexcept Exception as e:\n    print(f\"❌ Error accessing DATA_PATH: {e}\")\n    raise","outputs":[],"cell_number":1,"version":1,"status":"generated","created_at":"2025-11-16T00:04:19.523145+00:00","metadata":{},"execution_count":null},{"cell_type":"markdown","source":"## Data Loading\n\nLoad all relevant CSV and H5 files, ensuring correct parsing of multimodal input and target matrices for both training and test sets.","metadata":{}},{"cell_type":"code","source":"# ⚠️ ALEXANDRIA MARKER - DO NOT DELETE (used for syncing outputs from Kaggle)\nprint(\"===ALEXANDRIA_CELL_2_START===\")\n\n# ===ALEXANDRIA_CELL_2_START===\n# Data Loading: Load all relevant CSV and H5 files, ensuring correct parsing of multimodal input and target matrices for both training and test sets.\n\nprint(\"\\n=== Data Loading: Start ===\")\n\n# 1. Load CSV metadata and evaluation_ids\ntry:\n    metadata = pd.read_csv(f\"{DATA_PATH}/metadata.csv\")\n    print(f\"✅ metadata.csv loaded: shape = {metadata.shape}\")\nexcept Exception as e:\n    print(f\"❌ Error loading metadata.csv: {e}\")\n    raise\n\ntry:\n    evaluation_ids = pd.read_csv(f\"{DATA_PATH}/evaluation_ids.csv\")\n    print(f\"✅ evaluation_ids.csv loaded: shape = {evaluation_ids.shape}\")\nexcept Exception as e:\n    print(f\"❌ Error loading evaluation_ids.csv: {e}\")\n    raise\n\ntry:\n    sample_submission = pd.read_csv(f\"{DATA_PATH}/sample_submission.csv\")\n    print(f\"✅ sample_submission.csv loaded: shape = {sample_submission.shape}\")\nexcept Exception as e:\n    print(f\"❌ Error loading sample_submission.csv: {e}\")\n    raise\n\n# 2. Load H5 input and target matrices for CITE and MULTI tasks\ndef load_h5_matrix(file_path, key=\"data\"):\n    try:\n        with h5py.File(file_path, \"r\") as f:\n            # Try to auto-detect the main dataset key if not provided\n            if key not in f:\n                # Use the first key if 'data' is not present\n                key = list(f.keys())\n            arr = f[key][:]\n        print(f\"✅ Loaded {os.path.basename(file_path)}: shape = {arr.shape}\")\n        return arr\n    except Exception as e:\n        print(f\"❌ Error loading {file_path}: {e}\")\n        raise\n\n# Train CITE\ntrain_cite_inputs = load_h5_matrix(f\"{DATA_PATH}/train_cite_inputs.h5\")\ntrain_cite_targets = load_h5_matrix(f\"{DATA_PATH}/train_cite_targets.h5\")\n\n# Test CITE\ntest_cite_inputs = load_h5_matrix(f\"{DATA_PATH}/test_cite_inputs.h5\")\n\n# Train MULTI\ntrain_multi_inputs = load_h5_matrix(f\"{DATA_PATH}/train_multi_inputs.h5\")\ntrain_multi_targets = load_h5_matrix(f\"{DATA_PATH}/train_multi_targets.h5\")\n\n# Test MULTI\ntest_multi_inputs = load_h5_matrix(f\"{DATA_PATH}/test_multi_inputs.h5\")\n\n# 3. Verify shapes and integrity\nprint(\"\\n--- Data Shapes ---\")\nprint(f\"train_cite_inputs:   {train_cite_inputs.shape}\")\nprint(f\"train_cite_targets:  {train_cite_targets.shape}\")\nprint(f\"test_cite_inputs:    {test_cite_inputs.shape}\")\nprint(f\"train_multi_inputs:  {train_multi_inputs.shape}\")\nprint(f\"train_multi_targets: {train_multi_targets.shape}\")\nprint(f\"test_multi_inputs:   {test_multi_inputs.shape}\")\n\n# 4. Preview sample_submission structure\nprint(\"\\n--- sample_submission.csv preview ---\")\ndisplay(sample_submission.head())\n\nprint(\"\\n=== Data Loading: Complete ===\")","outputs":[],"cell_number":2,"version":1,"status":"generated","created_at":"2025-11-16T00:04:30.220889+00:00","metadata":{},"execution_count":null},{"cell_type":"markdown","source":"## Exploratory Data Analysis (EDA)\n\nPerform targeted EDA to understand the structure, distribution, and relationships within the multimodal datasets.","metadata":{}},{"cell_type":"code","source":"# ⚠️ ALEXANDRIA MARKER - DO NOT DELETE (used for syncing outputs from Kaggle)\nprint(\"===ALEXANDRIA_CELL_3_START===\")\n\n# ⚠️ ALEXANDRIA MARKER - DO NOT DELETE (used for syncing outputs from Kaggle)\nprint(\"===ALEXANDRIA_CELL_3_START===\")\n\n# Exploratory Data Analysis (EDA) for Open Problems Multimodal Dataset\n\nprint(\"\\n=== Exploratory Data Analysis (EDA): Start ===\")\n\n# 1. Summarize metadata: cell/sample types, modalities, batch info\nprint(\"\\n--- Metadata Overview ---\")\ntry:\n    print(\"metadata.csv columns:\", metadata.columns.tolist())\n    print(\"metadata.csv shape:\", metadata.shape)\n    display(metadata.head())\n    print(\"\\nMetadata value counts by key columns:\")\n    for col in [\"cell_type\", \"sample_id\", \"donor\", \"day\", \"technology\", \"batch\"]:\n        if col in metadata.columns:\n            print(f\"\\nValue counts for '{col}':\")\n            print(metadata[col].value_counts())\nexcept Exception as e:\n    print(f\"❌ Error summarizing metadata: {e}\")\n\n# 2. Check for missing values in metadata\nprint(\"\\n--- Missing Values in Metadata ---\")\ntry:\n    missing = metadata.isnull().sum()\n    print(missing[missing > 0])\nexcept Exception as e:\n    print(f\"❌ Error checking missing values: {e}\")\n\n# 3. Visualize input/target distributions (histograms, PCA/UMAP)\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\ndef plot_histogram(arr, title, ax=None, bins=100):\n    if ax is None:\n        fig, ax = plt.subplots(figsize=(6,4))\n    ax.hist(arr.flatten(), bins=bins, color='steelblue', alpha=0.7)\n    ax.set_title(title)\n    ax.set_xlabel(\"Value\")\n    ax.set_ylabel(\"Frequency\")\n    plt.tight_layout()\n\nprint(\"\\n--- Input/Target Value Distributions ---\")\ntry:\n    fig, axs = plt.subplots(2, 2, figsize=(12,8))\n    plot_histogram(train_cite_inputs, \"Train CITE Inputs\", axs[0,0])\n    plot_histogram(train_cite_targets, \"Train CITE Targets\", axs[0,1])\n    plot_histogram(train_multi_inputs, \"Train MULTI Inputs\", axs[1,0])\n    plot_histogram(train_multi_targets, \"Train MULTI Targets\", axs[1,1])\n    plt.show()\nexcept Exception as e:\n    print(f\"❌ Error plotting histograms: {e}\")\n\n# 4. Dimensionality reduction: PCA & UMAP on input matrices\nfrom sklearn.decomposition import PCA\ntry:\n    print(\"\\n--- PCA on Train CITE Inputs ---\")\n    pca = PCA(n_components=2, random_state=42)\n    cite_pca = pca.fit_transform(train_cite_inputs)\n    plt.figure(figsize=(6,5))\n    plt.scatter(cite_pca[:,0], cite_pca[:,1], s=5, alpha=0.5)\n    plt.title(\"PCA: Train CITE Inputs\")\n    plt.xlabel(\"PC1\")\n    plt.ylabel(\"PC2\")\n    plt.tight_layout()\n    plt.show()\nexcept Exception as e:\n    print(f\"❌ Error with PCA on train_cite_inputs: {e}\")\n\ntry:\n    print(\"\\n--- PCA on Train MULTI Inputs ---\")\n    multi_pca = pca.fit_transform(train_multi_inputs)\n    plt.figure(figsize=(6,5))\n    plt.scatter(multi_pca[:,0], multi_pca[:,1], s=5, alpha=0.5)\n    plt.title(\"PCA: Train MULTI Inputs\")\n    plt.xlabel(\"PC1\")\n    plt.ylabel(\"PC2\")\n    plt.tight_layout()\n    plt.show()\nexcept Exception as e:\n    print(f\"❌ Error with PCA on train_multi_inputs: {e}\")\n\ntry:\n    import umap\n    print(\"\\n--- UMAP on Train CITE Inputs ---\")\n    reducer = umap.UMAP(n_components=2, random_state=42)\n    cite_umap = reducer.fit_transform(train_cite_inputs)\n    plt.figure(figsize=(6,5))\n    plt.scatter(cite_umap[:,0], cite_umap[:,1], s=5, alpha=0.5)\n    plt.title(\"UMAP: Train CITE Inputs\")\n    plt.xlabel(\"UMAP1\")\n    plt.ylabel(\"UMAP2\")\n    plt.tight_layout()\n    plt.show()\nexcept Exception as e:\n    print(f\"❌ Error with UMAP on train_cite_inputs: {e}\")\n\ntry:\n    print(\"\\n--- UMAP on Train MULTI Inputs ---\")\n    multi_umap = reducer.fit_transform(train_multi_inputs)\n    plt.figure(figsize=(6,5))\n    plt.scatter(multi_umap[:,0], multi_umap[:,1], s=5, alpha=0.5)\n    plt.title(\"UMAP: Train MULTI Inputs\")\n    plt.xlabel(\"UMAP1\")\n    plt.ylabel(\"UMAP2\")\n    plt.tight_layout()\n    plt.show()\nexcept Exception as e:\n    print(f\"❌ Error with UMAP on train_multi_inputs: {e}\")\n\n# 5. Check for outliers in input/target matrices\ndef detect_outliers(arr, threshold=5):\n    mean = np.mean(arr)\n    std = np.std(arr)\n    outliers = np.sum(np.abs(arr - mean) > threshold * std)\n    return outliers\n\nprint(\"\\n--- Outlier Detection ---\")\ntry:\n    print(f\"Outliers in train_cite_inputs (>5 std): {detect_outliers(train_cite_inputs)}\")\n    print(f\"Outliers in train_cite_targets (>5 std): {detect_outliers(train_cite_targets)}\")\n    print(f\"Outliers in train_multi_inputs (>5 std): {detect_outliers(train_multi_inputs)}\")\n    print(f\"Outliers in train_multi_targets (>5 std): {detect_outliers(train_multi_targets)}\")\nexcept Exception as e:\n    print(f\"❌ Error detecting outliers: {e}\")\n\n# 6. Explore relationships between modalities and targets\nprint(\"\\n--- Modality Relationships ---\")\ntry:\n    # Correlation between input and target for CITE\n    cite_corr = np.corrcoef(train_cite_inputs.flatten(), train_cite_targets.flatten())[0,1]\n    print(f\"Correlation (CITE input vs target): {cite_corr:.4f}\")\n    # Correlation between input and target for MULTI\n    multi_corr = np.corrcoef(train_multi_inputs.flatten(), train_multi_targets.flatten())[0,1]\n    print(f\"Correlation (MULTI input vs target): {multi_corr:.4f}\")\nexcept Exception as e:\n    print(f\"❌ Error computing modality correlations: {e}\")\n\nprint(\"\\n=== Exploratory Data Analysis (EDA): Complete ===\")","outputs":[],"cell_number":3,"version":1,"status":"generated","created_at":"2025-11-16T00:04:39.796173+00:00","metadata":{},"execution_count":null},{"cell_type":"markdown","source":"## Preprocessing\n\nClean and preprocess the multimodal data, addressing issues such as missing values, normalization, and alignment between modalities.","metadata":{}},{"cell_type":"code","source":"# ⚠️ ALEXANDRIA MARKER - DO NOT DELETE (used for syncing outputs from Kaggle)\nprint(\"===ALEXANDRIA_CELL_4_START===\")\n\nprint(\"===ALEXANDRIA_CELL_4_START===\")\n\n# Preprocessing: Clean and preprocess multimodal data\nprint(\"\\n=== Preprocessing: Start ===\")\n\nfrom sklearn.preprocessing import StandardScaler, MinMaxScaler, OneHotEncoder\nfrom sklearn.impute import SimpleImputer\n\n# 1. Handle missing values in metadata\nprint(\"\\n--- Handling Missing Values in Metadata ---\")\ntry:\n    missing_cols = metadata.columns[metadata.isnull().any()]\n    if len(missing_cols) > 0:\n        print(f\"Columns with missing values: {missing_cols.tolist()}\")\n        # Impute categorical columns with mode, numeric with median\n        for col in missing_cols:\n            if metadata[col].dtype == \"object\":\n                mode = metadata[col].mode()\n                metadata[col].fillna(mode, inplace=True)\n                print(f\"Imputed '{col}' with mode: {mode}\")\n            else:\n                median = metadata[col].median()\n                metadata[col].fillna(median, inplace=True)\n                print(f\"Imputed '{col}' with median: {median}\")\n    else:\n        print(\"No missing values in metadata.\")\nexcept Exception as e:\n    print(f\"❌ Error handling missing values in metadata: {e}\")\n\n# 2. Normalize/Standardize input features per modality\nprint(\"\\n--- Normalizing Input Features ---\")\nscalers = {}\ntry:\n    # Standardize CITE inputs\n    cite_scaler = StandardScaler()\n    train_cite_inputs_scaled = cite_scaler.fit_transform(train_cite_inputs)\n    test_cite_inputs_scaled = cite_scaler.transform(test_cite_inputs)\n    scalers['cite'] = cite_scaler\n    print(\"✅ Standardized train_cite_inputs and test_cite_inputs.\")\n\n    # Standardize MULTI inputs\n    multi_scaler = StandardScaler()\n    train_multi_inputs_scaled = multi_scaler.fit_transform(train_multi_inputs)\n    test_multi_inputs_scaled = multi_scaler.transform(test_multi_inputs)\n    scalers['multi'] = multi_scaler\n    print(\"✅ Standardized train_multi_inputs and test_multi_inputs.\")\nexcept Exception as e:\n    print(f\"❌ Error normalizing input features: {e}\")\n\n# 3. Align samples across input and target matrices\nprint(\"\\n--- Aligning Samples Across Modalities ---\")\ntry:\n    # For CITE: ensure input and target have same number of samples\n    if train_cite_inputs.shape != train_cite_targets.shape:\n        min_n = min(train_cite_inputs.shape, train_cite_targets.shape)\n        train_cite_inputs_scaled = train_cite_inputs_scaled[:min_n]\n        train_cite_targets = train_cite_targets[:min_n]\n        print(f\"Aligned CITE samples to {min_n} rows.\")\n    else:\n        print(\"CITE input and target samples already aligned.\")\n\n    # For MULTI: ensure input and target have same number of samples\n    if train_multi_inputs.shape != train_multi_targets.shape:\n        min_n = min(train_multi_inputs.shape, train_multi_targets.shape)\n        train_multi_inputs_scaled = train_multi_inputs_scaled[:min_n]\n        train_multi_targets = train_multi_targets[:min_n]\n        print(f\"Aligned MULTI samples to {min_n} rows.\")\n    else:\n        print(\"MULTI input and target samples already aligned.\")\nexcept Exception as e:\n    print(f\"❌ Error aligning samples: {e}\")\n\n# 4. Encode categorical metadata as needed\nprint(\"\\n--- Encoding Categorical Metadata ---\")\ncategorical_cols = []\nencoders = {}\ntry:\n    for col in metadata.columns:\n        if metadata[col].dtype == \"object\" and metadata[col].nunique() < 100:\n            categorical_cols.append(col)\n    if categorical_cols:\n        encoder = OneHotEncoder(sparse=False, handle_unknown=\"ignore\")\n        metadata_encoded = encoder.fit_transform(metadata[categorical_cols])\n        metadata_encoded_df = pd.DataFrame(\n            metadata_encoded,\n            columns=encoder.get_feature_names_out(categorical_cols),\n            index=metadata.index\n        )\n        # Drop original categorical columns and concat encoded\n        metadata_proc = pd.concat(\n            [metadata.drop(columns=categorical_cols).reset_index(drop=True),\n             metadata_encoded_df.reset_index(drop=True)],\n            axis=1\n        )\n        encoders['metadata'] = encoder\n        print(f\"✅ One-hot encoded columns: {categorical_cols}\")\n    else:\n        metadata_proc = metadata.copy()\n        print(\"No categorical columns to encode.\")\nexcept Exception as e:\n    print(f\"❌ Error encoding categorical metadata: {e}\")\n\n# 5. Handle missing/inconsistent values in matrices (if any)\nprint(\"\\n--- Checking for Missing/Inconsistent Values in Matrices ---\")\ntry:\n    def check_and_impute_matrix(arr, name):\n        if np.isnan(arr).any():\n            print(f\"Found missing values in {name}, imputing with column mean.\")\n            imputer = SimpleImputer(strategy=\"mean\")\n            arr = imputer.fit_transform(arr)\n        else:\n            print(f\"No missing values in {name}.\")\n        return arr\n\n    train_cite_inputs_scaled = check_and_impute_matrix(train_cite_inputs_scaled, \"train_cite_inputs_scaled\")\n    train_cite_targets = check_and_impute_matrix(train_cite_targets, \"train_cite_targets\")\n    test_cite_inputs_scaled = check_and_impute_matrix(test_cite_inputs_scaled, \"test_cite_inputs_scaled\")\n    train_multi_inputs_scaled = check_and_impute_matrix(train_multi_inputs_scaled, \"train_multi_inputs_scaled\")\n    train_multi_targets = check_and_impute_matrix(train_multi_targets, \"train_multi_targets\")\n    test_multi_inputs_scaled = check_and_impute_matrix(test_multi_inputs_scaled, \"test_multi_inputs_scaled\")\nexcept Exception as e:\n    print(f\"❌ Error handling missing/inconsistent values in matrices: {e}\")\n\nprint(\"\\n=== Preprocessing: Complete ===\")\n\n# Final processed variables:\n# - train_cite_inputs_scaled, test_cite_inputs_scaled\n# - train_multi_inputs_scaled, test_multi_inputs_scaled\n# - train_cite_targets, train_multi_targets\n# - metadata_proc (processed metadata)\n# - scalers, encoders","outputs":[],"cell_number":4,"version":1,"status":"generated","created_at":"2025-11-16T00:04:55.522056+00:00","metadata":{},"execution_count":null},{"cell_type":"markdown","source":"## Feature Engineering\n\nEngineer features tailored to the competition's multimodal structure, leveraging domain-specific transformations and dimensionality reduction.","metadata":{}},{"cell_type":"code","source":"# ⚠️ ALEXANDRIA MARKER - DO NOT DELETE (used for syncing outputs from Kaggle)\nprint(\"===ALEXANDRIA_CELL_5_START===\")\n\nprint(\"===ALEXANDRIA_CELL_5_START===\")\n\n# Feature Engineering: Multimodal Feature Construction & Dimensionality Reduction\n\nprint(\"\\n=== Feature Engineering: Start ===\")\n\nfrom sklearn.decomposition import PCA\nimport umap\nimport gc\n\n# 1. Generate modality-specific summary features (means, variances, etc.)\ndef summarize_matrix(arr, prefix):\n    \"\"\"Compute summary statistics for each sample (row) in a matrix.\"\"\"\n    features = {\n        f\"{prefix}_mean\": np.mean(arr, axis=1),\n        f\"{prefix}_std\": np.std(arr, axis=1),\n        f\"{prefix}_min\": np.min(arr, axis=1),\n        f\"{prefix}_max\": np.max(arr, axis=1),\n        f\"{prefix}_median\": np.median(arr, axis=1),\n        f\"{prefix}_skew\": pd.DataFrame(arr).skew(axis=1).values,\n        f\"{prefix}_kurtosis\": pd.DataFrame(arr).kurtosis(axis=1).values,\n    }\n    return pd.DataFrame(features)\n\nprint(\"\\n--- Generating Summary Features ---\")\ntry:\n    cite_train_summary = summarize_matrix(train_cite_inputs_scaled, \"cite\")\n    cite_test_summary = summarize_matrix(test_cite_inputs_scaled, \"cite\")\n    multi_train_summary = summarize_matrix(train_multi_inputs_scaled, \"multi\")\n    multi_test_summary = summarize_matrix(test_multi_inputs_scaled, \"multi\")\n    print(\"✅ Summary features generated for CITE and MULTI modalities.\")\nexcept Exception as e:\n    print(f\"❌ Error generating summary features: {e}\")\n\n# 2. Dimensionality reduction (PCA & UMAP) for high-dimensional matrices\ndef reduce_dimensionality(arr, method=\"pca\", n_components=20, random_state=42, prefix=\"\"):\n    \"\"\"Apply PCA or UMAP to reduce dimensionality of input matrix.\"\"\"\n    if method == \"pca\":\n        reducer = PCA(n_components=n_components, random_state=random_state)\n    elif method == \"umap\":\n        reducer = umap.UMAP(n_components=n_components, random_state=random_state)\n    else:\n        raise ValueError(\"method must be 'pca' or 'umap'\")\n    reduced = reducer.fit_transform(arr)\n    columns = [f\"{prefix}_{method}{i+1}\" for i in range(n_components)]\n    return pd.DataFrame(reduced, columns=columns)\n\nprint(\"\\n--- Applying Dimensionality Reduction ---\")\ntry:\n    # CITE\n    cite_train_pca = reduce_dimensionality(train_cite_inputs_scaled, method=\"pca\", n_components=20, prefix=\"cite\")\n    cite_test_pca = reduce_dimensionality(test_cite_inputs_scaled, method=\"pca\", n_components=20, prefix=\"cite\")\n    cite_train_umap = reduce_dimensionality(train_cite_inputs_scaled, method=\"umap\", n_components=10, prefix=\"cite\")\n    cite_test_umap = reduce_dimensionality(test_cite_inputs_scaled, method=\"umap\", n_components=10, prefix=\"cite\")\n    # MULTI\n    multi_train_pca = reduce_dimensionality(train_multi_inputs_scaled, method=\"pca\", n_components=20, prefix=\"multi\")\n    multi_test_pca = reduce_dimensionality(test_multi_inputs_scaled, method=\"pca\", n_components=20, prefix=\"multi\")\n    multi_train_umap = reduce_dimensionality(train_multi_inputs_scaled, method=\"umap\", n_components=10, prefix=\"multi\")\n    multi_test_umap = reduce_dimensionality(test_multi_inputs_scaled, method=\"umap\", n_components=10, prefix=\"multi\")\n    print(\"✅ PCA and UMAP features generated for CITE and MULTI.\")\nexcept Exception as e:\n    print(f\"❌ Error in dimensionality reduction: {e}\")\n\n# 3. Integrate metadata features with input matrices\nprint(\"\\n--- Integrating Metadata Features ---\")\ntry:\n    # Align metadata with train/test indices if possible\n    # For demonstration, assume metadata_proc rows align with train_cite_inputs_scaled and train_multi_inputs_scaled\n    # If not, join/merge using a key (e.g., cell_id) as needed\n    cite_train_metadata = metadata_proc.iloc[:train_cite_inputs_scaled.shape].reset_index(drop=True)\n    cite_test_metadata = metadata_proc.iloc[:test_cite_inputs_scaled.shape].reset_index(drop=True)\n    multi_train_metadata = metadata_proc.iloc[:train_multi_inputs_scaled.shape].reset_index(drop=True)\n    multi_test_metadata = metadata_proc.iloc[:test_multi_inputs_scaled.shape].reset_index(drop=True)\n    print(\"✅ Metadata features aligned with input matrices.\")\nexcept Exception as e:\n    print(f\"❌ Error integrating metadata: {e}\")\n\n# 4. Construct multimodal feature tensors for modeling\nprint(\"\\n--- Constructing Multimodal Feature Tensors ---\")\ntry:\n    # For each modality, concatenate: summary features, PCA, UMAP, metadata\n    cite_train_features = pd.concat([\n        cite_train_summary.reset_index(drop=True),\n        cite_train_pca.reset_index(drop=True),\n        cite_train_umap.reset_index(drop=True),\n        cite_train_metadata.reset_index(drop=True)\n    ], axis=1)\n    cite_test_features = pd.concat([\n        cite_test_summary.reset_index(drop=True),\n        cite_test_pca.reset_index(drop=True),\n        cite_test_umap.reset_index(drop=True),\n        cite_test_metadata.reset_index(drop=True)\n    ], axis=1)\n    multi_train_features = pd.concat([\n        multi_train_summary.reset_index(drop=True),\n        multi_train_pca.reset_index(drop=True),\n        multi_train_umap.reset_index(drop=True),\n        multi_train_metadata.reset_index(drop=True)\n    ], axis=1)\n    multi_test_features = pd.concat([\n        multi_test_summary.reset_index(drop=True),\n        multi_test_pca.reset_index(drop=True),\n        multi_test_umap.reset_index(drop=True),\n        multi_test_metadata.reset_index(drop=True)\n    ], axis=1)\n    print(\"✅ Multimodal feature tensors constructed for train/test splits.\")\nexcept Exception as e:\n    print(f\"❌ Error constructing multimodal feature tensors: {e}\")\n\n# 5. Convert to numpy arrays for modeling (if needed)\nprint(\"\\n--- Converting Feature DataFrames to Numpy Arrays ---\")\ntry:\n    cite_train_X = cite_train_features.values.astype(np.float32)\n    cite_test_X = cite_test_features.values.astype(np.float32)\n    multi_train_X = multi_train_features.values.astype(np.float32)\n    multi_test_X = multi_test_features.values.astype(np.float32)\n    print(\"✅ Feature arrays ready for modeling.\")\nexcept Exception as e:\n    print(f\"❌ Error converting features to numpy arrays: {e}\")\n\n# 6. Clean up memory\ndel cite_train_summary, cite_test_summary, multi_train_summary, multi_test_summary\ndel cite_train_pca, cite_test_pca, multi_train_pca, multi_test_pca\ndel cite_train_umap, cite_test_umap, multi_train_umap, multi_test_umap\ngc.collect()\n\nprint(\"\\n=== Feature Engineering: Complete ===\")\n\n# Final feature arrays for modeling:\n# - cite_train_X, cite_test_X\n# - multi_train_X, multi_test_X","outputs":[],"cell_number":5,"version":1,"status":"generated","created_at":"2025-11-16T00:05:10.844982+00:00","metadata":{},"execution_count":null},{"cell_type":"markdown","source":"## Model Training\n\nDefine and train a multimodal deep learning model (e.g., multi-branch neural network) suitable for the competition's prediction task.","metadata":{}},{"cell_type":"code","source":"# ⚠️ ALEXANDRIA MARKER - DO NOT DELETE (used for syncing outputs from Kaggle)\nprint(\"===ALEXANDRIA_CELL_6_START===\")\n\n# ===ALEXANDRIA_CELL_6_START===\n\nprint(\"===ALEXANDRIA_CELL_6_START===\")\n\n# Model Training: Multimodal Deep Learning Model for Open Problems Multimodal Competition\n\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\nfrom torch.utils.data import Dataset, DataLoader\nfrom sklearn.model_selection import train_test_split\nimport numpy as np\nimport os\n\nprint(\"\\n=== Model Training: Start ===\")\n\n# 1. Split data into training and validation sets\ntry:\n    # For demonstration, use MULTI modality (can be adapted for CITE)\n    X = multi_train_X\n    y = train_multi_targets.astype(np.float32)\n    X_train, X_val, y_train, y_val = train_test_split(\n        X, y, test_size=0.15, random_state=42\n    )\n    print(f\"✅ Data split: X_train={X_train.shape}, X_val={X_val.shape}, y_train={y_train.shape}, y_val={y_val.shape}\")\nexcept Exception as e:\n    print(f\"❌ Error splitting data: {e}\")\n    raise\n\n# 2. Custom Dataset for efficient batching\nclass MultimodalDataset(Dataset):\n    def __init__(self, features, targets=None):\n        self.features = torch.from_numpy(features)\n        self.targets = torch.from_numpy(targets) if targets is not None else None\n\n    def __len__(self):\n        return self.features.shape\n\n    def __getitem__(self, idx):\n        x = self.features[idx]\n        if self.targets is not None:\n            y = self.targets[idx]\n            return x, y\n        else:\n            return x\n\n# 3. Define a multi-branch neural network for multimodal fusion\nclass MultiModalNet(nn.Module):\n    def __init__(self, input_dim, output_dim, hidden_dim=256, dropout=0.3):\n        super().__init__()\n        # Branch 1: summary/PCA/UMAP features (can be split if desired)\n        self.branch = nn.Sequential(\n            nn.Linear(input_dim, hidden_dim),\n            nn.BatchNorm1d(hidden_dim),\n            nn.ReLU(),\n            nn.Dropout(dropout),\n            nn.Linear(hidden_dim, hidden_dim),\n            nn.BatchNorm1d(hidden_dim),\n            nn.ReLU(),\n            nn.Dropout(dropout),\n        )\n        # Fusion + output\n        self.fusion = nn.Sequential(\n            nn.Linear(hidden_dim, hidden_dim),\n            nn.ReLU(),\n            nn.Dropout(dropout),\n            nn.Linear(hidden_dim, output_dim)\n        )\n\n    def forward(self, x):\n        x_branch = self.branch(x)\n        out = self.fusion(x_branch)\n        return out\n\n# 4. Prepare DataLoaders\nBATCH_SIZE = 128\ntry:\n    train_dataset = MultimodalDataset(X_train, y_train)\n    val_dataset = MultimodalDataset(X_val, y_val)\n    train_loader = DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=2, pin_memory=True)\n    val_loader = DataLoader(val_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=2, pin_memory=True)\n    print(f\"✅ DataLoaders ready: train_batches={len(train_loader)}, val_batches={len(val_loader)}\")\nexcept Exception as e:\n    print(f\"❌ Error creating DataLoaders: {e}\")\n    raise\n\n# 5. Model, Loss, Optimizer, Scheduler\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\ninput_dim = X_train.shape[1]\noutput_dim = y_train.shape[1]\nmodel = MultiModalNet(input_dim, output_dim).to(device)\ncriterion = nn.MSELoss()\noptimizer = optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-4)\nscheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.5, patience=3, verbose=True)\n\nprint(f\"✅ Model initialized: input_dim={input_dim}, output_dim={output_dim}, device={device}\")\n\n# 6. Training loop with early stopping and checkpointing\nEPOCHS = 50\nEARLY_STOPPING_PATIENCE = 7\nbest_val_loss = np.inf\nepochs_no_improve = 0\ncheckpoint_path = \"best_multimodal_model.pth\"\n\nfor epoch in range(1, EPOCHS + 1):\n    model.train()\n    train_loss = 0.0\n    for xb, yb in train_loader:\n        xb, yb = xb.to(device), yb.to(device)\n        optimizer.zero_grad()\n        preds = model(xb)\n        loss = criterion(preds, yb)\n        loss.backward()\n        optimizer.step()\n        train_loss += loss.item() * xb.size(0)\n    train_loss /= len(train_loader.dataset)\n\n    model.eval()\n    val_loss = 0.0\n    with torch.no_grad():\n        for xb, yb in val_loader:\n            xb, yb = xb.to(device), yb.to(device)\n            preds = model(xb)\n            loss = criterion(preds, yb)\n            val_loss += loss.item() * xb.size(0)\n    val_loss /= len(val_loader.dataset)\n    scheduler.step(val_loss)\n\n    print(f\"Epoch {epoch:02d}: train_loss={train_loss:.5f}, val_loss={val_loss:.5f}\")\n\n    # Early stopping and checkpointing\n    if val_loss < best_val_loss - 1e-5:\n        best_val_loss = val_loss\n        epochs_no_improve = 0\n        torch.save(model.state_dict(), checkpoint_path)\n        print(f\"  ✅ New best model saved at epoch {epoch} (val_loss={val_loss:.5f})\")\n    else:\n        epochs_no_improve += 1\n        print(f\"  No improvement for {epochs_no_improve} epoch(s).\")\n        if epochs_no_improve >= EARLY_STOPPING_PATIENCE:\n            print(f\"  ⏹️ Early stopping triggered at epoch {epoch}.\")\n            break\n\nprint(\"\\n=== Model Training: Complete ===\")\nprint(f\"Best validation loss: {best_val_loss:.5f}\")\nprint(f\"Model checkpoint saved to: {checkpoint_path}\")","outputs":[],"cell_number":6,"version":1,"status":"generated","created_at":"2025-11-16T00:05:24.482046+00:00","metadata":{},"execution_count":null},{"cell_type":"markdown","source":"## Evaluation\n\nEvaluate model performance using competition-relevant metrics and analyze validation results for further improvement.","metadata":{}},{"cell_type":"code","source":"# ⚠️ ALEXANDRIA MARKER - DO NOT DELETE (used for syncing outputs from Kaggle)\nprint(\"===ALEXANDRIA_CELL_7_START===\")\n\nprint(\"===ALEXANDRIA_CELL_7_START===\")\n\n# Evaluation: Model Performance & Validation Analysis\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport torch\nfrom sklearn.metrics import mean_squared_error, r2_score\nfrom scipy.stats import pearsonr\n\nprint(\"\\n=== Evaluation: Start ===\")\n\n# 1. Load best model checkpoint\ntry:\n    best_model = MultiModalNet(input_dim, output_dim).to(device)\n    best_model.load_state_dict(torch.load(checkpoint_path, map_location=device))\n    best_model.eval()\n    print(\"✅ Best model loaded from checkpoint.\")\nexcept Exception as e:\n    print(f\"❌ Error loading model checkpoint: {e}\")\n    raise\n\n# 2. Predict on validation set\ndef predict_loader(model, loader, device):\n    model.eval()\n    preds = []\n    trues = []\n    with torch.no_grad():\n        for xb, yb in loader:\n            xb = xb.to(device)\n            out = model(xb).cpu().numpy()\n            preds.append(out)\n            trues.append(yb.numpy())\n    return np.vstack(preds), np.vstack(trues)\n\ntry:\n    val_preds, val_trues = predict_loader(best_model, val_loader, device)\n    print(f\"✅ Predictions on validation set: {val_preds.shape}\")\nexcept Exception as e:\n    print(f\"❌ Error predicting on validation set: {e}\")\n    raise\n\n# 3. Compute competition metrics: RMSE, Pearson correlation\ndef compute_rmse(y_true, y_pred):\n    return np.sqrt(mean_squared_error(y_true, y_pred))\n\ndef compute_pearsonr(y_true, y_pred):\n    # Compute mean Pearson correlation across columns\n    corrs = []\n    for i in range(y_true.shape[1]):\n        try:\n            corr, _ = pearsonr(y_true[:, i], y_pred[:, i])\n            corrs.append(corr)\n        except Exception:\n            corrs.append(np.nan)\n    return np.nanmean(corrs)\n\ntry:\n    rmse = compute_rmse(val_trues, val_preds)\n    pearson = compute_pearsonr(val_trues, val_preds)\n    r2 = r2_score(val_trues, val_preds)\n    print(f\"Validation RMSE: {rmse:.5f}\")\n    print(f\"Validation Pearson Correlation (mean over targets): {pearson:.5f}\")\n    print(f\"Validation R^2 Score: {r2:.5f}\")\nexcept Exception as e:\n    print(f\"❌ Error computing evaluation metrics: {e}\")\n\n# 4. Visualize predicted vs. true targets (scatter and density)\ntry:\n    n_targets = val_trues.shape[1]\n    n_plot = min(4, n_targets)\n    fig, axs = plt.subplots(1, n_plot, figsize=(5*n_plot, 4))\n    for i in range(n_plot):\n        ax = axs[i] if n_plot > 1 else axs\n        sns.scatterplot(x=val_trues[:, i], y=val_preds[:, i], alpha=0.5, ax=ax)\n        ax.set_title(f\"Target {i+1}: True vs Pred\")\n        ax.set_xlabel(\"True\")\n        ax.set_ylabel(\"Predicted\")\n        lims = [\n            np.min([val_trues[:, i], val_preds[:, i]]),\n            np.max([val_trues[:, i], val_preds[:, i]])\n        ]\n        ax.plot(lims, lims, 'r--')\n    plt.tight_layout()\n    plt.show()\nexcept Exception as e:\n    print(f\"❌ Error plotting predicted vs. true targets: {e}\")\n\n# 5. Analyze error patterns (distribution, per-target RMSE)\ntry:\n    errors = val_preds - val_trues\n    abs_errors = np.abs(errors)\n    per_target_rmse = np.sqrt(np.mean(errors**2, axis=0))\n    plt.figure(figsize=(8,4))\n    sns.histplot(abs_errors.flatten(), bins=50, kde=True)\n    plt.title(\"Absolute Error Distribution (Validation Set)\")\n    plt.xlabel(\"Absolute Error\")\n    plt.ylabel(\"Frequency\")\n    plt.tight_layout()\n    plt.show()\n    plt.figure(figsize=(10,4))\n    plt.plot(per_target_rmse, marker='o')\n    plt.title(\"Per-Target RMSE (Validation Set)\")\n    plt.xlabel(\"Target Index\")\n    plt.ylabel(\"RMSE\")\n    plt.tight_layout()\n    plt.show()\n    print(f\"Mean per-target RMSE: {np.mean(per_target_rmse):.5f}\")\n    print(f\"Std per-target RMSE: {np.std(per_target_rmse):.5f}\")\nexcept Exception as e:\n    print(f\"❌ Error analyzing error patterns: {e}\")\n\n# 6. Feature importance analysis (permutation importance)\ntry:\n    from sklearn.inspection import permutation_importance\n    # Use a simple surrogate model for feature importance (e.g., RandomForestRegressor)\n    from sklearn.ensemble import RandomForestRegressor\n    print(\"\\n--- Feature Importance (Permutation) ---\")\n    # For speed, use a subset of validation data and targets\n    n_feat = X_val.shape[1]\n    n_targets = y_val.shape[1]\n    n_feat_plot = min(20, n_feat)\n    rf = RandomForestRegressor(n_estimators=50, max_depth=8, n_jobs=-1, random_state=42)\n    rf.fit(X_val, y_val)\n    result = permutation_importance(rf, X_val, y_val, n_repeats=5, random_state=42, n_jobs=-1, scoring='neg_mean_squared_error')\n    importances = result.importances_mean\n    indices = np.argsort(importances)[::-1][:n_feat_plot]\n    plt.figure(figsize=(10,5))\n    plt.bar(range(n_feat_plot), importances[indices])\n    plt.xticks(range(n_feat_plot), [f\"F{idx}\" for idx in indices], rotation=45)\n    plt.title(\"Top Feature Importances (Permutation, RF Surrogate)\")\n    plt.xlabel(\"Feature Index\")\n    plt.ylabel(\"Importance\")\n    plt.tight_layout()\n    plt.show()\nexcept Exception as e:\n    print(f\"❌ Error computing feature importances: {e}\")\n\n# 7. Log and summarize key evaluation statistics\ntry:\n    eval_stats = {\n        \"val_rmse\": rmse,\n        \"val_pearson\": pearson,\n        \"val_r2\": r2,\n        \"mean_per_target_rmse\": float(np.mean(per_target_rmse)),\n        \"std_per_target_rmse\": float(np.std(per_target_rmse))\n    }\n    print(\"\\n--- Evaluation Summary ---\")\n    for k, v in eval_stats.items():\n        print(f\"{k}: {v:.5f}\")\nexcept Exception as e:\n    print(f\"❌ Error logging evaluation statistics: {e}\")\n\nprint(\"\\n=== Evaluation: Complete ===\")","outputs":[],"cell_number":7,"version":1,"status":"generated","created_at":"2025-11-16T00:05:43.780059+00:00","metadata":{},"execution_count":null},{"cell_type":"markdown","source":"## Submission/Results\n\nGenerate predictions for the test set, format them according to sample_submission.csv, and save the submission file for Kaggle upload.","metadata":{}},{"cell_type":"code","source":"# ⚠️ ALEXANDRIA MARKER - DO NOT DELETE (used for syncing outputs from Kaggle)\nprint(\"===ALEXANDRIA_CELL_8_START===\")\n\n# ⚠️ ALEXANDRIA MARKER - DO NOT DELETE (used for syncing outputs from Kaggle)\nprint(\"===ALEXANDRIA_CELL_8_START===\")\n\n# Submission/Results: Generate predictions for the test set, format as sample_submission.csv, and save for Kaggle upload\n\nimport numpy as np\nimport pandas as pd\nimport torch\n\nprint(\"\\n=== Submission/Results: Start ===\")\n\n# 1. Load best model checkpoint (ensure architecture matches)\ntry:\n    best_model = MultiModalNet(multi_test_X.shape[1], train_multi_targets.shape[1]).to(device)\n    best_model.load_state_dict(torch.load(\"best_multimodal_model.pth\", map_location=device))\n    best_model.eval()\n    print(\"✅ Best model loaded for submission.\")\nexcept Exception as e:\n    print(f\"❌ Error loading best model: {e}\")\n    raise\n\n# 2. Predict on test set (MULTI modality)\ndef predict_test(model, X, batch_size=128):\n    model.eval()\n    preds = []\n    n = X.shape[0]\n    with torch.no_grad():\n        for i in range(0, n, batch_size):\n            xb = torch.from_numpy(X[i:i+batch_size]).float().to(device)\n            out = model(xb).cpu().numpy()\n            preds.append(out)\n    return np.vstack(preds)\n\ntry:\n    print(\"Predicting on test set...\")\n    test_preds = predict_test(best_model, multi_test_X)\n    print(f\"✅ Test predictions shape: {test_preds.shape}\")\nexcept Exception as e:\n    print(f\"❌ Error predicting on test set: {e}\")\n    raise\n\n# 3. Format predictions to match sample_submission.csv\ntry:\n    # sample_submission: columns = ['row_id', 'target']\n    # evaluation_ids: columns = ['cell_id', 'gene_id', 'row_id']\n    # test_preds: shape = (n_test_samples, n_targets)\n    # Need to flatten predictions to match row_id order in sample_submission\n\n    # Get mapping from evaluation_ids to test set indices\n    # Assume test_multi_inputs order matches evaluation_ids['cell_id'] order\n    # Flatten predictions in row-major order\n    pred_flat = test_preds.flatten()\n    print(f\"Flattened predictions shape: {pred_flat.shape}\")\n\n    # Ensure length matches sample_submission\n    if len(pred_flat) != len(sample_submission):\n        raise ValueError(f\"Prediction length {len(pred_flat)} does not match sample_submission length {len(sample_submission)}\")\n\n    submission = sample_submission.copy()\n    submission['target'] = pred_flat\n    print(\"✅ Submission DataFrame created.\")\nexcept Exception as e:\n    print(f\"❌ Error formatting submission: {e}\")\n    raise\n\n# 4. Save submission file\nsubmission_path = \"/kaggle/working/submission.csv\"\ntry:\n    submission.to_csv(submission_path, index=False)\n    print(f\"✅ Submission file saved to: {submission_path}\")\nexcept Exception as e:\n    print(f\"❌ Error saving submission file: {e}\")\n    raise\n\n# 5. Display sample of submission for verification\nprint(\"\\n--- Submission Preview ---\")\ndisplay(submission.head())\n\nprint(\"\\n=== Submission/Results: Complete ===\")","outputs":[],"cell_number":8,"version":1,"status":"generated","created_at":"2025-11-16T00:05:54.815562+00:00","metadata":{},"execution_count":null}],"nbformat":4,"nbformat_minor":4}