{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"cells":[{"cell_type":"markdown","source":"# Competition: physionet-ecg-image-digitization\n\n**Generated by Alexandria Research Assistant**\n\n**Dataset:** physionet-ecg-image-digitization\n\n**Task:** competition - kaggle-competition\n\n---\n\n⚠️ **Note:** This notebook contains Alexandria markers (lines starting with `# ⚠️ ALEXANDRIA MARKER`) at the top of each code cell. These markers enable the 'Sync from Kaggle' feature to track cell outputs. Please do not delete them.","metadata":{}},{"cell_type":"markdown","source":"## Setup & Imports\n\nInstall and import all necessary libraries for image processing, data handling, and model development specific to ECG image digitization.","metadata":{}},{"cell_type":"code","source":"# ⚠️ ALEXANDRIA MARKER - DO NOT DELETE (used for syncing outputs from Kaggle)\nprint(\"===ALEXANDRIA_CELL_1_START===\")\n\n# Setup & Imports\n# Install required packages for image processing, data handling, and deep learning\n\n!pip install --quiet numpy pandas opencv-python torch torchvision scikit-learn matplotlib wfdb\n\nimport os\nimport random\n\nimport numpy as np\nimport pandas as pd\nimport cv2\nimport torch\nimport torchvision\nimport matplotlib.pyplot as plt\nimport wfdb\nfrom sklearn.model_selection import train_test_split\n\n# Set the data path as specified\nDATA_PATH = \"/kaggle/input/physionet-ecg-image-digitization\"\n\n# Set random seeds for reproducibility\nSEED = 42\nrandom.seed(SEED)\nnp.random.seed(SEED)\ntorch.manual_seed(SEED)\ntorch.cuda.manual_seed_all(SEED)\n\nprint(\"All libraries installed and imported successfully.\")\nprint(f\"Data path set to: {DATA_PATH}\")\nprint(f\"Random seed set to: {SEED}\")","outputs":[],"cell_number":1,"version":1,"status":"generated","created_at":"2025-11-16T18:57:00.351563+00:00","metadata":{},"execution_count":null},{"cell_type":"markdown","source":"## Data Loading\n\nLoad ECG images and corresponding ground truth time-series data from the physionet-ecg-image-digitization dataset.","metadata":{}},{"cell_type":"code","source":"# ⚠️ ALEXANDRIA MARKER - DO NOT DELETE (used for syncing outputs from Kaggle)\nprint(\"===ALEXANDRIA_CELL_2_START===\")\n\nprint(\"===ALEXANDRIA_CELL_2_START===\")\n\n# Data Loading: Load ECG images, ground truth time-series, and metadata\n\nimport glob\n\n# 1. List all subdirectories in the dataset root\nsubdirs = [os.path.join(DATA_PATH, d) for d in os.listdir(DATA_PATH) if os.path.isdir(os.path.join(DATA_PATH, d))]\nprint(f\"Found {len(subdirs)} subdirectories in dataset root.\")\n\n# 2. Find all ECG image files (PNG/JPG) recursively\nimage_extensions = (\"*.png\", \"*.jpg\", \"*.jpeg\")\nimage_files = []\nfor subdir in subdirs:\n    for ext in image_extensions:\n        image_files.extend(glob.glob(os.path.join(subdir, \"**\", ext), recursive=True))\nprint(f\"Found {len(image_files)} ECG image files (PNG/JPG).\")\n\n# 3. Find all ground truth time-series files (CSV/WFDB)\ncsv_files = []\nwfdb_record_bases = []\nfor subdir in subdirs:\n    # CSV files\n    csv_files.extend(glob.glob(os.path.join(subdir, \"**\", \"*.csv\"), recursive=True))\n    # WFDB: look for .hea header files (WFDB records are .hea + .dat)\n    wfdb_headers = glob.glob(os.path.join(subdir, \"**\", \"*.hea\"), recursive=True)\n    for hea in wfdb_headers:\n        wfdb_record_bases.append(hea[:-4])  # strip .hea to get record base\n\nprint(f\"Found {len(csv_files)} ground truth CSV files.\")\nprint(f\"Found {len(wfdb_record_bases)} WFDB record bases.\")\n\n# 4. Find metadata/demographic CSV/header files\nmetadata_files = glob.glob(os.path.join(DATA_PATH, \"*.csv\"))\nprint(f\"Found {len(metadata_files)} metadata/demographic CSV files at dataset root.\")\n\n# 5. Preview: Show a sample ECG image and corresponding signal (if available)\nif image_files:\n    sample_img_path = image_files\n    img = cv2.imread(sample_img_path)\n    img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    plt.figure(figsize=(8, 4))\n    plt.imshow(img_rgb)\n    plt.title(f\"Sample ECG Image: {os.path.basename(sample_img_path)}\")\n    plt.axis('off')\n    plt.show()\nelse:\n    print(\"No ECG images found for preview.\")\n\n# Preview a sample ground truth signal (CSV or WFDB)\nif csv_files:\n    sample_csv = csv_files\n    df = pd.read_csv(sample_csv)\n    plt.figure(figsize=(10, 3))\n    plt.plot(df.iloc[:,0], label=df.columns)\n    if df.shape[1] > 1:\n        plt.plot(df.iloc[:,1], label=df.columns[1])\n    plt.title(f\"Sample Ground Truth Signal (CSV): {os.path.basename(sample_csv)}\")\n    plt.xlabel(\"Sample Index\")\n    plt.ylabel(\"Amplitude\")\n    plt.legend()\n    plt.show()\nelif wfdb_record_bases:\n    sample_record = wfdb_record_bases\n    record = wfdb.rdrecord(sample_record)\n    plt.figure(figsize=(10, 3))\n    plt.plot(record.p_signal[:,0], label=record.sig_name)\n    if record.p_signal.shape[1] > 1:\n        plt.plot(record.p_signal[:,1], label=record.sig_name[1])\n    plt.title(f\"Sample Ground Truth Signal (WFDB): {os.path.basename(sample_record)}\")\n    plt.xlabel(\"Sample Index\")\n    plt.ylabel(\"Amplitude\")\n    plt.legend()\n    plt.show()\nelse:\n    print(\"No ground truth signal files found for preview.\")\n\n# 6. Preview metadata/demographics (if available)\nif metadata_files:\n    meta_df = pd.read_csv(metadata_files)\n    print(\"Sample metadata/demographic info:\")\n    display(meta_df.head())\nelse:\n    print(\"No metadata/demographic CSV files found.\")","outputs":[],"cell_number":2,"version":1,"status":"generated","created_at":"2025-11-16T18:57:13.536917+00:00","metadata":{},"execution_count":null},{"cell_type":"markdown","source":"## Exploratory Data Analysis (EDA)\n\nPerform exploratory analysis to understand image artifacts, signal characteristics, and dataset distribution.","metadata":{}},{"cell_type":"code","source":"# ⚠️ ALEXANDRIA MARKER - DO NOT DELETE (used for syncing outputs from Kaggle)\nprint(\"===ALEXANDRIA_CELL_3_START===\")\n\n# ⚠️ ALEXANDRIA MARKER - DO NOT DELETE (used for syncing outputs from Kaggle)\nprint(\"===ALEXANDRIA_CELL_3_START===\")\n\n# Exploratory Data Analysis (EDA)\nprint(\"=== Exploratory Data Analysis (EDA) ===\")\n\nimport os\nimport glob\nimport cv2\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport wfdb\nfrom collections import defaultdict\n\n# Use the data path from prior cells\nDATA_PATH = \"/kaggle/input/physionet-ecg-image-digitization\"\n\n# 1. Visualize sample ECG images and overlay ground truth signals\nprint(\"Visualizing sample ECG images and overlaying ground truth signals...\")\n\n# Select a few sample image files for visualization\nsample_image_files = image_files[:3] if image_files else []\nsample_csv_files = csv_files[:3] if csv_files else []\nsample_wfdb_bases = wfdb_record_bases[:3] if wfdb_record_bases else []\n\nfor i, img_path in enumerate(sample_image_files):\n    # Read and display the ECG image\n    img = cv2.imread(img_path)\n    img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    plt.figure(figsize=(12, 6))\n    plt.subplot(1, 2, 1)\n    plt.imshow(img_rgb)\n    plt.title(f\"Sample ECG Image: {os.path.basename(img_path)}\")\n    plt.axis('off')\n    \n    # Try to find corresponding ground truth signal (CSV or WFDB)\n    base_name = os.path.splitext(os.path.basename(img_path))[0]\n    signal_found = False\n    \n    # Check for CSV signal\n    for csv_path in csv_files:\n        if base_name in csv_path:\n            df = pd.read_csv(csv_path)\n            plt.subplot(1, 2, 2)\n            for col in df.columns:\n                plt.plot(df[col], label=col)\n            plt.title(f\"Ground Truth Signal (CSV): {os.path.basename(csv_path)}\")\n            plt.xlabel(\"Sample Index\")\n            plt.ylabel(\"Amplitude\")\n            plt.legend()\n            signal_found = True\n            break\n    \n    # If no CSV, check for WFDB\n    if not signal_found:\n        for wfdb_base in wfdb_record_bases:\n            if base_name in wfdb_base:\n                record = wfdb.rdrecord(wfdb_base)\n                plt.subplot(1, 2, 2)\n                for i_lead in range(record.p_signal.shape[1]):\n                    plt.plot(record.p_signal[:, i_lead], label=record.sig_name[i_lead])\n                plt.title(f\"Ground Truth Signal (WFDB): {os.path.basename(wfdb_base)}\")\n                plt.xlabel(\"Sample Index\")\n                plt.ylabel(\"Amplitude\")\n                plt.legend()\n                signal_found = True\n                break\n    \n    if not signal_found:\n        plt.subplot(1, 2, 2)\n        plt.text(0.5, 0.5, 'No matching signal found', horizontalalignment='center', verticalalignment='center', transform=plt.gca().transAxes)\n        plt.title(\"No Ground Truth Signal\")\n        plt.axis('off')\n    \n    plt.tight_layout()\n    plt.show()\n\n# 2. Summarize demographic and diagnostic metadata\nprint(\"Summarizing demographic and diagnostic metadata...\")\n\nif metadata_files:\n    meta_df = pd.read_csv(metadata_files[0])\n    print(\"Dataset metadata summary:\")\n    print(meta_df.info())\n    print(\"\\nFirst few rows:\")\n    display(meta_df.head())\n    print(\"\\nBasic statistics:\")\n    display(meta_df.describe())\nelse:\n    print(\"No metadata/demographic CSV files found.\")\n\n# 3. Analyze image quality, artifacts, and lead configurations\nprint(\"Analyzing image quality, artifacts, and lead configurations...\")\n\nimage_stats = defaultdict(list)\nfor img_path in image_files:\n    img = cv2.imread(img_path)\n    if img is not None:\n        image_stats['width'].append(img.shape[1])\n        image_stats['height'].append(img.shape[0])\n        image_stats['channels'].append(img.shape[2] if len(img.shape) > 2 else 1)\n        # Simple artifact detection: check for uniform regions (potential artifacts)\n        gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n        _, binary = cv2.threshold(gray, 240, 255, cv2.THRESH_BINARY)\n        artifact_ratio = np.sum(binary > 0) / binary.size\n        image_stats['artifact_ratio'].append(artifact_ratio)\n\nif image_stats:\n    print(f\"Image count: {len(image_stats['width'])}\")\n    print(f\"Average image size: {np.mean(image_stats['width']):.1f} x {np.mean(image_stats['height']):.1f}\")\n    print(f\"Channel distribution: {pd.Series(image_stats['channels']).value_counts().to_dict()}\")\n    print(f\"Average artifact ratio: {np.mean(image_stats['artifact_ratio']):.3f}\")\nelse:\n    print(\"No images found for quality analysis.\")\n\n# 4. Plot signal statistics (amplitude, duration, noise)\nprint(\"Plotting signal statistics (amplitude, duration, noise)...\")\n\nsignal_stats = defaultdict(list)\nfor csv_path in csv_files:\n    df = pd.read_csv(csv_path)\n    for col in df.columns:\n        signal = df[col].dropna().values\n        if len(signal) > 0:\n            signal_stats['amplitude'].append(np.max(signal) - np.min(signal))\n            signal_stats['duration'].append(len(signal))\n            signal_stats['noise'].append(np.std(signal))\n\nfor wfdb_base in wfdb_record_bases:\n    record = wfdb.rdrecord(wfdb_base)\n    for i_lead in range(record.p_signal.shape[1]):\n        signal = record.p_signal[:, i_lead]\n        if len(signal) > 0:\n            signal_stats['amplitude'].append(np.max(signal) - np.min(signal))\n            signal_stats['duration'].append(len(signal))\n            signal_stats['noise'].append(np.std(signal))\n\nif signal_stats:\n    fig, axes = plt.subplots(1, 3, figsize=(15, 5))\n    axes[0].hist(signal_stats['amplitude'], bins=20, alpha=0.7)\n    axes[0].set_title('Signal Amplitude Distribution')\n    axes[0].set_xlabel('Amplitude')\n    axes[0].set_ylabel('Frequency')\n    \n    axes[1].hist(signal_stats['duration'], bins=20, alpha=0.7)\n    axes[1].set_title('Signal Duration Distribution')\n    axes[1].set_xlabel('Duration (samples)')\n    axes[1].set_ylabel('Frequency')\n    \n    axes[2].hist(signal_stats['noise'], bins=20, alpha=0.7)\n    axes[2].set_title('Signal Noise (Std) Distribution')\n    axes[2].set_xlabel('Noise (Std)')\n    axes[2].set_ylabel('Frequency')\n    \n    plt.tight_layout()\n    plt.show()\nelse:\n    print(\"No signal files found for statistics.\")\n\nprint(\"EDA completed.\")","outputs":[],"cell_number":3,"version":1,"status":"generated","created_at":"2025-11-16T18:57:36.505535+00:00","metadata":{},"execution_count":null},{"cell_type":"markdown","source":"## Preprocessing\n\nClean and standardize ECG images and signals to prepare for feature extraction and modeling.","metadata":{}},{"cell_type":"code","source":"# ⚠️ ALEXANDRIA MARKER - DO NOT DELETE (used for syncing outputs from Kaggle)\nprint(\"===ALEXANDRIA_CELL_4_START===\")\n\n# ⚠️ ALEXANDRIA MARKER - DO NOT DELETE (used for syncing outputs from Kaggle)\nprint(\"===ALEXANDRIA_CELL_4_START===\")\n\nprint(\"=== Preprocessing: Cleaning and standardizing ECG images and signals ===\")\n\nimport os\nimport cv2\nimport numpy as np\nimport pandas as pd\nimport wfdb\nfrom glob import glob\nfrom sklearn.preprocessing import StandardScaler\n\n# Use the data path from prior cells\nDATA_PATH = \"/kaggle/input/physionet-ecg-image-digitization\"\n\n# Define preprocessing parameters\nIMAGE_SIZE = (1600, 512)  # Standardize to 1600x512 as in research\nGRID_THRESHOLD = 230  # For binary thresholding to remove grid lines\nMORPH_KERNEL_SIZE = 3  # For morphological operations\n\n# Output directories for preprocessed data\nPREPROCESSED_IMAGES_DIR = \"/kaggle/working/preprocessed_images\"\nPREPROCESSED_SIGNALS_DIR = \"/kaggle/working/preprocessed_signals\"\nos.makedirs(PREPROCESSED_IMAGES_DIR, exist_ok=True)\nos.makedirs(PREPROCESSED_SIGNALS_DIR, exist_ok=True)\n\n# 1. Preprocess ECG images\nprint(\"Preprocessing ECG images...\")\nfor img_path in image_files:\n    try:\n        # Read image\n        img = cv2.imread(img_path)\n        if img is None:\n            print(f\"Warning: Could not read image {img_path}\")\n            continue\n        \n        # Convert to grayscale\n        gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n        \n        # Apply thresholding to remove grid lines and artifacts\n        _, binary = cv2.threshold(gray, GRID_THRESHOLD, 255, cv2.THRESH_BINARY)\n        \n        # Apply morphological operations to remove small artifacts\n        kernel = np.ones((MORPH_KERNEL_SIZE, MORPH_KERNEL_SIZE), np.uint8)\n        binary = cv2.morphologyEx(binary, cv2.MORPH_OPEN, kernel)\n        binary = cv2.morphologyEx(binary, cv2.MORPH_CLOSE, kernel)\n        \n        # Resize to standard size\n        resized = cv2.resize(binary, IMAGE_SIZE, interpolation=cv2.INTER_AREA)\n        \n        # Save preprocessed image\n        output_path = os.path.join(PREPROCESSED_IMAGES_DIR, os.path.basename(img_path))\n        cv2.imwrite(output_path, resized)\n        \n    except Exception as e:\n        print(f\"Error processing image {img_path}: {e}\")\n\nprint(f\"Preprocessed {len(image_files)} ECG images.\")\n\n# 2. Preprocess ECG signals\nprint(\"Preprocessing ECG signals...\")\nscaler = StandardScaler()\n\nfor csv_path in csv_files:\n    try:\n        # Read CSV signal\n        df = pd.read_csv(csv_path)\n        \n        # Normalize signal amplitudes\n        normalized = scaler.fit_transform(df.values)\n        normalized_df = pd.DataFrame(normalized, columns=df.columns)\n        \n        # Save preprocessed signal\n        output_path = os.path.join(PREPROCESSED_SIGNALS_DIR, os.path.basename(csv_path))\n        normalized_df.to_csv(output_path, index=False)\n        \n    except Exception as e:\n        print(f\"Error processing signal {csv_path}: {e}\")\n\nfor wfdb_base in wfdb_record_bases:\n    try:\n        # Read WFDB signal\n        record = wfdb.rdrecord(wfdb_base)\n        \n        # Normalize signal amplitudes\n        normalized = scaler.fit_transform(record.p_signal)\n        \n        # Save preprocessed signal as CSV\n        output_path = os.path.join(PREPROCESSED_SIGNALS_DIR, os.path.basename(wfdb_base) + \".csv\")\n        df = pd.DataFrame(normalized, columns=record.sig_name)\n        df.to_csv(output_path, index=False)\n        \n    except Exception as e:\n        print(f\"Error processing WFDB record {wfdb_base}: {e}\")\n\nprint(f\"Preprocessed {len(csv_files) + len(wfdb_record_bases)} ECG signals.\")\n\n# 3. Handle missing or corrupted data\nprint(\"Handling missing or corrupted data...\")\npreprocessed_image_files = glob(os.path.join(PREPROCESSED_IMAGES_DIR, \"*.png\")) + glob(os.path.join(PREPROCESSED_IMAGES_DIR, \"*.jpg\"))\npreprocessed_signal_files = glob(os.path.join(PREPROCESSED_SIGNALS_DIR, \"*.csv\"))\n\n# Remove any files that are empty or corrupted\nvalid_image_files = []\nfor img_path in preprocessed_image_files:\n    try:\n        img = cv2.imread(img_path)\n        if img is not None and img.size > 0:\n            valid_image_files.append(img_path)\n        else:\n            os.remove(img_path)\n    except:\n        os.remove(img_path)\n\nvalid_signal_files = []\nfor sig_path in preprocessed_signal_files:\n    try:\n        df = pd.read_csv(sig_path)\n        if df.shape[0] > 0 and df.shape[1] > 0:\n            valid_signal_files.append(sig_path)\n        else:\n            os.remove(sig_path)\n    except:\n        os.remove(sig_path)\n\nprint(f\"Valid preprocessed images: {len(valid_image_files)}\")\nprint(f\"Valid preprocessed signals: {len(valid_signal_files)}\")\n\nprint(\"Preprocessing completed.\")","outputs":[],"cell_number":4,"version":1,"status":"generated","created_at":"2025-11-16T18:57:53.248271+00:00","metadata":{},"execution_count":null},{"cell_type":"markdown","source":"## Feature Engineering\n\nExtract features from ECG images and signals relevant for digitization and classification tasks.","metadata":{}},{"cell_type":"code","source":"# ⚠️ ALEXANDRIA MARKER - DO NOT DELETE (used for syncing outputs from Kaggle)\nprint(\"===ALEXANDRIA_CELL_5_START===\")\n\n# ⚠️ ALEXANDRIA MARKER - DO NOT DELETE (used for syncing outputs from Kaggle)\nprint(\"===ALEXANDRIA_CELL_5_START===\")\n\nprint(\"=== Feature Engineering: Extracting features from ECG images and signals ===\")\n\nimport os\nimport cv2\nimport numpy as np\nimport pandas as pd\nimport torch\nimport matplotlib.pyplot as plt\nfrom scipy import ndimage\nfrom skimage.transform import hough_line, hough_line_peaks\nfrom glob import glob\n\n# Use the data path from prior cells\nDATA_PATH = \"/kaggle/input/physionet-ecg-image-digitization\"\n\n# Output directories for feature-engineered data\nFEATURES_IMAGES_DIR = \"/kaggle/working/features_images\"\nFEATURES_SIGNALS_DIR = \"/kaggle/working/features_signals\"\nos.makedirs(FEATURES_IMAGES_DIR, exist_ok=True)\nos.makedirs(FEATURES_SIGNALS_DIR, exist_ok=True)\n\n# 1. Segment ECG waveforms using deep learning (nnU-Net segmentation)\n# For demonstration, we'll simulate segmentation using morphological operations\n# In practice, load a pretrained nnU-Net model and run inference\nprint(\"Segmenting ECG waveforms using deep learning (simulated with morphological operations)...\")\n\nfor img_path in glob(os.path.join(PREPROCESSED_IMAGES_DIR, \"*.png\")):\n    try:\n        # Read preprocessed image\n        img = cv2.imread(img_path, cv2.IMREAD_GRAYSCALE)\n        \n        # Simulate segmentation: remove small artifacts and enhance lines\n        kernel = np.ones((3, 3), np.uint8)\n        segmented = cv2.morphologyEx(img, cv2.MORPH_CLOSE, kernel)\n        segmented = cv2.morphologyEx(segmented, cv2.MORPH_OPEN, kernel)\n        \n        # Save segmented image\n        output_path = os.path.join(FEATURES_IMAGES_DIR, \"seg_\" + os.path.basename(img_path))\n        cv2.imwrite(output_path, segmented)\n        \n    except Exception as e:\n        print(f\"Error segmenting image {img_path}: {e}\")\n\nprint(f\"Segmented {len(glob(os.path.join(PREPROCESSED_IMAGES_DIR, '*.png')))} ECG images.\")\n\n# 2. Extract classical features using Hough Transform for line detection\nprint(\"Extracting classical features using Hough Transform for line detection...\")\n\nfor img_path in glob(os.path.join(FEATURES_IMAGES_DIR, \"seg_*.png\")):\n    try:\n        # Read segmented image\n        img = cv2.imread(img_path, cv2.IMREAD_GRAYSCALE)\n        \n        # Apply Hough Transform\n        h, theta, d = hough_line(img)\n        _, angles, dists = hough_line_peaks(h, theta, d)\n        \n        # Save Hough features as CSV\n        hough_features = pd.DataFrame({'angle': angles, 'distance': dists})\n        output_path = os.path.join(FEATURES_SIGNALS_DIR, \"hough_\" + os.path.splitext(os.path.basename(img_path))[0] + \".csv\")\n        hough_features.to_csv(output_path, index=False)\n        \n    except Exception as e:\n        print(f\"Error extracting Hough features from {img_path}: {e}\")\n\nprint(f\"Extracted Hough features from {len(glob(os.path.join(FEATURES_IMAGES_DIR, 'seg_*.png')))} segmented images.\")\n\n# 3. Generate synthetic data for augmentation using ECG-Image-Kit (simulated)\n# For demonstration, we'll simulate synthetic data generation by adding noise\nprint(\"Generating synthetic data for augmentation (simulated by adding noise)...\")\n\nfor img_path in glob(os.path.join(FEATURES_IMAGES_DIR, \"seg_*.png\")):\n    try:\n        # Read segmented image\n        img = cv2.imread(img_path, cv2.IMREAD_GRAYSCALE)\n        \n        # Add Gaussian noise\n        noise = np.random.normal(0, 10, img.shape)\n        synthetic = np.clip(img + noise, 0, 255).astype(np.uint8)\n        \n        # Save synthetic image\n        output_path = os.path.join(FEATURES_IMAGES_DIR, \"synth_\" + os.path.basename(img_path))\n        cv2.imwrite(output_path, synthetic)\n        \n    except Exception as e:\n        print(f\"Error generating synthetic data for {img_path}: {e}\")\n\nprint(f\"Generated synthetic data for {len(glob(os.path.join(FEATURES_IMAGES_DIR, 'seg_*.png')))} segmented images.\")\n\n# 4. Prepare input tensors for model training\nprint(\"Preparing input tensors for model training...\")\n\nfor img_path in glob(os.path.join(FEATURES_IMAGES_DIR, \"seg_*.png\")):\n    try:\n        # Read segmented image\n        img = cv2.imread(img_path, cv2.IMREAD_GRAYSCALE)\n        \n        # Normalize and convert to tensor\n        img_tensor = torch.from_numpy(img / 255.0).float().unsqueeze(0)\n        \n        # Save tensor\n        output_path = os.path.join(FEATURES_SIGNALS_DIR, \"tensor_\" + os.path.splitext(os.path.basename(img_path))[0] + \".pt\")\n        torch.save(img_tensor, output_path)\n        \n    except Exception as e:\n        print(f\"Error preparing tensor for {img_path}: {e}\")\n\nprint(f\"Prepared input tensors for {len(glob(os.path.join(FEATURES_IMAGES_DIR, 'seg_*.png')))} segmented images.\")\n\nprint(\"Feature engineering completed.\")","outputs":[],"cell_number":5,"version":1,"status":"generated","created_at":"2025-11-16T18:58:11.426508+00:00","metadata":{},"execution_count":null},{"cell_type":"markdown","source":"## Model Training\n\nTrain a deep segmentation model (e.g., nnU-Net) and/or classical algorithms to digitize ECG waveforms from images.","metadata":{}},{"cell_type":"code","source":"# ⚠️ ALEXANDRIA MARKER - DO NOT DELETE (used for syncing outputs from Kaggle)\nprint(\"===ALEXANDRIA_CELL_6_START===\")\n\nprint(\"===ALEXANDRIA_CELL_6_START===\")\n\n# Title: Model Training\n# Description: Train a deep segmentation model (U-Net) and classical algorithms to digitize ECG waveforms from images.\n\nimport os\nimport numpy as np\nimport pandas as pd\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\nfrom torch.utils.data import Dataset, DataLoader, random_split\nimport torchvision.transforms as T\nimport cv2\nfrom glob import glob\nimport matplotlib.pyplot as plt\n\n# Paths from previous cells\nPREPROCESSED_IMAGES_DIR = \"/kaggle/working/preprocessed_images\"\nFEATURES_IMAGES_DIR = \"/kaggle/working/features_images\"\nFEATURES_SIGNALS_DIR = \"/kaggle/working/features_signals\"\n\n# Hyperparameters\nBATCH_SIZE = 8\nNUM_EPOCHS = 20\nLEARNING_RATE = 1e-3\nIMAGE_SIZE = (1600, 512)\nDEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nVAL_SPLIT = 0.15\n\n# --- Dataset Definition ---\nclass ECGSegmentationDataset(Dataset):\n    def __init__(self, image_dir, mask_dir, transform=None):\n        self.image_paths = sorted(glob(os.path.join(image_dir, \"seg_*.png\")))\n        self.mask_paths = sorted(glob(os.path.join(mask_dir, \"*.png\")))\n        # Match masks to images by basename\n        self.mask_dict = {os.path.basename(p): p for p in self.mask_paths}\n        self.transform = transform\n        self.valid_pairs = []\n        for img_path in self.image_paths:\n            base = os.path.basename(img_path)\n            if base in self.mask_dict:\n                self.valid_pairs.append((img_path, self.mask_dict[base]))\n        self.length = len(self.valid_pairs)\n\n    def __len__(self):\n        return self.length\n\n    def __getitem__(self, idx):\n        img_path, mask_path = self.valid_pairs[idx]\n        img = cv2.imread(img_path, cv2.IMREAD_GRAYSCALE)\n        mask = cv2.imread(mask_path, cv2.IMREAD_GRAYSCALE)\n        img = cv2.resize(img, IMAGE_SIZE, interpolation=cv2.INTER_AREA)\n        mask = cv2.resize(mask, IMAGE_SIZE, interpolation=cv2.INTER_NEAREST)\n        img = img.astype(np.float32) / 255.0\n        mask = (mask > 127).astype(np.float32)\n        img = np.expand_dims(img, axis=0)\n        mask = np.expand_dims(mask, axis=0)\n        if self.transform:\n            img = self.transform(torch.from_numpy(img))\n            mask = self.transform(torch.from_numpy(mask))\n        return torch.from_numpy(img), torch.from_numpy(mask)\n\n# --- U-Net Model Definition ---\nclass DoubleConv(nn.Module):\n    def __init__(self, in_channels, out_channels):\n        super().__init__()\n        self.double_conv = nn.Sequential(\n            nn.Conv2d(in_channels, out_channels, 3, padding=1),\n            nn.InstanceNorm2d(out_channels),\n            nn.LeakyReLU(inplace=True),\n            nn.Conv2d(out_channels, out_channels, 3, padding=1),\n            nn.InstanceNorm2d(out_channels),\n            nn.LeakyReLU(inplace=True)\n        )\n\n    def forward(self, x):\n        return self.double_conv(x)\n\nclass UNet(nn.Module):\n    def __init__(self, in_channels=1, out_channels=1, features=[32, 64, 128, 256]):\n        super().__init__()\n        self.downs = nn.ModuleList()\n        self.ups = nn.ModuleList()\n        # Down path\n        for feature in features:\n            self.downs.append(DoubleConv(in_channels, feature))\n            in_channels = feature\n        # Up path\n        for feature in reversed(features):\n            self.ups.append(\n                nn.ConvTranspose2d(feature*2, feature, kernel_size=2, stride=2)\n            )\n            self.ups.append(DoubleConv(feature*2, feature))\n        self.bottleneck = DoubleConv(features[-1], features[-1]*2)\n        self.final_conv = nn.Conv2d(features, out_channels, kernel_size=1)\n\n    def forward(self, x):\n        skip_connections = []\n        for down in self.downs:\n            x = down(x)\n            skip_connections.append(x)\n            x = nn.functional.max_pool2d(x, 2)\n        x = self.bottleneck(x)\n        skip_connections = skip_connections[::-1]\n        for idx in range(0, len(self.ups), 2):\n            x = self.ups[idx](x)\n            skip = skip_connections[idx//2]\n            if x.shape != skip.shape:\n                x = T.functional.resize(x, skip.shape[2:])\n            x = torch.cat((skip, x), dim=1)\n            x = self.ups[idx+1](x)\n        return torch.sigmoid(self.final_conv(x))\n\n# --- Loss and Metrics ---\ndef dice_loss(pred, target, smooth=1.):\n    pred = pred.contiguous()\n    target = target.contiguous()\n    intersection = (pred * target).sum(dim=(2,3))\n    loss = 1 - ((2. * intersection + smooth) / (pred.sum(dim=(2,3)) + target.sum(dim=(2,3)) + smooth))\n    return loss.mean()\n\ndef iou_score(pred, target, threshold=0.5):\n    pred = (pred > threshold).float()\n    intersection = (pred * target).sum(dim=(2,3))\n    union = ((pred + target) > 0).float().sum(dim=(2,3))\n    score = (intersection + 1e-6) / (union + 1e-6)\n    return score.mean().item()\n\n# --- Data Preparation ---\nprint(\"Preparing dataset splits...\")\ndataset = ECGSegmentationDataset(FEATURES_IMAGES_DIR, FEATURES_IMAGES_DIR)\nn_val = int(len(dataset) * VAL_SPLIT)\nn_train = len(dataset) - n_val\ntrain_set, val_set = random_split(dataset, [n_train, n_val], generator=torch.Generator().manual_seed(42))\ntrain_loader = DataLoader(train_set, batch_size=BATCH_SIZE, shuffle=True, num_workers=2, pin_memory=True)\nval_loader = DataLoader(val_set, batch_size=BATCH_SIZE, shuffle=False, num_workers=2, pin_memory=True)\nprint(f\"Train samples: {n_train}, Validation samples: {n_val}\")\n\n# --- Model, Optimizer, Scheduler ---\nmodel = UNet().to(DEVICE)\noptimizer = optim.Adam(model.parameters(), lr=LEARNING_RATE)\nscheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.5, patience=3, verbose=True)\n\n# --- Training Loop ---\nbest_val_loss = float('inf')\nhistory = {'train_loss': [], 'val_loss': [], 'val_iou': []}\n\nprint(\"Starting model training...\")\nfor epoch in range(NUM_EPOCHS):\n    model.train()\n    train_losses = []\n    for imgs, masks in train_loader:\n        imgs, masks = imgs.to(DEVICE), masks.to(DEVICE)\n        optimizer.zero_grad()\n        outputs = model(imgs)\n        loss = dice_loss(outputs, masks)\n        loss.backward()\n        optimizer.step()\n        train_losses.append(loss.item())\n    avg_train_loss = np.mean(train_losses)\n    model.eval()\n    val_losses = []\n    val_ious = []\n    with torch.no_grad():\n        for imgs, masks in val_loader:\n            imgs, masks = imgs.to(DEVICE), masks.to(DEVICE)\n            outputs = model(imgs)\n            loss = dice_loss(outputs, masks)\n            val_losses.append(loss.item())\n            val_ious.append(iou_score(outputs, masks))\n    avg_val_loss = np.mean(val_losses)\n    avg_val_iou = np.mean(val_ious)\n    scheduler.step(avg_val_loss)\n    history['train_loss'].append(avg_train_loss)\n    history['val_loss'].append(avg_val_loss)\n    history['val_iou'].append(avg_val_iou)\n    print(f\"Epoch {epoch+1}/{NUM_EPOCHS} | Train Loss: {avg_train_loss:.4f} | Val Loss: {avg_val_loss:.4f} | Val IoU: {avg_val_iou:.4f}\")\n    # Save best model\n    if avg_val_loss < best_val_loss:\n        best_val_loss = avg_val_loss\n        torch.save(model.state_dict(), \"/kaggle/working/best_unet_ecg.pth\")\n        print(\"Saved new best model.\")\n\n# --- Plot Training Curves ---\nplt.figure(figsize=(12,4))\nplt.subplot(1,2,1)\nplt.plot(history['train_loss'], label='Train Loss')\nplt.plot(history['val_loss'], label='Val Loss')\nplt.xlabel('Epoch')\nplt.ylabel('Loss')\nplt.title('Loss Curve')\nplt.legend()\nplt.subplot(1,2,2)\nplt.plot(history['val_iou'], label='Val IoU')\nplt.xlabel('Epoch')\nplt.ylabel('IoU')\nplt.title('Validation IoU')\nplt.legend()\nplt.tight_layout()\nplt.show()\n\nprint(\"Training complete. Best model weights saved to /kaggle/working/best_unet_ecg.pth\")\n\n# --- Validate Model on Hold-out Data ---\nprint(\"Validating model on hold-out data...\")\nmodel.load_state_dict(torch.load(\"/kaggle/working/best_unet_ecg.pth\"))\nmodel.eval()\nious = []\nwith torch.no_grad():\n    for imgs, masks in val_loader:\n        imgs, masks = imgs.to(DEVICE), masks.to(DEVICE)\n        outputs = model(imgs)\n        ious.append(iou_score(outputs, masks))\nmean_iou = np.mean(ious)\nprint(f\"Final Validation IoU: {mean_iou:.4f}\")\n\n# --- Save Model Weights ---\ntorch.save(model.state_dict(), \"/kaggle/working/final_unet_ecg.pth\")\nprint(\"Final model weights saved to /kaggle/working/final_unet_ecg.pth\")","outputs":[],"cell_number":6,"version":1,"status":"generated","created_at":"2025-11-16T18:59:04.663044+00:00","metadata":{},"execution_count":null},{"cell_type":"markdown","source":"## Evaluation\n\nEvaluate model predictions against ground truth using competition metrics and visualize results.","metadata":{}},{"cell_type":"code","source":"# ⚠️ ALEXANDRIA MARKER - DO NOT DELETE (used for syncing outputs from Kaggle)\nprint(\"===ALEXANDRIA_CELL_7_START===\")\n\nprint(\"===ALEXANDRIA_CELL_7_START===\")\n\n# Title: Evaluation\n# Description: Evaluate model predictions against ground truth using competition metrics and visualize results.\n\nimport os\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom glob import glob\nfrom sklearn.metrics import f1_score, precision_score, recall_score\nfrom scipy.signal import find_peaks\nfrom scipy.spatial.distance import directed_hausdorff\n\n# For clinical parameter extraction\nimport neurokit2 as nk\n\n# Paths from previous cells\nPREPROCESSED_SIGNALS_DIR = \"/kaggle/working/preprocessed_signals\"\nFEATURES_SIGNALS_DIR = \"/kaggle/working/features_signals\"\nDATA_PATH = \"/kaggle/input/physionet-ecg-image-digitization\"\n\n# 1. Load predictions and ground truth signals\nprint(\"Loading predicted and ground truth signals...\")\n\n# For demonstration, assume predictions are stored as CSVs in FEATURES_SIGNALS_DIR with prefix \"pred_\"\npred_signal_files = sorted(glob(os.path.join(FEATURES_SIGNALS_DIR, \"pred_*.csv\")))\ngt_signal_files = sorted(glob(os.path.join(PREPROCESSED_SIGNALS_DIR, \"*.csv\")))\n\n# Match predictions to ground truth by basename (after removing \"pred_\")\ngt_dict = {os.path.basename(f).replace(\".csv\", \"\"): f for f in gt_signal_files}\npairs = []\nfor pred_path in pred_signal_files:\n    base = os.path.basename(pred_path).replace(\"pred_\", \"\").replace(\".csv\", \"\")\n    if base in gt_dict:\n        pairs.append((pred_path, gt_dict[base]))\n\nprint(f\"Found {len(pairs)} prediction-ground truth pairs for evaluation.\")\n\n# 2. Define evaluation metrics\n\ndef compute_rmse(y_true, y_pred):\n    return np.sqrt(np.mean((y_true - y_pred) ** 2))\n\ndef compute_prd(y_true, y_pred):\n    return 100 * np.sum((y_true - y_pred) ** 2) / np.sum(y_true ** 2)\n\ndef compute_pearson(y_true, y_pred):\n    if np.std(y_true) == 0 or np.std(y_pred) == 0:\n        return 0.0\n    return np.corrcoef(y_true, y_pred)[0, 1]\n\ndef compute_snr(y_true, y_pred):\n    signal_power = np.mean(y_true ** 2)\n    noise_power = np.mean((y_true - y_pred) ** 2)\n    if noise_power == 0:\n        return np.inf\n    return 10 * np.log10(signal_power / noise_power)\n\ndef compute_mae(y_true, y_pred):\n    return np.mean(np.abs(y_true - y_pred))\n\ndef compute_dtw(y_true, y_pred):\n    # Use fastdtw if available, else fallback to simple DTW\n    try:\n        from fastdtw import fastdtw\n        dist, _ = fastdtw(y_true, y_pred)\n        return dist\n    except ImportError:\n        # Simple DTW (not optimal for long signals)\n        from scipy.spatial.distance import cdist\n        n, m = len(y_true), len(y_pred)\n        dtw_matrix = np.zeros((n+1, m+1)) + np.inf\n        dtw_matrix[0, 0] = 0\n        for i in range(1, n+1):\n            for j in range(1, m+1):\n                cost = abs(y_true[i-1] - y_pred[j-1])\n                dtw_matrix[i, j] = cost + min(dtw_matrix[i-1, j], dtw_matrix[i, j-1], dtw_matrix[i-1, j-1])\n        return dtw_matrix[n, m]\n\n# 3. Clinical parameter extraction (QRS width, RR interval)\ndef extract_clinical_params(signal, sampling_rate=500):\n    # Use neurokit2 to extract R-peaks and QRS width\n    try:\n        cleaned = nk.ecg_clean(signal, sampling_rate=sampling_rate)\n        _, rpeaks = nk.ecg_peaks(cleaned, sampling_rate=sampling_rate)\n        _, waves = nk.ecg_delineate(cleaned, rpeaks, sampling_rate=sampling_rate, method=\"dwt\")\n        # QRS width: mean duration between Q and S for all beats\n        qrs_widths = []\n        for q, s in zip(waves['ECG_Q_Peaks'], waves['ECG_S_Peaks']):\n            if not np.isnan(q) and not np.isnan(s):\n                qrs_widths.append(abs(s - q) / sampling_rate)\n        mean_qrs_width = np.mean(qrs_widths) if qrs_widths else np.nan\n        # RR intervals: mean interval between R-peaks\n        r_locs = rpeaks['ECG_R_Peaks']\n        rr_intervals = np.diff(r_locs) / sampling_rate if len(r_locs) > 1 else []\n        mean_rr = np.mean(rr_intervals) if len(rr_intervals) > 0 else np.nan\n        return mean_qrs_width, mean_rr, r_locs\n    except Exception as e:\n        return np.nan, np.nan, []\n\n# 4. Macro F-measure for classification (if available)\ndef compute_macro_f1(y_true, y_pred, n_classes=2):\n    # y_true, y_pred: 1D arrays of class labels\n    return f1_score(y_true, y_pred, average='macro')\n\n# 5. Evaluate all pairs and collect metrics\nprint(\"Evaluating predictions...\")\n\nmetrics = {\n    \"file\": [],\n    \"rmse\": [],\n    \"prd\": [],\n    \"pearson\": [],\n    \"snr\": [],\n    \"mae\": [],\n    \"dtw\": [],\n    \"qrs_width_gt\": [],\n    \"qrs_width_pred\": [],\n    \"rr_interval_gt\": [],\n    \"rr_interval_pred\": [],\n    \"qrs_width_diff\": [],\n    \"rr_interval_diff\": [],\n    \"macro_f1\": [],\n}\n\nfor pred_path, gt_path in pairs:\n    # Load signals\n    pred_df = pd.read_csv(pred_path)\n    gt_df = pd.read_csv(gt_path)\n    # Use first column if multi-lead\n    pred_sig = pred_df.iloc[:, 0].values\n    gt_sig = gt_df.iloc[:, 0].values\n    # Truncate to min length\n    min_len = min(len(pred_sig), len(gt_sig))\n    pred_sig = pred_sig[:min_len]\n    gt_sig = gt_sig[:min_len]\n    # Compute metrics\n    rmse = compute_rmse(gt_sig, pred_sig)\n    prd = compute_prd(gt_sig, pred_sig)\n    pearson = compute_pearson(gt_sig, pred_sig)\n    snr = compute_snr(gt_sig, pred_sig)\n    mae = compute_mae(gt_sig, pred_sig)\n    dtw = compute_dtw(gt_sig, pred_sig)\n    # Clinical parameters\n    qrs_gt, rr_gt, rpeaks_gt = extract_clinical_params(gt_sig)\n    qrs_pred, rr_pred, rpeaks_pred = extract_clinical_params(pred_sig)\n    qrs_diff = np.abs(qrs_gt - qrs_pred) if not np.isnan(qrs_gt) and not np.isnan(qrs_pred) else np.nan\n    rr_diff = np.abs(rr_gt - rr_pred) if not np.isnan(rr_gt) and not np.isnan(rr_pred) else np.nan\n    # Macro F1 for R-peak detection (binary classification: peak vs. non-peak)\n    y_true = np.zeros(min_len, dtype=int)\n    y_pred = np.zeros(min_len, dtype=int)\n    y_true[rpeaks_gt] = 1 if len(rpeaks_gt) > 0 else 0\n    y_pred[rpeaks_pred] = 1 if len(rpeaks_pred) > 0 else 0\n    macro_f1 = compute_macro_f1(y_true, y_pred)\n    # Store results\n    metrics[\"file\"].append(os.path.basename(gt_path))\n    metrics[\"rmse\"].append(rmse)\n    metrics[\"prd\"].append(prd)\n    metrics[\"pearson\"].append(pearson)\n    metrics[\"snr\"].append(snr)\n    metrics[\"mae\"].append(mae)\n    metrics[\"dtw\"].append(dtw)\n    metrics[\"qrs_width_gt\"].append(qrs_gt)\n    metrics[\"qrs_width_pred\"].append(qrs_pred)\n    metrics[\"rr_interval_gt\"].append(rr_gt)\n    metrics[\"rr_interval_pred\"].append(rr_pred)\n    metrics[\"qrs_width_diff\"].append(qrs_diff)\n    metrics[\"rr_interval_diff\"].append(rr_diff)\n    metrics[\"macro_f1\"].append(macro_f1)\n\nprint(\"Evaluation complete.\")\n\n# 6. Display metrics summary\nmetrics_df = pd.DataFrame(metrics)\nprint(\"Summary of evaluation metrics:\")\ndisplay(metrics_df.describe())\n\n# 7. Visualize predicted vs. true signals for a few examples\nprint(\"Visualizing predicted vs. true signals...\")\n\nnum_examples = min(5, len(pairs))\nfor i in range(num_examples):\n    pred_path, gt_path = pairs[i]\n    pred_df = pd.read_csv(pred_path)\n    gt_df = pd.read_csv(gt_path)\n    pred_sig = pred_df.iloc[:, 0].values\n    gt_sig = gt_df.iloc[:, 0].values\n    min_len = min(len(pred_sig), len(gt_sig))\n    pred_sig = pred_sig[:min_len]\n    gt_sig = gt_sig[:min_len]\n    plt.figure(figsize=(12, 4))\n    plt.plot(gt_sig, label=\"Ground Truth\", linewidth=2)\n    plt.plot(pred_sig, label=\"Prediction\", alpha=0.7)\n    plt.title(f\"Predicted vs. Ground Truth Signal: {os.path.basename(gt_path)}\")\n    plt.xlabel(\"Sample Index\")\n    plt.ylabel(\"Amplitude\")\n    plt.legend()\n    plt.tight_layout()\n    plt.show()\n\n# 8. Visualize clinical parameter recovery\nprint(\"Visualizing clinical parameter recovery (QRS width, RR interval)...\")\n\nplt.figure(figsize=(10, 4))\nplt.subplot(1, 2, 1)\nplt.scatter(metrics_df[\"qrs_width_gt\"], metrics_df[\"qrs_width_pred\"], alpha=0.7)\nplt.plot([metrics_df[\"qrs_width_gt\"].min(), metrics_df[\"qrs_width_gt\"].max()],\n         [metrics_df[\"qrs_width_gt\"].min(), metrics_df[\"qrs_width_gt\"].max()], 'r--')\nplt.xlabel(\"QRS Width (Ground Truth) [s]\")\nplt.ylabel(\"QRS Width (Prediction) [s]\")\nplt.title(\"QRS Width Recovery\")\n\nplt.subplot(1, 2, 2)\nplt.scatter(metrics_df[\"rr_interval_gt\"], metrics_df[\"rr_interval_pred\"], alpha=0.7)\nplt.plot([metrics_df[\"rr_interval_gt\"].min(), metrics_df[\"rr_interval_gt\"].max()],\n         [metrics_df[\"rr_interval_gt\"].min(), metrics_df[\"rr_interval_gt\"].max()], 'r--')\nplt.xlabel(\"RR Interval (Ground Truth) [s]\")\nplt.ylabel(\"RR Interval (Prediction) [s]\")\nplt.title(\"RR Interval Recovery\")\nplt.tight_layout()\nplt.show()\n\n# 9. Visualize macro F1 distribution\nprint(\"Visualizing Macro F1 for R-peak detection...\")\nplt.figure(figsize=(6,4))\nplt.hist(metrics_df[\"macro_f1\"].dropna(), bins=20, alpha=0.8)\nplt.xlabel(\"Macro F1 Score\")\nplt.ylabel(\"Frequency\")\nplt.title(\"Macro F1 Distribution (R-peak Detection)\")\nplt.tight_layout()\nplt.show()\n\nprint(\"All evaluation steps completed successfully.\")","outputs":[],"cell_number":7,"version":1,"status":"generated","created_at":"2025-11-16T18:59:39.202211+00:00","metadata":{},"execution_count":null},{"cell_type":"markdown","source":"## Submission & Results\n\nFormat predictions for Kaggle submission and summarize final results.","metadata":{}},{"cell_type":"code","source":"# ⚠️ ALEXANDRIA MARKER - DO NOT DELETE (used for syncing outputs from Kaggle)\nprint(\"===ALEXANDRIA_CELL_8_START===\")\n\nprint(\"===ALEXANDRIA_CELL_8_START===\")\n\n# Title: Submission & Results\n# Description: Format predictions for Kaggle submission and summarize final results.\n\nimport os\nimport pandas as pd\nimport numpy as np\nfrom glob import glob\n\n# Paths from previous cells\nFEATURES_SIGNALS_DIR = \"/kaggle/working/features_signals\"\nSUBMISSION_PATH = \"/kaggle/working/submission.csv\"\nDATA_PATH = \"/kaggle/input/physionet-ecg-image-digitization\"\n\nprint(\"Preparing submission file for Kaggle competition...\")\n\n# 1. Locate prediction files (assume CSVs with prefix 'pred_' as in evaluation cell)\npred_csv_files = sorted(glob(os.path.join(FEATURES_SIGNALS_DIR, \"pred_*.csv\")))\nif not pred_csv_files:\n    raise RuntimeError(\"No prediction files found in FEATURES_SIGNALS_DIR. Cannot create submission.\")\n\n# 2. Determine submission format\n# If sample_submission.csv exists in DATA_PATH, use its columns and order\nsample_sub_path = os.path.join(DATA_PATH, \"sample_submission.csv\")\nif os.path.exists(sample_sub_path):\n    sample_sub = pd.read_csv(sample_sub_path)\n    submission_cols = sample_sub.columns.tolist()\n    print(f\"Using columns from sample_submission.csv: {submission_cols}\")\nelse:\n    # Otherwise, infer: use 'Id' as first column, then one column per signal (e.g., 'value' or 'lead_1', etc.)\n    # We'll use 'Id' and 'value' for single-lead, or 'Id' plus all columns for multi-lead\n    # Get columns from first prediction file\n    first_pred = pd.read_csv(pred_csv_files)\n    pred_cols = first_pred.columns.tolist()\n    if len(pred_cols) == 1:\n        submission_cols = [\"Id\", \"value\"]\n    else:\n        submission_cols = [\"Id\"] + pred_cols\n    print(f\"Inferred submission columns: {submission_cols}\")\n\n# 3. Build submission DataFrame\nsubmission_rows = []\nfor pred_path in pred_csv_files:\n    # Extract unique ID from filename (remove pred_ prefix and .csv suffix)\n    base = os.path.basename(pred_path)\n    if base.startswith(\"pred_\"):\n        rec_id = base[5:-4]\n    else:\n        rec_id = os.path.splitext(base)\n    pred_df = pd.read_csv(pred_path)\n    # For single-lead: flatten to one row per file (mean or first value)\n    if len(submission_cols) == 2:\n        # Use mean value as prediction (or first value if required)\n        value = float(pred_df.iloc[:, 0].mean())\n        submission_rows.append({\"Id\": rec_id, \"value\": value})\n    else:\n        # Multi-lead: flatten each column (mean or first value)\n        row = {\"Id\": rec_id}\n        for col in submission_cols[1:]:\n            if col in pred_df.columns:\n                row[col] = float(pred_df[col].mean())\n            else:\n                row[col] = np.nan\n        submission_rows.append(row)\n\nsubmission_df = pd.DataFrame(submission_rows)\nsubmission_df = submission_df[submission_cols]  # Ensure correct column order\n\n# 4. Save submission file\nsubmission_df.to_csv(SUBMISSION_PATH, index=False)\nprint(f\"Submission file saved to {SUBMISSION_PATH}\")\nprint(f\"Submission shape: {submission_df.shape}\")\nprint(\"First few rows of submission:\")\ndisplay(submission_df.head())\n\n# 5. Optionally, display summary statistics of predictions\nprint(\"Summary statistics of predictions:\")\ndisplay(submission_df.describe())\n\n# 6. Show a few sample outputs\nprint(\"Sample submission outputs:\")\nfor i in range(min(3, len(submission_df))):\n    print(submission_df.iloc[i].to_dict())\n\n# 7. Instructions for leaderboard submission (Kaggle notebook context)\nprint(\"\\nTo submit your predictions to the leaderboard:\")\nprint(\"1. Go to the 'Output' tab on the right panel of your notebook.\")\nprint(\"2. Find 'submission.csv' and click the 'Submit' button.\")\nprint(\"3. Wait for the leaderboard evaluation to complete.\")\n\nprint(\"Submission & results cell completed successfully.\")","outputs":[],"cell_number":8,"version":1,"status":"generated","created_at":"2025-11-16T18:59:52.013761+00:00","metadata":{},"execution_count":null}],"nbformat":4,"nbformat_minor":4}