{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.10.0"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":97984,"databundleVersionId":14096757,"sourceType":"competition"}],"dockerImageVersionId":31192,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# PhysioNet ECG Baseline - Complete Submission Format Guide\n\n## Overview\nThis notebook provides:\n- ✅ **Correct submission format** (learned through trial and error)\n- ✅ **Working baseline** (Score: 0.09, Rank: ~647)\n- ✅ **Common mistakes** and how to avoid them\n- ✅ **Future improvement** directions\n\n## Key Learnings\n1. **Submission file**: Must be `submission.csv` (NOT .parquet!)\n2. **ID format**: `{ecg_id}_{sample_index}_{lead}` (order matters!)\n3. **Column names**: `['id', 'value']` (NOT 'voltage'!)\n4. **Data type**: value must be numeric (int64 or float64)\n\n---\n\n## Table of Contents\n1. Understanding the Submission Format\n2. Common Mistakes to Avoid\n3. Baseline Implementation\n4. Results and Next Steps","metadata":{}},{"cell_type":"markdown","source":"# 1. Understanding the Submission Format\n\n## Critical Discovery: sample_submission.parquet\n\nThe competition provides `sample_submission.parquet` (NOT .csv) in the data directory.\nHowever, **your submission must be a CSV file**!\n\n### Why This Matters\n- Many competitions provide `sample_submission.csv`\n- This competition only has `sample_submission.parquet`\n- You must read the parquet file to understand the format\n- Then generate a CSV file for submission","metadata":{}},{"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nfrom pathlib import Path\n\nDATA_DIR = Path('/kaggle/input/physionet-ecg-image-digitization')\nOUTPUT_DIR = Path('/kaggle/working')\n\nprint('=== Environment Setup ===')\nprint(f'Data directory: {DATA_DIR}')\nprint(f'Output directory: {OUTPUT_DIR}')","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Step 1: Read sample_submission.parquet\n\n**IMPORTANT**: Read this file to understand the exact format required!","metadata":{}},{"cell_type":"code","source":"# Read sample submission to understand format\nsample = pd.read_parquet(DATA_DIR / 'sample_submission.parquet')\n\nprint('=== SAMPLE SUBMISSION FORMAT ===')\nprint(f'Shape: {sample.shape}')\nprint(f'Columns: {sample.columns.tolist()}')\nprint(f'Data types:\\n{sample.dtypes}')\nprint(f'\\nFirst 30 rows:')\nprint(sample.head(30))\nprint(f'\\nLast 10 rows:')\nprint(sample.tail(10))","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Key Observations from Sample Submission\n\n1. **Shape**: (75000, 2) - 75,000 rows, 2 columns\n2. **Columns**: `['id', 'value']` - NOT 'voltage' or 'prediction'!\n3. **ID Format**: `{ecg_id}_{sample_index}_{lead}`\n   - Example: `1053922973_0_I` means:\n     - ECG ID: 1053922973\n     - Sample index: 0 (first sample)\n     - Lead: I\n4. **Data Type**: value is int64 (can also be float64)","metadata":{}},{"cell_type":"markdown","source":"## Step 2: Analyze ID Format Pattern","metadata":{}},{"cell_type":"code","source":"# Analyze ID pattern\nprint('=== ID FORMAT ANALYSIS ===')\nsample_ids = sample['id'].head(20)\nfor i, id_val in enumerate(sample_ids):\n    parts = id_val.split('_')\n    print(f'{i:2d}: {id_val:25s} -> ECG:{parts[0]:10s} Sample:{parts[1]:4s} Lead:{parts[2]}')","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 2. Common Mistakes to Avoid\n\n## Mistake 1: Wrong Column Name ❌\n```python\n# WRONG - will cause submission error\nsubmission = pd.DataFrame({'id': ids, 'voltage': values})\n\n# CORRECT\nsubmission = pd.DataFrame({'id': ids, 'value': values})\n```\n\n## Mistake 2: Wrong ID Format ❌\n```python\n# WRONG - incorrect order\nsubmission_id = f\"{ecg_id}_{lead}_{sample_index}\"  # 1053922973_I_0\n\n# CORRECT - sample_index comes before lead\nsubmission_id = f\"{ecg_id}_{sample_index}_{lead}\"  # 1053922973_0_I\n```\n\n## Mistake 3: Wrong File Format ❌\n```python\n# WRONG - competition wants CSV, not parquet\nsubmission.to_parquet('submission.parquet', index=False)\n\n# CORRECT\nsubmission.to_csv('submission.csv', index=False)\n```\n\n## Mistake 4: Wrong Row Count ❌\nThe total number of rows must equal the sum of `number_of_rows` in test.csv.\nFor this test set: **75,000 rows exactly**.","metadata":{}},{"cell_type":"markdown","source":"# 3. Baseline Implementation\n\n## Strategy\n- **Goal**: Get a valid submission first, optimize later\n- **Method**: Zero baseline (all predictions = 0)\n- **Why**: Verifies format is correct before investing in complex algorithms\n- **Result**: Score 0.09, Rank ~647 (proof it works!)","metadata":{}},{"cell_type":"code","source":"# Load test metadata\ntest_meta = pd.read_csv(DATA_DIR / 'test.csv')\n\nprint('=== TEST METADATA ===')\nprint(test_meta.head(15))\nprint(f'\\nShape: {test_meta.shape}')\nprint(f'Unique ECG IDs: {test_meta[\"id\"].nunique()}')\nprint(f'Leads per ECG: {len(test_meta[\"lead\"].unique())}')\nprint(f'Total samples needed: {test_meta[\"number_of_rows\"].sum():,}')","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Generate Baseline Predictions\n\nFor each (ECG, Lead) combination:\n1. Get the number of samples from test.csv\n2. Generate predictions (zeros for baseline)\n3. Create IDs in correct format: `{ecg_id}_{sample_index}_{lead}`\n4. Build submission dataframe","metadata":{}},{"cell_type":"code","source":"print('=' * 80)\nprint('Generating submission.csv...')\nprint('=' * 80)\n\n# Create submission rows\nsubmission_rows = []\n\nfor idx, row in test_meta.iterrows():\n    ecg_id = row['id']\n    lead = row['lead']\n    n_rows = row['number_of_rows']\n    \n    # Baseline: predict zeros\n    predictions = np.zeros(n_rows, dtype=np.int64)\n    \n    # Create submission rows with CORRECT format\n    # CRITICAL: Format is {ecg_id}_{sample_index}_{lead}\n    for i, pred_value in enumerate(predictions):\n        submission_id = f\"{ecg_id}_{i}_{lead}\"\n        submission_rows.append({\n            'id': submission_id,\n            'value': int(pred_value)\n        })\n    \n    if (idx + 1) % 5 == 0:\n        print(f'Processed {idx + 1}/{len(test_meta)} test entries...')\n\n# Create submission DataFrame\nsubmission = pd.DataFrame(submission_rows)\n\n# Ensure correct data types\nsubmission['value'] = submission['value'].astype(np.int64)\n\nprint(f'\\n✅ Submission created!')\nprint(f'Shape: {submission.shape}')\nprint(f'Columns: {submission.columns.tolist()}')","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Verify Submission Format\n\n**CRITICAL**: Always verify your submission matches the sample format!","metadata":{}},{"cell_type":"code","source":"print('=' * 80)\nprint('SUBMISSION VERIFICATION')\nprint('=' * 80)\n\n# Check 1: Shape\nprint(f'\\n[1] Shape Check')\nprint(f'    Expected: {sample.shape}')\nprint(f'    Actual:   {submission.shape}')\nif submission.shape == sample.shape:\n    print('    ✅ PASS')\nelse:\n    print('    ❌ FAIL')\n\n# Check 2: Columns\nprint(f'\\n[2] Column Check')\nprint(f'    Expected: {sample.columns.tolist()}')\nprint(f'    Actual:   {submission.columns.tolist()}')\nif submission.columns.tolist() == sample.columns.tolist():\n    print('    ✅ PASS')\nelse:\n    print('    ❌ FAIL')\n\n# Check 3: Data Types\nprint(f'\\n[3] Data Type Check')\nprint(f'    Expected:\\n{sample.dtypes}')\nprint(f'    Actual:\\n{submission.dtypes}')\nif submission.dtypes['value'] in [np.int64, np.float64]:\n    print('    ✅ PASS (value is numeric)')\nelse:\n    print('    ❌ FAIL (value must be numeric)')\n\n# Check 4: No Missing Values\nprint(f'\\n[4] Missing Value Check')\nmissing = submission.isnull().sum().sum()\nprint(f'    Missing values: {missing}')\nif missing == 0:\n    print('    ✅ PASS')\nelse:\n    print('    ❌ FAIL')\n\n# Check 5: ID Format\nprint(f'\\n[5] ID Format Check')\nprint(f'    Sample IDs:')\nfor i in range(5):\n    print(f'      {sample[\"id\"].iloc[i]}')\nprint(f'    Submission IDs:')\nfor i in range(5):\n    print(f'      {submission[\"id\"].iloc[i]}')\n    \nsample_parts = len(sample['id'].iloc[0].split('_'))\nsubmission_parts = len(submission['id'].iloc[0].split('_'))\nif sample_parts == submission_parts == 3:\n    print('    ✅ PASS (ID has 3 parts: ecg_id, sample_index, lead)')\nelse:\n    print(f'    ❌ FAIL (Expected 3 parts, got {submission_parts})')","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Preview Submission","metadata":{}},{"cell_type":"code","source":"print('=== SUBMISSION PREVIEW ===')\nprint('\\nFirst 30 rows:')\nprint(submission.head(30))\n\nprint('\\nLast 20 rows:')\nprint(submission.tail(20))\n\nprint('\\nValue statistics:')\nprint(submission['value'].describe())","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Save Submission File\n\n**IMPORTANT**: File must be named exactly `submission.csv`","metadata":{}},{"cell_type":"code","source":"# Save submission\nsubmission_path = OUTPUT_DIR / 'submission.csv'\nsubmission.to_csv(submission_path, index=False)\n\nprint('=' * 80)\nprint('✅ SUBMISSION SAVED!')\nprint('=' * 80)\nprint(f'File: {submission_path}')\nprint(f'Size: {submission_path.stat().st_size / (1024*1024):.2f} MB')\nprint(f'Rows: {len(submission):,}')\n\n# Verify file is readable\nif submission_path.exists():\n    test_load = pd.read_csv(submission_path)\n    print(f'\\n✅ File verified (loaded {len(test_load):,} rows)')\n    print(f'   Columns: {test_load.columns.tolist()}')\n    print('\\n🚀 READY FOR SUBMISSION!')\nelse:\n    print('\\n❌ ERROR: File not found!')","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 4. Results and Next Steps\n\n## Baseline Results\n- **Strategy**: Zero baseline (all predictions = 0)\n- **Score**: 0.09\n- **Rank**: ~647 (out of all participants)\n- **Status**: ✅ Valid submission, format verified\n\n## Why This Baseline Is Valuable\n1. **Confirms submission format** - No more format errors!\n2. **Establishes minimum score** - Any real algorithm should beat 0.09\n3. **Saves submission attempts** - Format issues waste precious submissions\n4. **Provides working template** - Easy to improve upon\n\n## Next Steps for Improvement\n\n### Short-term (Easy wins)\n1. **Use mean/median from training data** instead of zeros\n2. **Lead-specific baselines** - Different leads have different ranges\n3. **Simple heuristics** - Use typical ECG patterns\n\n### Medium-term (Moderate effort)\n1. **Image processing**:\n   - Extract grid lines\n   - Detect ECG traces\n   - Convert pixels to voltage values\n2. **Calibration**:\n   - Use calibration pulses in images\n   - Scale pixel values to mV\n3. **Signal processing**:\n   - Denoise extracted signals\n   - Interpolate missing points\n\n### Long-term (Advanced)\n1. **Deep learning**:\n   - U-Net for trace segmentation\n   - CNN for signal extraction\n   - Seq2Seq models for time-series\n2. **Ensemble methods**:\n   - Combine multiple extraction methods\n   - Weighted averaging based on confidence\n3. **Post-processing**:\n   - Enforce physiological constraints\n   - Smooth transitions between leads\n\n## Key Takeaways\n\n### Format Requirements (Don't forget!)\n- ✅ File: `submission.csv` (not .parquet)\n- ✅ Columns: `['id', 'value']` (not 'voltage')\n- ✅ ID format: `{ecg_id}_{sample_index}_{lead}` (order matters!)\n- ✅ Rows: Must match sum of `number_of_rows` in test.csv\n\n### Debugging Tips\n1. Always read `sample_submission.parquet` first\n2. Compare your submission shape/columns with sample\n3. Check first/last rows for ID format issues\n4. Verify no missing values\n5. Test with baseline before complex algorithms\n\n## References\n- [Competition Page](https://www.kaggle.com/competitions/physionet-ecg-image-digitization)\n- [George B. Moody PhysioNet Challenge 2024](https://moody-challenge.physionet.org/2024/)\n- [Winning Solution (GitHub)](https://github.com/felixkrones/ECG-Digitiser)\n\n---\n\n## Acknowledgments\n\nGood luck with your submissions! 🚀","metadata":{}}]}