{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.12.13"},"papermill":{"default_parameters":{},"duration":88.037565,"end_time":"2026-08-31T20:44:56.671924+00:00","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2026-08-31T20:43:28.634359+00:00","version":"2.7.0"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceType":"competition","sourceId":106809}],"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"eade717c","cell_type":"markdown","source":"# Brain-to-Text '25 — Notebook 1 / 3: Data, Checks, Preprocessing, Splits\n\n**Pipeline**: this notebook -> (saves `/kaggle/working/nb1_output/`) -> attach as a Kaggle\nDataset -> **Notebook 2** (training) reads it, and it's also read directly in **Notebook 3**\n(visualization) alongside Notebook 2's results.\n\n**What this notebook does, in order:**\n1. Environment + path setup (auto-detects real Kaggle competition data vs. this notebook's\n   synthetic fallback, so it's fully runnable and checkable before you have data access)\n2. Dataset schema (electrode/region layout, phoneme vocabulary) -- from the official\n   Dataset_Details / GitHub repo documentation\n3. Session scan + integrity checks (every train/val trial must carry complete labels;\n   malformed sessions are flagged, never silently dropped)\n4. Corpus and speaking-session distribution analysis\n5. Before/after-preprocessing feature statistics\n6. Two splits: stratified dev split (model selection) and chronological split (continual\n   learning) -- kept distinct from Kaggle's own held-out test partition\n7. Save manifest + stats for Notebooks 2 and 3","metadata":{"papermill":{"duration":0.003377,"end_time":"2026-08-31T20:43:31.108328+00:00","exception":false,"start_time":"2026-08-31T20:43:31.104951+00:00","status":"completed"},"tags":[]}},{"id":"8972f8a7-b47d-4b89-bbd2-a984679d0490","cell_type":"markdown","source":"## Fix log (applied by Claude, see chat for full diagnosis)\n\n**Bug: `corpus` was `\"unknown\"` for 100% of real trials.** `scan_dataset` only ever read a\nper-trial HDF5 attribute (`g.attrs[\"corpus\"]`) that your *synthetic* generator populates but\nthe real competition files do not set -- corpus is block/session-level metadata that lives\nonly in `t15_copyTaskData_description.csv`. `CORPUS_CSV` already pointed at that file, but\nnothing ever opened it, so the \"stratify dev split by corpus\" step silently collapsed into\none unstratified bucket.\n\n**Fix**: added `load_corpus_lookup()`, which reads that CSV and joins corpus back onto each\ntrial by session (falling back to the HDF5 attribute first, in case it's ever present, then\nto \"unknown\" only if truly nothing matches). `scan_dataset` now prints how many records were\nresolved via each source, so this stays visible instead of silently reverting. If your real\nCSV's column names differ from `session`/`corpus`, the warning it prints will tell you exactly\nwhat it found so you can adjust the two column-name lines.\n\n**Update**: `REAL_CORPUS_CSV`'s exact nested path (`t15_copyTask_neuralData/data/...`) was\nalready in the notebook before any of this -- it was a guess (by whoever wrote the original\nscaffold) about Kaggle's exact folder layout, not something invented for this fix. Since that\nguess can't be verified from outside Kaggle, `load_corpus_lookup` now tries it first, then\n`find_corpus_csv` searches outward (`KAGGLE_COMPETITION_ROOT`, then all of `/kaggle/input`) for\na plausibly-matching CSV before giving up -- and if it still can't find one, it prints every\n`.csv` file that *does* exist nearby so you can see the real layout instead of guessing again.\nEither way this cannot make things worse than before: on failure it falls back to \"unknown\",\nexactly like the original code.\n","metadata":{}},{"id":"edcd6c95","cell_type":"code","source":"import os, sys, json, csv, random\nfrom pathlib import Path\nfrom collections import defaultdict, Counter\nfrom dataclasses import dataclass, asdict\n\nimport numpy as np\nimport h5py\n\nprint(\"Python OK. :\")\ntry:\n    import torch\n    print(\"torch:\", torch.__version__, \"| CUDA available:\", torch.cuda.is_available(),\n          \"| device count:\", torch.cuda.device_count() if torch.cuda.is_available() else 0)\nexcept ImportError:\n    print(\"torch not installed in this environment (fine -- this notebook doesn't need it)\")","metadata":{"execution":{"iopub.execute_input":"2026-08-31T20:43:31.114984Z","iopub.status.busy":"2026-08-31T20:43:31.114219Z","iopub.status.idle":"2026-08-31T20:43:36.258233Z","shell.execute_reply":"2026-08-31T20:43:36.257241Z"},"papermill":{"duration":5.149265,"end_time":"2026-08-31T20:43:36.260127+00:00","exception":false,"start_time":"2026-08-31T20:43:31.110862+00:00","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"c6ca0b58","cell_type":"markdown","source":"## 1. Paths\n\nOn Kaggle, attach the competition as a data source and this resolves automatically. Outside\nKaggle (e.g. this sandbox), `USE_SYNTHETIC=True` builds a small dataset matching the exact\ndocumented schema, so every check and split below is exercised for real either way.","metadata":{"papermill":{"duration":0.002743,"end_time":"2026-08-31T20:43:36.26574+00:00","exception":false,"start_time":"2026-08-31T20:43:36.262997+00:00","status":"completed"},"tags":[]}},{"id":"33a01ded","cell_type":"code","source":"KAGGLE_COMPETITION_ROOT = Path(\"/kaggle/input/competitions/brain-to-text-25\")\nREAL_HDF5_ROOT = KAGGLE_COMPETITION_ROOT / \"t15_copyTask_neuralData\" / \"hdf5_data_final\"\nREAL_CORPUS_CSV = KAGGLE_COMPETITION_ROOT / \"t15_copyTask_neuralData\" / \"data\" / \"t15_copyTaskData_description.csv\"\n\nOUTPUT_DIR = Path(\"/kaggle/working/nb1_output\") if Path(\"/kaggle/working\").exists() else Path(\"./nb1_output\")\nOUTPUT_DIR.mkdir(parents=True, exist_ok=True)\n\nUSE_SYNTHETIC = not REAL_HDF5_ROOT.exists()\nif USE_SYNTHETIC:\n    print(f\"Real competition data not found at {REAL_HDF5_ROOT} -- building a small synthetic \"\n          f\"dataset with the identical schema so this notebook is fully runnable/checkable now.\")\n    SYNTH_ROOT = Path(\"./synthetic_data\")\nelse:\n    print(f\"Found real competition data at {REAL_HDF5_ROOT}\")\n\nprint(\"Output directory:\", OUTPUT_DIR.resolve())","metadata":{"execution":{"iopub.execute_input":"2026-08-31T20:43:36.272848Z","iopub.status.busy":"2026-08-31T20:43:36.271978Z","iopub.status.idle":"2026-08-31T20:43:36.287099Z","shell.execute_reply":"2026-08-31T20:43:36.286392Z"},"papermill":{"duration":0.020223,"end_time":"2026-08-31T20:43:36.288562+00:00","exception":false,"start_time":"2026-08-31T20:43:36.268339+00:00","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"908b953d","cell_type":"markdown","source":"## 2. Dataset schema\n\nTranscribed directly from `Dataset_Details.pdf` and cross-checked against the official\n`Neuroprosthetics-Lab/nejm-brain-to-text` GitHub repo (both agree). Getting this exactly right\nmatters: an off-by-one here silently mislabels which cortical region a channel belongs to.","metadata":{"papermill":{"duration":0.002427,"end_time":"2026-08-31T20:43:36.293509+00:00","exception":false,"start_time":"2026-08-31T20:43:36.291082+00:00","status":"completed"},"tags":[]}},{"id":"7a090543","cell_type":"code","source":"LOGIT_TO_PHONEME = [\n    \"BLANK\", \"AA\", \"AE\", \"AH\", \"AO\", \"AW\", \"AY\", \"B\", \"CH\", \"D\", \"DH\",\n    \"EH\", \"ER\", \"EY\", \"F\", \"G\", \"HH\", \"IH\", \"IY\", \"JH\", \"K\", \"L\", \"M\",\n    \"N\", \"NG\", \"OW\", \"OY\", \"P\", \"R\", \"S\", \"SH\", \"T\", \"TH\", \"UH\", \"UW\",\n    \"V\", \"W\", \"Y\", \"Z\", \"ZH\", \"|\",  # \"|\" = inter-word silence\n]\nN_PHONEME_CLASSES = len(LOGIT_TO_PHONEME)\nCTC_BLANK_ID, SILENCE_ID = 0, LOGIT_TO_PHONEME.index(\"|\")\n\nREGION_NAMES = [\"ventral_6v\", \"area_4\", \"55b\", \"dorsal_6v\"]\nN_REGIONS = len(REGION_NAMES)\nN_ELECTRODES_PER_REGION = 64\nN_ELECTRODES = N_REGIONS * N_ELECTRODES_PER_REGION      # 256\nN_RAW_FEATURES = 2 * N_ELECTRODES                          # 512: [0:256]=threshold crossings, [256:512]=spike band power\nT15_CORPORA = [\"switchboard\", \"openwebtext2\", \"fifty_word\", \"frequent_word\", \"random_word\"]\n\nprint(f\"{N_PHONEME_CLASSES} phoneme classes, {N_REGIONS} regions x {N_ELECTRODES_PER_REGION} \"\n      f\"electrodes = {N_ELECTRODES} electrodes, {N_RAW_FEATURES} raw features per timestep\")\nassert N_PHONEME_CLASSES == 41 and N_ELECTRODES == 256 and N_RAW_FEATURES == 512","metadata":{"execution":{"iopub.execute_input":"2026-08-31T20:43:36.299806Z","iopub.status.busy":"2026-08-31T20:43:36.299244Z","iopub.status.idle":"2026-08-31T20:43:36.305394Z","shell.execute_reply":"2026-08-31T20:43:36.304603Z"},"papermill":{"duration":0.01086,"end_time":"2026-08-31T20:43:36.306892+00:00","exception":false,"start_time":"2026-08-31T20:43:36.296032+00:00","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"eda5c9b9","cell_type":"markdown","source":"## 3. Synthetic fallback data (skipped automatically when real data is present)\n\nMatches the real HDF5 structure exactly: same group/attribute/dataset names, same\ntrain/val/test convention (test carries no labels), same corpus-description CSV format.","metadata":{"papermill":{"duration":0.00242,"end_time":"2026-08-31T20:43:36.311849+00:00","exception":false,"start_time":"2026-08-31T20:43:36.309429+00:00","status":"completed"},"tags":[]}},{"id":"0ae54d6b","cell_type":"code","source":"_RNG_WORDS = [\"hello\", \"world\", \"please\", \"bring\", \"water\", \"computer\", \"family\", \"today\",\n              \"tomorrow\", \"speak\", \"listen\", \"thank\", \"you\", \"help\", \"time\", \"yes\", \"no\",\n              \"good\", \"morning\", \"night\", \"friend\", \"home\", \"walk\", \"read\", \"write\"]\n\n\ndef _fake_sentence(rng, min_words=3, max_words=9):\n    n_words = rng.integers(min_words, max_words + 1)\n    words = [rng.choice(_RNG_WORDS) for _ in range(n_words)]\n    sentence = \" \".join(words).capitalize() + \".\"\n    phon_ids = []\n    for w in words:\n        n_ph = max(1, len(w) // 2)\n        phon_ids.extend(rng.integers(1, N_PHONEME_CLASSES - 1, size=n_ph).tolist())\n        phon_ids.append(SILENCE_ID)\n    return sentence, phon_ids\n\n\ndef _fake_neural_features(rng, n_timesteps, session_drift):\n    base = rng.standard_normal((n_timesteps, N_RAW_FEATURES)).astype(np.float32) + session_drift\n    base[:, N_RAW_FEATURES // 2:] = np.abs(base[:, N_RAW_FEATURES // 2:])  # spike band power >= 0\n    return base\n\n\ndef build_synthetic_dataset(root: Path, n_sessions=8, seed=0, trials_per_split=(14, 5, 4)):\n    root = Path(root)\n    hdf5_root = root / \"hdf5_data_final\"\n    rng = np.random.default_rng(seed)\n    base_dates = [f\"2023.{m:02d}.{d:02d}\" for m in range(8, 13) for d in (1, 10, 20)][:n_sessions]\n    csv_rows = []\n    for i, date in enumerate(base_dates):\n        session_id = f\"t15.{date}\"\n        sdir = hdf5_root / session_id\n        sdir.mkdir(parents=True, exist_ok=True)\n        n_train, n_val, n_test = trials_per_split\n        corpus_mix = list(rng.choice(T15_CORPORA, size=rng.integers(2, 4), replace=False))\n        session_drift = 0.05 * i + rng.normal(0, 0.02)\n        for split, n_trials in [(\"train\", n_train), (\"val\", n_val), (\"test\", n_test)]:\n            with h5py.File(sdir / f\"data_{split}.hdf5\", \"w\") as f:\n                for t in range(n_trials):\n                    n_ts = int(rng.integers(80, 220))\n                    feats = _fake_neural_features(rng, n_ts, session_drift)\n                    g = f.create_group(f\"trial_{t:04d}\")\n                    g.create_dataset(\"input_features\", data=feats)\n                    g.attrs[\"n_time_steps\"] = n_ts\n                    g.attrs[\"session\"] = session_id\n                    g.attrs[\"block_num\"] = int(rng.integers(1, 6))\n                    g.attrs[\"trial_num\"] = t\n                    g.attrs[\"corpus\"] = str(rng.choice(corpus_mix))\n                    if split in (\"train\", \"val\"):\n                        sentence, phon_ids = _fake_sentence(rng)\n                        g.create_dataset(\"seq_class_ids\", data=np.array(phon_ids, dtype=np.int64))\n                        g.attrs[\"seq_len\"] = len(phon_ids)\n                        g.attrs[\"sentence_label\"] = sentence\n        csv_rows.append({\"session\": session_id, \"block_num\": 1,\n                          \"n_trials\": sum(trials_per_split), \"corpus\": \"|\".join(corpus_mix),\n                          \"split\": \"train\" if i < n_sessions - 2 else \"val\"})\n    with open(root / \"t15_copyTaskData_description.csv\", \"w\", newline=\"\") as f:\n        w = csv.DictWriter(f, fieldnames=[\"session\", \"block_num\", \"n_trials\", \"corpus\", \"split\"])\n        w.writeheader(); w.writerows(csv_rows)\n    return sorted(p.name for p in hdf5_root.iterdir() if p.is_dir())\n\n\nif USE_SYNTHETIC:\n    session_ids = build_synthetic_dataset(SYNTH_ROOT, n_sessions=8, seed=0)\n    HDF5_ROOT = SYNTH_ROOT / \"hdf5_data_final\"\n    CORPUS_CSV = SYNTH_ROOT / \"t15_copyTaskData_description.csv\"\n    print(f\"Built {len(session_ids)} synthetic sessions.\")\nelse:\n    HDF5_ROOT = REAL_HDF5_ROOT\n    CORPUS_CSV = REAL_CORPUS_CSV","metadata":{"execution":{"iopub.execute_input":"2026-08-31T20:43:36.318524Z","iopub.status.busy":"2026-08-31T20:43:36.317949Z","iopub.status.idle":"2026-08-31T20:43:36.331141Z","shell.execute_reply":"2026-08-31T20:43:36.330466Z"},"papermill":{"duration":0.018264,"end_time":"2026-08-31T20:43:36.332582+00:00","exception":false,"start_time":"2026-08-31T20:43:36.314318+00:00","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"2ae02850","cell_type":"markdown","source":"## 4. Session scan + integrity checks\n\nEvery session directory is scanned; every trial in every train/val file is checked for a\ncomplete `sentence_label` + `seq_class_ids` pair. Malformed sessions are **flagged**, never\nsilently dropped -- a silent drop would quietly shrink the training set without anyone\nnoticing until results looked strange much later.","metadata":{"papermill":{"duration":0.002623,"end_time":"2026-08-31T20:43:36.337815+00:00","exception":false,"start_time":"2026-08-31T20:43:36.335192+00:00","status":"completed"},"tags":[]}},{"id":"e0f1d018","cell_type":"code","source":"@dataclass\nclass TrialRecord:\n    session_id: str\n    split: str\n    path: str\n    key: str\n    corpus: str = \"unknown\"\n    n_time_steps: int = 0\n    seq_len: int = 0\n\n\ndef find_corpus_csv(primary_path: Path):\n    \"\"\"The exact nested folder structure Kaggle uses for a competition's data files is\n    something we can only guess at from outside Kaggle -- `REAL_CORPUS_CSV` above is\n    one reasonable guess (it matches the path the real HDF5 data was actually found at,\n    just swapping in this competition's documented CSV filename), but if your copy of\n    the dataset lays it out even slightly differently, that guess can be wrong. Rather\n    than hard-fail on one hardcoded path, try it first, then search outward -- first\n    under KAGGLE_COMPETITION_ROOT, then the whole /kaggle/input tree -- for a file that\n    looks like the right one, before giving up.\n    \"\"\"\n    if primary_path.exists():\n        return primary_path\n    for search_root in (KAGGLE_COMPETITION_ROOT, Path(\"/kaggle/input\")):\n        if not search_root.exists():\n            continue\n        candidates = sorted(search_root.rglob(\"*copyTaskData_description*.csv\"))\n        if not candidates:\n            candidates = sorted(\n                p for p in search_root.rglob(\"*.csv\")\n                if \"corpus\" in p.name.lower() or \"description\" in p.name.lower()\n            )\n        if candidates:\n            print(f\"  [corpus] '{primary_path}' not found; using '{candidates[0]}' \"\n                  f\"found by searching {search_root} instead.\")\n            return candidates[0]\n    return primary_path\n\n\ndef load_corpus_lookup(csv_path: Path):\n    \"\"\"Read the block-by-block corpus description CSV (session, block_num, n_trials,\n    corpus, split -- see the official B2T'25 data card) into a per-session lookup.\n\n    BUG FIX: the original code only ever looked at a per-trial HDF5 attribute\n    (`g.attrs[\"corpus\"]`). That attribute is populated by this notebook's own\n    *synthetic* data generator, but it is NOT present on the real competition\n    HDF5 files -- corpus is block/session-level metadata that lives only in this\n    CSV. On real data every single trial therefore fell back to the \"unknown\"\n    default, so `dev_test_split`'s \"stratify by corpus\" logic silently collapsed\n    into one big unstratified bucket. We now read the CSV (which `CORPUS_CSV`\n    already pointed at, but nothing ever opened) and join it back to trials by\n    session. A session can span multiple corpora (multiple CSV rows), so we keep\n    the sorted, \"|\"-joined set of corpora seen for that session -- this mirrors\n    exactly how the synthetic generator itself encodes a multi-corpus session.\n    \"\"\"\n    lookup = {}\n    csv_path = find_corpus_csv(csv_path)\n    if not csv_path.exists():\n        print(f\"  [corpus] WARNING: could not find a corpus CSV (tried '{csv_path}', and \"\n              f\"searched under {KAGGLE_COMPETITION_ROOT}). Corpus stays 'unknown' for every \"\n              f\"trial -- this is EXACTLY the original notebook's behavior, so nothing is \"\n              f\"worse than before; the enrichment below just can't kick in until the right \"\n              f\"file is located (see the file list printed below, or run \"\n              f\"`!find /kaggle/input -iname \\'*.csv\\'` in a new cell).\")\n        if KAGGLE_COMPETITION_ROOT.exists():\n            nearby = sorted(str(p) for p in KAGGLE_COMPETITION_ROOT.rglob(\"*.csv\"))\n            if nearby:\n                print(f\"  [corpus] .csv files that DO exist under {KAGGLE_COMPETITION_ROOT}:\")\n                for p in nearby[:20]:\n                    print(\"   -\", p)\n            else:\n                print(f\"  [corpus] No .csv files found anywhere under {KAGGLE_COMPETITION_ROOT}.\")\n        else:\n            print(f\"  [corpus] {KAGGLE_COMPETITION_ROOT} itself doesn't exist on this machine.\")\n        return lookup\n    per_session = defaultdict(set)\n    with open(csv_path, newline=\"\") as f:\n        reader = csv.DictReader(f)\n        fieldnames = reader.fieldnames or []\n        session_col = next((c for c in (\"session\", \"session_id\") if c in fieldnames), None)\n        corpus_col = next((c for c in (\"corpus\", \"corpus_name\") if c in fieldnames), None)\n        if session_col is None or corpus_col is None:\n            print(f\"  [corpus] WARNING: {csv_path} does not have the expected \"\n                  f\"'session'/'corpus' columns (found: {fieldnames}). Corpus will stay \"\n                  f\"'unknown'. Update `session_col`/`corpus_col` above to match.\")\n            return lookup\n        for row in reader:\n            sid = row[session_col]\n            for c in str(row[corpus_col]).split(\"|\"):\n                c = c.strip()\n                if c:\n                    per_session[sid].add(c)\n    for sid, corpora in per_session.items():\n        lookup[sid] = \"|\".join(sorted(corpora))\n    return lookup\n\n\ndef scan_dataset(hdf5_root: Path, corpus_lookup: dict):\n    session_dirs = sorted(p for p in hdf5_root.iterdir() if p.is_dir())\n    records, session_ids, integrity_issues = [], [], []\n    corpus_source_counts = Counter()  # diagnostic: where did `corpus` actually come from?\n    for sdir in session_dirs:\n        session_id = sdir.name\n        session_ids.append(session_id)\n        for split in (\"train\", \"val\", \"test\"):\n            fpath = sdir / f\"data_{split}.hdf5\"\n            if not fpath.exists():\n                continue\n            try:\n                with h5py.File(fpath, \"r\") as f:\n                    for key in f.keys():\n                        g = f[key]\n                        has_labels = \"seq_class_ids\" in g and \"sentence_label\" in g.attrs\n                        if split in (\"train\", \"val\") and not has_labels:\n                            integrity_issues.append(f\"{session_id}/{split}/{key}: missing labels\")\n                            continue\n                        if \"corpus\" in g.attrs:\n                            corpus = str(g.attrs[\"corpus\"])\n                            corpus_source_counts[\"hdf5_attr\"] += 1\n                        elif session_id in corpus_lookup:\n                            corpus = corpus_lookup[session_id]\n                            corpus_source_counts[\"csv_lookup\"] += 1\n                        else:\n                            corpus = \"unknown\"\n                            corpus_source_counts[\"unknown\"] += 1\n                        records.append(TrialRecord(\n                            session_id=session_id, split=split, path=str(fpath), key=key,\n                            corpus=corpus,\n                            n_time_steps=int(g.attrs.get(\"n_time_steps\", g[\"input_features\"].shape[0])),\n                            seq_len=int(g.attrs.get(\"seq_len\", 0)) if has_labels else 0,\n                        ))\n            except Exception as e:\n                integrity_issues.append(f\"{session_id}/{split}: FAILED TO OPEN ({e})\")\n    print(f\"  [corpus] resolved via HDF5 attribute: {corpus_source_counts['hdf5_attr']}, \"\n          f\"via CSV session lookup: {corpus_source_counts['csv_lookup']}, \"\n          f\"still unknown: {corpus_source_counts['unknown']}\")\n    return records, session_ids, integrity_issues\n\n\ncorpus_lookup = load_corpus_lookup(CORPUS_CSV)\nrecords, session_ids, integrity_issues = scan_dataset(HDF5_ROOT, corpus_lookup)\nprint(f\"Sessions found (chronological order): {len(session_ids)}\")\nprint(f\"Total trial records indexed: {len(records)}\")\nprint(f\"Integrity issues flagged: {len(integrity_issues)}\")\nfor issue in integrity_issues[:20]:\n    print(\"  ISSUE:\", issue)\nassert session_ids == sorted(session_ids), \"session directory names must sort chronologically (t15.YYYY.MM.DD)\"\n\nby_split = Counter(r.split for r in records)\nprint(\"Records per split:\", dict(by_split))","metadata":{"execution":{"iopub.execute_input":"2026-08-31T20:43:36.344286Z","iopub.status.busy":"2026-08-31T20:43:36.344085Z","iopub.status.idle":"2026-08-31T20:44:50.767688Z","shell.execute_reply":"2026-08-31T20:44:50.76685Z"},"papermill":{"duration":74.431325,"end_time":"2026-08-31T20:44:50.771915+00:00","exception":false,"start_time":"2026-08-31T20:43:36.34059+00:00","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"f6a84739","cell_type":"markdown","source":"## 5. Corpus and session distribution","metadata":{"papermill":{"duration":0.00264,"end_time":"2026-08-31T20:44:50.777466+00:00","exception":false,"start_time":"2026-08-31T20:44:50.774826+00:00","status":"completed"},"tags":[]}},{"id":"bf10df26","cell_type":"code","source":"trials_per_session = Counter(r.session_id for r in records)\ncorpus_counts = Counter(r.corpus for r in records if r.split != \"test\")\nprint(\"Trials per session:\")\nfor sid in session_ids:\n    print(f\"  {sid}: {trials_per_session.get(sid, 0)}\")\nprint(\"\\nCorpus distribution (train+val):\", dict(corpus_counts))\n\nseq_lens = [r.seq_len for r in records if r.seq_len > 0]\ntime_steps = [r.n_time_steps for r in records]\nprint(f\"\\nPhoneme sequence length: min={min(seq_lens)} max={max(seq_lens)} mean={np.mean(seq_lens):.1f}\")\nprint(f\"Trial duration (timesteps @ 20ms): min={min(time_steps)} max={max(time_steps)} mean={np.mean(time_steps):.1f}\")","metadata":{"execution":{"iopub.execute_input":"2026-08-31T20:44:50.784028Z","iopub.status.busy":"2026-08-31T20:44:50.783738Z","iopub.status.idle":"2026-08-31T20:44:50.795335Z","shell.execute_reply":"2026-08-31T20:44:50.794289Z"},"papermill":{"duration":0.016716,"end_time":"2026-08-31T20:44:50.796796+00:00","exception":false,"start_time":"2026-08-31T20:44:50.78008+00:00","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"974d22ca","cell_type":"markdown","source":"## 6. Before/after preprocessing feature statistics\n\nThe dataset ships already z-scored per-session against the preceding 20 trials. This section\nrecords the RAW per-channel statistics (as found in the files) so Notebook 3 can plot a\nbefore/after comparison once Notebook 2 applies the model's own `InputNormalization`. This\nnotebook only measures and records what's already there -- it does not re-normalize.","metadata":{"papermill":{"duration":0.002632,"end_time":"2026-08-31T20:44:50.8021+00:00","exception":false,"start_time":"2026-08-31T20:44:50.799468+00:00","status":"completed"},"tags":[]}},{"id":"a51bf995","cell_type":"code","source":"def sample_raw_feature_stats(records, n_samples=40, seed=0):\n    rng = random.Random(seed)\n    sample = rng.sample(records, min(n_samples, len(records)))\n    tx_vals, sbp_vals = [], []\n    for r in sample:\n        with h5py.File(r.path, \"r\") as f:\n            feats = f[r.key][\"input_features\"][:]\n        tx_vals.append(feats[:, :N_ELECTRODES].flatten())\n        sbp_vals.append(feats[:, N_ELECTRODES:].flatten())\n    tx_all = np.concatenate(tx_vals)\n    sbp_all = np.concatenate(sbp_vals)\n    return {\n        \"threshold_crossings\": {\"mean\": float(tx_all.mean()), \"std\": float(tx_all.std()),\n                                  \"min\": float(tx_all.min()), \"max\": float(tx_all.max())},\n        \"spike_band_power\": {\"mean\": float(sbp_all.mean()), \"std\": float(sbp_all.std()),\n                               \"min\": float(sbp_all.min()), \"max\": float(sbp_all.max())},\n        \"sample_histogram_tx\": np.histogram(tx_all, bins=40)[0].tolist(),\n        \"sample_histogram_tx_edges\": np.histogram(tx_all, bins=40)[1].tolist(),\n        \"sample_histogram_sbp\": np.histogram(sbp_all, bins=40)[0].tolist(),\n        \"sample_histogram_sbp_edges\": np.histogram(sbp_all, bins=40)[1].tolist(),\n    }\n\n\nraw_feature_stats = sample_raw_feature_stats(records, n_samples=40)\nprint(\"Raw threshold-crossing stats:\", raw_feature_stats[\"threshold_crossings\"])\nprint(\"Raw spike-band-power stats:\", raw_feature_stats[\"spike_band_power\"])","metadata":{"execution":{"iopub.execute_input":"2026-08-31T20:44:50.808777Z","iopub.status.busy":"2026-08-31T20:44:50.808232Z","iopub.status.idle":"2026-08-31T20:44:53.243303Z","shell.execute_reply":"2026-08-31T20:44:53.242405Z"},"papermill":{"duration":2.440025,"end_time":"2026-08-31T20:44:53.244863+00:00","exception":false,"start_time":"2026-08-31T20:44:50.804838+00:00","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"da30670a","cell_type":"markdown","source":"## 7. Two splits\n\n- **`dev_test_split`**: stratified by corpus, carved from the *labeled* train+val pool only,\n  for ordinary model selection. Kaggle's own unlabeled `data_test.hdf5` partition is never\n  touched here -- kept as `kaggle_test` conceptually distinct throughout.\n- **`chronological_cl_split`**: all sessions in date order; the first `n_base_sessions` form\n  the base pool, every session after that is one sequential continual-learning task.","metadata":{"papermill":{"duration":0.00283,"end_time":"2026-08-31T20:44:53.250662+00:00","exception":false,"start_time":"2026-08-31T20:44:53.247832+00:00","status":"completed"},"tags":[]}},{"id":"41d42cff","cell_type":"code","source":"def dev_test_split(records, val_fraction=0.15, test_fraction=0.15, seed=1337):\n    rng = random.Random(seed)\n    by_corpus = defaultdict(list)\n    for r in records:\n        by_corpus[r.corpus].append(r)\n    out = {\"train\": [], \"val\": [], \"dev_test\": []}\n    for corpus, recs in by_corpus.items():\n        recs = recs[:]\n        rng.shuffle(recs)\n        n = len(recs)\n        n_test = max(1, round(n * test_fraction)) if n >= 3 else 0\n        n_val = max(1, round(n * val_fraction)) if n - n_test >= 2 else 0\n        out[\"dev_test\"].extend(recs[:n_test])\n        out[\"val\"].extend(recs[n_test:n_test + n_val])\n        out[\"train\"].extend(recs[n_test + n_val:])\n    return out\n\n\ndef chronological_cl_split(session_ids, n_base_sessions):\n    if n_base_sessions >= len(session_ids):\n        n_base_sessions = max(1, len(session_ids) - 1)\n        print(f\"WARNING: n_base_sessions too large for {len(session_ids)} sessions; clamped to {n_base_sessions}\")\n    return {\"base_sessions\": session_ids[:n_base_sessions], \"cl_task_sessions\": session_ids[n_base_sessions:]}\n\n\nlabeled_records = [r for r in records if r.split in (\"train\", \"val\")]\ndev_splits = dev_test_split(labeled_records, val_fraction=0.15, test_fraction=0.15, seed=1337)\nprint(\"Dev-split sizes:\", {k: len(v) for k, v in dev_splits.items()})\n\nN_BASE_SESSIONS = max(1, len(session_ids) - 3)\ncl_split = chronological_cl_split(session_ids, n_base_sessions=N_BASE_SESSIONS)\nprint(\"CL base sessions:\", cl_split[\"base_sessions\"])\nprint(\"CL task sessions (chronological order):\", cl_split[\"cl_task_sessions\"])","metadata":{"execution":{"iopub.execute_input":"2026-08-31T20:44:53.257218Z","iopub.status.busy":"2026-08-31T20:44:53.256994Z","iopub.status.idle":"2026-08-31T20:44:53.269616Z","shell.execute_reply":"2026-08-31T20:44:53.268885Z"},"papermill":{"duration":0.017658,"end_time":"2026-08-31T20:44:53.271007+00:00","exception":false,"start_time":"2026-08-31T20:44:53.253349+00:00","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"id":"180e9369","cell_type":"markdown","source":"## 8. Save manifest + stats for Notebooks 2 and 3\n\nRecords are saved as **relative paths** (`session_id/split_filename.hdf5` + trial key) rather\nthan absolute paths, since the absolute mount point can differ slightly between where this\nnotebook runs and where Notebook 2 runs. Notebook 2 rejoins these against its own `HDF5_ROOT`.","metadata":{"papermill":{"duration":0.002744,"end_time":"2026-08-31T20:44:53.276626+00:00","exception":false,"start_time":"2026-08-31T20:44:53.273882+00:00","status":"completed"},"tags":[]}},{"id":"4b455f2b","cell_type":"code","source":"def record_to_portable(r: TrialRecord) -> dict:\n    rel_path = f\"{r.session_id}/{Path(r.path).name}\"\n    return {\"session_id\": r.session_id, \"split\": r.split, \"rel_path\": rel_path, \"key\": r.key,\n            \"corpus\": r.corpus, \"n_time_steps\": r.n_time_steps, \"seq_len\": r.seq_len}\n\n\nmanifest = {\n    \"schema\": {\n        \"n_phoneme_classes\": N_PHONEME_CLASSES, \"ctc_blank_id\": CTC_BLANK_ID, \"silence_id\": SILENCE_ID,\n        \"logit_to_phoneme\": LOGIT_TO_PHONEME, \"region_names\": REGION_NAMES,\n        \"n_electrodes_per_region\": N_ELECTRODES_PER_REGION, \"n_electrodes\": N_ELECTRODES,\n        \"n_raw_features\": N_RAW_FEATURES,\n    },\n    \"session_ids\": session_ids,\n    \"trials_per_session\": {sid: trials_per_session.get(sid, 0) for sid in session_ids},\n    \"corpus_distribution\": dict(corpus_counts),\n    \"integrity_issues\": integrity_issues,\n    \"sequence_length_stats\": {\"min\": int(min(seq_lens)), \"max\": int(max(seq_lens)), \"mean\": float(np.mean(seq_lens))},\n    \"trial_duration_stats\": {\"min\": int(min(time_steps)), \"max\": int(max(time_steps)), \"mean\": float(np.mean(time_steps))},\n    \"raw_feature_stats\": raw_feature_stats,\n    \"splits\": {\n        \"dev_test_sizes\": {k: len(v) for k, v in dev_splits.items()},\n        \"continual_learning\": {\"base_sessions\": cl_split[\"base_sessions\"],\n                                  \"cl_task_sessions\": cl_split[\"cl_task_sessions\"],\n                                  \"n_base_sessions\": N_BASE_SESSIONS},\n    },\n    \"used_synthetic_data\": USE_SYNTHETIC,\n}\n\nwith open(OUTPUT_DIR / \"manifest.json\", \"w\") as f:\n    json.dump(manifest, f, indent=2)\n\nall_records_portable = {\n    \"train\": [record_to_portable(r) for r in dev_splits[\"train\"]],\n    \"val\": [record_to_portable(r) for r in dev_splits[\"val\"]],\n    \"dev_test\": [record_to_portable(r) for r in dev_splits[\"dev_test\"]],\n    \"kaggle_test\": [record_to_portable(r) for r in records if r.split == \"test\"],\n    \"cl_base_pool\": [record_to_portable(r) for r in labeled_records if r.session_id in cl_split[\"base_sessions\"]],\n    \"cl_tasks\": {\n        sid: [record_to_portable(r) for r in labeled_records if r.session_id == sid]\n        for sid in cl_split[\"cl_task_sessions\"]\n    },\n}\nwith open(OUTPUT_DIR / \"records.json\", \"w\") as f:\n    json.dump(all_records_portable, f)\n\nprint(f\"\\nSaved manifest.json ({(OUTPUT_DIR / 'manifest.json').stat().st_size / 1024:.1f} KB) \"\n      f\"and records.json ({(OUTPUT_DIR / 'records.json').stat().st_size / 1024:.1f} KB) to {OUTPUT_DIR}\")\nprint(\"\\nNotebook 1 complete. Attach this notebook's output as a Kaggle Dataset, \"\n      \"then add it (alongside the original competition dataset) as input to Notebook 2.\")","metadata":{"execution":{"iopub.execute_input":"2026-08-31T20:44:53.283879Z","iopub.status.busy":"2026-08-31T20:44:53.283624Z","iopub.status.idle":"2026-08-31T20:44:53.557787Z","shell.execute_reply":"2026-08-31T20:44:53.556945Z"},"papermill":{"duration":0.279673,"end_time":"2026-08-31T20:44:53.55929+00:00","exception":false,"start_time":"2026-08-31T20:44:53.279617+00:00","status":"completed"},"tags":[]},"outputs":[],"execution_count":null}]}