{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.11"}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"aee53347-6591-421a-85d2-b14e6be22e48","cell_type":"markdown","source":"# GISLR — Stratified 80/20 Split + Parquet → NPZ Conversion\n\n1. Splits the official GISLR (`asl-signs`) `train.csv` 80/20, stratified on `sign`.\n2. Converts each sequence's long-format parquet landmarks into the same `(T, 543, 3)` npz\n   array layout used for the PopSign extraction — `[0:33]=pose, [33:501]=face,\n   [501:522]=left_hand, [522:543]=right_hand` — so the two datasets share one schema and can\n   be merged later.\n\nOutput:\n```\n/kaggle/working/\n├── train.csv   # uid, sign, participant_id, sequence_id, split, npz_relpath\n├── test.csv\n├── train/      # <uid>.npz\n└── test/       # <uid>.npz\n```\n\n**Note:** stratifies on `sign` only, as requested — the same `participant_id` can appear in\nboth splits. If participant-disjoint evaluation matters later, that needs a different strategy\n(e.g. `GroupShuffleSplit` on `participant_id`) — flagging since it's a common generalization\ngotcha, not because it was asked for here.","metadata":{}},{"id":"b59ddc05-2097-4dcc-b2da-8ea863dd9fe2","cell_type":"code","source":"!pip install -q scikit-learn tqdm\n\nimport os\nimport numpy as np\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\nfrom multiprocessing import Pool, cpu_count\nfrom tqdm import tqdm\n\nBASE = \"/kaggle/input/competitions/asl-signs\"\nTRAIN_CSV = os.path.join(BASE, \"train.csv\")\n\nOUTPUT_ROOT = \"/kaggle/working\"\nTRAIN_DIR = os.path.join(OUTPUT_ROOT, \"train\")\nTEST_DIR = os.path.join(OUTPUT_ROOT, \"test\")\nos.makedirs(TRAIN_DIR, exist_ok=True)\nos.makedirs(TEST_DIR, exist_ok=True)\n\ndf = pd.read_csv(TRAIN_CSV)\nprint(f\"total sequences: {len(df)}\")\nprint(f\"unique signs: {df['sign'].nunique()}\")","metadata":{},"outputs":[],"execution_count":null},{"id":"4a96b30b-f1df-4102-ae35-96edf38284f7","cell_type":"markdown","source":"## Stratified split","metadata":{}},{"id":"6952e522-d2ce-4e14-bc32-4b4c1a891873","cell_type":"code","source":"counts = df[\"sign\"].value_counts()\ntoo_rare = counts[counts < 2]  # sklearn can't stratify a class with <2 members\nif len(too_rare):\n    print(f\"{len(too_rare)} sign(s) have <2 samples — can't be split across both sets:\")\n    print(too_rare)\nelse:\n    print(\"every sign has >=2 samples — safe to stratify across all classes\")\n\nstratifiable = df[df[\"sign\"].isin(counts[counts >= 2].index)]\nunstratifiable = df[df[\"sign\"].isin(too_rare.index)]\n\ntrain_df, test_df = train_test_split(\n    stratifiable, test_size=0.2, stratify=stratifiable[\"sign\"], random_state=42,\n)\ntrain_df = pd.concat([train_df, unstratifiable], ignore_index=True)  # keep rare signs, in train only\n\nprint(f\"train: {len(train_df)} ({len(train_df) / len(df):.1%})\")\nprint(f\"test:  {len(test_df)} ({len(test_df) / len(df):.1%})\")\n\ncheck = pd.DataFrame({\n    \"train_count\": train_df[\"sign\"].value_counts(),\n    \"test_count\": test_df[\"sign\"].value_counts(),\n}).fillna(0)\ncheck[\"test_ratio\"] = check[\"test_count\"] / (check[\"train_count\"] + check[\"test_count\"])\nprint(\"\\nper-class test ratio — min / median / max (expect all close to 0.20):\")\nprint(check[\"test_ratio\"].agg([\"min\", \"median\", \"max\"]))","metadata":{},"outputs":[],"execution_count":null},{"id":"904c40b3-b556-492b-b9cb-34b9590a69ae","cell_type":"markdown","source":"## Parquet → NPZ conversion\n\nEach parquet file is long-format: one row per `(frame, type, landmark_index)` with `x, y, z`\nalready in that row. `landmark_index` is 0-indexed within its `type`, so\n`LAYOUT[type][0] + landmark_index` maps directly onto the same 543-slot layout as PopSign —\nno model inference needed, just a vectorized reshape.","metadata":{}},{"id":"1ecdcd08-24a4-42db-a088-191638379838","cell_type":"code","source":"N_POSE, N_FACE, N_LH, N_RH = 33, 468, 21, 21\nTOTAL = N_POSE + N_FACE + N_LH + N_RH  # 543\nLAYOUT = {\"pose\": (0, 33), \"face\": (33, 501), \"left_hand\": (501, 522), \"right_hand\": (522, 543)}\nTYPE_START = {k: v[0] for k, v in LAYOUT.items()}\n\ndef parquet_to_array(parquet_path):\n    pdf = pd.read_parquet(parquet_path, columns=[\"frame\", \"type\", \"landmark_index\", \"x\", \"y\", \"z\"])\n    frames = np.sort(pdf[\"frame\"].unique())\n    frame_to_idx = {f: i for i, f in enumerate(frames)}\n    T = len(frames)\n    arr = np.full((T, TOTAL, 3), np.nan, dtype=np.float32)\n\n    row_idx = pdf[\"frame\"].map(frame_to_idx).to_numpy()\n    col_idx = pdf[\"type\"].map(TYPE_START).to_numpy() + pdf[\"landmark_index\"].to_numpy()\n    arr[row_idx, col_idx, 0] = pdf[\"x\"].to_numpy()\n    arr[row_idx, col_idx, 1] = pdf[\"y\"].to_numpy()\n    arr[row_idx, col_idx, 2] = pdf[\"z\"].to_numpy()\n    return arr\n\ndef process_row(args):\n    uid, rel_path, split = args\n    out_path = os.path.join(OUTPUT_ROOT, split, f\"{uid}.npz\")\n    if os.path.exists(out_path):\n        return uid, out_path, \"cached\"\n    try:\n        arr = parquet_to_array(os.path.join(BASE, rel_path))\n        np.savez_compressed(out_path, landmarks=arr)\n        return uid, out_path, \"ok\"\n    except Exception as e:\n        return uid, None, f\"error: {e}\"","metadata":{},"outputs":[],"execution_count":null},{"id":"e0d76683-7498-4cf5-b180-e8983105c37f","cell_type":"markdown","source":"### Quick sanity check on one sequence before converting all ~94k","metadata":{}},{"id":"9a3a819a-6d46-4454-8b04-7a0a0722a112","cell_type":"code","source":"_sample_path = os.path.join(BASE, train_df.iloc[0][\"path\"])\n_sample_arr = parquet_to_array(_sample_path)\nprint(f\"shape: {_sample_arr.shape}  (T, 543, 3)\")\nprint(f\"NaN fraction per region:\")\nfor name, (s, e) in LAYOUT.items():\n    frac = np.isnan(_sample_arr[:, s:e, :]).mean()\n    print(f\"  {name}: {frac:.1%} NaN\")","metadata":{},"outputs":[],"execution_count":null},{"id":"a1daf82b-1858-4578-9784-351d6053fcfc","cell_type":"markdown","source":"## Run conversion\n\nCheckpointed via `os.path.exists(out_path)` — safe to re-run after a session timeout.","metadata":{}},{"id":"c747679f-4444-4058-93dd-2831f0ba8e60","cell_type":"code","source":"train_df[\"uid\"] = \"gislr_\" + train_df[\"sequence_id\"].astype(str)\ntest_df[\"uid\"] = \"gislr_\" + test_df[\"sequence_id\"].astype(str)\n\njobs = (\n    [(u, p, \"train\") for u, p in zip(train_df[\"uid\"], train_df[\"path\"])] +\n    [(u, p, \"test\") for u, p in zip(test_df[\"uid\"], test_df[\"path\"])]\n)\n\nresults = []\nwith Pool(processes=max(1, cpu_count() - 1)) as pool:\n    for r in tqdm(pool.imap_unordered(process_row, jobs), total=len(jobs)):\n        results.append(r)\n\nstatus = pd.DataFrame(results, columns=[\"uid\", \"npz_path\", \"status\"])\nprint(status[\"status\"].value_counts())","metadata":{},"outputs":[],"execution_count":null},{"id":"cfa7b987-ee6f-43e2-9bbd-ed58cd141a6f","cell_type":"code","source":"all_meta = pd.concat([\n    train_df.assign(split=\"train\"),\n    test_df.assign(split=\"test\"),\n], ignore_index=True).merge(status, on=\"uid\")\n\nok = all_meta[all_meta[\"status\"].isin([\"ok\", \"cached\"])].copy()\nok[\"npz_relpath\"] = ok[\"split\"] + \"/\" + ok[\"uid\"] + \".npz\"\n\ncols = [\"uid\", \"sign\", \"participant_id\", \"sequence_id\", \"split\", \"npz_relpath\"]\nok[ok[\"split\"] == \"train\"][cols].to_csv(os.path.join(OUTPUT_ROOT, \"train.csv\"), index=False)\nok[ok[\"split\"] == \"test\"][cols].to_csv(os.path.join(OUTPUT_ROOT, \"test.csv\"), index=False)\n\nprint(\"train.csv rows:\", (ok[\"split\"] == \"train\").sum())\nprint(\"test.csv rows:\", (ok[\"split\"] == \"test\").sum())\n\nfailed = all_meta[~all_meta[\"status\"].isin([\"ok\", \"cached\"])]\nif len(failed):\n    print(f\"\\n{len(failed)} sequences failed — inspect before publishing:\")\n    print(failed[\"status\"].value_counts())","metadata":{},"outputs":[],"execution_count":null}]}