{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.12.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"databundleVersionId":11294684,"isSourceIdPinned":false,"sourceId":91249,"sourceType":"competition"}],"dockerImageVersionId":31328,"isGpuEnabled":false,"isInternetEnabled":true,"language":"python","sourceType":"notebook"},"papermill":{"default_parameters":{},"duration":3.798347,"end_time":"2026-05-02T03:32:51.061891+00:00","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2026-05-02T03:32:47.263544+00:00","version":"2.7.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"84bcf530","cell_type":"markdown","source":"\"Kaggle\"","metadata":{}},{"id":"ab8e853b","cell_type":"markdown","source":"# 目次\n\n- [1. 環境設定と入力探索](#sec-setup)\n- [2. tomo 抽出](#sec-select)\n- [3. 前処理（raw Image -> .npy）](#sec-preprocess)\n- [4. CustomDataset と座標→ラベルマップ変換](#sec-dataset)\n- [5. モデル構築](#sec-build-model)\n - [5-1. BasicBlock](#sec-build-basicblock)\n - [5-2. Bottleneck](#sec-build-bottleneck)\n - [5-3. Encoder](#sec-build-encoder)\n - [5-4. Decoder Common Block](#sec-build-decoder-common)\n - [5-5. Decoder](#sec-build-decoder)\n - [5-6. Net](#sec-build-net)\n - [5-7. BaseModel](#sec-build-basemodel)\n - [5-8. Utility](#sec-build-utility)\n- [6. 学習最適化と拡張（損失・拡張・最適化戦略）](#sec-train-optimization)\n - [6-1. Custom Losses](#sec-custom-losses)\n - [6-2. Optimizer/Scheduler Utils](#sec-optim-utils)\n - [6-3. 3D Spatial Augmentations](#sec-aug3d)\n- [7. 推論・後処理・評価](#sec-train)\n - [7-1. Sliding Window Eval](#sec-eval-5-1)\n - [7-2. Postprocess (NMS)](#sec-postprocess-5-2)\n - [7-3. Metric Scoring](#sec-metric-5-3)\n - [7-4. Flip TTA](#sec-flip-tta-5-4)\n- [8. 大規模化と運用（フェーズ6）](#sec-scale-ops)\n - [8-1. AMP/DDP/Fold Loop (WBS 6-1)](#sec-scale-ops-6-1)\n - [8-2. Multi-GPU Run Script (WBS 6-2)](#sec-scale-ops-6-2)\n - [8-3. WandB Logger (WBS 6-3)](#sec-scale-ops-6-3)\n - [8-4. CFG/Torch Utilities (WBS 6-4)](#sec-scale-ops-6-4)\n - [8-5. E2E Smoke Test (WBS 6-5)](#sec-scale-ops-6-5)\n- [9. 実データを用いた学習](#sec-real-train)\n\nこの目次から各章へ移動できます。","metadata":{}},{"id":"43783292","cell_type":"markdown","source":"## 1. 環境設定と入力探索\n\n\nライブラリ読み込み、`BaseConfig`（2-4）で学習設定を一元化、seed 固定、入力ディレクトリ探索を行います。","metadata":{}},{"id":"c47cf537","cell_type":"code","source":"from my_implementation.chapter1_setup import BaseConfig, run_chapter1_setup\n\n# Chapter 1 logic + assertions are externalized in my_implementation/chapter1_setup.py\nchapter1_ctx = run_chapter1_setup(cfg=BaseConfig())\n\nCFG = chapter1_ctx[\"CFG\"]\nSEED = chapter1_ctx[\"SEED\"]\nNUM_TOMOS = chapter1_ctx[\"NUM_TOMOS\"]\nDEPTH = chapter1_ctx[\"DEPTH\"]\nIMG_SIZE = chapter1_ctx[\"IMG_SIZE\"]\nBATCH_SIZE = chapter1_ctx[\"BATCH_SIZE\"]\nNUM_WORKERS = chapter1_ctx[\"NUM_WORKERS\"]\nEPOCHS = chapter1_ctx[\"EPOCHS\"]\nLR = chapter1_ctx[\"LR\"]\nLABEL_RADIUS = chapter1_ctx[\"LABEL_RADIUS\"]\ncandidate_roots = chapter1_ctx[\"candidate_roots\"]\ninput_root = chapter1_ctx[\"input_root\"]","metadata":{},"outputs":[],"execution_count":null},{"id":"577b7183","cell_type":"markdown","source":"## 2. tomo 抽出\n\n\n利用可能な tomo から seed 固定で学習対象を抽出します。","metadata":{}},{"id":"e7910e38","cell_type":"code","source":"from my_implementation.chapter2_select_tomos import run_chapter2_select_tomos\n\n# Chapter 2 logic + assertions are externalized in my_implementation/chapter2_select_tomos.py\nall_tomos, selected_tomos = run_chapter2_select_tomos(\n input_root=input_root,\n num_tomos=NUM_TOMOS,\n seed=SEED,\n)","metadata":{},"outputs":[],"execution_count":null},{"id":"522b57b2","cell_type":"markdown","source":"## 3. 前処理（raw Image -> .npy）\n\n\n生データのスライス画像を読み込み、学習で使いやすい `.npy` ボリュームとして保存します。","metadata":{}},{"id":"6c957e07","cell_type":"code","source":"from my_implementation.chapter3_preprocess import run_chapter3_preprocess\n\n# Chapter 3 logic + assertions are externalized in my_implementation/chapter3_preprocess.py\nprocessed_root = run_chapter3_preprocess(\n selected_tomos=selected_tomos,\n img_size=IMG_SIZE,\n)","metadata":{},"outputs":[],"execution_count":null},{"id":"00eb406b","cell_type":"markdown","source":"## 4. CustomDataset と座標→ラベルマップ変換\n\n\nCSV 座標を読み取り、3D ラベルマップへ変換して input/target を返す Dataset を作成し、`get_dataset` / `get_dataloader` で DataLoader を組み立てます。","metadata":{}},{"id":"8e09064c","cell_type":"code","source":"from my_implementation.chapter4_dataset import (\n CustomDataset,\n build_dataloader_cfg,\n get_dataset,\n get_dataloader,\n run_chapter4_dataset_pipeline,\n)\n\n# Chapter 4 logic + assertions are externalized in my_implementation/chapter4_dataset.py\nchapter4_ctx = run_chapter4_dataset_pipeline(\n selected_tomos=selected_tomos,\n depth=DEPTH,\n img_size=IMG_SIZE,\n label_radius=LABEL_RADIUS,\n batch_size=BATCH_SIZE,\n num_workers=NUM_WORKERS,\n processed_root=processed_root,\n)\n\nDATALOADER_CFG = chapter4_ctx[\"DATALOADER_CFG\"]\ntrain_tomos = chapter4_ctx[\"train_tomos\"]\nval_tomos = chapter4_ctx[\"val_tomos\"]\ntrain_ds = chapter4_ctx[\"train_ds\"]\nval_ds = chapter4_ctx[\"val_ds\"]\ntrain_loader = chapter4_ctx[\"train_loader\"]\nval_loader = chapter4_ctx[\"val_loader\"]","metadata":{},"outputs":[],"execution_count":null},{"id":"2f05d6a2","cell_type":"markdown","source":"## 5. モデル構築\n","metadata":{}},{"id":"b8742609","cell_type":"markdown","source":"### 5-1. BasicBlock\n\n\nBasicBlock の実装と downsample 分岐の検証を行います。","metadata":{}},{"id":"6241285d","cell_type":"code","source":"# ===== 3-1-3: BasicBlock downsample branch implementation =====\nfrom my_implementation.chapter5_1_basicblock import (\n BasicBlock,\n conv3x3x3,\n run_section_5_1_assertions,\n)\n\n# Section 5-1 logic + assertions are externalized in my_implementation/chapter5_1_basicblock.py\nrun_section_5_1_assertions()","metadata":{},"outputs":[],"execution_count":null},{"id":"cba8407a","cell_type":"markdown","source":"### 5-2. Bottleneck\n\n\nBottleneck ブロックの実装と shape 整合性を検証します。","metadata":{}},{"id":"8ddb4b5d","cell_type":"code","source":"# ===== 3-1-4: Bottleneck block implementation =====\nfrom my_implementation.chapter5_2_bottleneck import (\n Bottleneck,\n run_section_5_2_assertions,\n)\n\n# Section 5-2 logic + assertions are externalized in my_implementation/chapter5_2_bottleneck.py\nrun_section_5_2_assertions()","metadata":{},"outputs":[],"execution_count":null},{"id":"d4479464","cell_type":"markdown","source":"### 5-3. Encoder\n\n\nResNet3D Encoder 本体（backbone 設定、DropPath、pretrained、checkpoint、forward_features、channels）を実装します。","metadata":{}},{"id":"8d603ef7","cell_type":"code","source":"# ===== 3-1-6/3-1-7/3-1-8/3-1-9/3-1-10/3-1-11/3-1-12/3-1-13/3-1-15: backbone + DropPath + pretrained + input channels + checkpoint + features + channels =====\nfrom my_implementation.chapter5_3_encoder import (\n LoadRecorder,\n ResnetEncoder3d,\n build_linear_drop_path_rates,\n conv_out_dim,\n gather_block_drop_probs,\n load_weights_stub,\n resolve_backbone,\n resolve_pretrained_path,\n run_section_5_3_assertions,\n)\n\n# Section 5-3 logic + assertions are externalized in my_implementation/chapter5_3_encoder.py\nrun_section_5_3_assertions()","metadata":{},"outputs":[],"execution_count":null},{"id":"c395f3a2","cell_type":"markdown","source":"### 5-4. Decoder Common Block (3-2-1)\n\n\n3-2-1として、UNetデコーダで使う `Conv3d -> BatchNorm3d -> ReLU` の共通ブロック `ConvBnAct3d` を実装します。","metadata":{}},{"id":"a6950f61","cell_type":"code","source":"# ===== 3-2-1: ConvBnAct3d implementation =====\nfrom my_implementation.chapter5_4_decoder_common import (\n ConvBnAct3d,\n run_section_5_4_assertions,\n)\n\n# Section 5-4 logic + assertions are externalized in my_implementation/chapter5_4_decoder_common.py\nrun_section_5_4_assertions()","metadata":{},"outputs":[],"execution_count":null},{"id":"4e370f37","cell_type":"markdown","source":"### 5-5. Decoder\n\n\n3-2-2 と 3-2-3 を統合し、upsample経路とskip接続を持つ `DecoderBlock3d` を実装します。","metadata":{}},{"id":"7c5a7bb0","cell_type":"code","source":"# ===== 3-2-2 + 3-2-3: DecoderBlock3d (upsample + skip) implementation =====\nfrom my_implementation.chapter5_5_decoder import (\n DecoderBlock3d,\n run_section_5_5_decoder_block_assertions,\n)\n\n# Section 5-5 (DecoderBlock3d) logic + assertions are externalized in my_implementation/chapter5_5_decoder.py\nrun_section_5_5_decoder_block_assertions()","metadata":{},"outputs":[],"execution_count":null},{"id":"00c12976","cell_type":"code","source":"# ===== 3-2-4 + 3-2-5 + 3-2-6: channel design + block assembly + forward =====\nfrom my_implementation.chapter5_5_decoder import (\n UnetDecoder3d,\n build_decoder_channel_plan,\n extract_block_specs,\n run_section_5_5_unet_decoder_assertions,\n)\n\n# Section 5-5 (UnetDecoder3d) logic + assertions are externalized in my_implementation/chapter5_5_decoder.py\nrun_section_5_5_unet_decoder_assertions()","metadata":{},"outputs":[],"execution_count":null},{"id":"27ac81fc","cell_type":"code","source":"# ===== 3-2-7: SegmentationHead3d implementation =====\nfrom my_implementation.chapter5_5_decoder import (\n SegmentationHead3d,\n run_section_5_5_seg_head_assertions,\n)\n\n# Section 5-5 (SegmentationHead3d) logic + assertions are externalized in my_implementation/chapter5_5_decoder.py\nrun_section_5_5_seg_head_assertions()","metadata":{},"outputs":[],"execution_count":null},{"id":"0a1542f9","cell_type":"markdown","source":"### 5-6. Net\n\n\n3-3-1〜3-3-5として、`Net` クラスのコンストラクタI/O仕様、encoder接続、main head接続、基本`forward`（encoder→decoder→head）を実装します。","metadata":{}},{"id":"89f8a46b","cell_type":"code","source":"# ===== 3-3-1 + 3-3-2 + 3-3-5 + 3-3-6 + 3-3-7: Net constructor I/O + encoder connection + forward + deep supervision =====\nimport importlib\nimport my_implementation.chapter5_6_net as chapter5_6_net\n\n# Always reload to reflect latest implementation changes in notebook execution.\nchapter5_6_net = importlib.reload(chapter5_6_net)\nbuild_net_class = chapter5_6_net.build_net_class\nrun_section_5_6_assertions = chapter5_6_net.run_section_5_6_assertions\n\nNet = build_net_class(\n ResnetEncoder3d=ResnetEncoder3d,\n UnetDecoder3d=UnetDecoder3d,\n SegmentationHead3d=SegmentationHead3d,\n)\n\n# Run section 5-6 assertions from the externalized implementation module.\nrun_section_5_6_assertions(Net)","metadata":{},"outputs":[],"execution_count":null},{"id":"5a93ef30","cell_type":"markdown","source":"### 5-7. BaseModel (`_base.py`)\n\n\nWBS 3-4として、`BaseModel` に損失関数の動的接続と重みロード機能を実装します。","metadata":{}},{"id":"c108d602","cell_type":"code","source":"# ===== 3-4: BaseModel loss wiring + weight loading =====\nimport importlib\nimport my_implementation.chapter5_7_base_model as chapter5_7_base_model\n\nchapter5_7_base_model = importlib.reload(chapter5_7_base_model)\nrun_section_5_7_assertions = chapter5_7_base_model.run_section_5_7_assertions\n\n# Run section 5-7 assertions from the externalized implementation module.\nrun_section_5_7_assertions()","metadata":{},"outputs":[],"execution_count":null},{"id":"6011d43a","cell_type":"markdown","source":"### 5-8. Utility (`utils.py`)\n\n\nWBS 3-5として、モデル生成・パラメータ数算出・重みロードを担う `get_model` を実装します。","metadata":{}},{"id":"dd63bdea","cell_type":"code","source":"# ===== 3-5: utility.get_model (build + param count + weight load) =====\nimport importlib\nimport my_implementation.chapter5_8_utility as chapter5_8_utility\n\nchapter5_8_utility = importlib.reload(chapter5_8_utility)\nget_model = chapter5_8_utility.get_model\nrun_section_5_8_assertions = chapter5_8_utility.run_section_5_8_assertions\n\n# Run section 5-8 assertions from the externalized implementation module.\nrun_section_5_8_assertions()","metadata":{},"outputs":[],"execution_count":null},{"id":"d8bbad84","cell_type":"markdown","source":"## 6. 学習最適化と拡張（損失・拡張・最適化戦略）\n\n","metadata":{}},{"id":"e2c16cf4","cell_type":"markdown","source":"### 6-1. Custom Losses (`base.py`)\n\n\nWBS 4-1として、Dice Lossを中心にカスタム損失（SmoothBCE / BCE+Dice）を実装します。","metadata":{}},{"id":"145951f6","cell_type":"code","source":"# ===== 4-1: custom losses (Dice / SmoothBCE / BCE+Dice) =====\nimport importlib\nimport my_implementation.chapter6_1_losses as chapter6_1_losses\n\nchapter6_1_losses = importlib.reload(chapter6_1_losses)\nDiceLoss = chapter6_1_losses.DiceLoss\nSmoothBCE = chapter6_1_losses.SmoothBCE\nBCEDiceLoss = chapter6_1_losses.BCEDiceLoss\nrun_section_6_1_assertions = chapter6_1_losses.run_section_6_1_assertions\n\n# Run section 6-1 assertions from the externalized implementation module.\nrun_section_6_1_assertions()","metadata":{},"outputs":[],"execution_count":null},{"id":"6d08ae1e","cell_type":"markdown","source":"### 6-2. Optimizer/Scheduler Utils (`utils.py`)\n\n\nWBS 4-2として、optimizer/scheduler構築とgrad norm計測・clip補助を実装します。","metadata":{}},{"id":"9940bb3f","cell_type":"code","source":"# ===== 4-2: optimizer/scheduler + grad norm utils =====\nimport importlib\nimport my_implementation.chapter6_2_optim_utils as chapter6_2_optim_utils\n\nchapter6_2_optim_utils = importlib.reload(chapter6_2_optim_utils)\ncalc_grad_norm = chapter6_2_optim_utils.calc_grad_norm\nget_optimizer = chapter6_2_optim_utils.get_optimizer\nget_scheduler = chapter6_2_optim_utils.get_scheduler\nclip_grad_and_measure = chapter6_2_optim_utils.clip_grad_and_measure\nrun_section_6_2_assertions = chapter6_2_optim_utils.run_section_6_2_assertions\n\n# Run section 6-2 assertions from the externalized implementation module.\nrun_section_6_2_assertions()","metadata":{},"outputs":[],"execution_count":null},{"id":"c3b4cb6b","cell_type":"markdown","source":"### 6-3. 3D Spatial Augmentations (`aug3d.py`)\n\n\nWBS 4-3として、3D回転・反転・軸入れ替え・coarse dropoutを実装します。","metadata":{}},{"id":"57de247c","cell_type":"code","source":"# ===== 4-3: 3D spatial augmentations (aug3d) =====\nimport importlib\nimport my_implementation.chapter6_3_aug3d as chapter6_3_aug3d\n\nchapter6_3_aug3d = importlib.reload(chapter6_3_aug3d)\nrotate = chapter6_3_aug3d.rotate\nflip_3d = chapter6_3_aug3d.flip_3d\nswap_dims = chapter6_3_aug3d.swap_dims\ncoarse_dropout_3d = chapter6_3_aug3d.coarse_dropout_3d\nrun_section_6_3_assertions = chapter6_3_aug3d.run_section_6_3_assertions\n\n# Run section 6-3 assertions from the externalized implementation module.\nrun_section_6_3_assertions()","metadata":{},"outputs":[],"execution_count":null},{"id":"cf512f77","cell_type":"markdown","source":"### 6-4. Mixup/CutMix (`mixup.py` / `cutmix.py`)\n\n\nWBS 4-4として、3D版 Mixup / CutMix を実装します。","metadata":{}},{"id":"5a87d39d","cell_type":"code","source":"# ===== 4-4: 3D Mixup / CutMix =====\nimport importlib\nimport my_implementation.chapter6_4_mixup_cutmix as chapter6_4_mixup_cutmix\n\nchapter6_4_mixup_cutmix = importlib.reload(chapter6_4_mixup_cutmix)\nMixup = chapter6_4_mixup_cutmix.Mixup\nCutmixSimple = chapter6_4_mixup_cutmix.CutmixSimple\nrun_section_6_4_assertions = chapter6_4_mixup_cutmix.run_section_6_4_assertions\n\n# Run section 6-4 assertions from the externalized implementation module.\nrun_section_6_4_assertions()","metadata":{},"outputs":[],"execution_count":null},{"id":"b805f81f","cell_type":"markdown","source":"## 7. 推論・後処理・評価\n\n\nWBS 5-1として、`run_eval`（sliding window 推論）を notebook から実行できる状態にします。\n加えて、train/valid を分けて loss と最小評価指標（precision, recall, fbeta）を比較します。","metadata":{}},{"id":"c1654eba","cell_type":"code","source":"# ===== 5-1: run_eval (sliding window inference) =====\nimport importlib\nfrom types import SimpleNamespace\n\nimport torch\nimport my_implementation.chapter7_1_eval as chapter7_1_eval\n\nchapter7_1_eval = importlib.reload(chapter7_1_eval)\nrun_eval = chapter7_1_eval.run_eval\nrun_section_5_1_assertions = chapter7_1_eval.run_section_5_1_assertions\n\n# Section 5-1 self-check from externalized implementation.\nrun_section_5_1_assertions()\n\n# Optional: run_eval on current val_loader with a tiny eval model.\n# You can swap this with your Net-based model once train/eval loop wiring is ready.\ntiny_eval_model = chapter7_1_eval._TinyEvalModel().to(device if 'device' in globals() else 'cpu').eval()\neval_cfg = SimpleNamespace(\n device=torch.device(device) if 'device' in globals() else torch.device('cpu'),\n roi_size=(DEPTH, IMG_SIZE, IMG_SIZE),\n overlap=0.25,\n sw_batch_size=1,\n mixed_precision=False,\n disable_tqdm=True,\n )\n\n# Run only when chapter4 val_loader is ready.\nif 'val_loader' in globals():\n eval_out = run_eval(tiny_eval_model, val_loader, eval_cfg, val_metrics={'val': {}})\n print('[5-1/notebook_eval_samples]', eval_out['val'].get('n_samples'))\n print('[5-1/notebook_eval_loss]', eval_out['val'].get('loss'))\nelse:\n print('[INFO] val_loader is not defined yet. Run Chapter 4 cells first.')","metadata":{},"outputs":[],"execution_count":null},{"id":"38180f12","cell_type":"markdown","source":"### 7-2. Postprocess (NMS)\n\n\nWBS 5-2として、`nms_2d` / `nms_3d` と Top-kピーク抽出を実装し、5-1の推論出力に後処理を適用します。","metadata":{}},{"id":"a06e5feb","cell_type":"code","source":"# ===== 5-2: postprocess utils (NMS + top-k peak extraction) =====\nimport importlib\nimport torch\nimport my_implementation.chapter7_2_postprocess as chapter7_2_postprocess\n\nchapter7_2_postprocess = importlib.reload(chapter7_2_postprocess)\nnms_2d = chapter7_2_postprocess.nms_2d\nnms_3d = chapter7_2_postprocess.nms_3d\nextract_topk_after_nms_3d = chapter7_2_postprocess.extract_topk_after_nms_3d\nrun_section_7_2_assertions = chapter7_2_postprocess.run_section_7_2_assertions\nbuild_postprocess_cfg = chapter7_2_postprocess.build_postprocess_cfg\n\n# Section 5-2 self-check\nrun_section_7_2_assertions()\n\n# Example: apply NMS postprocess to 5-1 eval output\npost_cfg = build_postprocess_cfg(nms_radius=2, topk=1, score_threshold=0.0)\n\nif 'eval_out' in globals() and 'max_preds' in eval_out and len(eval_out['max_preds']) > 0:\n # Tiny demo map around first prediction coordinate\n first = eval_out['max_preds'][0]\n d = int(DEPTH) if 'DEPTH' in globals() else 16\n h = int(IMG_SIZE) if 'IMG_SIZE' in globals() else 96\n w = int(IMG_SIZE) if 'IMG_SIZE' in globals() else 96\n demo_scores = torch.zeros(d, h, w)\n z = min(max(int(first['z']), 0), d - 1)\n y = min(max(int(first['y']), 0), h - 1)\n x = min(max(int(first['x']), 0), w - 1)\n demo_scores[z, y, x] = float(first['prob'])\n if x + 1 < w:\n demo_scores[z, y, x + 1] = max(0.0, float(first['prob']) - 0.05)\n\n peaks = extract_topk_after_nms_3d(\n demo_scores,\n nms_radius=post_cfg.nms_radius,\n topk=post_cfg.topk,\n score_threshold=post_cfg.score_threshold,\n )\n print('[5-2/postprocess_peaks]', peaks)\nelse:\n print('[INFO] eval_out is not ready. Run Cell 40 (5-1) first.')","metadata":{},"outputs":[],"execution_count":null},{"id":"632fad1d","cell_type":"markdown","source":"### 7-3. Metric Scoring\n\n\nWBS 5-3として、`metric.py` の距離ベース判定 + precision/recall/fbeta のスコア計算を実装し、ノートブックから呼び出します。","metadata":{}},{"id":"74a5fc05","cell_type":"code","source":"# ===== 5-3: metric scoring (distance-based matching + F-beta) =====\nimport importlib\nimport pandas as pd\nimport my_implementation.chapter7_3_metric as chapter7_3_metric\n\nchapter7_3_metric = importlib.reload(chapter7_3_metric)\ndistance_metric = chapter7_3_metric.distance_metric\nbinary_classification_metrics = chapter7_3_metric.binary_classification_metrics\nscore = chapter7_3_metric.score\nrun_section_7_3_assertions = chapter7_3_metric.run_section_7_3_assertions\n\n# Section 5-3 self-check\nrun_section_7_3_assertions()\n\n# Example: compute score on tiny synthetic solution/submission\nsol_demo = pd.DataFrame(\n {\n 'tomo_id': [0, 1, 2, 3],\n 'Motor axis 0': [-1, 250, 100, 200],\n 'Motor axis 1': [-1, 250, 100, 200],\n 'Motor axis 2': [-1, 250, 100, 200],\n 'Voxel spacing': [10, 10, 10, 10],\n 'Has motor': [0, 1, 1, 1],\n }\n)\nsub_demo = pd.DataFrame(\n {\n 'tomo_id': [0, 1, 2, 3],\n 'Motor axis 0': [100, 251, 600, -1],\n 'Motor axis 1': [100, 251, 600, -1],\n 'Motor axis 2': [100, 251, 600, -1],\n }\n)\n\nmetric_out = score(sol_demo, sub_demo, min_radius=1000, beta=2.0)\nprint('[5-3/metric_demo]', metric_out)","metadata":{},"outputs":[],"execution_count":null},{"id":"799e2602","cell_type":"markdown","source":"### 7-4. Flip TTA\n\n\nWBS 5-4として、推論時のFlip TTA（反転推論のアンサンブル）を `run_eval` から有効化し、ON/OFFの差分を比較します。","metadata":{}},{"id":"5261a61a","cell_type":"code","source":"# ===== 5-4: Flip TTA in run_eval =====\nimport importlib\nfrom types import SimpleNamespace\n\nimport torch\nimport my_implementation.chapter7_1_eval as chapter7_1_eval\n\nchapter7_1_eval = importlib.reload(chapter7_1_eval)\nrun_eval = chapter7_1_eval.run_eval\n\n# Tiny model with predict(use_tta=...) path for explicit WBS 5-4 validation\ntta_model = chapter7_1_eval._TinyEvalModelWithPredict().to(device if 'device' in globals() else 'cpu').eval()\ntta_ds = chapter7_1_eval._TinyEvalDataset(n=4, d=12, h=32, w=32)\ntta_dl = torch.utils.data.DataLoader(tta_ds, batch_size=2, shuffle=False, num_workers=0)\n\ncfg_base = SimpleNamespace(\n device=torch.device(device) if 'device' in globals() else torch.device('cpu'),\n roi_size=(8, 16, 16),\n overlap=0.25,\n sw_batch_size=1,\n mixed_precision=False,\n disable_tqdm=True,\n)\n\ncfg_no_tta = SimpleNamespace(**vars(cfg_base), use_tta=False)\ncfg_tta = SimpleNamespace(**vars(cfg_base), use_tta=True, tta_flip_dims=((2,), (3,), (4,)))\n\nout_no_tta = run_eval(tta_model, tta_dl, cfg_no_tta, val_metrics={'val': {}})\nout_tta = run_eval(tta_model, tta_dl, cfg_tta, val_metrics={'val': {}})\n\nprint('[5-4/no_tta_used_tta]', out_no_tta['val'].get('used_tta'))\nprint('[5-4/tta_used_tta]', out_tta['val'].get('used_tta'))\nprint('[5-4/mean_max_prob_no_tta]', out_no_tta['val'].get('mean_max_prob'))\nprint('[5-4/mean_max_prob_tta]', out_tta['val'].get('mean_max_prob'))\nprint('[5-4/predict_calls]', int(tta_model.predict_calls))","metadata":{},"outputs":[],"execution_count":null},{"id":"cf056650","cell_type":"markdown","source":"## 8. 大規模化と運用（フェーズ6）\n\n\nWBS 6-1〜6-3として、AMP/DDP・マルチGPU起動・WandBロギングを notebook から検証可能な形で実装します。","metadata":{}},{"id":"e4875bb6","cell_type":"markdown","source":"### 8-1. AMP/DDP/Fold Loop (WBS 6-1)\n\n\nAMP混合精度学習、DDPの安全な有効化判定、fold分割ループを実装し、自己検証を実行します。","metadata":{}},{"id":"d2b8ac10","cell_type":"code","source":"# ===== 6-1: AMP + DDP gate + fold loop =====\nimport importlib\nfrom types import SimpleNamespace\n\nimport torch\nimport my_implementation.chapter8_1_amp_ddp_fold as chapter8_1_amp_ddp_fold\n\nchapter8_1_amp_ddp_fold = importlib.reload(chapter8_1_amp_ddp_fold)\nrun_fold_loop = chapter8_1_amp_ddp_fold.run_fold_loop\nrun_section_8_1_assertions = chapter8_1_amp_ddp_fold.run_section_8_1_assertions\n\n# Section 6-1 self-check\nrun_section_8_1_assertions()\n\n# Optional: explicit tiny fold-loop run\ncfg_6_1 = SimpleNamespace(\n device=torch.device(device) if 'device' in globals() else torch.device('cpu'),\n use_amp=True,\n use_ddp=False, # Notebook default: keep False unless process group is initialized.\n local_rank=0,\n fold_ids=[0, 1],\n n_folds=2,\n epochs=1,\n batch_size=3,\n num_workers=0,\n lr=1e-3,\n)\n\ntiny_ds_6_1 = chapter8_1_amp_ddp_fold._TinyFoldDataset(n=12, d=8, h=16, w=16)\nout_6_1 = run_fold_loop(\n model_factory=chapter8_1_amp_ddp_fold._TinyFoldModel,\n dataset=tiny_ds_6_1,\n cfg=cfg_6_1,\n )\n\nfor fold_id, rec in sorted(out_6_1.items()):\n row = rec['history'][-1]\n print(\n f\"[6-1/fold={fold_id}] train_loss={row['train']['loss']:.6f} \"\n f\"val_loss={row['val']['loss']:.6f} ddp={rec['ddp']['enabled']}\"\n )","metadata":{},"outputs":[],"execution_count":null},{"id":"8c529a19","cell_type":"markdown","source":"### 8-2. Multi-GPU Run Script (WBS 6-2)\n\n\nWBS 6-2として、`run.sh` のマルチGPU起動コマンドをテンプレート化し、`torchrun` ベースで安全に生成・検証できるようにします。","metadata":{}},{"id":"ba0ef287","cell_type":"code","source":"# ===== 6-2: run.sh multi-GPU launcher =====\nimport importlib\nfrom pathlib import Path\nfrom types import SimpleNamespace\n\nimport my_implementation.chapter8_2_run_sh as chapter8_2_run_sh\n\nchapter8_2_run_sh = importlib.reload(chapter8_2_run_sh)\nbuild_torchrun_command = chapter8_2_run_sh.build_torchrun_command\nbuild_run_sh_text = chapter8_2_run_sh.build_run_sh_text\nvalidate_run_sh_text = chapter8_2_run_sh.validate_run_sh_text\nrun_section_8_2_assertions = chapter8_2_run_sh.run_section_8_2_assertions\n\n# Section 6-2 self-check\nrun_section_8_2_assertions()\n\n# Example: generate run.sh text for 2-GPU local training and save it under tmp/.\ncfg_6_2 = SimpleNamespace(\n gpus=\"0,1\",\n nproc_per_node=2,\n nnodes=1,\n node_rank=0,\n master_addr=\"127.0.0.1\",\n master_port=29510,\n train_script=\"train.py\",\n config=\"r3d200\",\n overrides={\n \"epochs\": 5,\n \"fold\": 999,\n \"save_weights\": True,\n },\n)\n\ncmd_6_2 = build_torchrun_command(cfg_6_2)\nrun_sh_6_2 = build_run_sh_text(cfg_6_2)\ncheck_6_2 = validate_run_sh_text(run_sh_6_2)\n\ntmp_dir = Path(\"tmp\")\ntmp_dir.mkdir(parents=True, exist_ok=True)\nrun_sh_path = tmp_dir / \"run_6_2.sh\"\nrun_sh_path.write_text(run_sh_6_2, encoding=\"utf-8\")\n\nprint(\"[6-2/cmd]\", cmd_6_2)\nprint(\"[6-2/valid]\", check_6_2[\"is_valid\"])\nprint(\"[6-2/path]\", str(run_sh_path))\nprint(\"[6-2/content]\\n\" + run_sh_6_2)","metadata":{},"outputs":[],"execution_count":null},{"id":"b2c265a5","cell_type":"markdown","source":"### 8-3. WandB Logger (WBS 6-3)\n\n\nWBS 6-3として、main process のみが WandB を初期化する安全な logger を実装し、wandb 未導入環境では NoLogger にフォールバックできるようにします。","metadata":{}},{"id":"9bdf2ad1","cell_type":"code","source":"# ===== 6-3: WandB logger =====\nimport importlib\nfrom types import SimpleNamespace\n\nimport my_implementation.chapter8_3_wandb_logger as chapter8_3_wandb_logger\n\nchapter8_3_wandb_logger = importlib.reload(chapter8_3_wandb_logger)\nBaseLogger = chapter8_3_wandb_logger.BaseLogger\nNoLogger = chapter8_3_wandb_logger.NoLogger\nWandbLogger = chapter8_3_wandb_logger.WandbLogger\nflatten_config = chapter8_3_wandb_logger.flatten_config\nget_logger = chapter8_3_wandb_logger.get_logger\nrun_section_8_3_assertions = chapter8_3_wandb_logger.run_section_8_3_assertions\n\n# Section 6-3 self-check\nrun_section_8_3_assertions()\n\n# Notebook-safe demo: wandb module is optional and automatically falls back to NoLogger.\ncfg_6_3 = SimpleNamespace(\n project=\"byu-kaggle\",\n logger=\"wandb\",\n local_rank=0,\n run_name=\"phase6-wbs6-3-demo\",\n group=\"fold-0\",\n fold=0,\n epochs=1,\n nested={\"batch_size\": BATCH_SIZE if 'BATCH_SIZE' in globals() else 2},\n)\n\nlogger_6_3 = get_logger(cfg_6_3)\nlogger_6_3.log({\"demo/loss\": 0.1234, \"demo/fbeta\": 0.5678}, commit=True)\nlogger_6_3.finish()\n\nprint(\"[6-3/logger_type]\", type(logger_6_3).__name__)\nprint(\"[6-3/hparams_keys]\", len(getattr(logger_6_3, \"hparams\", {})))","metadata":{},"outputs":[],"execution_count":null},{"id":"7d5d2831","cell_type":"markdown","source":"### 8-4. CFG/Torch Utilities (WBS 6-4)\n\n\nWBS 6-4として、設定上書き（CFG update）と PyTorch 実行補助（seed固定・device移送・state_dict CPU化・パラメータ数算出）をユーティリティとして整備します。","metadata":{}},{"id":"e13a6ef3","cell_type":"code","source":"# ===== 6-4: cfg/torch utilities =====\nimport importlib\nfrom types import SimpleNamespace\n\nimport torch\nimport my_implementation.chapter8_4_cfg_torch_utils as chapter8_4_cfg_torch_utils\n\nchapter8_4_cfg_torch_utils = importlib.reload(chapter8_4_cfg_torch_utils)\nupdate_cfg = chapter8_4_cfg_torch_utils.update_cfg\nseed_everything = chapter8_4_cfg_torch_utils.seed_everything\nresolve_device = chapter8_4_cfg_torch_utils.resolve_device\nto_device = chapter8_4_cfg_torch_utils.to_device\nstate_dict_to_cpu = chapter8_4_cfg_torch_utils.state_dict_to_cpu\ncount_trainable_params = chapter8_4_cfg_torch_utils.count_trainable_params\nrun_section_8_4_assertions = chapter8_4_cfg_torch_utils.run_section_8_4_assertions\n\n# Section 6-4 self-check\nrun_section_8_4_assertions()\n\n# Notebook demo: cfg override + device helper + parameter count.\ncfg_6_4 = SimpleNamespace(\n lr=1e-3,\n epochs=2,\n use_amp=False,\n model=SimpleNamespace(depth=16, name=\"tiny\"),\n)\nupdate_cfg(cfg_6_4, {\"lr\": \"0.0005\", \"epochs\": \"5\", \"model.depth\": \"24\", \"use_amp\": \"True\"})\n\ndevice_6_4 = resolve_device(\"cpu\")\nseed_everything(42, deterministic=True)\ndemo_batch_6_4 = {\"input\": torch.randn(1, 1, 8, 16, 16), \"meta\": {\"name\": \"demo\"}}\ndemo_batch_6_4 = to_device(demo_batch_6_4, device_6_4)\n\ndemo_model_6_4 = torch.nn.Conv3d(1, 2, kernel_size=3, padding=1)\nn_params_6_4 = count_trainable_params(demo_model_6_4)\ncpu_state_6_4 = state_dict_to_cpu(demo_model_6_4.state_dict())\n\nprint(\"[6-4/cfg]\", {\"lr\": cfg_6_4.lr, \"epochs\": cfg_6_4.epochs, \"use_amp\": cfg_6_4.use_amp, \"depth\": cfg_6_4.model.depth})\nprint(\"[6-4/device]\", str(device_6_4))\nprint(\"[6-4/input_device]\", demo_batch_6_4[\"input\"].device.type)\nprint(\"[6-4/n_params]\", n_params_6_4)\nprint(\"[6-4/state_cpu]\", all((not torch.is_tensor(v)) or (v.device.type == \"cpu\") for v in cpu_state_6_4.values()))","metadata":{},"outputs":[],"execution_count":null},{"id":"5bff491a","cell_type":"markdown","source":"### 8-5. E2E Smoke Test (WBS 6-5)\n\n\nWBS 6-5として、小サブセットデータで train/valid を fold ループで実行し、loss が有限値で返ることを end-to-end で確認します。","metadata":{}},{"id":"91ee3429","cell_type":"code","source":"# ===== 6-5: e2e smoke test =====\nimport importlib\nfrom types import SimpleNamespace\n\nimport my_implementation.chapter8_5_e2e_smoke as chapter8_5_e2e_smoke\n\nchapter8_5_e2e_smoke = importlib.reload(chapter8_5_e2e_smoke)\nrun_e2e_smoke = chapter8_5_e2e_smoke.run_e2e_smoke\nrun_section_8_5_assertions = chapter8_5_e2e_smoke.run_section_8_5_assertions\n\n# Section 6-5 self-check\nrun_section_8_5_assertions()\n\n# Notebook demo: tiny end-to-end run (train + valid on small subset).\ncfg_6_5 = SimpleNamespace(\n seed=123,\n logger=\"none\",\n local_rank=0,\n project=\"byu-kaggle\",\n run_name=\"phase6-wbs6-5-notebook\",\n dataset=SimpleNamespace(n=10, d=8, h=16, w=16),\n train=SimpleNamespace(\n use_amp=True,\n use_ddp=False,\n fold_ids=[0, 1],\n n_folds=2,\n epochs=1,\n batch_size=2,\n num_workers=0,\n lr=1e-3,\n ),\n)\n\nout_6_5 = run_e2e_smoke(cfg=cfg_6_5)\nsummary_6_5 = out_6_5[\"summary\"]\nprint(\"[6-5/device]\", out_6_5[\"device\"])\nprint(\"[6-5/summary]\", summary_6_5)\nfor fold_id, rec in sorted(out_6_5[\"per_fold\"].items()):\n print(\n f\"[6-5/fold={fold_id}] train_loss={rec['train_loss']:.6f} \"\n f\"val_loss={rec['val_loss']:.6f} n_train={rec['n_train']} n_val={rec['n_val']}\"\n )","metadata":{},"outputs":[],"execution_count":null},{"id":"8755cbee","cell_type":"markdown","source":"## 9. 実データを用いた学習\n\n\nここでは、`train_data` で学習したモデルを `eval_data` で評価し、\n`folds_all.csv` を正解データとして tomo 単位の正解率（accuracy）を出力します。\n`processed_tomos` が存在すれば `.npy` を優先利用し、無ければ画像スライスを読み込みます。","metadata":{}},{"id":"feddd365","cell_type":"markdown","source":"### ローカル用","metadata":{}},{"id":"03c80926","cell_type":"code","source":"# ===== 9: train on train_data, evaluate on eval_data, report accuracy =====\nimport importlib\nfrom pathlib import Path\nfrom types import SimpleNamespace\n\nimport matplotlib.pyplot as plt\nimport pandas as pd\nimport torch\nimport torch.nn as nn\nfrom IPython.display import clear_output\nfrom torch.utils.data import DataLoader\n\nimport my_implementation.chapter4_dataset as chapter4_dataset\nimport my_implementation.chapter8_1_amp_ddp_fold as chapter8_1_amp_ddp_fold\n\nchapter4_dataset = importlib.reload(chapter4_dataset)\nchapter8_1_amp_ddp_fold = importlib.reload(chapter8_1_amp_ddp_fold)\nCustomDataset = chapter4_dataset.CustomDataset\ntrain_one_epoch_amp_9 = chapter8_1_amp_ddp_fold.train_one_epoch_amp\nrun_eval_simple_9 = chapter8_1_amp_ddp_fold.run_eval_simple\nresolve_device_9 = chapter8_1_amp_ddp_fold.resolve_device\n\ntrain_root_9 = Path('train_data')\neval_root_9 = Path('eval_data')\nlabel_csv_9 = Path('folds_all.csv')\n\ntrain_tomo_dirs_9 = sorted([p for p in train_root_9.iterdir() if p.is_dir()]) if train_root_9.exists() else []\neval_tomo_dirs_9 = sorted([p for p in eval_root_9.iterdir() if p.is_dir()]) if eval_root_9.exists() else []\n\nif len(train_tomo_dirs_9) < 1:\n print('[INFO] train_data に学習用 tomo が見つかりません。')\nelif len(eval_tomo_dirs_9) < 1:\n print('[INFO] eval_data に評価用 tomo が見つかりません。')\nelif not label_csv_9.exists():\n print('[INFO] folds_all.csv が見つかりません。正解率を計算できません。')\nelse:\n depth_9 = int(DEPTH) if 'DEPTH' in globals() else 16\n img_size_9 = int(IMG_SIZE) if 'IMG_SIZE' in globals() else 96\n label_radius_9 = int(LABEL_RADIUS) if 'LABEL_RADIUS' in globals() else 1\n batch_size_9 = int(BATCH_SIZE) if 'BATCH_SIZE' in globals() else 2\n num_workers_9 = int(NUM_WORKERS) if 'NUM_WORKERS' in globals() else 0\n epochs_9 = 20\n\n processed_root_9 = Path('processed_tomos')\n if not processed_root_9.exists():\n processed_root_9 = None\n\n train_ds_9 = CustomDataset(\n train_tomo_dirs_9,\n depth=depth_9,\n img_size=img_size_9,\n label_radius=label_radius_9,\n processed_root=processed_root_9,\n )\n eval_ds_9 = CustomDataset(\n eval_tomo_dirs_9,\n depth=depth_9,\n img_size=img_size_9,\n label_radius=label_radius_9,\n processed_root=processed_root_9,\n )\n\n train_loader_9 = DataLoader(train_ds_9, batch_size=batch_size_9, shuffle=True, num_workers=num_workers_9)\n eval_loader_9 = DataLoader(eval_ds_9, batch_size=batch_size_9, shuffle=False, num_workers=num_workers_9)\n\n cfg_9 = SimpleNamespace(\n device=torch.device('cuda' if torch.cuda.is_available() else 'cpu'),\n use_amp=True,\n )\n device_9 = resolve_device_9(cfg_9)\n\n model_name_9 = '_TinyFoldModel (fallback)'\n model_9 = chapter8_1_amp_ddp_fold._TinyFoldModel().to(device_9)\n if 'Net' in globals():\n try:\n model_9 = Net().to(device_9)\n model_name_9 = 'Net'\n except Exception as ex:\n print(f'[WARN] Net の初期化に失敗したため fallback を使用します: {ex}')\n\n optimizer_9 = torch.optim.Adam(model_9.parameters(), lr=1e-3)\n loss_fn_9 = nn.BCEWithLogitsLoss()\n\n history_9 = {'epoch': [], 'train_loss': [], 'eval_loss': []}\n best_eval_9 = float('inf')\n best_epoch_9 = 0\n\n for epoch_9 in range(1, epochs_9 + 1):\n train_out_9 = train_one_epoch_amp_9(\n model=model_9,\n loader=train_loader_9,\n optimizer=optimizer_9,\n loss_fn=loss_fn_9,\n cfg=cfg_9,\n )\n eval_out_9 = run_eval_simple_9(\n model=model_9,\n loader=eval_loader_9,\n loss_fn=loss_fn_9,\n cfg=cfg_9,\n )\n\n train_loss_9 = float(train_out_9['loss'])\n eval_loss_9 = float(eval_out_9['loss'])\n\n history_9['epoch'].append(epoch_9)\n history_9['train_loss'].append(train_loss_9)\n history_9['eval_loss'].append(eval_loss_9)\n\n if eval_loss_9 < best_eval_9:\n best_eval_9 = eval_loss_9\n best_epoch_9 = epoch_9\n\n progress_pct_9 = 100.0 * epoch_9 / epochs_9\n\n clear_output(wait=True)\n fig, ax = plt.subplots(1, 1, figsize=(7, 4))\n ax.plot(history_9['epoch'], history_9['train_loss'], marker='o', label='train_loss')\n ax.plot(history_9['epoch'], history_9['eval_loss'], marker='s', label='eval_loss')\n ax.set_xlabel('Epoch')\n ax.set_ylabel('Loss')\n ax.set_title('Chapter 9 Progress (train_data -> eval_data)')\n ax.grid(True, alpha=0.3)\n ax.legend(loc='best')\n plt.show()\n\n print(f\"[9/progress] epoch={epoch_9}/{epochs_9} ({progress_pct_9:.1f}%)\")\n print(f\"[9/current] train_loss={train_loss_9:.6f} eval_loss={eval_loss_9:.6f}\")\n print(f\"[9/best] epoch={best_epoch_9} best_eval_loss={best_eval_9:.6f}\")\n\n # ----- tomo単位の正解率（Has motor分類） -----\n label_df_9 = pd.read_csv(label_csv_9)\n required_cols_9 = ['tomo_id', 'Motor axis 0', 'Motor axis 1', 'Motor axis 2']\n for c in required_cols_9:\n if c not in label_df_9.columns:\n raise ValueError(f\"folds_all.csv に必要列がありません: {c}\")\n\n gt_has_motor_9 = {}\n for tomo_id_9, g_9 in label_df_9.groupby('tomo_id'):\n arr_9 = g_9[['Motor axis 0', 'Motor axis 1', 'Motor axis 2']].to_numpy(dtype=float)\n has_motor_9 = bool(((arr_9 >= 0).all(axis=1)).any())\n gt_has_motor_9[str(tomo_id_9)] = int(has_motor_9)\n\n model_9.eval()\n pred_records_9 = []\n with torch.no_grad():\n for batch_9 in eval_loader_9:\n x_9 = batch_9['input'].to(device_9).float()\n logits_9 = model_9(x_9)\n probs_9 = torch.sigmoid(logits_9).amax(dim=(1, 2, 3, 4))\n pred_has_motor_9 = (probs_9 >= 0.5).long().cpu().tolist()\n\n tomo_ids_9 = batch_9['tomo_id']\n if not isinstance(tomo_ids_9, list):\n tomo_ids_9 = list(tomo_ids_9)\n\n for tomo_id_9, pred_9, prob_9 in zip(tomo_ids_9, pred_has_motor_9, probs_9.cpu().tolist()):\n pred_records_9.append(\n {\n 'tomo_id': str(tomo_id_9),\n 'pred_has_motor': int(pred_9),\n 'pred_max_prob': float(prob_9),\n 'gt_has_motor': gt_has_motor_9.get(str(tomo_id_9), None),\n }\n )\n\n pred_df_9 = pd.DataFrame(pred_records_9)\n pred_df_9_valid = pred_df_9[pred_df_9['gt_has_motor'].notna()].copy()\n\n if len(pred_df_9_valid) == 0:\n print('[WARN] eval_data の tomo_id と folds_all.csv の対応が取れず、正解率を算出できませんでした。')\n accuracy_9 = float('nan')\n else:\n pred_df_9_valid['gt_has_motor'] = pred_df_9_valid['gt_has_motor'].astype(int)\n accuracy_9 = float((pred_df_9_valid['pred_has_motor'] == pred_df_9_valid['gt_has_motor']).mean())\n\n print('[9/model]', model_name_9)\n print('[9/device]', str(device_9))\n print('[9/n_train_samples]', len(train_ds_9))\n print('[9/n_eval_samples]', len(eval_ds_9))\n print('[9/final_train_loss]', history_9['train_loss'][-1])\n print('[9/final_eval_loss]', history_9['eval_loss'][-1])\n print('[9/accuracy]', accuracy_9)\n print('[9/eval_preview]')\n display(pred_df_9.head(10))","metadata":{},"outputs":[],"execution_count":null},{"id":"e6cd0dd9","cell_type":"markdown","source":"### kaggle用（GPUを用いた学習）","metadata":{}},{"id":"885ffab8","cell_type":"code","source":"# ===== 9 (kaggle): train/valid split from official train data =====\nimport importlib\nimport importlib.machinery\nimport importlib.util\nfrom pathlib import Path\nfrom types import SimpleNamespace\n\nimport matplotlib.pyplot as plt\nimport pandas as pd\nimport torch\nimport torch.nn as nn\nfrom IPython.display import clear_output\nfrom torch.utils.data import DataLoader\n\nimport sys\n\n# 0) Ensure repository exists in Kaggle working directory before training.\nrepo_dir_9 = Path('/kaggle/working/BYU-competition')\nif not repo_dir_9.exists():\n print('[9/repo_setup] cloning repository to /kaggle/working ...')\n get_ipython().system('GIT_SSH_COMMAND=\"ssh -o StrictHostKeyChecking=no\" git clone https://github.com/UgoKo-Rubeast/BYU-competition.git')\nelse:\n print('[9/repo_setup] repository already exists, skip clone')\n\n\ndef _add_to_syspath_9(path_obj):\n p = Path(path_obj).resolve()\n ps = str(p)\n if ps not in sys.path:\n sys.path.insert(0, ps)\n\n\n# 1) Add high-confidence candidate repo roots first.\ncandidate_roots_9 = [\n Path('/kaggle/working/BYU-competition'),\n Path('/kaggle/working'),\n Path.cwd(),\n Path.cwd() / 'BYU-competition',\n]\nfor p_9 in candidate_roots_9:\n _add_to_syspath_9(p_9)\n\n# 2) Search for my_implementation directory and add its parent to sys.path.\ndiscovered_repo_roots_9 = []\nsearch_bases_9 = [Path('/kaggle/working'), Path('/kaggle/input')]\nsearch_patterns_9 = [\n 'my_implementation',\n '*/my_implementation',\n '*/*/my_implementation',\n '*/*/*/my_implementation',\n]\nfor base_9 in search_bases_9:\n if not base_9.exists():\n continue\n for pat_9 in search_patterns_9:\n for pkg_dir_9 in base_9.glob(pat_9):\n if pkg_dir_9.is_dir() and pkg_dir_9.name == 'my_implementation':\n repo_root_9 = pkg_dir_9.parent.resolve()\n discovered_repo_roots_9.append(str(repo_root_9))\n _add_to_syspath_9(repo_root_9)\n\n# 3) Force canonical Kaggle working repo root.\nexpected_repo_root_9 = Path('/kaggle/working/BYU-competition').resolve()\nexpected_pkg_dir_9 = expected_repo_root_9 / 'my_implementation'\n_add_to_syspath_9(expected_repo_root_9)\n\n# 4) Clear stale module/importer cache to avoid false-negative resolution.\nfor k_9 in list(sys.modules.keys()):\n if k_9 == 'my_implementation' or k_9.startswith('my_implementation.'):\n sys.modules.pop(k_9, None)\n\nimportlib.invalidate_caches()\nsys.path_importer_cache.pop(str(expected_repo_root_9), None)\n\n# 5) Resolve package spec with normal resolver, then explicit PathFinder fallback.\nspec_9 = importlib.util.find_spec('my_implementation')\nif spec_9 is None:\n spec_9 = importlib.machinery.PathFinder.find_spec('my_implementation', [str(expected_repo_root_9)])\n\nprint('[9/cwd]', Path.cwd())\nprint('[9/discovered_repo_roots]', discovered_repo_roots_9[:8])\nprint('[9/expected_repo_root_exists]', expected_repo_root_9.exists())\nprint('[9/expected_pkg_exists]', expected_pkg_dir_9.exists())\nprint('[9/expected_init_exists]', (expected_pkg_dir_9 / '__init__.py').exists())\nprint('[9/spec_found]', spec_9 is not None)\nprint('[9/sys_path_head]', sys.path[:8])\n\nif spec_9 is None:\n raise ModuleNotFoundError(\n 'my_implementation import failed after cache reset. '\n 'Confirm /kaggle/working/BYU-competition/my_implementation exists and contains __init__.py, '\n 'then restart the notebook kernel and run this cell again.'\n )\n\nimport my_implementation.chapter4_dataset as chapter4_dataset\nimport my_implementation.chapter8_1_amp_ddp_fold as chapter8_1_amp_ddp_fold\n\nchapter4_dataset = importlib.reload(chapter4_dataset)\nchapter8_1_amp_ddp_fold = importlib.reload(chapter8_1_amp_ddp_fold)\nCustomDataset = chapter4_dataset.CustomDataset\ntrain_one_epoch_amp_9 = chapter8_1_amp_ddp_fold.train_one_epoch_amp\nrun_eval_simple_9 = chapter8_1_amp_ddp_fold.run_eval_simple\nresolve_device_9 = chapter8_1_amp_ddp_fold.resolve_device\n\ntrain_root_9 = Path('/kaggle/input/competitions/byu-locating-bacterial-flagellar-motors-2025/train')\nlabel_csv_9 = Path('/kaggle/input/competitions/byu-locating-bacterial-flagellar-motors-2025/train_labels.csv')\n\nall_tomo_dirs_9 = sorted([p for p in train_root_9.iterdir() if p.is_dir()]) if train_root_9.exists() else []\n\nif len(all_tomo_dirs_9) < 40:\n print(f'[INFO] train directory has fewer than 40 tomos: {len(all_tomo_dirs_9)}')\nelif not label_csv_9.exists():\n print('[INFO] train_labels.csv was not found; accuracy cannot be computed.')\nelse:\n # ---- random split: 30 train + 10 valid ----\n split_seed_9 = 42\n g_9 = torch.Generator().manual_seed(split_seed_9)\n perm_9 = torch.randperm(len(all_tomo_dirs_9), generator=g_9).tolist()\n selected_40_9 = [all_tomo_dirs_9[i] for i in perm_9[:40]]\n train_tomo_dirs_9 = selected_40_9[:30]\n val_tomo_dirs_9 = selected_40_9[30:40]\n\n depth_9 = int(DEPTH) if 'DEPTH' in globals() else 16\n img_size_9 = int(IMG_SIZE) if 'IMG_SIZE' in globals() else 96\n label_radius_9 = int(LABEL_RADIUS) if 'LABEL_RADIUS' in globals() else 1\n batch_size_9 = int(BATCH_SIZE) if 'BATCH_SIZE' in globals() else 2\n num_workers_9 = int(NUM_WORKERS) if 'NUM_WORKERS' in globals() else 0\n epochs_9 = 20\n\n label_df_9 = pd.read_csv(label_csv_9)\n required_cols_9 = ['tomo_id', 'Motor axis 0', 'Motor axis 1', 'Motor axis 2']\n for c in required_cols_9:\n if c not in label_df_9.columns:\n raise ValueError(f'train_labels.csv is missing required column: {c}')\n\n def _normalize_tomo_id_9(x):\n s = str(x)\n if s.startswith('tomo_'):\n s = s[5:]\n return s\n\n gt_has_motor_9 = {}\n for tomo_id_9, g_9 in label_df_9.groupby('tomo_id'):\n arr_9 = g_9[['Motor axis 0', 'Motor axis 1', 'Motor axis 2']].to_numpy(dtype=float)\n has_motor_9 = bool(((arr_9 >= 0).all(axis=1)).any())\n gt_has_motor_9[_normalize_tomo_id_9(tomo_id_9)] = int(has_motor_9)\n\n train_ds_9 = CustomDataset(\n train_tomo_dirs_9,\n depth=depth_9,\n img_size=img_size_9,\n label_radius=label_radius_9,\n processed_root=None,\n )\n val_ds_9 = CustomDataset(\n val_tomo_dirs_9,\n depth=depth_9,\n img_size=img_size_9,\n label_radius=label_radius_9,\n processed_root=None,\n )\n\n train_loader_9 = DataLoader(train_ds_9, batch_size=batch_size_9, shuffle=True, num_workers=num_workers_9)\n val_loader_9 = DataLoader(val_ds_9, batch_size=batch_size_9, shuffle=False, num_workers=num_workers_9)\n\n cfg_9 = SimpleNamespace(\n device=torch.device('cuda' if torch.cuda.is_available() else 'cpu'),\n use_amp=True,\n )\n device_9 = resolve_device_9(cfg_9)\n\n # If CUDA exists but training device is not CUDA, force-switch to GPU.\n if torch.cuda.is_available() and getattr(device_9, 'type', 'cpu') != 'cuda':\n device_9 = torch.device('cuda')\n cfg_9.device = device_9\n print('[9/device_override] GPU available -> force using CUDA')\n elif not torch.cuda.is_available():\n print('[9/device_info] CUDA is not available. Training runs on CPU.')\n\n # ---- explicit GPU/runtime diagnostics for Kaggle logs ----\n cuda_available_9 = torch.cuda.is_available()\n gpu_count_9 = torch.cuda.device_count()\n print('[9/cuda_available]', cuda_available_9)\n print('[9/gpu_count]', gpu_count_9)\n print('[9/torch_cuda_version]', torch.version.cuda)\n print('[9/selected_device]', str(device_9))\n print('[9/amp_enabled]', bool(cfg_9.use_amp))\n\n if cuda_available_9:\n cuda_idx_9 = device_9.index if getattr(device_9, 'index', None) is not None else torch.cuda.current_device()\n print('[9/gpu_name]', torch.cuda.get_device_name(cuda_idx_9))\n print('[9/gpu_capability]', torch.cuda.get_device_capability(cuda_idx_9))\n print('[9/cudnn_enabled]', torch.backends.cudnn.enabled)\n\n model_name_9 = '_TinyFoldModel (fallback)'\n model_9 = chapter8_1_amp_ddp_fold._TinyFoldModel().to(device_9)\n if 'Net' in globals():\n try:\n model_9 = Net().to(device_9)\n model_name_9 = 'Net'\n except Exception as ex:\n print(f'[WARN] Net initialization failed; using fallback: {ex}')\n\n first_param_device_9 = next(model_9.parameters()).device\n print('[9/model_param_device]', str(first_param_device_9))\n\n # One-step runtime proof: input and output tensors are on the selected device.\n sanity_batch_9 = next(iter(train_loader_9))\n sanity_x_9 = sanity_batch_9['input'][:1].to(device_9).float()\n with torch.no_grad():\n sanity_y_9 = model_9(sanity_x_9)\n print('[9/runtime_input_device]', str(sanity_x_9.device))\n print('[9/runtime_output_device]', str(sanity_y_9.device))\n\n if device_9.type == 'cuda':\n print('[9/cuda_mem_allocated_mb]', round(torch.cuda.memory_allocated(device_9) / (1024 ** 2), 2))\n print('[9/cuda_mem_reserved_mb]', round(torch.cuda.memory_reserved(device_9) / (1024 ** 2), 2))\n\n def _compute_tomo_accuracy_on_loader_9(model, loader, device, gt_map):\n pred_records = []\n model.eval()\n with torch.no_grad():\n for batch in loader:\n x = batch['input'].to(device).float()\n logits = model(x)\n probs = torch.sigmoid(logits).amax(dim=(1, 2, 3, 4))\n pred_has_motor = (probs >= 0.5).long().cpu().tolist()\n\n tomo_ids = batch['tomo_id']\n if not isinstance(tomo_ids, list):\n tomo_ids = list(tomo_ids)\n\n for tomo_id, pred, prob in zip(tomo_ids, pred_has_motor, probs.cpu().tolist()):\n norm_id = _normalize_tomo_id_9(tomo_id)\n pred_records.append(\n {\n 'tomo_id': str(tomo_id),\n 'norm_tomo_id': norm_id,\n 'pred_has_motor': int(pred),\n 'pred_max_prob': float(prob),\n 'gt_has_motor': gt_map.get(norm_id, None),\n }\n )\n\n pred_df = pd.DataFrame(pred_records)\n pred_df_valid = pred_df[pred_df['gt_has_motor'].notna()].copy()\n\n if len(pred_df_valid) == 0:\n return float('nan'), pred_df, 0\n\n pred_df_valid['gt_has_motor'] = pred_df_valid['gt_has_motor'].astype(int)\n acc = float((pred_df_valid['pred_has_motor'] == pred_df_valid['gt_has_motor']).mean())\n return acc, pred_df, len(pred_df_valid)\n\n optimizer_9 = torch.optim.Adam(model_9.parameters(), lr=1e-3)\n loss_fn_9 = nn.BCEWithLogitsLoss()\n\n history_9 = {'epoch': [], 'train_loss': [], 'val_loss': [], 'val_acc': []}\n best_val_9 = float('inf')\n best_epoch_9 = 0\n\n for epoch_9 in range(1, epochs_9 + 1):\n train_out_9 = train_one_epoch_amp_9(\n model=model_9,\n loader=train_loader_9,\n optimizer=optimizer_9,\n loss_fn=loss_fn_9,\n cfg=cfg_9,\n )\n val_out_9 = run_eval_simple_9(\n model=model_9,\n loader=val_loader_9,\n loss_fn=loss_fn_9,\n cfg=cfg_9,\n )\n\n val_acc_9, _, n_eval_match_9 = _compute_tomo_accuracy_on_loader_9(\n model=model_9,\n loader=val_loader_9,\n device=device_9,\n gt_map=gt_has_motor_9,\n )\n\n train_loss_9 = float(train_out_9['loss'])\n val_loss_9 = float(val_out_9['loss'])\n\n history_9['epoch'].append(epoch_9)\n history_9['train_loss'].append(train_loss_9)\n history_9['val_loss'].append(val_loss_9)\n history_9['val_acc'].append(val_acc_9)\n\n if val_loss_9 < best_val_9:\n best_val_9 = val_loss_9\n best_epoch_9 = epoch_9\n\n progress_pct_9 = 100.0 * epoch_9 / epochs_9\n\n clear_output(wait=True)\n fig, axes = plt.subplots(1, 2, figsize=(12, 4))\n\n axes[0].plot(history_9['epoch'], history_9['train_loss'], marker='o', label='train_loss')\n axes[0].plot(history_9['epoch'], history_9['val_loss'], marker='s', label='val_loss')\n axes[0].set_xlabel('Epoch')\n axes[0].set_ylabel('Loss')\n axes[0].set_title('Chapter 9 Kaggle Loss')\n axes[0].grid(True, alpha=0.3)\n axes[0].legend(loc='best')\n\n axes[1].plot(history_9['epoch'], history_9['val_acc'], marker='^', color='tab:green', label='val_accuracy')\n axes[1].set_xlabel('Epoch')\n axes[1].set_ylabel('Accuracy')\n axes[1].set_ylim(0.0, 1.0)\n axes[1].set_title('Chapter 9 Kaggle Accuracy')\n axes[1].grid(True, alpha=0.3)\n axes[1].legend(loc='best')\n\n plt.tight_layout()\n plt.show()\n\n print(f'[9/progress] epoch={epoch_9}/{epochs_9} ({progress_pct_9:.1f}%)')\n print(f'[9/current] train_loss={train_loss_9:.6f} val_loss={val_loss_9:.6f} val_acc={val_acc_9:.6f}')\n print(f'[9/best] epoch={best_epoch_9} best_val_loss={best_val_9:.6f}')\n print(f'[9/acc_eval_matched_tomos] {n_eval_match_9}')\n print(f'[9/train_device_epoch] {device_9}')\n\n # ---- final tomo-level accuracy on validation split ----\n accuracy_9, pred_df_9, n_eval_match_final_9 = _compute_tomo_accuracy_on_loader_9(\n model=model_9,\n loader=val_loader_9,\n device=device_9,\n gt_map=gt_has_motor_9,\n )\n\n if n_eval_match_final_9 == 0:\n print('[WARN] Could not align validation tomo_ids with train_labels.csv; accuracy was not computed.')\n\n print('[9/model]', model_name_9)\n print('[9/device]', str(device_9))\n print('[9/train_root]', str(train_root_9))\n print('[9/labels]', str(label_csv_9))\n print('[9/split_seed]', split_seed_9)\n print('[9/epochs]', epochs_9)\n print('[9/n_all_tomos]', len(all_tomo_dirs_9))\n print('[9/n_train_samples]', len(train_ds_9))\n print('[9/n_val_samples]', len(val_ds_9))\n print('[9/final_train_loss]', history_9['train_loss'][-1])\n print('[9/final_val_loss]', history_9['val_loss'][-1])\n print('[9/final_val_acc_from_curve]', history_9['val_acc'][-1])\n print('[9/accuracy]', accuracy_9)\n print('[9/val_preview]')\n display(pred_df_9.head(10))","metadata":{},"outputs":[],"execution_count":null},{"id":"ce189386","cell_type":"markdown","source":"### GPU使用可能か確認","metadata":{}},{"id":"ae2f5f95","cell_type":"code","source":"import torch, subprocess\n\nprint(\"torch.cuda.is_available():\", torch.cuda.is_available())\nprint(\"torch.cuda.device_count():\", torch.cuda.device_count())\n\nif torch.cuda.is_available():\n print(\"device:\", torch.cuda.get_device_name(0))\n print(\"cuda version (torch):\", torch.version.cuda)\nelse:\n print(\"GPU is not available from PyTorch\")\n\n# nvidia-smi でも確認\ntry:\n out = subprocess.check_output([\"nvidia-smi\"], text=True)\n print(\"\\n[nvidia-smi]\")\n print(out[:1200]) # 長すぎるので先頭だけ表示\nexcept Exception as e:\n print(\"nvidia-smi failed:\", e)","metadata":{},"outputs":[],"execution_count":null}]}