{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.x"}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"fd2a82be-413f-48da-b729-2962a702bed1","cell_type":"markdown","source":"# APTOS 2019 — Offline Checkpoint XAI Analysis (Kaggle)\n\nNotebook ini menjalankan **inference dan XAI tanpa retraining** pada checkpoint ekspor APTOS v8. Model awal adalah model C / SE-ResNeXt-50 fold 0. Grad-CAM dan Grad-CAM++ diterapkan hanya pada sampel kecil yang representatif. Heatmap menunjukkan area yang memengaruhi keputusan model; heatmap bukan bukti diagnosis atau bukti lokasi lesi klinis.\n","metadata":{}},{"id":"aa93419f-55c2-4997-be6b-f1ed100ffc7a","cell_type":"markdown","source":"## 1. Environment and Path Setup\n\nSemua output ditulis ke `/kaggle/working/outputs`; folder `/kaggle/input` hanya dibaca.","metadata":{}},{"id":"da0e2a72-7198-47c7-bc5a-e996c67e1f18","cell_type":"code","source":"from pathlib import Path\nimport os\nimport random\n\nIN_KAGGLE = Path('/kaggle').exists()\nif not IN_KAGGLE:\n    raise EnvironmentError('Notebook ini ditujukan untuk Kaggle. Folder /kaggle tidak ditemukan.')\n\nINPUT_ROOT = Path('/kaggle/input')\nWORKING_ROOT = Path('/kaggle/working')\nDEFAULT_DATA_ROOT = INPUT_ROOT / 'aptos2019-blindness-detection'\n\ndef discover_aptos_root():\n    if (DEFAULT_DATA_ROOT / 'train.csv').is_file() and (DEFAULT_DATA_ROOT / 'train_images').is_dir():\n        return DEFAULT_DATA_ROOT\n    candidates = []\n    for train_csv in INPUT_ROOT.rglob('train.csv'):\n        root = train_csv.parent\n        if (root / 'train_images').is_dir():\n            candidates.append(root)\n    if len(candidates) == 1:\n        return candidates[0]\n    raise FileNotFoundError(\n        'APTOS input tidak ditemukan. Harapkan train.csv dan train_images di ' + str(DEFAULT_DATA_ROOT) +\n        '. Kandidat yang ditemukan: ' + str([str(x) for x in candidates])\n    )\n\nDATA_ROOT = discover_aptos_root()\nTRAIN_IMAGES_DIR = DATA_ROOT / 'train_images'\nWORKING_WEIGHTS_ROOT = WORKING_ROOT / 'weights'\nWEIGHT_BUNDLE_DIR = WORKING_WEIGHTS_ROOT / 'aptos_v8_offline_weights'\nOUTPUT_ROOT = WORKING_ROOT / 'outputs'\n\nOUTPUT_DIRS = {\n    'reports': OUTPUT_ROOT / 'reports',\n    'predictions': OUTPUT_ROOT / 'predictions',\n    'gradcam': OUTPUT_ROOT / 'xai' / 'gradcam',\n    'gradcam_pp': OUTPUT_ROOT / 'xai' / 'gradcam_pp',\n    'overlays': OUTPUT_ROOT / 'xai' / 'overlays',\n    'per_class': OUTPUT_ROOT / 'xai' / 'per_class',\n    'correct': OUTPUT_ROOT / 'xai' / 'correct_predictions',\n    'incorrect': OUTPUT_ROOT / 'xai' / 'incorrect_predictions',\n    'metrics': OUTPUT_ROOT / 'xai' / 'metrics',\n    'comparison': OUTPUT_ROOT / 'figures' / 'xai_comparison',\n    'gallery': OUTPUT_ROOT / 'figures' / 'xai_gallery',\n    'confidence': OUTPUT_ROOT / 'figures' / 'confidence_drop',\n    'deletion': OUTPUT_ROOT / 'figures' / 'deletion_insertion_curves',\n}\nfor path in [WORKING_WEIGHTS_ROOT, *OUTPUT_DIRS.values()]:\n    path.mkdir(parents=True, exist_ok=True)\n\nCLASS_NAMES = ['No_DR', 'Mild', 'Moderate', 'Severe', 'Proliferative_DR']\nIMAGE_SIZE = 384\nIMAGENET_MEAN = (0.485, 0.456, 0.406)\nIMAGENET_STD = (0.229, 0.224, 0.225)\n# XAI_MODE keeps the fast single-checkpoint analysis available, while the\n# default fold-aware mode uses the checkpoint whose validation fold contains the image.\nXAI_MODE = 'fold_aware'  # options: 'single_fold_baseline', 'fold_aware'\nTARGET_MODEL_KEY = 'C'\nTARGET_FOLD = 0\nMAX_XAI_SAMPLES_PER_CLASS = 5\nMAX_XAI_PER_CLASS = MAX_XAI_SAMPLES_PER_CLASS\nCANDIDATES_PER_CLASS = 12\nMAX_EVALUATION_SAMPLES = 20\nMAX_CONSISTENCY_PER_CLASS = 2\nSEED = 42\nif XAI_MODE not in {'single_fold_baseline', 'fold_aware'}:\n    raise ValueError(f'Unsupported XAI_MODE: {XAI_MODE}')\nrandom.seed(SEED)\n\nprint('DATA_ROOT:', DATA_ROOT)\nprint('WORKING_ROOT:', WORKING_ROOT)\nprint('OUTPUT_ROOT:', OUTPUT_ROOT)\n","metadata":{},"outputs":[],"execution_count":null},{"id":"300047ef-6150-4347-b2a4-a68c02b26702","cell_type":"markdown","source":"## 2. Library Import\n\nTidak ada instalasi package atau download pretrained weight. Jika `timm` tidak tersedia, notebook berhenti dengan pesan yang jelas.","metadata":{}},{"id":"fb01849d-55f5-4164-9fbb-97304e49580d","cell_type":"code","source":"import json\nimport math\nimport shutil\nimport time\nimport zipfile\nfrom collections import defaultdict\nfrom datetime import datetime, timezone\n\nimport cv2\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.nn import Parameter\n\ntry:\n    import timm\nexcept ImportError as exc:\n    raise ImportError('Package timm tidak tersedia. Tambahkan timm ke Kaggle environment lalu jalankan ulang.') from exc\n\ntry:\n    import torchvision\nexcept ImportError as exc:\n    raise ImportError('Package torchvision tidak tersedia di Kaggle environment.') from exc\n\nDEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\nprint('torch:', torch.__version__)\nprint('torchvision:', torchvision.__version__)\nprint('timm:', timm.__version__)\nprint('opencv:', cv2.__version__)\nprint('device:', DEVICE)\nif DEVICE.type != 'cuda':\n    print('WARNING: GPU tidak tersedia. Notebook tetap dapat berjalan, tetapi Grad-CAM++ lebih lambat.')\n","metadata":{},"outputs":[],"execution_count":null},{"id":"db78cb62-7b2b-48d5-b999-f68b662d0dec","cell_type":"markdown","source":"## 3. Load APTOS Metadata\n\nKolom dan file citra diverifikasi sebelum model atau XAI dijalankan.","metadata":{}},{"id":"1b34a880-2fd4-4e69-a261-0ce99f665ec7","cell_type":"code","source":"train_csv = DATA_ROOT / 'train.csv'\ntrain_df = pd.read_csv(train_csv)\nrequired_columns = {'id_code', 'diagnosis'}\nmissing_columns = required_columns.difference(train_df.columns)\nif missing_columns:\n    raise ValueError('train.csv tidak memiliki kolom wajib: ' + str(sorted(missing_columns)))\n\ntrain_df['diagnosis'] = train_df['diagnosis'].astype(int)\nif not train_df['diagnosis'].between(0, 4).all():\n    raise ValueError('Diagnosis harus berada pada rentang 0 sampai 4.')\ntrain_df['image_path'] = train_df['id_code'].astype(str).map(lambda x: str(TRAIN_IMAGES_DIR / f'{x}.png'))\ntrain_df['class_name'] = train_df['diagnosis'].map(dict(enumerate(CLASS_NAMES)))\n\nprobe = train_df.sample(n=min(10, len(train_df)), random_state=SEED)\nmissing_images = [path for path in probe['image_path'] if not Path(path).is_file()]\nif missing_images:\n    raise FileNotFoundError('Sebagian contoh train image tidak ditemukan: ' + str(missing_images[:3]))\n\nclass_distribution = train_df['diagnosis'].value_counts().sort_index().rename(index=dict(enumerate(CLASS_NAMES)))\ndisplay(class_distribution.to_frame('count'))\nprint('Train rows:', len(train_df), '| verified image probes:', len(probe))\n\nfig, ax = plt.subplots(figsize=(7, 4))\nclass_distribution.plot.bar(ax=ax, color='#3274a1')\nax.set_title('APTOS 2019 training class distribution')\nax.set_ylabel('Number of images')\nplt.tight_layout()\nfig.savefig(OUTPUT_DIRS['comparison'] / 'class_distribution.png', dpi=200, bbox_inches='tight')\nplt.show()\n","metadata":{},"outputs":[],"execution_count":null},{"id":"53936254-36e0-4df4-8b6d-2e092076389d","cell_type":"markdown","source":"## 4. Load and Validate Exported Weights\n\nZIP dicari secara fleksibel pada Kaggle input. Bila input sudah berupa folder bundle, folder tersebut disalin ke working directory agar seluruh artefak output dan proses extraction tetap berada di area writeable.","metadata":{}},{"id":"e4f82e14-d020-42a8-b005-f925a282dc5d","cell_type":"code","source":"def safe_extract_zip(zip_path, target_dir):\n    target_dir = target_dir.resolve()\n    with zipfile.ZipFile(zip_path) as archive:\n        for member in archive.infolist():\n            destination = (target_dir / member.filename).resolve()\n            if destination != target_dir and target_dir not in destination.parents:\n                raise ValueError('Unsafe archive member: ' + member.filename)\n        archive.extractall(target_dir)\n\ndef discover_weight_source():\n    zip_candidates = sorted(INPUT_ROOT.rglob('aptos_v8_offline_weights.zip'))\n    if zip_candidates:\n        return 'zip', zip_candidates[0]\n    manifest_candidates = sorted(INPUT_ROOT.rglob('offline_weight_manifest.json'))\n    if manifest_candidates:\n        return 'directory', manifest_candidates[0].parent\n    raise FileNotFoundError(\n        'Weight bundle tidak ditemukan di /kaggle/input. Tambahkan aptos_v8_offline_weights.zip atau folder dengan offline_weight_manifest.json.'\n    )\n\nsource_kind, source_path = discover_weight_source()\nmanifest_target = WEIGHT_BUNDLE_DIR / 'offline_weight_manifest.json'\nif not manifest_target.is_file():\n    if WEIGHT_BUNDLE_DIR.exists():\n        shutil.rmtree(WEIGHT_BUNDLE_DIR)\n    WEIGHT_BUNDLE_DIR.mkdir(parents=True, exist_ok=True)\n    if source_kind == 'zip':\n        safe_extract_zip(source_path, WEIGHT_BUNDLE_DIR)\n    else:\n        shutil.copytree(source_path, WEIGHT_BUNDLE_DIR, dirs_exist_ok=True)\n\nmanifest_path = WEIGHT_BUNDLE_DIR / 'offline_weight_manifest.json'\nif not manifest_path.is_file():\n    raise FileNotFoundError('Extraction selesai tetapi manifest tidak ditemukan: ' + str(manifest_path))\nmanifest = json.loads(manifest_path.read_text())\ncheckpoint_records = manifest.get('checkpoints', [])\nstate_files = sorted((WEIGHT_BUNDLE_DIR / 'checkpoints').glob('*_best_state.pth'))\nmeta_files = sorted((WEIGHT_BUNDLE_DIR / 'checkpoints').glob('*_best_meta.json'))\nmissing_manifest_files = []\navailable_models = defaultdict(set)\navailable_folds = defaultdict(list)\nfor record in checkpoint_records:\n    state_path = WEIGHT_BUNDLE_DIR / record['state_file']\n    meta_path = WEIGHT_BUNDLE_DIR / record['meta_file']\n    if not state_path.is_file() or not meta_path.is_file():\n        missing_manifest_files.append(record['experiment_id'])\n        continue\n    meta = json.loads(meta_path.read_text())\n    cfg = meta['cfg']\n    available_models[cfg['model_key']].add(meta.get('resolved_model_name', cfg['model_name']))\n    available_folds[cfg['model_key']].append(int(cfg['fold']))\n\noof_prediction_files = list((WEIGHT_BUNDLE_DIR / 'metadata').glob('*oof*prediction*'))\nartifact_lines = [\n    '# Weight Artifact Validation',\n    '',\n    f'- Source: `{source_path}` ({source_kind})',\n    f'- Working bundle: `{WEIGHT_BUNDLE_DIR}`',\n    f'- Manifest present: {manifest_path.is_file()}',\n    f'- Checkpoint records: {len(checkpoint_records)}',\n    f'- State dict files: {len(state_files)}',\n    f'- Metadata JSON files: {len(meta_files)}',\n    f'- Missing manifest references: {missing_manifest_files}',\n    f'- OOF prediction files: {[p.name for p in oof_prediction_files]}',\n    '',\n    '## Models and folds',\n]\nfor key in sorted(available_models):\n    artifact_lines.append(f'- {key}: {sorted(available_models[key])}; folds={sorted(available_folds[key])}')\nartifact_lines.extend([\n    '',\n## Notes',\n    '- Class mapping and full ImageNet normalization are reconstructed from the original training pipeline, not independently stored in this manifest.',\n    '- Per-image OOF predictions are not required for this small Kaggle XAI run; fold-0 qualitative analysis is clearly labeled as fold-0 inference.',\n])\n(OUTPUT_DIRS['reports'] / 'weight_artifact_validation.md').write_text('\\n'.join(artifact_lines))\n\nif len(state_files) != len(checkpoint_records) or len(meta_files) != len(checkpoint_records) or missing_manifest_files:\n    raise RuntimeError('Weight artifact validation failed. See report for details.')\nprint('Weight source:', source_path)\nprint('Weight bundle:', WEIGHT_BUNDLE_DIR)\nprint('Checkpoints:', len(state_files), '| Metadata:', len(meta_files))\nprint('Models:', {k: sorted(v) for k, v in available_models.items()})\nprint('Folds:', {k: sorted(v) for k, v in available_folds.items()})\n","metadata":{},"outputs":[],"execution_count":null},{"id":"1bdab8ff-0095-473f-935f-9a0704f60d84","cell_type":"markdown","source":"## 5. Model Reconstruction\n\nWrapper berikut mereplikasi `TimmFeatureModel` dan GeM pooling dari notebook training. Arsitektur dibangun dengan `pretrained=False`, kemudian state dict checkpoint lokal dimuat secara strict.","metadata":{}},{"id":"35e4ea5e-227a-48fd-a87d-ae5c4de978df","cell_type":"code","source":"def gem(x, p, eps=1e-6):\n    return F.avg_pool2d(x.clamp(min=eps).pow(p), (x.size(-2), x.size(-1))).pow(1.0 / p)\n\nclass GeM(nn.Module):\n    def __init__(self, p=3.0, eps=1e-6):\n        super().__init__()\n        self.p = Parameter(torch.ones(1) * p)\n        self.eps = eps\n\n    def forward(self, x):\n        return gem(x, self.p, self.eps)\n\nclass TimmFeatureModel(nn.Module):\n    def __init__(self, model_name, input_size, out_dim=5, pooling_type='gem', pretrained=False):\n        super().__init__()\n        if pretrained:\n            raise ValueError('Offline exported checkpoint harus direbuild dengan pretrained=False.')\n        if model_name not in set(timm.list_models(pretrained=False)):\n            raise ValueError(f'Model {model_name} tidak tersedia pada timm {timm.__version__}.')\n        self.backbone = timm.create_model(model_name, pretrained=False, num_classes=0, global_pool='')\n        self.pool = GeM() if pooling_type == 'gem' else nn.AdaptiveAvgPool2d(1)\n        self.feature_dim = self._infer_feature_dim(input_size)\n        self.head = nn.Linear(self.feature_dim, out_dim)\n\n    def _forward_features(self, x):\n        feats = self.backbone.forward_features(x)\n        return feats[-1] if isinstance(feats, (tuple, list)) else feats\n\n    def _pool_features(self, feats):\n        if feats.ndim == 4:\n            return self.pool(feats).flatten(1)\n        if feats.ndim == 3:\n            return feats.mean(dim=1)\n        if feats.ndim == 2:\n            return feats\n        raise RuntimeError(f'Unexpected feature shape: {tuple(feats.shape)}')\n\n    def _infer_feature_dim(self, input_size):\n        self.backbone.eval()\n        with torch.no_grad():\n            features = self._pool_features(self._forward_features(torch.zeros(1, 3, input_size, input_size)))\n        return int(features.shape[1])\n\n    def forward(self, x):\n        return self.head(self._pool_features(self._forward_features(x)))\n\ndef checkpoint_for(model_key='C', fold=0):\n    matches = []\n    for record in checkpoint_records:\n        meta_path = WEIGHT_BUNDLE_DIR / record['meta_file']\n        meta = json.loads(meta_path.read_text())\n        cfg = meta['cfg']\n        if cfg['model_key'] == model_key and int(cfg['fold']) == fold and cfg['loss_type'] == 'cross_entropy':\n            matches.append((record, meta))\n    if len(matches) != 1:\n        raise LookupError(f'Expected one CE checkpoint for {model_key} fold {fold}; found {len(matches)}.')\n    return matches[0]\n\ntarget_record, checkpoint_meta = checkpoint_for(TARGET_MODEL_KEY, TARGET_FOLD)\ntarget_cfg = checkpoint_meta['cfg']\nresolved_name = checkpoint_meta['resolved_model_name']\nstate_path = WEIGHT_BUNDLE_DIR / target_record['state_file']\nmodel = TimmFeatureModel(resolved_name, int(target_cfg['input_size']), out_dim=5, pooling_type=target_cfg['pooling_type'], pretrained=False)\ntry:\n    state_dict = torch.load(state_path, map_location='cpu', weights_only=True)\nexcept TypeError:\n    state_dict = torch.load(state_path, map_location='cpu')\nmodel.load_state_dict(state_dict, strict=True)\nmodel = model.to(DEVICE).eval()\nassert model.head.out_features == 5\nprint('Checkpoint strict load: PASS')\nprint('Model:', resolved_name, '| Fold:', target_cfg['fold'], '| Head:', model.head.in_features, '->', model.head.out_features)\nprint('Checkpoint:', state_path)\n","metadata":{},"outputs":[],"execution_count":null},{"id":"ae34f4bc-d195-4baa-830a-f3d9ad4c0a0b","cell_type":"markdown","source":"## 6. Preprocessing and Real-Image Inference Sanity Check\n\nPreprocessing inference identik dengan notebook asli: OpenCV BGR→RGB, `INTER_AREA` plain resize 384×384, lalu normalisasi ImageNet. Tidak ada augmentasi training.","metadata":{}},{"id":"5b7abb59-82d5-4be1-add0-1ec798954994","cell_type":"code","source":"def preprocess_image(image_path):\n    image_bgr = cv2.imread(str(image_path), cv2.IMREAD_COLOR)\n    if image_bgr is None:\n        raise FileNotFoundError(f'Cannot read image: {image_path}')\n    image_rgb = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB)\n    return cv2.resize(image_rgb, (IMAGE_SIZE, IMAGE_SIZE), interpolation=cv2.INTER_AREA).astype(np.uint8)\n\ndef prepare_input_tensor(image_rgb):\n    if image_rgb.shape != (IMAGE_SIZE, IMAGE_SIZE, 3):\n        raise ValueError(f'Expected RGB {IMAGE_SIZE}x{IMAGE_SIZE}, got {image_rgb.shape}')\n    arr = image_rgb.astype(np.float32) / 255.0\n    arr = (arr - np.asarray(IMAGENET_MEAN, dtype=np.float32)) / np.asarray(IMAGENET_STD, dtype=np.float32)\n    return torch.from_numpy(np.ascontiguousarray(arr.transpose(2, 0, 1))).unsqueeze(0).to(DEVICE)\n\n@torch.inference_mode()\ndef predict_rgb(image_rgb):\n    logits = model(prepare_input_tensor(image_rgb))\n    probs = torch.softmax(logits.float(), dim=1)\n    if logits.shape != (1, 5):\n        raise RuntimeError(f'Unexpected logits shape: {tuple(logits.shape)}')\n    if not torch.isfinite(probs).all() or not torch.isclose(probs.sum(), torch.tensor(1.0, device=DEVICE), atol=1e-5):\n        raise RuntimeError('Invalid softmax output.')\n    pred = int(probs.argmax(dim=1).item())\n    return logits.detach().cpu().numpy()[0], probs.detach().cpu().numpy()[0], pred\n\ndef predict_row(row):\n    image_rgb = preprocess_image(row.image_path)\n    logits, probs, pred = predict_rgb(image_rgb)\n    return {\n        'image_id': row.id_code, 'true_label': int(row.diagnosis), 'true_class': row.class_name,\n        'predicted_label': pred, 'predicted_class': CLASS_NAMES[pred],\n        'confidence': float(probs[pred]), 'probability_sum': float(probs.sum()),\n        'logits_shape': '[1, 5]', 'probabilities': json.dumps([float(x) for x in probs]),\n    }\n\nsanity_rows = train_df.sample(n=min(3, len(train_df)), random_state=SEED)\nsanity_predictions = pd.DataFrame([predict_row(row) for row in sanity_rows.itertuples(index=False)])\nassert sanity_predictions['predicted_label'].between(0, 4).all()\nassert np.isfinite(sanity_predictions['confidence']).all()\nassert np.allclose(sanity_predictions['probability_sum'], 1.0, atol=1e-5)\nsanity_predictions.to_csv(OUTPUT_DIRS['predictions'] / 'inference_sanity_predictions.csv', index=False)\nsanity_report = [\n    '# Real APTOS Inference Sanity Check', '',\n    '- Status: PASS',\n    '- Model: ' + resolved_name,\n    '- Checkpoint: `' + str(state_path) + '`',\n    '- Preprocessing: OpenCV BGR→RGB, plain INTER_AREA resize 384×384, ImageNet normalization.',\n    '- Samples: ' + str(len(sanity_predictions)),\n    '- Logits: [1, 5] for every sample.',\n    '- Softmax: finite and sums to 1 within tolerance.',\n    '', sanity_predictions.to_string(index=False)\n]\n(OUTPUT_DIRS['reports'] / 'inference_sanity_check.md').write_text('\\n'.join(sanity_report))\ndisplay(sanity_predictions)\n","metadata":{},"outputs":[],"execution_count":null},{"id":"68e5ece0-7165-4e41-aaa3-788d6ed9271e","cell_type":"markdown","source":"## 7. Representative Sample Selection\n\nHanya kandidat kecil per kelas yang diinferensikan. Prediksi pada tahap ini berasal dari model C fold 0 dan bukan evaluasi OOF penuh.","metadata":{}},{"id":"859839e1-e032-4708-b990-d2341d5bc5dc","cell_type":"code","source":"candidate_frames = []\nfor label in range(5):\n    group = train_df[train_df.diagnosis.eq(label)]\n    candidate_frames.append(group.sample(n=min(CANDIDATES_PER_CLASS, len(group)), random_state=SEED + label))\ncandidate_df = pd.concat(candidate_frames, ignore_index=True)\ncandidate_predictions = pd.DataFrame([predict_row(row) for row in candidate_df.itertuples(index=False)])\ncandidate_predictions['correct'] = candidate_predictions.true_label.eq(candidate_predictions.predicted_label)\n\ndef add_choice(chosen, record, note):\n    record_id = None if record is None else (record['image_id'] if isinstance(record, pd.Series) else record.image_id)\n    if record is not None and record_id not in {(item['image_id'] if isinstance(item, pd.Series) else item.image_id) for item, _ in chosen}:\n        chosen.append((record, note))\n\nselected_parts = []\nfor label in range(5):\n    group = candidate_predictions[candidate_predictions.true_label.eq(label)].sort_values('confidence').reset_index(drop=True)\n    correct_group = group[group.correct].reset_index(drop=True)\n    wrong_group = group[~group.correct].reset_index(drop=True)\n    chosen = []\n    # Include confidence extremes, a central-confidence case, and an error when present.\n    if len(correct_group):\n        add_choice(chosen, correct_group.iloc[0], 'fold0_correct_low_confidence')\n        add_choice(chosen, correct_group.iloc[-1], 'fold0_correct_high_confidence')\n        add_choice(chosen, correct_group.iloc[len(correct_group) // 2], 'fold0_correct_mid_confidence')\n    if len(wrong_group):\n        add_choice(chosen, wrong_group.iloc[-1], 'fold0_incorrect_high_confidence_failure_case')\n        add_choice(chosen, wrong_group.iloc[0], 'fold0_incorrect_low_confidence_failure_case')\n    for record in group.sort_values('confidence', ascending=False).itertuples(index=False):\n        add_choice(chosen, record, 'fold0_representative_fill')\n        if len(chosen) >= MAX_XAI_PER_CLASS:\n            break\n    class_selected = pd.DataFrame([(record.to_dict() if isinstance(record, pd.Series) else record._asdict()) for record, _ in chosen[:MAX_XAI_PER_CLASS]])\n    if len(class_selected):\n        class_selected['selection_note'] = [note for _, note in chosen[:MAX_XAI_PER_CLASS]]\n        selected_parts.append(class_selected)\n\nselected_xai_df = pd.concat(selected_parts, ignore_index=True)\nselected_xai_df.to_csv(OUTPUT_DIRS['predictions'] / 'selected_xai_samples.csv', index=False)\nif selected_xai_df.empty:\n    raise RuntimeError('Tidak ada sampel XAI yang dapat dipilih.')\ndisplay(selected_xai_df[['image_id', 'true_class', 'predicted_class', 'confidence', 'correct', 'selection_note']])\nprint('Selected XAI samples:', len(selected_xai_df), '| correct:', int(selected_xai_df.correct.sum()), '| incorrect:', int((~selected_xai_df.correct).sum()))\n","metadata":{},"outputs":[],"execution_count":null},{"id":"d00ccad5-857e-4d88-885c-44fe9461f005","cell_type":"markdown","source":"## 8. Grad-CAM and Grad-CAM++\n\nTarget layer dipilih sebagai convolution terakhir dari backbone, bukan linear head. Target explanation default adalah kelas yang diprediksi model.","metadata":{}},{"id":"c41fd5da-53dd-43da-8a47-740d8e56e90d","cell_type":"code","source":"CAM_EPS = 1e-8\n\ndef tensor_stats(prefix, tensor):\n    if tensor is None:\n        return {f'{prefix}_shape': None, f'{prefix}_min': np.nan, f'{prefix}_max': np.nan, f'{prefix}_mean': np.nan}\n    value = tensor.detach().float().cpu()\n    return {\n        f'{prefix}_shape': str(list(value.shape)),\n        f'{prefix}_min': float(value.min()), f'{prefix}_max': float(value.max()), f'{prefix}_mean': float(value.mean()),\n    }\n\ndef get_target_layer(net):\n    # The model wrapper exposes the final backbone feature map directly. This avoids selecting SE fc2, which is 1x1.\n    with torch.no_grad():\n        probe = torch.zeros(1, 3, IMAGE_SIZE, IMAGE_SIZE, device=DEVICE)\n        features = net._forward_features(probe)\n    if features.ndim != 4 or features.shape[-2] <= 1 or features.shape[-1] <= 1:\n        raise RuntimeError(f'Final feature map is not spatially valid for CAM: {tuple(features.shape)}')\n    print('Grad-CAM activation source: net._forward_features | feature shape:', tuple(features.shape))\n    return 'net._forward_features'\n\nCAM_FEATURE_SOURCE = get_target_layer(model)\n\ndef _forward_features_for_cam(net, input_tensor, target_class):\n    net.eval()\n    net.zero_grad(set_to_none=True)\n    with torch.enable_grad():\n        features = net._forward_features(input_tensor)\n        if features.ndim != 4 or features.shape[-2] <= 1 or features.shape[-1] <= 1:\n            raise RuntimeError(f'Invalid CAM feature shape: {tuple(features.shape)}')\n        if not features.requires_grad:\n            raise RuntimeError('Feature map does not require gradients; verify that CAM is not inside torch.no_grad().')\n        features.retain_grad()\n        logits = net.head(net._pool_features(features))\n        score = logits[0, int(target_class)]\n        score.backward()\n    gradients = features.grad\n    if gradients is None:\n        raise RuntimeError('Feature-map gradient was not retained.')\n    return features.detach(), gradients.detach(), logits.detach()\n\ndef normalize_heatmap(cam):\n    cam = np.asarray(cam, dtype=np.float32)\n    if not np.isfinite(cam).all():\n        return None\n    low, high = float(cam.min()), float(cam.max())\n    if high <= low + CAM_EPS:\n        return None\n    return (cam - low) / (high - low + CAM_EPS)\n\ndef _finalize_cam(raw_cam, activations, gradients, logits, method):\n    diagnostics = {**tensor_stats('activation', activations), **tensor_stats('gradient', gradients), **tensor_stats('raw_cam', raw_cam)}\n    raw_cam = raw_cam.detach()\n    if not torch.isfinite(raw_cam).all():\n        diagnostics.update({'normalized_heatmap_min': np.nan, 'normalized_heatmap_max': np.nan})\n        return {'status': 'failed', 'heatmap': None, 'logits': logits, 'diagnostic': diagnostics}\n    positive_cam = F.relu(raw_cam)\n    if float(positive_cam.max()) > CAM_EPS:\n        selected_cam, status = positive_cam, 'success'\n    else:\n        absolute_cam = raw_cam.abs()\n        if float(absolute_cam.max()) <= CAM_EPS:\n            diagnostics.update({'normalized_heatmap_min': np.nan, 'normalized_heatmap_max': np.nan})\n            return {'status': 'failed', 'heatmap': None, 'logits': logits, 'diagnostic': diagnostics}\n        selected_cam, status = absolute_cam, 'fallback_raw_abs_cam'\n    upsampled = F.interpolate(selected_cam.unsqueeze(1), size=(IMAGE_SIZE, IMAGE_SIZE), mode='bilinear', align_corners=False)[0, 0]\n    heatmap = normalize_heatmap(upsampled.detach().float().cpu().numpy())\n    if heatmap is None or not np.isfinite(heatmap).all():\n        diagnostics.update({'normalized_heatmap_min': np.nan, 'normalized_heatmap_max': np.nan})\n        return {'status': 'failed', 'heatmap': None, 'logits': logits, 'diagnostic': diagnostics}\n    diagnostics.update({'normalized_heatmap_min': float(heatmap.min()), 'normalized_heatmap_max': float(heatmap.max())})\n    return {'status': status, 'heatmap': heatmap, 'logits': logits, 'diagnostic': diagnostics}\n\ndef generate_gradcam(net, input_tensor, target_class):\n    activations, gradients, logits = _forward_features_for_cam(net, input_tensor, target_class)\n    weights = gradients.mean(dim=(2, 3), keepdim=True)\n    raw_cam = (weights * activations).sum(dim=1)\n    return _finalize_cam(raw_cam, activations, gradients, logits, 'gradcam')\n\ndef generate_gradcam_plus_plus(net, input_tensor, target_class):\n    activations, gradients, logits = _forward_features_for_cam(net, input_tensor, target_class)\n    gradients2, gradients3 = gradients.pow(2), gradients.pow(3)\n    denominator = 2 * gradients2 + activations.sum(dim=(2, 3), keepdim=True) * gradients3\n    safe = denominator.abs() > CAM_EPS\n    alphas = torch.where(safe, gradients2 / torch.where(safe, denominator, torch.ones_like(denominator)), torch.zeros_like(denominator))\n    alphas = torch.where(torch.isfinite(alphas), alphas, torch.zeros_like(alphas))\n    weights = (alphas * F.relu(gradients)).sum(dim=(2, 3), keepdim=True)\n    raw_cam = (weights * activations).sum(dim=1)\n    return _finalize_cam(raw_cam, activations, gradients, logits, 'gradcam_pp')\n\ndef overlay_heatmap(image_rgb, heatmap, alpha=0.45):\n    color_bgr = cv2.applyColorMap(np.uint8(np.clip(heatmap, 0, 1) * 255), cv2.COLORMAP_JET)\n    color_rgb = cv2.cvtColor(color_bgr, cv2.COLOR_BGR2RGB)\n    return cv2.addWeighted(image_rgb, 1 - alpha, color_rgb, alpha, 0)\n\ndef save_xai_figure(image_rgb, heatmap, metadata, method, figure_path):\n    overlay = overlay_heatmap(image_rgb, heatmap)\n    fig, axes = plt.subplots(1, 3, figsize=(13, 4.5))\n    axes[0].imshow(image_rgb); axes[0].set_title('Original'); axes[0].axis('off')\n    axes[1].imshow(heatmap, cmap='jet', vmin=0, vmax=1); axes[1].set_title(method + ' heatmap'); axes[1].axis('off')\n    axes[2].imshow(overlay); axes[2].set_title('Overlay'); axes[2].axis('off')\n    title = (f\"{metadata['image_id']} | true={metadata['true_class']} | pred={metadata['predicted_class']} \"\n             f\"({metadata['confidence']:.3f}) | {method} | C fold {TARGET_FOLD} | target={metadata['predicted_class']}\")\n    fig.suptitle(title, fontsize=10)\n    fig.tight_layout()\n    fig.savefig(figure_path, dpi=220, bbox_inches='tight')\n    plt.close(fig)\n    return overlay\n","metadata":{},"outputs":[],"execution_count":null},{"id":"dabdf6a0-cb52-47d4-8334-8456fe685115","cell_type":"markdown","source":"## 9. Generate and Save XAI Visualizations\n\nHeatmap mentah disimpan sebagai `.npy`; figure laporan disimpan sebagai PNG. Sampel benar dan salah dipisahkan tanpa menyembunyikan failure case.","metadata":{}},{"id":"d2eaca66-74cc-4e68-bd90-fbd4d3012a10","cell_type":"code","source":"CAM_GENERATORS = {'gradcam': generate_gradcam, 'gradcam_pp': generate_gradcam_plus_plus}\nVALID_CAM_STATUSES = {'success', 'fallback_raw_abs_cam'}\nxai_records, cam_debug_rows, processed_samples = [], [], []\nheatmap_registry = {}\n\ndef process_xai_sample(sample, retry_sample=False):\n    sample_dict = sample._asdict()\n    image_id = sample_dict['image_id']\n    try:\n        image_rgb = preprocess_image(TRAIN_IMAGES_DIR / f'{image_id}.png')\n        input_tensor = prepare_input_tensor(image_rgb)\n    except Exception as exc:\n        for method in CAM_GENERATORS:\n            cam_debug_rows.append({**sample_dict, 'method': method, 'target_class': sample_dict['predicted_label'], 'status': 'failed', 'status_group': 'failed', 'error': str(exc), 'retry_sample': retry_sample})\n        return\n    processed_samples.append({**sample_dict, 'retry_sample': retry_sample})\n    target_class = int(sample_dict['predicted_label'])\n    for method, generator in CAM_GENERATORS.items():\n        started = time.perf_counter()\n        try:\n            result = generator(model, input_tensor, target_class)\n            runtime_sec = time.perf_counter() - started\n            status = result['status']\n            diagnostic = result['diagnostic']\n            diagnostic.update({\n                **sample_dict, 'method': method, 'target_class': target_class, 'predicted_class': sample_dict['predicted_class'],\n                'confidence': sample_dict['confidence'], 'status': status,\n                'status_group': 'success' if status == 'success' else ('fallback' if status.startswith('fallback') else 'failed'),\n                'runtime_seconds': runtime_sec, 'error': '', 'retry_sample': retry_sample,\n            })\n            cam_debug_rows.append(diagnostic)\n            if status not in VALID_CAM_STATUSES:\n                continue\n            heatmap = result['heatmap']\n            stem = f'{image_id}_{method}_C_fold{TARGET_FOLD}'\n            raw_path = OUTPUT_DIRS[method] / f'{stem}.npy'\n            figure_path = OUTPUT_DIRS['overlays'] / f'{stem}.png'\n            np.save(raw_path, heatmap.astype(np.float32), allow_pickle=False)\n            overlay = save_xai_figure(image_rgb, heatmap, sample_dict, method + (' [fallback]' if status != 'success' else ''), figure_path)\n            cv2.imwrite(str(OUTPUT_DIRS['overlays'] / f'{stem}_overlay.png'), cv2.cvtColor(overlay, cv2.COLOR_RGB2BGR))\n            class_dir = OUTPUT_DIRS['per_class'] / sample_dict['true_class']; class_dir.mkdir(parents=True, exist_ok=True)\n            shutil.copy2(figure_path, class_dir / figure_path.name)\n            outcome_dir = OUTPUT_DIRS['correct'] if bool(sample_dict['correct']) else OUTPUT_DIRS['incorrect']\n            shutil.copy2(figure_path, outcome_dir / figure_path.name)\n            heatmap_registry[(image_id, method)] = {'heatmap': heatmap, 'image_rgb': image_rgb, 'target_class': target_class, 'status': status}\n            xai_records.append({**sample_dict, 'method': method, 'target_class': target_class, 'cam_status': status, 'is_fallback': status != 'success', 'retry_sample': retry_sample, 'runtime_seconds': runtime_sec, 'raw_heatmap_path': str(raw_path), 'figure_path': str(figure_path)})\n        except Exception as exc:\n            runtime_sec = time.perf_counter() - started\n            cam_debug_rows.append({**sample_dict, 'method': method, 'target_class': target_class, 'status': 'failed', 'status_group': 'failed', 'runtime_seconds': runtime_sec, 'error': repr(exc), 'retry_sample': retry_sample})\n\nfor sample in selected_xai_df.itertuples(index=False):\n    process_xai_sample(sample)\n\n# Retry at most two extra candidates for a class with no valid standard Grad-CAM.\ninitial_ids = set(selected_xai_df.image_id)\nfor label in range(5):\n    valid_gradcam_ids = {row['image_id'] for row in xai_records if row['method'] == 'gradcam' and row['true_label'] == label}\n    if valid_gradcam_ids:\n        continue\n    retries = candidate_predictions[(candidate_predictions.true_label.eq(label)) & (~candidate_predictions.image_id.isin(initial_ids))].head(2).copy()\n    if len(retries):\n        retries['selection_note'] = 'retry_after_initial_cam_failure'\n        for retry in retries.itertuples(index=False):\n            process_xai_sample(retry, retry_sample=True)\n            if any(row['method'] == 'gradcam' and row['true_label'] == label for row in xai_records):\n                break\n\nxai_records_df = pd.DataFrame(xai_records)\ncam_debug_df = pd.DataFrame(cam_debug_rows)\nprocessed_samples_df = pd.DataFrame(processed_samples)\nif len(processed_samples_df):\n    processed_samples_df = processed_samples_df.drop_duplicates('image_id')\nelse:\n    processed_samples_df = pd.DataFrame(columns=list(selected_xai_df.columns) + ['retry_sample'])\nxai_records_df.to_csv(OUTPUT_DIRS['metrics'] / 'xai_generation_records.csv', index=False)\ncam_debug_df.to_csv(OUTPUT_DIRS['metrics'] / 'cam_debug_log.csv', index=False)\nprocessed_samples_df.to_csv(OUTPUT_DIRS['predictions'] / 'xai_processed_samples.csv', index=False)\n\ncommon_ids = [image_id for image_id in processed_samples_df.image_id if (image_id, 'gradcam') in heatmap_registry and (image_id, 'gradcam_pp') in heatmap_registry]\nif common_ids:\n    comparison_id = common_ids[0]\n    comparison_sample = processed_samples_df[processed_samples_df.image_id.eq(comparison_id)].iloc[0]\n    comparison_image = heatmap_registry[(comparison_id, 'gradcam')]['image_rgb']\n    fig, axes = plt.subplots(2, 3, figsize=(12, 8))\n    for row_idx, method in enumerate(['gradcam', 'gradcam_pp']):\n        comparison_heatmap = heatmap_registry[(comparison_id, method)]['heatmap']\n        axes[row_idx, 0].imshow(comparison_image); axes[row_idx, 0].set_title('Original'); axes[row_idx, 0].axis('off')\n        axes[row_idx, 1].imshow(comparison_heatmap, cmap='jet', vmin=0, vmax=1); axes[row_idx, 1].set_title(method); axes[row_idx, 1].axis('off')\n        axes[row_idx, 2].imshow(overlay_heatmap(comparison_image, comparison_heatmap)); axes[row_idx, 2].set_title(method + ' overlay'); axes[row_idx, 2].axis('off')\n    fig.suptitle(f'Grad-CAM comparison | {comparison_id} | true={comparison_sample.true_class} | pred={comparison_sample.predicted_class}')\n    fig.tight_layout(); fig.savefig(OUTPUT_DIRS['comparison'] / f'gradcam_comparison_{comparison_id}.png', dpi=220, bbox_inches='tight'); plt.show()\nelse:\n    print('No sample has both valid Grad-CAM and Grad-CAM++ outputs; comparison figure skipped.')\n\nstatus_counts = cam_debug_df.status_group.value_counts().to_dict() if len(cam_debug_df) else {}\nprint('CAM status counts:', status_counts)\ndisplay(xai_records_df[['image_id', 'method', 'cam_status', 'true_class', 'predicted_class', 'confidence', 'correct', 'runtime_seconds']].head() if len(xai_records_df) else cam_debug_df.head())\n","metadata":{},"outputs":[],"execution_count":null},{"id":"71b38c22-1922-4088-ae36-2cf7c7fcc868","cell_type":"markdown","source":"## 10. Lightweight Perturbation Evaluation\n\nTidak ada IoU, Dice, atau pixel accuracy karena APTOS tidak menyediakan lesion mask. Evaluasi memakai perubahan probabilitas kelas target setelah masking area penting, random, atau least-important region.","metadata":{}},{"id":"9dc86a45-e83d-4988-9b32-604256dc2d68","cell_type":"code","source":"def target_probability(image_rgb, target_class):\n    _, probs, _ = predict_rgb(image_rgb)\n    return float(probs[int(target_class)])\n\ndef safe_trapezoid(y, x):\n    y = np.asarray(y, dtype=np.float64)\n    x = np.asarray(x, dtype=np.float64)\n    if y.size < 2 or x.size < 2:\n        return np.nan\n    if y.size != x.size:\n        raise ValueError(f'safe_trapezoid expects equal lengths, got y={y.size}, x={x.size}')\n    if not np.isfinite(y).all() or not np.isfinite(x).all():\n        return np.nan\n    if hasattr(np, 'trapezoid'):\n        return float(np.trapezoid(y, x))\n    return float(np.sum((x[1:] - x[:-1]) * (y[1:] + y[:-1]) * 0.5))\n\ndef mask_by_heatmap(image_rgb, heatmap, fraction, mode, rng=None):\n    flat = heatmap.reshape(-1)\n    count = max(1, int(round(float(fraction) * flat.size)))\n    if mode == 'most':\n        indices = np.argpartition(flat, -count)[-count:]\n    elif mode == 'least':\n        indices = np.argpartition(flat, count - 1)[:count]\n    elif mode == 'random':\n        if rng is None:\n            rng = np.random.default_rng(SEED)\n        indices = rng.choice(flat.size, size=count, replace=False)\n    else:\n        raise ValueError('Unknown masking mode: ' + mode)\n    mask = np.zeros(flat.size, dtype=bool)\n    mask[indices] = True\n    mask = mask.reshape(heatmap.shape)\n    baseline = cv2.GaussianBlur(image_rgb, (0, 0), sigmaX=10)\n    masked = image_rgb.copy()\n    masked[mask] = baseline[mask]\n    return masked\n\ndef heatmap_entropy(heatmap):\n    p = heatmap.reshape(-1).astype(np.float64)\n    p = p / (p.sum() + 1e-12)\n    return float(-(p * np.log(p + 1e-12)).sum())\n\ndef select_evaluation_subset(frame, maximum=10):\n    first_per_class = frame.sort_values('confidence', ascending=False).groupby('true_label', group_keys=False).head(1)\n    remaining = frame.drop(index=first_per_class.index).sort_values('confidence', ascending=False)\n    return pd.concat([first_per_class, remaining]).drop_duplicates('image_id').head(maximum)\n\nvalid_sample_metadata = xai_records_df.drop_duplicates('image_id').copy() if len(xai_records_df) else pd.DataFrame(columns=selected_xai_df.columns)\nevaluation_samples = select_evaluation_subset(valid_sample_metadata, MAX_EVALUATION_SAMPLES) if len(valid_sample_metadata) else valid_sample_metadata\ndeletion_steps = [0.10, 0.20, 0.30, 0.40, 0.50]\nmetric_columns = ['image_id', 'true_label', 'true_class', 'predicted_label', 'predicted_class', 'correct', 'method', 'cam_status', 'is_fallback', 'target_class', 'original_target_probability', 'important_mask_probability', 'random_mask_probability', 'least_mask_probability', 'confidence_drop_important', 'confidence_drop_random', 'confidence_drop_least', 'deletion_auc_probability', 'aopc_deletion', 'heatmap_entropy', 'active_area_ratio_ge_0_5', 'generation_runtime_seconds', 'metric_status', 'metric_error']\nmetric_rows, curve_rows = [], []\nfor sample in evaluation_samples.itertuples(index=False):\n    for method in CAM_GENERATORS:\n        if (sample.image_id, method) not in heatmap_registry:\n            continue\n        item = heatmap_registry[(sample.image_id, method)]\n        base_row = {\n            'image_id': sample.image_id, 'true_label': sample.true_label, 'true_class': sample.true_class,\n            'predicted_label': sample.predicted_label, 'predicted_class': sample.predicted_class, 'correct': sample.correct,\n            'method': method, 'cam_status': item['status'], 'is_fallback': item['status'] != 'success', 'target_class': item['target_class'],\n            'generation_runtime_seconds': float(xai_records_df[(xai_records_df.image_id == sample.image_id) & (xai_records_df.method == method)].runtime_seconds.iloc[0]),\n        }\n        try:\n            original_prob = target_probability(item['image_rgb'], item['target_class'])\n            important_prob = target_probability(mask_by_heatmap(item['image_rgb'], item['heatmap'], 0.20, 'most'), item['target_class'])\n            random_prob = target_probability(mask_by_heatmap(item['image_rgb'], item['heatmap'], 0.20, 'random', np.random.default_rng(SEED + len(metric_rows))), item['target_class'])\n            least_prob = target_probability(mask_by_heatmap(item['image_rgb'], item['heatmap'], 0.20, 'least'), item['target_class'])\n            deletion_probs = [float(target_probability(mask_by_heatmap(item['image_rgb'], item['heatmap'], fraction, 'most'), item['target_class'])) for fraction in deletion_steps]\n            if len(deletion_probs) != len(deletion_steps) or not np.isfinite(deletion_probs).all():\n                raise ValueError('Deletion curve contains invalid probabilities.')\n            denom = float(deletion_steps[-1] - deletion_steps[0])\n            deletion_auc = np.nan if denom <= 0 else safe_trapezoid(deletion_probs, deletion_steps) / denom\n            local_curves = [{'image_id': sample.image_id, 'method': method, 'cam_status': item['status'], 'fraction_removed': fraction, 'target_probability': probability} for fraction, probability in zip(deletion_steps, deletion_probs)]\n            curve_rows.extend(local_curves)\n            metric_rows.append({**base_row,\n                'original_target_probability': original_prob, 'important_mask_probability': important_prob, 'random_mask_probability': random_prob, 'least_mask_probability': least_prob,\n                'confidence_drop_important': original_prob - important_prob, 'confidence_drop_random': original_prob - random_prob, 'confidence_drop_least': original_prob - least_prob,\n                'deletion_auc_probability': deletion_auc, 'aopc_deletion': float(np.mean([original_prob - value for value in deletion_probs])),\n                'heatmap_entropy': heatmap_entropy(item['heatmap']), 'active_area_ratio_ge_0_5': float((item['heatmap'] >= 0.5).mean()),\n                'metric_status': 'success', 'metric_error': '',\n            })\n        except Exception as exc:\n            metric_rows.append({**base_row, **{column: np.nan for column in metric_columns if column not in base_row and column not in {'metric_status', 'metric_error'}}, 'metric_status': 'failed', 'metric_error': repr(exc)})\n\nmetrics_df = pd.DataFrame(metric_rows, columns=metric_columns)\ncurves_df = pd.DataFrame(curve_rows, columns=['image_id', 'method', 'cam_status', 'fraction_removed', 'target_probability'])\nmetrics_df.to_csv(OUTPUT_DIRS['metrics'] / 'xai_metrics_per_image.csv', index=False)\ncurves_df.to_csv(OUTPUT_DIRS['metrics'] / 'deletion_curves.csv', index=False)\nmetric_success_df = metrics_df[metrics_df.metric_status.eq('success')].copy()\nmetric_failure_df = metrics_df[metrics_df.metric_status.eq('failed')].copy()\nmetric_failure_df.to_csv(OUTPUT_DIRS['metrics'] / 'xai_metric_failures.csv', index=False)\nsummary_df = metric_success_df.groupby(['method', 'cam_status', 'is_fallback'], as_index=False).agg(\n    samples=('image_id', 'count'), important_drop_mean=('confidence_drop_important', 'mean'),\n    random_drop_mean=('confidence_drop_random', 'mean'), least_drop_mean=('confidence_drop_least', 'mean'),\n    deletion_auc_probability_mean=('deletion_auc_probability', 'mean'), aopc_deletion_mean=('aopc_deletion', 'mean'), entropy_mean=('heatmap_entropy', 'mean'),\n    active_area_ratio_mean=('active_area_ratio_ge_0_5', 'mean'), runtime_seconds_mean=('generation_runtime_seconds', 'mean'),\n)\nper_class_df = metric_success_df.groupby(['method', 'cam_status', 'true_label', 'true_class'], as_index=False).agg(\n    samples=('image_id', 'count'), important_drop_mean=('confidence_drop_important', 'mean'),\n    random_drop_mean=('confidence_drop_random', 'mean'), entropy_mean=('heatmap_entropy', 'mean')\n)\ncorrectness_df = metric_success_df.groupby(['method', 'cam_status', 'correct'], as_index=False).agg(\n    samples=('image_id', 'count'), important_drop_mean=('confidence_drop_important', 'mean'),\n    entropy_mean=('heatmap_entropy', 'mean'), runtime_seconds_mean=('generation_runtime_seconds', 'mean')\n)\nsummary_df.to_csv(OUTPUT_DIRS['metrics'] / 'xai_metrics_summary.csv', index=False)\nper_class_df.to_csv(OUTPUT_DIRS['metrics'] / 'xai_metrics_per_class.csv', index=False)\ncorrectness_df.to_csv(OUTPUT_DIRS['metrics'] / 'xai_metrics_correct_incorrect.csv', index=False)\nprint('Metric success:', len(metric_success_df), '| metric failed:', len(metric_failure_df), '| skipped failed CAM records:', int((cam_debug_df.status_group.eq('failed')).sum()) if len(cam_debug_df) else 0)\ndisplay(summary_df)\n","metadata":{},"outputs":[],"execution_count":null},{"id":"2c090fc4-43cd-4145-8267-d0c40564e3d8","cell_type":"markdown","source":"## 11. Save Lightweight Evaluation Figures\n\nA confidence drop yang lebih besar pada important masking daripada random/least masking mendukung faithfulness secara relatif; ini bukan bukti klinis atau ground-truth lesion localization.","metadata":{}},{"id":"d359ec32-bed5-4b34-bb29-30dafbc43bea","cell_type":"code","source":"if summary_df.empty or curves_df.empty:\n    print('No valid/fallback CAM heatmap is available for perturbation figures. Empty metric CSV files were still saved.')\nelse:\n    drop_plot = summary_df.assign(method_status=summary_df.method + ' [' + summary_df.cam_status + ']').set_index('method_status')[['important_drop_mean', 'random_drop_mean', 'least_drop_mean']]\n    ax = drop_plot.plot.bar(figsize=(8, 4), color=['#c44e52', '#8172b3', '#64b5cd'])\n    ax.axhline(0, color='black', linewidth=0.8)\n    ax.set_ylabel('Mean target probability drop')\n    ax.set_title('Confidence drop: important vs random vs least-important masking')\n    plt.tight_layout(); plt.savefig(OUTPUT_DIRS['confidence'] / 'confidence_drop_comparison.png', dpi=220, bbox_inches='tight'); plt.show()\n    fig, ax = plt.subplots(figsize=(7, 4))\n    for (method, cam_status), group in curves_df.groupby(['method', 'cam_status']):\n        curve = group.groupby('fraction_removed', as_index=False).target_probability.mean()\n        ax.plot(curve.fraction_removed, curve.target_probability, marker='o', label=f'{method} [{cam_status}]')\n    ax.set_xlabel('Fraction of highest-heatmap area blurred'); ax.set_ylabel('Mean target-class probability')\n    ax.set_title('Deletion curve (small XAI subset)'); ax.legend(); plt.tight_layout()\n    plt.savefig(OUTPUT_DIRS['deletion'] / 'deletion_curve_comparison.png', dpi=220, bbox_inches='tight'); plt.show()\n","metadata":{},"outputs":[],"execution_count":null},{"id":"82b02d8a-c669-4485-81de-d1bfd4b94b3e","cell_type":"markdown","source":"## 12. XAI Result Analysis\n\nBagian ini membaca CSV hasil run yang sudah tersimpan. Tidak ada inference, CAM, atau perturbation ulang pada section analisis.","metadata":{}},{"id":"df8c6104-2b3d-4533-9d06-568ba7a5dfc8","cell_type":"code","source":"ANALYSIS_FILES = {\n    'generation': OUTPUT_DIRS['metrics'] / 'xai_generation_records.csv',\n    'cam_debug': OUTPUT_DIRS['metrics'] / 'cam_debug_log.csv',\n    'per_image': OUTPUT_DIRS['metrics'] / 'xai_metrics_per_image.csv',\n    'summary': OUTPUT_DIRS['metrics'] / 'xai_metrics_summary.csv',\n    'per_class': OUTPUT_DIRS['metrics'] / 'xai_metrics_per_class.csv',\n    'correctness': OUTPUT_DIRS['metrics'] / 'xai_metrics_correct_incorrect.csv',\n    'curves': OUTPUT_DIRS['metrics'] / 'deletion_curves.csv',\n    'final_summary': OUTPUT_DIRS['reports'] / 'xai_final_summary.md',\n}\nmissing_analysis_files = [name for name, path in ANALYSIS_FILES.items() if not path.exists() and name != 'final_summary']\nif missing_analysis_files:\n    raise FileNotFoundError('Run cells sebelumnya terlebih dahulu. Missing outputs: ' + str(missing_analysis_files))\n\nanalysis_generation_df = pd.read_csv(ANALYSIS_FILES['generation'])\nanalysis_debug_df = pd.read_csv(ANALYSIS_FILES['cam_debug'])\nanalysis_metrics_df = pd.read_csv(ANALYSIS_FILES['per_image'])\nanalysis_summary_df = pd.read_csv(ANALYSIS_FILES['summary'])\nanalysis_curves_df = pd.read_csv(ANALYSIS_FILES['curves'])\n\nsample_df = analysis_generation_df.drop_duplicates('image_id').copy()\nsample_distribution_df = sample_df.groupby('true_class', as_index=False).agg(\n    samples=('image_id', 'count'), correct=('correct', 'sum')\n)\nsample_distribution_df['incorrect'] = sample_distribution_df['samples'] - sample_distribution_df['correct']\nsample_distribution_df = sample_distribution_df.set_index('true_class').reindex(CLASS_NAMES, fill_value=0).reset_index()\n\ncam_status_df = analysis_debug_df.assign(status_group=analysis_debug_df.get('status_group', analysis_debug_df.get('status', 'failed'))).pivot_table(\n    index='method', columns='status_group', values='image_id', aggfunc='count', fill_value=0\n).reset_index()\nfor column in ['success', 'fallback', 'failed']:\n    if column not in cam_status_df:\n        cam_status_df[column] = 0\ncam_status_df = cam_status_df[['method', 'success', 'fallback', 'failed']]\n\nmetric_success_df = analysis_metrics_df[analysis_metrics_df.metric_status.eq('success')].copy() if 'metric_status' in analysis_metrics_df else analysis_metrics_df.copy()\nmetric_failed_df = analysis_metrics_df[analysis_metrics_df.metric_status.eq('failed')].copy() if 'metric_status' in analysis_metrics_df else analysis_metrics_df.iloc[0:0].copy()\nper_class_analysis_df = metric_success_df.groupby(['true_class', 'method', 'cam_status'], as_index=False).agg(\n    samples=('image_id', 'count'), important_drop_mean=('confidence_drop_important', 'mean'),\n    random_drop_mean=('confidence_drop_random', 'mean'), least_drop_mean=('confidence_drop_least', 'mean'),\n    deletion_auc_probability_mean=('deletion_auc_probability', 'mean'), aopc_deletion_mean=('aopc_deletion', 'mean'),\n    active_area_ratio_mean=('active_area_ratio_ge_0_5', 'mean'), entropy_mean=('heatmap_entropy', 'mean'),\n)\nper_class_analysis_df.to_csv(OUTPUT_DIRS['metrics'] / 'xai_metrics_per_class_enriched.csv', index=False)\n\ncam_total = int(len(analysis_generation_df))\nmetric_total = int(len(metric_success_df))\ncam_image_count = int(sample_df.image_id.nunique())\nmetric_image_count = int(metric_success_df.image_id.nunique()) if len(metric_success_df) else 0\nexcluded_images = max(0, cam_image_count - metric_image_count)\nconfidence_summary = sample_df.groupby('correct', as_index=False).agg(samples=('image_id', 'count'), confidence_mean=('confidence', 'mean'), confidence_median=('confidence', 'median'))\n\nresult_analysis_lines = [\n    '## XAI Result Analysis',\n    f'- Unique XAI images processed: {cam_image_count}; method-image CAM records: {cam_total}.',\n    f'- CAM status: success={int(cam_status_df.success.sum())}, fallback={int(cam_status_df.fallback.sum())}, failed={int(cam_status_df.failed.sum())}.',\n    f'- Perturbation metrics: success={metric_total}, failed={len(metric_failed_df)}, evaluated images={metric_image_count}.',\n    f'- CAM/metric count mismatch: {cam_total} CAM records versus {metric_total} successful metric records is expected when MAX_EVALUATION_SAMPLES limits perturbation evaluation to {metric_image_count} images; {excluded_images} selected image(s) were not evaluated to keep the Kaggle run lightweight.',\n    f'- Prediction outcomes in the XAI subset: correct={int(sample_df.correct.sum())}, incorrect={int((~sample_df.correct.astype(bool)).sum())}.',\n    '- Metric interpretation: larger important-region confidence drop and AOPC, together with lower deletion probability/AUC, indicate stronger relative dependence on the highlighted area. These are model-faithfulness indicators, not lesion-localization or clinical validation.',\n]\n(OUTPUT_DIRS['reports'] / 'xai_result_analysis.md').write_text('\\n'.join(result_analysis_lines + ['', '## Sample distribution', sample_distribution_df.to_string(index=False), '', '## CAM status', cam_status_df.to_string(index=False), '', '## Metric summary', analysis_summary_df.to_string(index=False), '', '## Per-class metrics', per_class_analysis_df.to_string(index=False)]))\n\ndisplay(sample_distribution_df)\ndisplay(cam_status_df)\ndisplay(confidence_summary)\ndisplay(analysis_summary_df)\ndisplay(per_class_analysis_df)\nprint('\\n'.join(result_analysis_lines))\n","metadata":{},"outputs":[],"execution_count":null},{"id":"6ee19d74-baad-4a74-be11-ee4464b2b11e","cell_type":"markdown","source":"## 13. XAI Visual Gallery\n\nGallery memakai overlay dan heatmap yang sudah disimpan pada output sebelumnya. Section ini tidak menghitung ulang Grad-CAM atau Grad-CAM++.","metadata":{}},{"id":"f265e0f7-82db-44e2-b84e-da2787d6750e","cell_type":"code","source":"gallery_records_df = analysis_generation_df[analysis_generation_df.cam_status.isin(['success', 'fallback_raw_abs_cam'])].copy()\ngallery_ids = gallery_records_df.groupby('image_id').method.nunique()\ngallery_ids = gallery_ids[gallery_ids >= 2].index\ngallery_image_df = gallery_records_df[gallery_records_df.image_id.isin(gallery_ids)].drop_duplicates('image_id').copy()\n\ndef load_saved_overlay(image_id, method):\n    overlay_path = OUTPUT_DIRS['overlays'] / f'{image_id}_{method}_C_fold{TARGET_FOLD}_overlay.png'\n    overlay_bgr = cv2.imread(str(overlay_path), cv2.IMREAD_COLOR)\n    if overlay_bgr is not None:\n        return cv2.cvtColor(overlay_bgr, cv2.COLOR_BGR2RGB)\n    raw_path = OUTPUT_DIRS[method] / f'{image_id}_{method}_C_fold{TARGET_FOLD}.npy'\n    if raw_path.exists():\n        return overlay_heatmap(preprocess_image(TRAIN_IMAGES_DIR / f'{image_id}.png'), np.load(raw_path))\n    raise FileNotFoundError(f'Missing saved overlay/heatmap for {image_id}, {method}')\n\ndef save_gallery(rows, title, output_path):\n    rows = rows.head(3).reset_index(drop=True)\n    if rows.empty:\n        print('Gallery skipped (no records):', title)\n        return\n    fig, axes = plt.subplots(len(rows), 3, figsize=(13, 4.3 * len(rows)), squeeze=False)\n    for idx, row in rows.iterrows():\n        image_rgb = preprocess_image(TRAIN_IMAGES_DIR / f\"{row['image_id']}.png\")\n        gradcam_overlay = load_saved_overlay(row['image_id'], 'gradcam')\n        gradcampp_overlay = load_saved_overlay(row['image_id'], 'gradcam_pp')\n        panels = [(image_rgb, 'Original'), (gradcam_overlay, 'Grad-CAM overlay'), (gradcampp_overlay, 'Grad-CAM++ overlay')]\n        for ax, (image, label) in zip(axes[idx], panels):\n            ax.imshow(image); ax.set_title(label); ax.axis('off')\n        axes[idx, 0].set_ylabel(f\"{row['image_id']}\\ntrue={row['true_class']} | pred={row['predicted_class']}\\nconf={row['confidence']:.3f}\", fontsize=9)\n    fig.suptitle(title, fontsize=13)\n    fig.tight_layout(); fig.savefig(output_path, dpi=220, bbox_inches='tight'); plt.show(); plt.close(fig)\n\ngallery_manifest = []\nfor class_name in CLASS_NAMES:\n    class_rows = gallery_image_df[gallery_image_df.true_class.eq(class_name)]\n    output_path = OUTPUT_DIRS['gallery'] / f'xai_gallery_per_class_{class_name}.png'\n    save_gallery(class_rows, f'XAI Gallery — {class_name}', output_path)\n    gallery_manifest.append({'gallery': f'per_class_{class_name}', 'records': len(class_rows.head(3)), 'path': str(output_path)})\n\nsave_gallery(gallery_image_df[gallery_image_df.correct.astype(bool)], 'XAI Gallery — Correct Predictions', OUTPUT_DIRS['gallery'] / 'xai_gallery_correct_predictions.png')\nsave_gallery(gallery_image_df[~gallery_image_df.correct.astype(bool)], 'XAI Gallery — Incorrect Predictions', OUTPUT_DIRS['gallery'] / 'xai_gallery_incorrect_predictions.png')\nsave_gallery(gallery_image_df, 'Grad-CAM vs Grad-CAM++ — Representative Samples', OUTPUT_DIRS['gallery'] / 'xai_gallery_gradcam_vs_gradcampp.png')\npd.DataFrame(gallery_manifest).to_csv(OUTPUT_DIRS['gallery'] / 'xai_gallery_manifest.csv', index=False)\n","metadata":{},"outputs":[],"execution_count":null},{"id":"fd3777e4-ba31-459e-b762-92b191522f92","cell_type":"markdown","source":"## 14. Additional XAI Summary Figures\n\nGrafik berikut merangkum hasil yang sudah dihitung sebelumnya; tidak memicu CAM atau perturbation baru.","metadata":{}},{"id":"352b840a-f50c-4456-8dd6-b9fb4b4d722b","cell_type":"code","source":"plot_summary_df = analysis_summary_df.copy()\nplot_summary_df['label'] = plot_summary_df['method'] + ' [' + plot_summary_df['cam_status'] + ']'\nif not plot_summary_df.empty:\n    fig, axes = plt.subplots(2, 2, figsize=(12, 8))\n    axes[0, 0].bar(plot_summary_df.label, plot_summary_df.important_drop_mean, color='#c44e52')\n    axes[0, 0].set_title('Important-region confidence drop'); axes[0, 0].tick_params(axis='x', rotation=20)\n    comparison = plot_summary_df.set_index('label')[['important_drop_mean', 'random_drop_mean', 'least_drop_mean']]\n    comparison.plot.bar(ax=axes[0, 1], color=['#c44e52', '#8172b3', '#64b5cd'])\n    axes[0, 1].set_title('Important vs random vs least-important'); axes[0, 1].tick_params(axis='x', rotation=20)\n    axes[1, 0].bar(plot_summary_df.label, plot_summary_df.active_area_ratio_mean, color='#55a868')\n    axes[1, 0].set_title('Active area ratio'); axes[1, 0].tick_params(axis='x', rotation=20)\n    axes[1, 1].bar(plot_summary_df.label, plot_summary_df.runtime_seconds_mean, color='#dd8452')\n    axes[1, 1].set_title('Mean CAM runtime (seconds)'); axes[1, 1].tick_params(axis='x', rotation=20)\n    fig.tight_layout(); fig.savefig(OUTPUT_DIRS['comparison'] / 'xai_metric_overview.png', dpi=220, bbox_inches='tight'); plt.show(); plt.close(fig)\n    fig, ax = plt.subplots(figsize=(7, 4))\n    for (method, cam_status), group in analysis_curves_df.groupby(['method', 'cam_status']):\n        curve = group.groupby('fraction_removed', as_index=False).target_probability.mean()\n        ax.plot(curve.fraction_removed, curve.target_probability, marker='o', label=f'{method} [{cam_status}]')\n    ax.set_xlabel('Fraction of highest-heatmap area blurred'); ax.set_ylabel('Mean target probability')\n    ax.set_title('Mean deletion curves from evaluated XAI subset'); ax.legend(); fig.tight_layout()\n    fig.savefig(OUTPUT_DIRS['comparison'] / 'xai_deletion_curve_overview.png', dpi=220, bbox_inches='tight'); plt.show(); plt.close(fig)\nelse:\n    print('No successful metric rows available for summary figures.')\n","metadata":{},"outputs":[],"execution_count":null},{"id":"8fced517-2b82-4d17-b8a8-4229f3f769e0","cell_type":"markdown","source":"## 15. Clinical-Oriented XAI Quality Analysis\n\nAudit ini menggunakan retina mask, optic-disc estimate, dan bright/dark visual proxy pada sampel XAI yang sudah ada. Semua proxy bersifat non-klinis: bukan lesion mask dan bukan ground truth diagnosis.","metadata":{}},{"id":"1c00a1db-f165-47f7-ae33-ffa386f781ae","cell_type":"code","source":"QUALITY_DIR = OUTPUT_ROOT / 'figures' / 'xai_quality_audit'\nQUALITY_DIR.mkdir(parents=True, exist_ok=True)\nQUALITY_EPS = 1e-8\nMAX_RETINA_AWARE_EVAL_IMAGES = 10\n\ndef largest_component(mask):\n    count, labels, stats, _ = cv2.connectedComponentsWithStats(mask.astype(np.uint8), connectivity=8)\n    if count <= 1:\n        return np.zeros_like(mask, dtype=bool)\n    label = 1 + np.argmax(stats[1:, cv2.CC_STAT_AREA])\n    return labels == label\n\ndef create_retina_mask(image_rgb):\n    gray = cv2.cvtColor(image_rgb, cv2.COLOR_RGB2GRAY)\n    mask = (gray > 10).astype(np.uint8)\n    mask = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (15, 15)))\n    return largest_component(mask)\n\ndef heatmap_mass_inside_retina(heatmap, retina_mask):\n    total = float(np.asarray(heatmap, dtype=np.float64).sum())\n    return np.nan if total <= QUALITY_EPS else float(np.asarray(heatmap)[retina_mask].sum() / total)\n\ndef heatmap_mass_outside_retina(heatmap, retina_mask):\n    inside = heatmap_mass_inside_retina(heatmap, retina_mask)\n    return np.nan if not np.isfinite(inside) else float(1.0 - inside)\n\ndef border_focus_ratio(heatmap, retina_mask):\n    return heatmap_mass_outside_retina(heatmap, retina_mask)\n\ndef estimate_optic_disc_mask(image_rgb, retina_mask):\n    gray = cv2.cvtColor(image_rgb, cv2.COLOR_RGB2GRAY)\n    clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8)).apply(gray)\n    values = clahe[retina_mask]\n    if values.size == 0:\n        return np.zeros_like(retina_mask, dtype=bool)\n    threshold = np.percentile(values, 99.0)\n    candidate = ((clahe >= threshold) & retina_mask).astype(np.uint8)\n    candidate = cv2.morphologyEx(candidate, cv2.MORPH_CLOSE, cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (9, 9)))\n    count, labels, stats, _ = cv2.connectedComponentsWithStats(candidate, connectivity=8)\n    retina_area = max(int(retina_mask.sum()), 1)\n    choices = [idx for idx in range(1, count) if 20 <= stats[idx, cv2.CC_STAT_AREA] <= 0.08 * retina_area]\n    if not choices:\n        return np.zeros_like(retina_mask, dtype=bool)\n    chosen = max(choices, key=lambda idx: stats[idx, cv2.CC_STAT_AREA])\n    disc = (labels == chosen).astype(np.uint8)\n    disc = cv2.dilate(disc, cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (17, 17)))\n    return disc.astype(bool) & retina_mask\n\ndef heatmap_mass_on_optic_disc(heatmap, optic_disc_mask):\n    total = float(np.asarray(heatmap, dtype=np.float64).sum())\n    return np.nan if total <= QUALITY_EPS else float(np.asarray(heatmap)[optic_disc_mask].sum() / total)\n\ndef create_bright_lesion_proxy_mask(image_rgb, retina_mask, optic_disc_mask):\n    green = image_rgb[:, :, 1]\n    enhanced = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8)).apply(green)\n    top_hat = cv2.morphologyEx(enhanced, cv2.MORPH_TOPHAT, cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (15, 15)))\n    values = top_hat[retina_mask]\n    threshold = np.percentile(values, 97.0) if values.size else 255\n    proxy = ((top_hat >= threshold) & retina_mask & (~optic_disc_mask)).astype(np.uint8)\n    proxy = cv2.morphologyEx(proxy, cv2.MORPH_OPEN, cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (3, 3)))\n    return proxy.astype(bool)\n\ndef create_dark_lesion_proxy_mask(image_rgb, retina_mask):\n    gray = cv2.cvtColor(image_rgb, cv2.COLOR_RGB2GRAY).astype(np.float32)\n    local = cv2.GaussianBlur(gray, (0, 0), sigmaX=9)\n    dark_response = local - gray\n    values = dark_response[retina_mask]\n    threshold = np.percentile(values, 97.0) if values.size else np.inf\n    proxy = ((dark_response >= threshold) & retina_mask).astype(np.uint8)\n    proxy = cv2.morphologyEx(proxy, cv2.MORPH_OPEN, cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (3, 3)))\n    return proxy.astype(bool)\n\ndef heatmap_topk_overlap(heatmap, proxy_mask, retina_mask, top_percent=10):\n    allowed = np.flatnonzero(retina_mask.reshape(-1))\n    if allowed.size == 0:\n        return np.nan, np.nan\n    count = max(1, int(round(allowed.size * top_percent / 100.0)))\n    flat = heatmap.reshape(-1)\n    top_indices = allowed[np.argpartition(flat[allowed], -count)[-count:]]\n    top_mask = np.zeros(flat.size, dtype=bool); top_mask[top_indices] = True\n    top_mask = top_mask.reshape(heatmap.shape)\n    return float((top_mask & proxy_mask).sum() / count), float((top_mask & retina_mask).sum() / count)\n\ndef retina_aware_mask(image_rgb, heatmap, retina_mask, fraction, mode, rng=None):\n    allowed = np.flatnonzero(retina_mask.reshape(-1))\n    if allowed.size == 0:\n        raise ValueError('Empty retina mask.')\n    count = max(1, int(round(float(fraction) * allowed.size)))\n    flat = heatmap.reshape(-1)\n    if mode == 'most':\n        indices = allowed[np.argpartition(flat[allowed], -count)[-count:]]\n    elif mode == 'least':\n        indices = allowed[np.argpartition(flat[allowed], count - 1)[:count]]\n    elif mode == 'random':\n        indices = (rng or np.random.default_rng(SEED)).choice(allowed, size=count, replace=False)\n    else:\n        raise ValueError('Unknown retina-aware mask mode: ' + mode)\n    mask = np.zeros(flat.size, dtype=bool); mask[indices] = True; mask = mask.reshape(heatmap.shape)\n    baseline = cv2.GaussianBlur(image_rgb, (0, 0), sigmaX=10)\n    output = image_rgb.copy(); output[mask] = baseline[mask]\n    return output\n\nquality_source_df = analysis_generation_df[analysis_generation_df.cam_status.isin(['success', 'fallback_raw_abs_cam'])].copy()\nretina_eval_ids = set(metric_success_df.image_id.drop_duplicates().head(MAX_RETINA_AWARE_EVAL_IMAGES))\nquality_rows, quality_cache = [], {}\nfor row in quality_source_df.itertuples(index=False):\n    image_rgb = preprocess_image(TRAIN_IMAGES_DIR / f'{row.image_id}.png')\n    heatmap = np.load(row.raw_heatmap_path)\n    retina_mask = create_retina_mask(image_rgb)\n    optic_disc_mask = estimate_optic_disc_mask(image_rgb, retina_mask)\n    bright_proxy = create_bright_lesion_proxy_mask(image_rgb, retina_mask, optic_disc_mask)\n    dark_proxy = create_dark_lesion_proxy_mask(image_rgb, retina_mask)\n    bright_overlap, top_retina_ratio = heatmap_topk_overlap(heatmap, bright_proxy, retina_mask)\n    dark_overlap, _ = heatmap_topk_overlap(heatmap, dark_proxy, retina_mask)\n    lesion_overlap, _ = heatmap_topk_overlap(heatmap, bright_proxy | dark_proxy, retina_mask)\n    base_metrics = metric_success_df[(metric_success_df.image_id == row.image_id) & (metric_success_df.method == row.method)]\n    base = base_metrics.iloc[0].to_dict() if len(base_metrics) else {}\n    retina_metrics = {'retina_aware_important_drop': np.nan, 'retina_aware_random_drop': np.nan, 'retina_aware_least_drop': np.nan, 'retina_aware_deletion_auc': np.nan, 'retina_aware_aopc': np.nan, 'retina_aware_metric_status': 'not_evaluated'}\n    if row.image_id in retina_eval_ids:\n        try:\n            original = target_probability(image_rgb, int(row.target_class))\n            most = target_probability(retina_aware_mask(image_rgb, heatmap, retina_mask, 0.20, 'most'), int(row.target_class))\n            random_value = target_probability(retina_aware_mask(image_rgb, heatmap, retina_mask, 0.20, 'random', np.random.default_rng(SEED + len(quality_rows))), int(row.target_class))\n            least = target_probability(retina_aware_mask(image_rgb, heatmap, retina_mask, 0.20, 'least'), int(row.target_class))\n            curve = [target_probability(retina_aware_mask(image_rgb, heatmap, retina_mask, fraction, 'most'), int(row.target_class)) for fraction in deletion_steps]\n            denom = float(deletion_steps[-1] - deletion_steps[0])\n            retina_metrics = {'retina_aware_important_drop': original - most, 'retina_aware_random_drop': original - random_value, 'retina_aware_least_drop': original - least, 'retina_aware_deletion_auc': np.nan if denom <= 0 else safe_trapezoid(curve, deletion_steps) / denom, 'retina_aware_aopc': float(np.mean([original - value for value in curve])), 'retina_aware_metric_status': 'success'}\n        except Exception as exc:\n            retina_metrics['retina_aware_metric_status'] = 'failed:' + repr(exc)\n    active_ratio = float(base.get('active_area_ratio_ge_0_5', (heatmap >= 0.5).mean()))\n    important_drop = float(base.get('confidence_drop_important', np.nan))\n    random_drop = float(base.get('confidence_drop_random', np.nan))\n    retina_focus = heatmap_mass_inside_retina(heatmap, retina_mask)\n    background_focus = heatmap_mass_outside_retina(heatmap, retina_mask)\n    optic_focus = heatmap_mass_on_optic_disc(heatmap, optic_disc_mask)\n    flags = []\n    if np.isfinite(background_focus) and background_focus > 0.15: flags.append('border_artifact_risk')\n    if np.isfinite(optic_focus) and optic_focus > 0.20: flags.append('optic_disc_dominant')\n    if np.isfinite(important_drop) and np.isfinite(random_drop) and important_drop <= random_drop: flags.append('weak_faithfulness')\n    if active_ratio > 0.15: flags.append('diffuse_heatmap')\n    if not flags: flags.append('good_candidate')\n    if any(flag != 'good_candidate' for flag in flags): flags.append('needs_manual_review')\n    if 'border_artifact_risk' in flags:\n        visual_note = 'Potential border/background emphasis; manual review.'\n    elif 'optic_disc_dominant' in flags:\n        visual_note = 'Estimated optic-disc emphasis; potential confounding to review.'\n    elif np.isfinite(bright_overlap) and bright_overlap >= dark_overlap:\n        visual_note = 'Top heatmap area overlaps a bright visual proxy; not a lesion annotation.'\n    else:\n        visual_note = 'Predominantly retinal attribution; proxy overlap remains non-clinical.'\n    quality_rows.append({**row._asdict(), 'important_drop': important_drop, 'random_drop': random_drop, 'least_drop': float(base.get('confidence_drop_least', np.nan)), 'active_area_ratio': active_ratio, 'retina_focus_ratio': retina_focus, 'background_focus_ratio': background_focus, 'border_focus_flag': bool(np.isfinite(background_focus) and background_focus > 0.15), 'optic_disc_focus_ratio': optic_focus, 'optic_disc_focus_flag': bool(np.isfinite(optic_focus) and optic_focus > 0.20), 'bright_proxy_overlap_top10': bright_overlap, 'dark_proxy_overlap_top10': dark_overlap, 'lesion_proxy_overlap_top10': lesion_overlap, 'heatmap_top10_retina_ratio': top_retina_ratio, 'xai_quality_flag': '|'.join(flags), 'visual_audit_note': visual_note, **retina_metrics})\n    quality_cache[(row.image_id, row.method)] = {'image': image_rgb, 'heatmap': heatmap, 'retina': retina_mask, 'disc': optic_disc_mask, 'bright': bright_proxy, 'dark': dark_proxy}\n\nquality_df = pd.DataFrame(quality_rows)\nquality_df.to_csv(OUTPUT_DIRS['metrics'] / 'xai_clinical_quality_per_image.csv', index=False)\ndisplay(quality_df[['image_id', 'true_class', 'predicted_class', 'method', 'retina_focus_ratio', 'background_focus_ratio', 'optic_disc_focus_ratio', 'bright_proxy_overlap_top10', 'dark_proxy_overlap_top10', 'xai_quality_flag']])\n","metadata":{},"outputs":[],"execution_count":null},{"id":"0d141c14-a8a9-4d9a-a2a7-01abfda17103","cell_type":"code","source":"def top10_contour(image_rgb, heatmap, retina_mask):\n    allowed = np.flatnonzero(retina_mask.reshape(-1)); count = max(1, int(round(allowed.size * 0.10)))\n    flat = heatmap.reshape(-1); indices = allowed[np.argpartition(flat[allowed], -count)[-count:]]\n    mask = np.zeros(flat.size, dtype=np.uint8); mask[indices] = 255; mask = mask.reshape(heatmap.shape)\n    result = image_rgb.copy(); contours, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)\n    cv2.drawContours(result, contours, -1, (255, 0, 0), 1)\n    return result\n\ndef save_quality_audit(rows, title, output_path):\n    rows = rows.head(2).reset_index(drop=True)\n    if rows.empty:\n        fig, ax = plt.subplots(figsize=(7, 2)); ax.text(0.5, 0.5, 'No samples available for this audit category', ha='center', va='center'); ax.axis('off'); fig.savefig(output_path, dpi=180, bbox_inches='tight'); plt.close(fig); return\n    fig, axes = plt.subplots(len(rows), 7, figsize=(21, 3.6 * len(rows)), squeeze=False)\n    for idx, row in rows.iterrows():\n        grad = quality_cache[(row.image_id, 'gradcam')]; gradpp = quality_cache[(row.image_id, 'gradcam_pp')]\n        panels = [(grad['image'], 'Original'), (grad['retina'], 'Retina mask'), (grad['bright'], 'Bright proxy'), (grad['dark'], 'Dark proxy'), (overlay_heatmap(grad['image'], grad['heatmap']), 'Grad-CAM'), (overlay_heatmap(gradpp['image'], gradpp['heatmap']), 'Grad-CAM++'), (top10_contour(grad['image'], grad['heatmap'], grad['retina']), 'Top-10% contour')]\n        for ax, (image, label) in zip(axes[idx], panels):\n            ax.imshow(image, cmap='gray' if np.asarray(image).ndim == 2 else None); ax.set_title(label, fontsize=9); ax.axis('off')\n        axes[idx, 0].set_ylabel(f\"{row['image_id']}\\n{row['xai_quality_flag']}\\n{row['visual_audit_note']}\", fontsize=8)\n    fig.suptitle(title, fontsize=13); fig.tight_layout(); fig.savefig(output_path, dpi=220, bbox_inches='tight'); plt.show(); plt.close(fig)\n\nquality_by_image = quality_df[quality_df.method.eq('gradcam')].copy()\nfor class_name in CLASS_NAMES:\n    save_quality_audit(quality_by_image[quality_by_image.true_class.eq(class_name)], f'Clinical-oriented XAI audit — {class_name}', QUALITY_DIR / f'xai_quality_audit_per_class_{class_name}.png')\nsave_quality_audit(quality_by_image[quality_by_image.xai_quality_flag.eq('good_candidate')], 'XAI quality — good candidates', QUALITY_DIR / 'xai_quality_good_candidates.png')\nsave_quality_audit(quality_by_image[quality_by_image.xai_quality_flag.str.contains('needs_manual_review', na=False)], 'XAI quality — needs manual review', QUALITY_DIR / 'xai_quality_needs_review.png')\nsave_quality_audit(quality_by_image[quality_by_image.xai_quality_flag.str.contains('border_artifact_risk', na=False)], 'XAI quality — border artifact risk', QUALITY_DIR / 'xai_quality_border_artifact_risk.png')\nsave_quality_audit(quality_by_image[quality_by_image.xai_quality_flag.str.contains('optic_disc_dominant', na=False)], 'XAI quality — optic-disc dominant', QUALITY_DIR / 'xai_quality_optic_disc_dominant.png')\n\nquality_summary = quality_df.groupby('method', as_index=False).agg(samples=('image_id', 'count'), retina_focus_mean=('retina_focus_ratio', 'mean'), background_focus_mean=('background_focus_ratio', 'mean'), optic_disc_focus_mean=('optic_disc_focus_ratio', 'mean'), bright_proxy_overlap_mean=('bright_proxy_overlap_top10', 'mean'), dark_proxy_overlap_mean=('dark_proxy_overlap_top10', 'mean'), lesion_proxy_overlap_mean=('lesion_proxy_overlap_top10', 'mean'))\nquality_class_summary = quality_df.groupby(['true_class', 'method'], as_index=False).agg(samples=('image_id', 'count'), retina_focus_mean=('retina_focus_ratio', 'mean'), background_focus_mean=('background_focus_ratio', 'mean'), optic_disc_focus_mean=('optic_disc_focus_ratio', 'mean'), lesion_proxy_overlap_mean=('lesion_proxy_overlap_top10', 'mean'))\nquality_flags = quality_df.xai_quality_flag.value_counts().to_frame('count')\ndisplay(quality_summary); display(quality_class_summary); display(quality_flags)\nclinical_summary_lines = [\n    '## Clinical-Oriented XAI Quality Analysis',\n    f'- Analysed heatmap records: {len(quality_df)} from {quality_df.image_id.nunique()} images.',\n    f'- Mean retina focus and background focus are reported per method below; high background focus is treated as an artifact-risk flag, not as a clinical interpretation.',\n    f'- Border-risk records: {int(quality_df.border_focus_flag.sum())}; optic-disc dominant records: {int(quality_df.optic_disc_focus_flag.sum())}.',\n    f'- Good-candidate records: {int(quality_df.xai_quality_flag.eq('good_candidate').sum())}; manual-review records: {int(quality_df.xai_quality_flag.str.contains('needs_manual_review', na=False).sum())}.',\n    '- Bright/dark proxy overlap is a non-clinical visual audit only. It cannot substitute for lesion annotation, expert grading, IoU, Dice, or pixel-level validation.',\n]\n(OUTPUT_DIRS['reports'] / 'xai_clinical_quality_analysis.md').write_text('\\n'.join(clinical_summary_lines + ['', '## Method summary', quality_summary.to_string(index=False), '', '## Per-class summary', quality_class_summary.to_string(index=False), '', '## Quality flags', quality_flags.to_string()]))\nprint('\\n'.join(clinical_summary_lines))\n","metadata":{},"outputs":[],"execution_count":null},{"id":"4ed28c11-28d5-42b2-b802-b256555a3abe","cell_type":"markdown","source":"## 16. Discussion Notes and Clinical Safety\n\n- Model dimuat dari checkpoint ekspor dan tidak dilatih ulang.\n- Grad-CAM dan Grad-CAM++ menunjukkan area yang memengaruhi logit kelas target pada model, bukan lokasi lesi yang telah tervalidasi.\n- Karena APTOS tidak menyediakan lesion mask, evaluasi memakai perturbation-based confidence drop dan deletion curve, bukan IoU/Dice.\n- Periksa secara visual kemungkinan fokus pada border hitam, noise, watermark, atau artefak kamera.\n- Heatmap kosong dapat terjadi ketika weighted activation kelas target seluruhnya negatif lalu menjadi nol setelah ReLU.\n- Fallback absolute raw CAM hanya dipakai untuk visual/debug dan dicatat; analisis utama mengutamakan status `success`, bukan fallback.\n- Jangan menyimpulkan bahwa microaneurysm, hemorrhage, exudate, atau neovascularization terdeteksi pasti tanpa anotasi ahli.\n- Failure case dan prediksi salah tetap disimpan dan diringkas bila muncul pada subset.","metadata":{}},{"id":"f00202a4-2850-4d53-90d7-1b56a210e371","cell_type":"markdown","source":"## 17. Save Final Summary and Download Archive\n","metadata":{}},{"id":"02904b0a-9e1c-4dea-89b1-df86828e7810","cell_type":"code","source":"success_count = int((cam_debug_df.status_group == 'success').sum()) if len(cam_debug_df) else 0\nfallback_count = int((cam_debug_df.status_group == 'fallback').sum()) if len(cam_debug_df) else 0\nfailed_debug = cam_debug_df[cam_debug_df.status_group.eq('failed')] if len(cam_debug_df) else pd.DataFrame()\nfailed_items = [f\"{row.image_id}:{row.method}\" for row in failed_debug.itertuples(index=False)] if len(failed_debug) else []\nmetric_success_count = int(len(metric_success_df))\nmetric_failure_count = int(len(metric_failure_df))\nmetric_error_items = [f\"{row.image_id}:{row.method}:{row.metric_error}\" for row in metric_failure_df.itertuples(index=False)] if metric_failure_count else []\nxai_readiness = 'analyzable_with_success_heatmaps' if success_count else 'not_ready_no_standard_success_heatmap'\nanalysis_lines = globals().get('result_analysis_lines', ['## XAI Result Analysis', '- Analysis section was not executed.'])\nanalysis_metric_summary = globals().get('analysis_summary_df', summary_df)\nanalysis_per_class = globals().get('per_class_analysis_df', per_class_df)\nanalysis_confidence = globals().get('confidence_summary', correctness_df)\nclinical_lines = globals().get('clinical_summary_lines', ['## Clinical-Oriented XAI Quality Analysis', '- Clinical-oriented audit section was not executed.'])\nclinical_method_summary = globals().get('quality_summary', pd.DataFrame())\nclinical_class_summary = globals().get('quality_class_summary', pd.DataFrame())\narchive_path = WORKING_ROOT / 'xai_outputs.zip'\narchive_roots = [OUTPUT_DIRS['reports'], OUTPUT_DIRS['metrics'], OUTPUT_ROOT / 'figures', OUTPUT_DIRS['predictions'], OUTPUT_DIRS['overlays']]\n\nfinal_summary = [\n    '# APTOS v8 Kaggle XAI Final Summary', '',\n    '- Generated at: ' + datetime.now(timezone.utc).isoformat(),\n    '- Model: ' + resolved_name,\n    '- Model key / fold: ' + TARGET_MODEL_KEY + ' / ' + str(TARGET_FOLD),\n    '- Checkpoint: `' + str(state_path) + '`',\n    '- Pretrained weights downloaded: no (`pretrained=False`).',\n    '- Retraining performed: no.',\n    '- Inference sanity check: PASS on ' + str(len(sanity_predictions)) + ' real APTOS images.',\n    '- XAI selected samples: ' + str(len(selected_xai_df)),\n    '- Perturbation evaluation samples: ' + str(len(evaluation_samples)),\n    '- XAI methods: Grad-CAM, Grad-CAM++.',\n    '- CAM success count: ' + str(success_count),\n    '- CAM fallback count: ' + str(fallback_count),\n    '- CAM failed count: ' + str(len(failed_items)),\n    '- Failed image/method pairs: ' + (', '.join(failed_items) if failed_items else 'none'),\n    '- XAI metric success count: ' + str(metric_success_count),\n    '- XAI metric failed count: ' + str(metric_failure_count),\n    '- Metric errors: ' + (' | '.join(metric_error_items) if metric_error_items else 'none'),\n    '- XAI readiness: ' + xai_readiness,\n    '- Metrics: important/random/least confidence drop, deletion curve, deletion AUC probability, entropy, active-area ratio, and generation runtime.',\n    '- Outputs: `' + str(OUTPUT_ROOT) + '`.',\n    '- Download archive: `' + str(archive_path) + '`.',\n    '', *analysis_lines,\n    '', '## Grad-CAM vs Grad-CAM++ Summary', analysis_metric_summary.to_string(index=False),\n    '', '## Per-Class XAI Findings', analysis_per_class.to_string(index=False),\n    '', '## Correct vs Incorrect Prediction Findings', analysis_confidence.to_string(index=False),\n    '', *clinical_lines,\n    '', '## Retina, Border, Optic Disc, and Proxy Summary', clinical_method_summary.to_string(index=False),\n    '', '## Clinical-Oriented Per-Class Summary', clinical_class_summary.to_string(index=False),\n    '', '## Metric Interpretation',\n    '- Larger confidence drop after important-region masking indicates that the highlighted area contributed more strongly to the model target score than the perturbed baseline.',\n    '- Random and least-important masking provide relative baselines; a larger important-region effect is supportive but not proof of clinical relevance.',\n    '- Lower deletion probability/AUC and higher AOPC indicate a stronger model-confidence decline as highly weighted regions are blurred.',\n    '- Active area ratio and entropy describe concentration/spread, while runtime describes computational efficiency; neither establishes clinical correctness.',\n    '', '## Limitations',\n    '- The baseline gallery is a small fold-0 qualitative subset, not full-dataset validation; Sections 18–19 add a fold-aware pass when assignments are available.',\n    '- No lesion masks are available; localization metrics are intentionally not reported.',\n    '- A failed CAM can arise from non-spatial/incorrect activations, missing gradients, numerical instability, or all-negative weighted activations after ReLU.',\n    '- Fallback absolute raw CAM is flagged and not pooled with standard-success Grad-CAM in the primary interpretation.',\n    '- Heatmaps are model-attribution visualizations, not medical diagnostic evidence.',\n    '- Ensemble heatmaps are not computed in this notebook and require a separate, explicit methodology.',\n    '', '## Generated metric summary', summary_df.to_string(index=False),\n]\n(OUTPUT_DIRS['reports'] / 'xai_final_summary.md').write_text('\\n'.join(final_summary))\nwith zipfile.ZipFile(archive_path, 'w', compression=zipfile.ZIP_DEFLATED) as archive:\n    for root in archive_roots:\n        if root.exists():\n            for file_path in root.rglob('*'):\n                if file_path.is_file():\n                    archive.write(file_path, file_path.relative_to(OUTPUT_ROOT.parent))\nprint('Final summary:', OUTPUT_DIRS['reports'] / 'xai_final_summary.md')\nprint('Download archive:', archive_path)\nprint('All outputs:', OUTPUT_ROOT)\n","metadata":{},"outputs":[],"execution_count":null},{"id":"563fe7c7-6f72-42aa-bcde-5b6f78d5f312","cell_type":"markdown","source":"## 18. Advanced XAI Audit and Improvement\n\nThis section adds fold-aware attribution, a small multi-checkpoint consistency audit, and transparent report-case ranking. It does not retrain or alter the exported models.\n","metadata":{}},{"id":"637114b5-69ae-45b9-960f-26cabbafebb5","cell_type":"code","source":"# Advanced configuration, current-result audit, and fold assignment resolution\nFOLDAWARE_ROOT = OUTPUT_ROOT / 'xai_foldaware'\nFOLDAWARE_DIRS = {\n    'root': FOLDAWARE_ROOT,\n    'gradcam': FOLDAWARE_ROOT / 'gradcam',\n    'gradcam_pp': FOLDAWARE_ROOT / 'gradcam_pp',\n    'overlays': FOLDAWARE_ROOT / 'overlays',\n    'metrics': FOLDAWARE_ROOT / 'metrics',\n    'figures': OUTPUT_ROOT / 'figures' / 'xai_foldaware',\n    'consistency_figures': OUTPUT_ROOT / 'figures' / 'xai_consistency',\n    'report_examples': OUTPUT_ROOT / 'figures' / 'xai_report_examples',\n}\nfor directory in FOLDAWARE_DIRS.values():\n    directory.mkdir(parents=True, exist_ok=True)\n\nADVANCED_REQUIRED = [\n    OUTPUT_DIRS['metrics'] / 'xai_generation_records.csv',\n    OUTPUT_DIRS['metrics'] / 'xai_metrics_per_image.csv',\n    OUTPUT_DIRS['metrics'] / 'xai_clinical_quality_per_image.csv',\n]\nmissing_advanced_inputs = [str(path) for path in ADVANCED_REQUIRED if not path.exists()]\nif missing_advanced_inputs:\n    raise FileNotFoundError('Run Sections 9–15 before the advanced audit. Missing: ' + str(missing_advanced_inputs))\n\nadvanced_generation_df = pd.read_csv(OUTPUT_DIRS['metrics'] / 'xai_generation_records.csv')\nadvanced_metrics_df = pd.read_csv(OUTPUT_DIRS['metrics'] / 'xai_metrics_per_image.csv')\nadvanced_quality_df = pd.read_csv(OUTPUT_DIRS['metrics'] / 'xai_clinical_quality_per_image.csv')\nadvanced_quality_df['correct'] = advanced_quality_df['correct'].astype(str).str.lower().isin(['true', '1'])\n\ndef fold_assignment_candidates(bundle_dir):\n    preferred = bundle_dir / 'metadata' / 'kfold_metadata.csv'\n    candidates = [preferred] if preferred.is_file() else []\n    for path in sorted((bundle_dir / 'metadata').glob('*fold*.csv')):\n        if path not in candidates:\n            candidates.append(path)\n    return candidates\n\ndef load_fold_assignment(bundle_dir):\n    for path in fold_assignment_candidates(bundle_dir):\n        try:\n            frame = pd.read_csv(path)\n        except Exception:\n            continue\n        id_column = next((column for column in ['image_id', 'id_code'] if column in frame.columns), None)\n        if id_column and 'fold' in frame.columns:\n            result = frame[[id_column, 'fold']].dropna().copy()\n            result = result.rename(columns={id_column: 'image_id'})\n            result['image_id'] = result['image_id'].astype(str)\n            result['fold'] = pd.to_numeric(result['fold'], errors='coerce')\n            result = result.dropna(subset=['fold'])\n            result['fold'] = result['fold'].astype(int)\n            result = result[result['fold'].isin([0, 1, 2])].drop_duplicates('image_id')\n            if len(result):\n                return result, path, 'available'\n    return pd.DataFrame(columns=['image_id', 'fold']), None, 'unavailable'\n\nfold_assignment_df, fold_assignment_path, fold_assignment_status = load_fold_assignment(WEIGHT_BUNDLE_DIR)\nfold_map = dict(zip(fold_assignment_df.image_id, fold_assignment_df.fold))\navailable_c_folds = sorted(set(int(value) for value in available_folds.get(TARGET_MODEL_KEY, [])))\n\nbaseline_by_image = advanced_generation_df[advanced_generation_df.method.eq('gradcam')].drop_duplicates('image_id').copy()\nquality_for_selection = advanced_quality_df[advanced_quality_df.method.eq('gradcam')].drop_duplicates('image_id').copy()\nselection_pool = baseline_by_image.merge(\n    quality_for_selection[['image_id', 'retina_focus_ratio', 'background_focus_ratio', 'optic_disc_focus_ratio',\n                           'lesion_proxy_overlap_top10', 'active_area_ratio', 'xai_quality_flag']],\n    on='image_id', how='left'\n)\nselection_pool['correct'] = selection_pool['correct'].astype(str).str.lower().isin(['true', '1'])\nselection_pool['fold'] = selection_pool.image_id.map(fold_map)\nselection_pool['fold_assignment_status'] = np.where(selection_pool.fold.notna(), 'matched', 'fallback_fold0_no_assignment')\nselection_pool['fold'] = selection_pool.fold.fillna(TARGET_FOLD).astype(int)\nselection_pool.loc[~selection_pool.fold.isin(available_c_folds), 'fold'] = TARGET_FOLD\nselection_pool['checkpoint_used'] = selection_pool.fold.map(lambda fold: f'{TARGET_MODEL_KEY}_fold{fold}')\nselection_pool['selection_rank'] = (\n    selection_pool.retina_focus_ratio.fillna(0)\n    - selection_pool.background_focus_ratio.fillna(0)\n    + selection_pool.lesion_proxy_overlap_top10.fillna(0)\n    - selection_pool.optic_disc_focus_ratio.fillna(0)\n    - selection_pool.active_area_ratio.fillna(0)\n)\n\nenhanced_parts = []\nfor class_name in CLASS_NAMES:\n    group = selection_pool[selection_pool.true_class.eq(class_name)].copy()\n    selected = []\n    def select_group(frame, reason):\n        for row in frame.itertuples(index=False):\n            if row.image_id not in {item['image_id'] for item in selected} and len(selected) < MAX_XAI_SAMPLES_PER_CLASS:\n                item = row._asdict(); item['selected_reason'] = reason; selected.append(item)\n    select_group(group[group.correct & group.xai_quality_flag.eq('good_candidate')].sort_values(['selection_rank', 'confidence'], ascending=False), 'correct_good_candidate')\n    select_group(group[(~group.correct) | group.xai_quality_flag.str.contains('needs_manual_review', na=False)].sort_values(['selection_rank', 'confidence'], ascending=True), 'failure_or_manual_review')\n    median_confidence = group.confidence.median() if len(group) else np.nan\n    select_group(group.assign(_distance=(group.confidence - median_confidence).abs()).sort_values('_distance'), 'representative_mid_confidence')\n    select_group(group.sort_values('confidence', ascending=False), 'representative_fill')\n    enhanced_parts.extend(selected)\n\nenhanced_samples_df = pd.DataFrame(enhanced_parts)\nif enhanced_samples_df.empty:\n    raise RuntimeError('Advanced sample selection produced no samples.')\nenhanced_columns = ['image_id', 'true_class', 'predicted_class', 'confidence', 'correct', 'selected_reason', 'fold', 'checkpoint_used', 'xai_quality_flag']\nfor column in enhanced_columns:\n    if column not in enhanced_samples_df:\n        enhanced_samples_df[column] = np.nan\nenhanced_samples_df = enhanced_samples_df.drop_duplicates('image_id').reset_index(drop=True)\nenhanced_samples_df.to_csv(OUTPUT_DIRS['predictions'] / 'xai_selected_samples_enhanced.csv', index=False)\n\nadvanced_flag_counts = advanced_quality_df.xai_quality_flag.fillna('unknown').value_counts()\nadvanced_per_class_df = advanced_quality_df.groupby(['true_class', 'method'], as_index=False).agg(\n    records=('image_id', 'count'), correct=('correct', 'sum'), confidence_mean=('confidence', 'mean'),\n    good_candidates=('xai_quality_flag', lambda values: int((values == 'good_candidate').sum())),\n    manual_review=('xai_quality_flag', lambda values: int(values.str.contains('needs_manual_review', na=False).sum())),\n    border_risk=('xai_quality_flag', lambda values: int(values.str.contains('border_artifact_risk', na=False).sum())),\n    optic_disc_dominant=('xai_quality_flag', lambda values: int(values.str.contains('optic_disc_dominant', na=False).sum())),\n)\nadvanced_correctness_df = advanced_quality_df.groupby(['method', 'correct'], as_index=False).agg(\n    records=('image_id', 'count'), confidence_mean=('confidence', 'mean'), retina_focus_mean=('retina_focus_ratio', 'mean'),\n    important_drop_mean=('important_drop', 'mean'), lesion_proxy_overlap_mean=('lesion_proxy_overlap_top10', 'mean')\n)\nadvanced_method_df = advanced_quality_df.groupby('method', as_index=False).agg(\n    records=('image_id', 'count'), retina_focus_mean=('retina_focus_ratio', 'mean'), background_focus_mean=('background_focus_ratio', 'mean'),\n    optic_disc_focus_mean=('optic_disc_focus_ratio', 'mean'), important_drop_mean=('important_drop', 'mean'),\n    random_drop_mean=('random_drop', 'mean'), lesion_proxy_overlap_mean=('lesion_proxy_overlap_top10', 'mean'),\n    active_area_mean=('active_area_ratio', 'mean'), runtime_seconds_mean=('runtime_seconds', 'mean')\n)\nadvanced_overview = [\n    '## Advanced XAI Audit and Improvement',\n    f'- Baseline XAI records available: {len(advanced_generation_df)} CAM records from {advanced_generation_df.image_id.nunique()} images.',\n    f\"- Good-candidate records: {int(advanced_quality_df.xai_quality_flag.eq('good_candidate').sum())}; manual-review records: {int(advanced_quality_df.xai_quality_flag.str.contains('needs_manual_review', na=False).sum())}.\",\n    f\"- Border-risk records: {int(advanced_quality_df.xai_quality_flag.str.contains('border_artifact_risk', na=False).sum())}; optic-disc-dominant records: {int(advanced_quality_df.xai_quality_flag.str.contains('optic_disc_dominant', na=False).sum())}.\",\n    f\"- Correct records: {int(advanced_quality_df.correct.sum())}; incorrect records: {int((~advanced_quality_df.correct).sum())}.\",\n    f\"- Fold assignment: {fold_assignment_status}; source: `{fold_assignment_path}`; matched enhanced samples: {int(enhanced_samples_df.fold_assignment_status.eq('matched').sum())}/{len(enhanced_samples_df)}.\",\n    '- The earlier pipeline is technically successful, but a single fold-0 checkpoint can have seen an analysed image during training. The fold-aware pass below addresses that limitation when the exported fold assignment is available.',\n    '- Remaining limitations include a small purposive subset, no lesion masks, heuristic visual proxies, and expected variability between independently trained fold checkpoints.',\n]\n(OUTPUT_DIRS['reports'] / 'xai_advanced_preflight.md').write_text('\\n'.join(advanced_overview + ['', '## Baseline quality flags', advanced_flag_counts.to_string(), '', '## Baseline Grad-CAM vs Grad-CAM++ summary', advanced_method_df.to_string(index=False), '', '## Baseline per-class summary', advanced_per_class_df.to_string(index=False), '', '## Baseline correct vs incorrect', advanced_correctness_df.to_string(index=False), '', '## Enhanced sample selection', enhanced_samples_df[enhanced_columns].to_string(index=False)]))\ndisplay(enhanced_samples_df[enhanced_columns])\ndisplay(advanced_method_df); display(advanced_per_class_df); display(advanced_correctness_df)\nprint('\\n'.join(advanced_overview))\n","metadata":{},"outputs":[],"execution_count":null},{"id":"e16fccc3-9ca0-4c55-b3d8-a44a917842da","cell_type":"code","source":"# Fold-aware CAM generation, retina-aware perturbation metrics, and internal quality ranking\nfold_model_cache = {TARGET_FOLD: model}\nfold_checkpoint_cache = {TARGET_FOLD: (target_record, checkpoint_meta, state_path)}\n\ndef load_model_for_fold(fold):\n    fold = int(fold)\n    if fold not in available_c_folds:\n        raise LookupError(f'No {TARGET_MODEL_KEY} cross-entropy checkpoint for fold {fold}. Available: {available_c_folds}')\n    if fold in fold_model_cache:\n        return fold_model_cache[fold], fold_checkpoint_cache[fold]\n    record, meta = checkpoint_for(TARGET_MODEL_KEY, fold)\n    cfg = meta['cfg']\n    checkpoint_path = WEIGHT_BUNDLE_DIR / record['state_file']\n    rebuilt = TimmFeatureModel(meta['resolved_model_name'], int(cfg['input_size']), out_dim=5, pooling_type=cfg['pooling_type'], pretrained=False)\n    try:\n        state = torch.load(checkpoint_path, map_location='cpu', weights_only=True)\n    except TypeError:\n        state = torch.load(checkpoint_path, map_location='cpu')\n    rebuilt.load_state_dict(state, strict=True)\n    rebuilt = rebuilt.to(DEVICE).eval()\n    fold_model_cache[fold] = rebuilt\n    fold_checkpoint_cache[fold] = (record, meta, checkpoint_path)\n    return rebuilt, fold_checkpoint_cache[fold]\n\ndef predict_rgb_with_model(net, image_rgb):\n    tensor = prepare_input_tensor(image_rgb)\n    with torch.no_grad():\n        logits = net(tensor)\n        probabilities = torch.softmax(logits, dim=1)[0].detach().cpu().numpy()\n    if not np.isfinite(probabilities).all() or not np.isclose(probabilities.sum(), 1.0, atol=1e-5):\n        raise RuntimeError('Invalid probabilities from checkpoint inference.')\n    return tensor, probabilities, int(np.argmax(probabilities))\n\ndef target_probability_with_model(net, image_rgb, target_class):\n    _, probabilities, _ = predict_rgb_with_model(net, image_rgb)\n    return float(probabilities[int(target_class)])\n\ndef save_foldaware_figure(image_rgb, heatmap, metadata, method, output_path):\n    overlay = overlay_heatmap(image_rgb, heatmap)\n    fig, axes = plt.subplots(1, 3, figsize=(13, 4.5))\n    axes[0].imshow(image_rgb); axes[0].set_title('Original'); axes[0].axis('off')\n    axes[1].imshow(heatmap, cmap='jet', vmin=0, vmax=1); axes[1].set_title(method + ' heatmap'); axes[1].axis('off')\n    axes[2].imshow(overlay); axes[2].set_title('Overlay'); axes[2].axis('off')\n    fig.suptitle(f\"{metadata['image_id']} | true={metadata['true_class']} | pred={metadata['predicted_class']} ({metadata['confidence']:.3f}) | {method} | C fold {metadata['fold']}\", fontsize=10)\n    fig.tight_layout(); fig.savefig(output_path, dpi=220, bbox_inches='tight'); plt.close(fig)\n    return overlay\n\nfoldaware_records, foldaware_debug_rows, foldaware_registry = [], [], {}\nfor sample in enhanced_samples_df.itertuples(index=False):\n    sample_dict = sample._asdict()\n    requested_fold = int(sample_dict['fold']) if pd.notna(sample_dict['fold']) else TARGET_FOLD\n    use_fold = requested_fold if XAI_MODE == 'fold_aware' and requested_fold in available_c_folds else TARGET_FOLD\n    used_fallback = bool(XAI_MODE != 'fold_aware' or sample_dict.get('fold_assignment_status') != 'matched' or use_fold != requested_fold)\n    try:\n        net, (_, fold_meta, fold_checkpoint_path) = load_model_for_fold(use_fold)\n        image_rgb = preprocess_image(TRAIN_IMAGES_DIR / f\"{sample_dict['image_id']}.png\")\n        input_tensor, probabilities, predicted_label = predict_rgb_with_model(net, image_rgb)\n    except Exception as exc:\n        for method in CAM_GENERATORS:\n            foldaware_debug_rows.append({**sample_dict, 'method': method, 'fold_used': use_fold, 'status': 'failed', 'status_group': 'failed', 'error': repr(exc)})\n        continue\n    fold_metadata = {**sample_dict, 'fold': use_fold, 'predicted_label': predicted_label, 'predicted_class': CLASS_NAMES[predicted_label], 'confidence': float(probabilities[predicted_label]), 'correct': int(predicted_label) == int(sample_dict['true_label'])}\n    target_class = predicted_label\n    for method, generator in CAM_GENERATORS.items():\n        started = time.perf_counter()\n        try:\n            result = generator(net, input_tensor, target_class)\n            runtime_seconds = time.perf_counter() - started\n            status = result['status']\n            diagnostic = result['diagnostic']\n            diagnostic.update({**fold_metadata, 'method': method, 'target_class': target_class, 'fold_used': use_fold, 'checkpoint_path': str(fold_checkpoint_path), 'status': status, 'status_group': 'success' if status == 'success' else ('fallback' if status.startswith('fallback') else 'failed'), 'runtime_seconds': runtime_seconds, 'error': ''})\n            foldaware_debug_rows.append(diagnostic)\n            if status not in VALID_CAM_STATUSES:\n                continue\n            heatmap = result['heatmap']\n            stem = f\"{sample_dict['image_id']}_{method}_{TARGET_MODEL_KEY}_fold{use_fold}\"\n            raw_path = FOLDAWARE_DIRS[method] / f'{stem}.npy'\n            figure_path = FOLDAWARE_DIRS['overlays'] / f'{stem}.png'\n            overlay_path = FOLDAWARE_DIRS['overlays'] / f'{stem}_overlay.png'\n            np.save(raw_path, heatmap.astype(np.float32), allow_pickle=False)\n            overlay = save_foldaware_figure(image_rgb, heatmap, fold_metadata, method + (' [fallback]' if status != 'success' else ''), figure_path)\n            cv2.imwrite(str(overlay_path), cv2.cvtColor(overlay, cv2.COLOR_RGB2BGR))\n            foldaware_registry[(sample_dict['image_id'], method)] = {'image_rgb': image_rgb, 'heatmap': heatmap, 'target_class': target_class, 'model': net, 'fold_used': use_fold, 'status': status}\n            foldaware_records.append({**fold_metadata, 'method': method, 'target_class': target_class, 'cam_status': status, 'is_fallback': status != 'success', 'fold_used': use_fold, 'fold_assignment_status': sample_dict.get('fold_assignment_status', 'unknown'), 'checkpoint_path': str(fold_checkpoint_path), 'runtime_seconds': runtime_seconds, 'raw_heatmap_path': str(raw_path), 'figure_path': str(figure_path), 'overlay_path': str(overlay_path), 'xai_mode': XAI_MODE, 'used_fold0_fallback': used_fallback})\n        except Exception as exc:\n            foldaware_debug_rows.append({**fold_metadata, 'method': method, 'target_class': target_class, 'fold_used': use_fold, 'status': 'failed', 'status_group': 'failed', 'runtime_seconds': time.perf_counter() - started, 'error': repr(exc)})\n\nfoldaware_records_df = pd.DataFrame(foldaware_records)\nfoldaware_debug_df = pd.DataFrame(foldaware_debug_rows)\nfoldaware_records_df.to_csv(FOLDAWARE_DIRS['metrics'] / 'xai_foldaware_generation_records.csv', index=False)\nfoldaware_debug_df.to_csv(FOLDAWARE_DIRS['metrics'] / 'cam_debug_log.csv', index=False)\n\nfoldaware_samples = foldaware_records_df.drop_duplicates('image_id').copy() if len(foldaware_records_df) else pd.DataFrame()\nfoldaware_eval_samples = select_evaluation_subset(foldaware_samples, MAX_EVALUATION_SAMPLES) if len(foldaware_samples) else pd.DataFrame()\nfoldaware_metric_rows, foldaware_curve_rows = [], []\nfor sample in foldaware_eval_samples.itertuples(index=False):\n    for method in CAM_GENERATORS:\n        item = foldaware_registry.get((sample.image_id, method))\n        if item is None or item['status'] not in VALID_CAM_STATUSES:\n            continue\n        base = {'image_id': sample.image_id, 'true_label': sample.true_label, 'true_class': sample.true_class, 'predicted_label': sample.predicted_label, 'predicted_class': sample.predicted_class, 'confidence': sample.confidence, 'correct': sample.correct, 'method': method, 'fold_used': item['fold_used'], 'cam_status': item['status'], 'target_class': item['target_class'], 'runtime_seconds': sample.runtime_seconds}\n        try:\n            retina = create_retina_mask(item['image_rgb'])\n            original = target_probability_with_model(item['model'], item['image_rgb'], item['target_class'])\n            important = target_probability_with_model(item['model'], retina_aware_mask(item['image_rgb'], item['heatmap'], retina, 0.20, 'most'), item['target_class'])\n            random_value = target_probability_with_model(item['model'], retina_aware_mask(item['image_rgb'], item['heatmap'], retina, 0.20, 'random', np.random.default_rng(SEED + len(foldaware_metric_rows))), item['target_class'])\n            least = target_probability_with_model(item['model'], retina_aware_mask(item['image_rgb'], item['heatmap'], retina, 0.20, 'least'), item['target_class'])\n            curve = [target_probability_with_model(item['model'], retina_aware_mask(item['image_rgb'], item['heatmap'], retina, fraction, 'most'), item['target_class']) for fraction in deletion_steps]\n            denom = float(deletion_steps[-1] - deletion_steps[0])\n            foldaware_curve_rows.extend({'image_id': sample.image_id, 'method': method, 'fold_used': item['fold_used'], 'fraction_removed': fraction, 'target_probability': value} for fraction, value in zip(deletion_steps, curve))\n            foldaware_metric_rows.append({**base, 'original_target_probability': original, 'retina_aware_important_drop': original - important, 'retina_aware_random_drop': original - random_value, 'retina_aware_least_drop': original - least, 'retina_aware_deletion_auc': np.nan if denom <= 0 else safe_trapezoid(curve, deletion_steps) / denom, 'retina_aware_aopc': float(np.mean([original - value for value in curve])), 'metric_status': 'success', 'metric_error': ''})\n        except Exception as exc:\n            foldaware_metric_rows.append({**base, 'original_target_probability': np.nan, 'retina_aware_important_drop': np.nan, 'retina_aware_random_drop': np.nan, 'retina_aware_least_drop': np.nan, 'retina_aware_deletion_auc': np.nan, 'retina_aware_aopc': np.nan, 'metric_status': 'failed', 'metric_error': repr(exc)})\n\nfoldaware_metrics_df = pd.DataFrame(foldaware_metric_rows)\nfoldaware_curves_df = pd.DataFrame(foldaware_curve_rows)\nfoldaware_metrics_df.to_csv(FOLDAWARE_DIRS['metrics'] / 'xai_foldaware_metrics_per_image.csv', index=False)\nfoldaware_curves_df.to_csv(FOLDAWARE_DIRS['metrics'] / 'xai_foldaware_deletion_curves.csv', index=False)\n\nmetric_lookup = foldaware_metrics_df[foldaware_metrics_df.metric_status.eq('success')].set_index(['image_id', 'method']).to_dict('index') if len(foldaware_metrics_df) else {}\nfoldaware_quality_rows = []\nfor row in foldaware_records_df.itertuples(index=False):\n    if row.cam_status not in VALID_CAM_STATUSES:\n        continue\n    item = foldaware_registry[(row.image_id, row.method)]\n    retina = create_retina_mask(item['image_rgb'])\n    disc = estimate_optic_disc_mask(item['image_rgb'], retina)\n    bright = create_bright_lesion_proxy_mask(item['image_rgb'], retina, disc)\n    dark = create_dark_lesion_proxy_mask(item['image_rgb'], retina)\n    bright_overlap, top_retina = heatmap_topk_overlap(item['heatmap'], bright, retina)\n    dark_overlap, _ = heatmap_topk_overlap(item['heatmap'], dark, retina)\n    lesion_overlap, _ = heatmap_topk_overlap(item['heatmap'], bright | dark, retina)\n    values = metric_lookup.get((row.image_id, row.method), {})\n    retina_focus = heatmap_mass_inside_retina(item['heatmap'], retina)\n    background_focus = heatmap_mass_outside_retina(item['heatmap'], retina)\n    optic_focus = heatmap_mass_on_optic_disc(item['heatmap'], disc)\n    active_area = float((item['heatmap'] >= 0.5).mean())\n    important_drop = float(values.get('retina_aware_important_drop', np.nan))\n    random_drop = float(values.get('retina_aware_random_drop', np.nan))\n    flags = []\n    if np.isfinite(background_focus) and background_focus > 0.15: flags.append('border_artifact_risk')\n    if np.isfinite(optic_focus) and optic_focus > 0.20: flags.append('optic_disc_dominant')\n    if np.isfinite(important_drop) and np.isfinite(random_drop) and important_drop <= random_drop: flags.append('weak_faithfulness')\n    if active_area > 0.15: flags.append('diffuse_heatmap')\n    if not flags: flags.append('good_candidate')\n    if any(flag != 'good_candidate' for flag in flags): flags.append('needs_manual_review')\n    foldaware_quality_rows.append({**row._asdict(), 'retina_focus_ratio': retina_focus, 'background_focus_ratio': background_focus, 'optic_disc_focus_ratio': optic_focus, 'bright_proxy_overlap_top10': bright_overlap, 'dark_proxy_overlap_top10': dark_overlap, 'lesion_proxy_overlap_top10': lesion_overlap, 'heatmap_top10_retina_ratio': top_retina, 'active_area_ratio': active_area, 'important_drop': important_drop, 'random_drop': random_drop, 'retina_aware_important_drop': important_drop, 'retina_aware_random_drop': random_drop, 'retina_aware_least_drop': values.get('retina_aware_least_drop', np.nan), 'retina_aware_deletion_auc': values.get('retina_aware_deletion_auc', np.nan), 'retina_aware_aopc': values.get('retina_aware_aopc', np.nan), 'metric_status': values.get('metric_status', 'not_evaluated'), 'xai_quality_flag': '|'.join(flags)})\n\nfoldaware_quality_df = pd.DataFrame(foldaware_quality_rows)\nfoldaware_quality_df.to_csv(FOLDAWARE_DIRS['metrics'] / 'xai_foldaware_clinical_quality_per_image.csv', index=False)\n\ndef add_internal_quality_score(frame, mode_label):\n    scored = frame.copy()\n    scored['xai_mode'] = mode_label\n    scored['normalized_important_drop'] = 0.0\n    for method, indices in scored.groupby('method').groups.items():\n        values = pd.to_numeric(scored.loc[indices, 'important_drop'], errors='coerce')\n        finite = values[np.isfinite(values)]\n        if len(finite) and finite.max() > finite.min():\n            scored.loc[indices, 'normalized_important_drop'] = ((values - finite.min()) / (finite.max() - finite.min())).fillna(0.0)\n        elif len(finite):\n            scored.loc[indices, 'normalized_important_drop'] = 0.5\n    optic_penalty = np.clip(scored.optic_disc_focus_ratio.fillna(0) / 0.20, 0, 1)\n    diffuse_penalty = np.clip((scored.active_area_ratio.fillna(0) - 0.15) / 0.20, 0, 1)\n    scored['xai_quality_score'] = (\n        scored.retina_focus_ratio.fillna(0)\n        + scored.normalized_important_drop.fillna(0)\n        + scored.lesion_proxy_overlap_top10.fillna(0)\n        - scored.background_focus_ratio.fillna(0)\n        - optic_penalty\n        - diffuse_penalty\n    )\n    scored['quality_score_note'] = 'Internal non-clinical ranking only; not XAI accuracy or clinical validation.'\n    return scored\n\nbaseline_score_df = advanced_quality_df.copy()\nif 'important_drop' not in baseline_score_df:\n    baseline_score_df['important_drop'] = np.nan\nbaseline_scored_df = add_internal_quality_score(baseline_score_df, 'single_fold_baseline')\nfoldaware_scored_df = add_internal_quality_score(foldaware_quality_df, 'fold_aware') if len(foldaware_quality_df) else pd.DataFrame()\nquality_score_df = pd.concat([baseline_scored_df, foldaware_scored_df], ignore_index=True, sort=False)\nquality_score_df.to_csv(OUTPUT_DIRS['metrics'] / 'xai_quality_score.csv', index=False)\n\nfoldaware_status_counts = foldaware_debug_df.status_group.value_counts().to_dict() if len(foldaware_debug_df) else {}\nprint('Fold-aware CAM status:', foldaware_status_counts)\nprint('Fold-aware metrics:', int((foldaware_metrics_df.metric_status == 'success').sum()) if len(foldaware_metrics_df) else 0, 'success |', int((foldaware_metrics_df.metric_status == 'failed').sum()) if len(foldaware_metrics_df) else 0, 'failed')\ndisplay(foldaware_quality_df.head())\n","metadata":{},"outputs":[],"execution_count":null},{"id":"7c1432f9-d4fd-445d-8a26-887e28c73ba3","cell_type":"code","source":"# Single-fold versus fold-aware comparison and limited multi-checkpoint consistency audit\ndef pearson_similarity(left, right):\n    left = np.asarray(left, dtype=np.float64).reshape(-1); right = np.asarray(right, dtype=np.float64).reshape(-1)\n    if left.size != right.size or left.std() <= 1e-12 or right.std() <= 1e-12:\n        return np.nan\n    return float(np.corrcoef(left, right)[0, 1])\n\ndef cosine_similarity(left, right):\n    left = np.asarray(left, dtype=np.float64).reshape(-1); right = np.asarray(right, dtype=np.float64).reshape(-1)\n    denom = np.linalg.norm(left) * np.linalg.norm(right)\n    return np.nan if denom <= 1e-12 else float(np.dot(left, right) / denom)\n\ntry:\n    from skimage.metrics import structural_similarity as structural_similarity\n    SSIM_AVAILABLE = True\nexcept Exception:\n    structural_similarity = None\n    SSIM_AVAILABLE = False\n\ndef topk_overlap_ratio(left, right, retina_mask=None, top_percent=10):\n    allowed = np.flatnonzero(retina_mask.reshape(-1)) if retina_mask is not None else np.arange(left.size)\n    if not len(allowed):\n        return np.nan\n    count = max(1, int(round(len(allowed) * top_percent / 100.0)))\n    flat_left, flat_right = left.reshape(-1), right.reshape(-1)\n    left_set = set(allowed[np.argpartition(flat_left[allowed], -count)[-count:]])\n    right_set = set(allowed[np.argpartition(flat_right[allowed], -count)[-count:]])\n    return float(len(left_set & right_set) / max(len(left_set | right_set), 1))\n\ndef heatmap_similarity(left, right, retina_mask=None):\n    ssim = np.nan\n    if SSIM_AVAILABLE:\n        try:\n            ssim = float(structural_similarity(left.astype(np.float32), right.astype(np.float32), data_range=1.0))\n        except Exception:\n            pass\n    return {'pearson_correlation': pearson_similarity(left, right), 'cosine_similarity': cosine_similarity(left, right), 'ssim': ssim, 'top10_overlap_ratio': topk_overlap_ratio(left, right, retina_mask)}\n\nbaseline_record_lookup = advanced_generation_df.set_index(['image_id', 'method']).to_dict('index')\nbaseline_quality_lookup = advanced_quality_df.set_index(['image_id', 'method']).to_dict('index')\nfoldaware_quality_lookup = foldaware_quality_df.set_index(['image_id', 'method']).to_dict('index') if len(foldaware_quality_df) else {}\ncomparison_rows = []\nfor row in foldaware_records_df.itertuples(index=False):\n    baseline_record = baseline_record_lookup.get((row.image_id, row.method))\n    if baseline_record is None:\n        continue\n    try:\n        baseline_heatmap = np.load(baseline_record['raw_heatmap_path'])\n        foldaware_heatmap = np.load(row.raw_heatmap_path)\n        image_rgb = preprocess_image(TRAIN_IMAGES_DIR / f'{row.image_id}.png')\n        retina = create_retina_mask(image_rgb)\n        similarity = heatmap_similarity(baseline_heatmap, foldaware_heatmap, retina)\n    except Exception as exc:\n        similarity = {'pearson_correlation': np.nan, 'cosine_similarity': np.nan, 'ssim': np.nan, 'top10_overlap_ratio': np.nan, 'comparison_error': repr(exc)}\n    baseline_quality = baseline_quality_lookup.get((row.image_id, row.method), {})\n    fold_quality = foldaware_quality_lookup.get((row.image_id, row.method), {})\n    comparison_rows.append({\n        'image_id': row.image_id, 'method': row.method, 'true_class': row.true_class,\n        'single_fold': TARGET_FOLD, 'fold_aware_fold': row.fold_used,\n        'single_predicted_class': baseline_record.get('predicted_class'), 'foldaware_predicted_class': row.predicted_class,\n        'prediction_matches': baseline_record.get('predicted_label') == row.predicted_label,\n        'single_confidence': baseline_record.get('confidence'), 'foldaware_confidence': row.confidence,\n        'confidence_difference': float(row.confidence) - float(baseline_record.get('confidence', np.nan)),\n        'single_important_drop': baseline_quality.get('important_drop', np.nan), 'foldaware_important_drop': fold_quality.get('important_drop', np.nan),\n        'single_retina_focus_ratio': baseline_quality.get('retina_focus_ratio', np.nan), 'foldaware_retina_focus_ratio': fold_quality.get('retina_focus_ratio', np.nan),\n        'single_lesion_proxy_overlap_top10': baseline_quality.get('lesion_proxy_overlap_top10', np.nan), 'foldaware_lesion_proxy_overlap_top10': fold_quality.get('lesion_proxy_overlap_top10', np.nan),\n        'single_quality_flag': baseline_quality.get('xai_quality_flag', ''), 'foldaware_quality_flag': fold_quality.get('xai_quality_flag', ''),\n        **similarity,\n    })\ncomparison_df = pd.DataFrame(comparison_rows)\ncomparison_df.to_csv(OUTPUT_DIRS['metrics'] / 'xai_single_vs_foldaware_comparison.csv', index=False)\ncomparison_summary_df = comparison_df.groupby('method', as_index=False).agg(\n    records=('image_id', 'count'), prediction_match_rate=('prediction_matches', 'mean'),\n    pearson_mean=('pearson_correlation', 'mean'), cosine_mean=('cosine_similarity', 'mean'),\n    ssim_mean=('ssim', 'mean'), top10_overlap_mean=('top10_overlap_ratio', 'mean'),\n    confidence_difference_mean=('confidence_difference', 'mean')\n) if len(comparison_df) else pd.DataFrame()\ncomparison_summary_df.to_csv(FOLDAWARE_DIRS['metrics'] / 'xai_single_vs_foldaware_summary.csv', index=False)\n\ndef save_single_vs_foldaware_gallery(frame, output_path):\n    ids = frame.image_id.drop_duplicates().head(3).tolist()\n    if not ids:\n        fig, ax = plt.subplots(figsize=(7, 2)); ax.text(.5, .5, 'No comparable single-fold and fold-aware records', ha='center'); ax.axis('off'); fig.savefig(output_path, dpi=180, bbox_inches='tight'); plt.close(fig); return\n    fig, axes = plt.subplots(len(ids), 5, figsize=(18, 3.7 * len(ids)), squeeze=False)\n    for idx, image_id in enumerate(ids):\n        row = frame[frame.image_id.eq(image_id)].iloc[0]\n        image_rgb = preprocess_image(TRAIN_IMAGES_DIR / f'{image_id}.png')\n        panels = [(image_rgb, 'Original')]\n        for method in ['gradcam', 'gradcam_pp']:\n            baseline = baseline_record_lookup.get((image_id, method)); fold_record = foldaware_records_df[(foldaware_records_df.image_id.eq(image_id)) & (foldaware_records_df.method.eq(method))]\n            if baseline is not None and len(fold_record):\n                panels.append((overlay_heatmap(image_rgb, np.load(baseline['raw_heatmap_path'])), f'Single fold-0 {method}'))\n                panels.append((overlay_heatmap(image_rgb, np.load(fold_record.iloc[0].raw_heatmap_path)), f'Fold-aware fold-{int(fold_record.iloc[0].fold_used)} {method}'))\n        for ax, (image, title) in zip(axes[idx], panels):\n            ax.imshow(image); ax.set_title(title, fontsize=8); ax.axis('off')\n        axes[idx, 0].set_ylabel(f\"{image_id}\\ntrue={row.true_class}\\nPearson={row.pearson_correlation:.3f}\", fontsize=8)\n    fig.suptitle('Single-fold baseline vs fold-aware attribution', fontsize=13); fig.tight_layout(); fig.savefig(output_path, dpi=220, bbox_inches='tight'); plt.show(); plt.close(fig)\n\nsave_single_vs_foldaware_gallery(comparison_df, FOLDAWARE_DIRS['figures'] / 'xai_single_vs_foldaware_examples.png')\n\nconsistency_parts = []\nfor class_name in CLASS_NAMES:\n    subset = enhanced_samples_df[enhanced_samples_df.true_class.eq(class_name)].copy()\n    priority = subset.selected_reason.str.contains('failure|manual', case=False, na=False).astype(int)\n    consistency_parts.append(subset.assign(_priority=priority).sort_values(['_priority', 'confidence'], ascending=[False, False]).head(MAX_CONSISTENCY_PER_CLASS))\nconsistency_samples_df = pd.concat(consistency_parts, ignore_index=True).drop_duplicates('image_id') if consistency_parts else pd.DataFrame()\nconsistency_rows = []\nfor sample in consistency_samples_df.itertuples(index=False):\n    image_rgb = preprocess_image(TRAIN_IMAGES_DIR / f'{sample.image_id}.png')\n    retina = create_retina_mask(image_rgb)\n    _, _, fixed_target_class = predict_rgb_with_model(model, image_rgb)  # same class condition across folds\n    by_method = {method: {} for method in CAM_GENERATORS}\n    predicted_by_fold = {}\n    for fold in available_c_folds:\n        try:\n            net, _ = load_model_for_fold(fold)\n            tensor, probabilities, predicted = predict_rgb_with_model(net, image_rgb)\n            predicted_by_fold[fold] = predicted\n            for method, generator in CAM_GENERATORS.items():\n                result = generator(net, tensor, fixed_target_class)\n                if result['status'] in VALID_CAM_STATUSES:\n                    by_method[method][fold] = result['heatmap']\n        except Exception as exc:\n            for method in CAM_GENERATORS:\n                consistency_rows.append({'image_id': sample.image_id, 'true_class': sample.true_class, 'method': method, 'fold_a': fold, 'fold_b': np.nan, 'target_class_fixed_from_fold0': fixed_target_class, 'status': 'failed', 'error': repr(exc)})\n    for method, fold_maps in by_method.items():\n        for left_index, fold_a in enumerate(available_c_folds):\n            for fold_b in available_c_folds[left_index + 1:]:\n                if fold_a not in fold_maps or fold_b not in fold_maps:\n                    continue\n                similarity = heatmap_similarity(fold_maps[fold_a], fold_maps[fold_b], retina)\n                consistency_rows.append({'image_id': sample.image_id, 'true_class': sample.true_class, 'method': method, 'fold_a': fold_a, 'fold_b': fold_b, 'target_class_fixed_from_fold0': fixed_target_class, 'predicted_class_fold_a': CLASS_NAMES[predicted_by_fold.get(fold_a, fixed_target_class)], 'predicted_class_fold_b': CLASS_NAMES[predicted_by_fold.get(fold_b, fixed_target_class)], 'status': 'success', 'error': '', **similarity})\n    complete_methods = all(all(fold in by_method[method] for fold in available_c_folds) for method in ['gradcam', 'gradcam_pp'])\n    if complete_methods:\n        fig, axes = plt.subplots(2, 4, figsize=(14, 7))\n        for row_index, display_method in enumerate(['gradcam', 'gradcam_pp']):\n            axes[row_index, 0].imshow(image_rgb); axes[row_index, 0].set_title('Original'); axes[row_index, 0].axis('off')\n            for col, fold in enumerate(available_c_folds, start=1):\n                axes[row_index, col].imshow(overlay_heatmap(image_rgb, by_method[display_method][fold])); axes[row_index, col].set_title(f'{display_method} fold {fold}'); axes[row_index, col].axis('off')\n        fig.suptitle(f'{sample.image_id}: same target class {CLASS_NAMES[fixed_target_class]} across checkpoints', fontsize=10); fig.tight_layout(); fig.savefig(FOLDAWARE_DIRS['consistency_figures'] / f'{sample.image_id}_checkpoint_consistency.png', dpi=220, bbox_inches='tight'); plt.close(fig)\n\nconsistency_df = pd.DataFrame(consistency_rows)\nconsistency_df.to_csv(OUTPUT_DIRS['metrics'] / 'xai_checkpoint_consistency.csv', index=False)\nconsistency_success_df = consistency_df[consistency_df.status.eq('success')].copy() if len(consistency_df) else pd.DataFrame()\nconsistency_summary_df = consistency_success_df.groupby('method', as_index=False).agg(\n    comparisons=('image_id', 'count'), pearson_mean=('pearson_correlation', 'mean'), cosine_mean=('cosine_similarity', 'mean'), ssim_mean=('ssim', 'mean'), top10_overlap_mean=('top10_overlap_ratio', 'mean')\n) if len(consistency_success_df) else pd.DataFrame()\nconsistency_summary_df.to_csv(FOLDAWARE_DIRS['metrics'] / 'xai_checkpoint_consistency_summary.csv', index=False)\ndisplay(comparison_summary_df); display(consistency_summary_df)\n","metadata":{},"outputs":[],"execution_count":null},{"id":"febd532f-92c1-414c-8dc1-bd0ca2df09db","cell_type":"markdown","source":"## 19. Best XAI Examples and Failure Cases\n\nThe following galleries deliberately retain both internally ranked candidates and cases requiring manual review. The internal score is not a clinical XAI validation metric.\n","metadata":{}},{"id":"33185687-e372-4126-9a1d-494d23699842","cell_type":"code","source":"# Reporting galleries, advanced academic summaries, and a final archive refresh\ncase_quality_df = foldaware_scored_df.copy() if len(foldaware_scored_df) else baseline_scored_df.copy()\ncase_mode = 'fold_aware' if len(foldaware_scored_df) else 'single_fold_baseline_fallback'\ncase_grad_df = case_quality_df[case_quality_df.method.eq('gradcam')].copy()\ncase_grad_df['correct'] = case_grad_df['correct'].astype(str).str.lower().isin(['true', '1'])\nbest_case_df = case_grad_df[\n    case_grad_df.correct\n    & case_grad_df.xai_quality_flag.eq('good_candidate')\n    & (case_grad_df.retina_focus_ratio >= 0.85)\n    & (case_grad_df.background_focus_ratio <= 0.15)\n].sort_values('xai_quality_score', ascending=False).head(6)\nfailure_case_df = case_grad_df[\n    (~case_grad_df.correct)\n    | case_grad_df.xai_quality_flag.str.contains('needs_manual_review|border_artifact_risk|optic_disc_dominant|diffuse_heatmap', regex=True, na=False)\n].sort_values('xai_quality_score', ascending=True).head(6)\n\ndef overlay_for_case(image_id, method):\n    if case_mode == 'fold_aware':\n        record = foldaware_records_df[(foldaware_records_df.image_id.eq(image_id)) & (foldaware_records_df.method.eq(method))]\n        if len(record):\n            image_rgb = preprocess_image(TRAIN_IMAGES_DIR / f'{image_id}.png')\n            return overlay_heatmap(image_rgb, np.load(record.iloc[0].raw_heatmap_path))\n    record = advanced_generation_df[(advanced_generation_df.image_id.eq(image_id)) & (advanced_generation_df.method.eq(method))]\n    if len(record):\n        image_rgb = preprocess_image(TRAIN_IMAGES_DIR / f'{image_id}.png')\n        return overlay_heatmap(image_rgb, np.load(record.iloc[0].raw_heatmap_path))\n    return None\n\ndef save_case_gallery(frame, title, output_path):\n    if frame.empty:\n        fig, ax = plt.subplots(figsize=(7, 2)); ax.text(.5, .5, 'No cases match this selection rule', ha='center', va='center'); ax.axis('off'); fig.savefig(output_path, dpi=180, bbox_inches='tight'); plt.close(fig); return\n    frame = frame.head(6).reset_index(drop=True)\n    fig, axes = plt.subplots(len(frame), 3, figsize=(13, 4.0 * len(frame)), squeeze=False)\n    for index, row in frame.iterrows():\n        image_rgb = preprocess_image(TRAIN_IMAGES_DIR / f\"{row['image_id']}.png\")\n        panels = [(image_rgb, 'Original'), (overlay_for_case(row['image_id'], 'gradcam'), 'Grad-CAM overlay'), (overlay_for_case(row['image_id'], 'gradcam_pp'), 'Grad-CAM++ overlay')]\n        for ax, (image, label) in zip(axes[index], panels):\n            if image is not None:\n                ax.imshow(image)\n            ax.set_title(label); ax.axis('off')\n        notes = f\"true={row['true_class']} | pred={row['predicted_class']} | conf={row['confidence']:.3f}\\nscore={row['xai_quality_score']:.3f} | {row['xai_quality_flag']}\\nInternal ranking only; manual review remains required.\"\n        axes[index, 0].set_ylabel(notes, fontsize=8)\n    fig.suptitle(title, fontsize=13); fig.tight_layout(); fig.savefig(output_path, dpi=220, bbox_inches='tight'); plt.show(); plt.close(fig)\n\nsave_case_gallery(best_case_df, 'Best XAI candidates — internal, non-clinical selection', FOLDAWARE_DIRS['report_examples'] / 'best_xai_candidates.png')\nsave_case_gallery(failure_case_df, 'Failure / manual-review cases — deliberately retained', FOLDAWARE_DIRS['report_examples'] / 'xai_failure_cases.png')\n\nfoldaware_summary_df = foldaware_scored_df.groupby('method', as_index=False).agg(\n    records=('image_id', 'count'), retina_focus_mean=('retina_focus_ratio', 'mean'), background_focus_mean=('background_focus_ratio', 'mean'),\n    important_drop_mean=('important_drop', 'mean'), random_drop_mean=('random_drop', 'mean'), lesion_proxy_overlap_mean=('lesion_proxy_overlap_top10', 'mean'), quality_score_mean=('xai_quality_score', 'mean')\n) if len(foldaware_scored_df) else pd.DataFrame()\nfoldaware_per_class_df = foldaware_scored_df.groupby(['true_class', 'method'], as_index=False).agg(\n    records=('image_id', 'count'), retina_focus_mean=('retina_focus_ratio', 'mean'), important_drop_mean=('important_drop', 'mean'), quality_score_mean=('xai_quality_score', 'mean')\n) if len(foldaware_scored_df) else pd.DataFrame()\n\nfoldaware_report_lines = [\n    '# Fold-Aware / OOF-Style XAI Summary',\n    f'- Requested XAI mode: `{XAI_MODE}`.',\n    f'- Fold assignment status: {fold_assignment_status}; source: `{fold_assignment_path}`.',\n    f\"- Enhanced samples: {len(enhanced_samples_df)}; assignment matches: {int(enhanced_samples_df.fold_assignment_status.eq('matched').sum())}; fold-0 fallbacks: {int((~enhanced_samples_df.fold_assignment_status.eq('matched')).sum())}.\",\n    f\"- Fold-aware CAM success/fallback/failed: {int((foldaware_debug_df.status_group == 'success').sum()) if len(foldaware_debug_df) else 0}/{int((foldaware_debug_df.status_group == 'fallback').sum()) if len(foldaware_debug_df) else 0}/{int((foldaware_debug_df.status_group == 'failed').sum()) if len(foldaware_debug_df) else 0}.\",\n    f\"- Fold-aware perturbation metric success/failure: {int((foldaware_metrics_df.metric_status == 'success').sum()) if len(foldaware_metrics_df) else 0}/{int((foldaware_metrics_df.metric_status == 'failed').sum()) if len(foldaware_metrics_df) else 0}.\",\n    '- Fold-aware interpretation: for a matched image, the selected model is the checkpoint whose validation fold contains that image, so the checkpoint did not use that image in its own training split.',\n    '- Checkpoint consistency uses a fixed fold-0 target class across folds. This isolates attribution stability from class-target changes; it does not assert a clinical ground truth.',\n    '- Bright/dark proxy overlap and the quality score are non-clinical audit/ranking aids, not lesion annotations, XAI accuracy, or diagnostic evidence.',\n    '', '## Fold-aware method summary', foldaware_summary_df.to_string(index=False),\n    '', '## Fold-aware per-class summary', foldaware_per_class_df.to_string(index=False),\n    '', '## Single-fold versus fold-aware summary', comparison_summary_df.to_string(index=False),\n    '', '## Multi-checkpoint consistency summary', consistency_summary_df.to_string(index=False),\n]\n(OUTPUT_DIRS['reports'] / 'xai_foldaware_summary.md').write_text('\\n'.join(foldaware_report_lines))\n\ncase_report_lines = [\n    '# XAI Report Case Selection',\n    f'- Source mode: `{case_mode}`.',\n    f'- Best-candidate images: {len(best_case_df)}. Criteria: correct prediction, good-candidate flag, predominantly retinal attribution, low background focus, and high internal quality score.',\n    f'- Failure/manual-review images: {len(failure_case_df)}. Criteria deliberately include incorrect predictions, weak faithfulness, diffuse attribution, border/background risk, or optic-disc emphasis.',\n    '- The internal quality score ranks examples for reporting only. It is not an XAI accuracy score and cannot substitute for expert annotation or clinical review.',\n    '', '## Best candidates', best_case_df[['image_id', 'true_class', 'predicted_class', 'confidence', 'xai_quality_score', 'xai_quality_flag']].to_string(index=False),\n    '', '## Failure / manual-review cases', failure_case_df[['image_id', 'true_class', 'predicted_class', 'confidence', 'xai_quality_score', 'xai_quality_flag']].to_string(index=False),\n]\n(OUTPUT_DIRS['reports'] / 'xai_report_case_selection.md').write_text('\\n'.join(case_report_lines))\n\nadvanced_final_lines = [\n    '', '## Advanced XAI Audit and Improvement',\n    *advanced_overview,\n    '', '## Fold-Aware Interpretation',\n    '- Fold-aware XAI uses the checkpoint corresponding to the image validation fold when `kfold_metadata.csv` supplies a match. Unmatched images fall back explicitly to fold 0 and are labelled as such.',\n    '- Agreement between the fold-0 baseline and fold-aware attribution supports checkpoint robustness for this subset; disagreement indicates checkpoint-sensitive attribution and should be manually reviewed.',\n    '', '## Consistency Interpretation',\n    '- Pearson correlation, cosine similarity, SSIM when available, and top-10% overlap quantify similarity across independently trained C checkpoints for a small, fixed-target subset.',\n    '- Stable maps are a stronger model-level indication than a single-checkpoint visualization; unstable maps remain valid outputs but should not be over-interpreted.',\n    '', '## Faithfulness and Clinical Safety',\n    '- Confidence drop, random/least baselines, deletion AUC, and AOPC assess perturbation sensitivity rather than lesion localization.',\n    '- Retina-aware perturbation excludes black background from the masking candidate area. Bright/dark proxies are non-clinical visual cues, not lesion masks.',\n    '- Heatmaps show areas given high weight by the model; they do not prove a diagnosis, detect lesions with clinical certainty, or replace expert review.',\n    '', '## Internal Quality Ranking',\n    '- `xai_quality_score` combines retinal focus, normalized perturbation effect, proxy overlap, and penalties for background, optic-disc emphasis, and diffuse attribution. It is only a transparent reporting/ranking aid.',\n    '', '## Advanced Tables',\n    '### Fold-aware summary', foldaware_summary_df.to_string(index=False),\n    '', '### Single-fold vs fold-aware', comparison_summary_df.to_string(index=False),\n    '', '### Checkpoint consistency', consistency_summary_df.to_string(index=False),\n]\n\n# Extend the summary already prepared in Section 17, then rebuild the archive from all output artifacts.\nfinal_summary.extend(advanced_final_lines)\n(OUTPUT_DIRS['reports'] / 'xai_final_summary.md').write_text('\\n'.join(final_summary))\narchive_path = WORKING_ROOT / 'xai_outputs.zip'\nwith zipfile.ZipFile(archive_path, 'w', compression=zipfile.ZIP_DEFLATED) as archive:\n    for file_path in OUTPUT_ROOT.rglob('*'):\n        if file_path.is_file() and file_path.suffix not in {'.pth', '.zip'}:\n            archive.write(file_path, file_path.relative_to(OUTPUT_ROOT.parent))\nprint('Advanced fold-aware report:', OUTPUT_DIRS['reports'] / 'xai_foldaware_summary.md')\nprint('Case-selection report:', OUTPUT_DIRS['reports'] / 'xai_report_case_selection.md')\nprint('Refreshed download archive:', archive_path)\n","metadata":{},"outputs":[],"execution_count":null},{"id":"6203a2bb-cacc-4135-a633-b15c4ad80660","cell_type":"markdown","source":"## 20. Final Output Analysis\n\nThis read-only section inventories completed outputs without rerunning CAM or changing the model.\n","metadata":{}},{"id":"8ff0fa7c-d9d8-43d2-9a78-a28a125c5026","cell_type":"code","source":"\n# Final Output Analysis: inventory already-produced artefacts without regenerating CAM.\nFINAL_REPORT_DIR = OUTPUT_ROOT / 'reports'\nPREDICTION_FIG_DIR = OUTPUT_ROOT / 'figures' / 'prediction_analysis'\nJOINT_FIG_DIR = OUTPUT_ROOT / 'figures' / 'xai_prediction_joint'\nERROR_REVIEW_DIR = OUTPUT_ROOT / 'figures' / 'prediction_error_xai_review'\nREPORT_SELECTED_DIR = OUTPUT_ROOT / 'figures' / 'report_selected'\nfor output_dir in [FINAL_REPORT_DIR, PREDICTION_FIG_DIR, JOINT_FIG_DIR, ERROR_REVIEW_DIR, REPORT_SELECTED_DIR]:\n    output_dir.mkdir(parents=True, exist_ok=True)\n\nRUN_PREDICTION_MODE_COMPARISON = False\nREPORT_CLASS_NAMES = list(globals().get('CLASS_NAMES', ['No_DR', 'Mild', 'Moderate', 'Severe', 'Proliferative_DR']))\n\ndef read_report_csv(path):\n    path = Path(path)\n    if not path.is_file():\n        return pd.DataFrame()\n    try:\n        return pd.read_csv(path)\n    except Exception as exc:\n        print('Unreadable output:', path, repr(exc))\n        return pd.DataFrame()\n\ndef bool_column(frame, name='correct'):\n    return frame.get(name, pd.Series(False, index=frame.index)).astype(str).str.lower().isin(['true', '1', 'yes'])\n\ndef status_summary(frame):\n    if frame.empty:\n        return {'success': 0, 'fallback': 0, 'failed': 0}\n    column = 'status_group' if 'status_group' in frame else ('cam_status' if 'cam_status' in frame else 'status')\n    values = frame.get(column, pd.Series('failed', index=frame.index)).astype(str)\n    return {'success': int(values.eq('success').sum()),\n            'fallback': int(values.str.startswith('fallback').sum() | values.eq('fallback').sum()),\n            'failed': int(values.eq('failed').sum())}\n\nBASE_METRICS_DIR = OUTPUT_ROOT / 'xai' / 'metrics'\nFOLD_METRICS_DIR = OUTPUT_ROOT / 'xai_foldaware' / 'metrics'\nbase_generation_final = read_report_csv(BASE_METRICS_DIR / 'xai_generation_records.csv')\nbase_debug_final = read_report_csv(BASE_METRICS_DIR / 'cam_debug_log.csv')\nbase_metric_final = read_report_csv(BASE_METRICS_DIR / 'xai_metrics_per_image.csv')\nbase_quality_final = read_report_csv(BASE_METRICS_DIR / 'xai_clinical_quality_per_image.csv')\nfold_generation_final = read_report_csv(FOLD_METRICS_DIR / 'xai_foldaware_generation_records.csv')\nfold_debug_final = read_report_csv(FOLD_METRICS_DIR / 'cam_debug_log.csv')\nfold_metric_final = read_report_csv(FOLD_METRICS_DIR / 'xai_foldaware_metrics_per_image.csv')\nquality_score_final = read_report_csv(BASE_METRICS_DIR / 'xai_quality_score.csv')\nsingle_fold_comparison_final = read_report_csv(BASE_METRICS_DIR / 'xai_single_vs_foldaware_comparison.csv')\ncheckpoint_consistency_final = read_report_csv(BASE_METRICS_DIR / 'xai_checkpoint_consistency.csv')\n\nbase_status_final = status_summary(base_debug_final if len(base_debug_final) else base_generation_final)\nfold_status_final = status_summary(fold_debug_final if len(fold_debug_final) else fold_generation_final)\nbase_metric_ok_final = int(base_metric_final.get('metric_status', pd.Series(dtype=str)).eq('success').sum())\nbase_metric_fail_final = int(base_metric_final.get('metric_status', pd.Series(dtype=str)).eq('failed').sum())\nfold_metric_ok_final = int(fold_metric_final.get('metric_status', pd.Series(dtype=str)).eq('success').sum())\nfold_metric_fail_final = int(fold_metric_final.get('metric_status', pd.Series(dtype=str)).eq('failed').sum())\n\nquality_final = quality_score_final[quality_score_final.get('xai_mode', pd.Series('', index=quality_score_final.index)).eq('fold_aware')].copy()\nif quality_final.empty:\n    quality_final = base_quality_final.copy()\nif not quality_final.empty:\n    quality_final['correct'] = bool_column(quality_final)\n    quality_final['manual_review'] = quality_final.get('xai_quality_flag', pd.Series('', index=quality_final.index)).astype(str).str.contains('needs_manual_review', na=False)\n    quality_final['border_risk'] = quality_final.get('xai_quality_flag', pd.Series('', index=quality_final.index)).astype(str).str.contains('border_artifact_risk', na=False)\n    quality_final['optic_disc_risk'] = quality_final.get('xai_quality_flag', pd.Series('', index=quality_final.index)).astype(str).str.contains('optic_disc_dominant', na=False)\n\nquality_count_final = {\n    'good': int(quality_final.get('xai_quality_flag', pd.Series('', index=quality_final.index)).eq('good_candidate').sum()) if len(quality_final) else 0,\n    'manual': int(quality_final.manual_review.sum()) if len(quality_final) else 0,\n    'border': int(quality_final.border_risk.sum()) if len(quality_final) else 0,\n    'optic': int(quality_final.optic_disc_risk.sum()) if len(quality_final) else 0,\n}\nquality_class_final = quality_final.groupby(['true_class', 'method'], as_index=False).agg(\n    records=('image_id', 'count'), good=('xai_quality_flag', lambda x: int((x == 'good_candidate').sum())),\n    manual=('manual_review', 'sum'), retina_focus=('retina_focus_ratio', 'mean'),\n    important_drop=('important_drop', 'mean'), quality_score=('xai_quality_score', 'mean')\n) if len(quality_final) and 'xai_quality_score' in quality_final else pd.DataFrame()\n\nconsistency_ok_final = checkpoint_consistency_final[checkpoint_consistency_final.get('status', pd.Series('', index=checkpoint_consistency_final.index)).eq('success')].copy()\nconsistency_summary_final = consistency_ok_final.groupby('method', as_index=False).agg(\n    pairs=('image_id', 'count'), pearson=('pearson_correlation', 'mean'), cosine=('cosine_similarity', 'mean'),\n    ssim=('ssim', 'mean'), top10_overlap=('top10_overlap_ratio', 'mean')\n) if len(consistency_ok_final) else pd.DataFrame()\ncomparison_summary_final = single_fold_comparison_final.groupby('method', as_index=False).agg(\n    records=('image_id', 'count'), prediction_match=('prediction_matches', 'mean'),\n    pearson=('pearson_correlation', 'mean'), cosine=('cosine_similarity', 'mean'),\n    ssim=('ssim', 'mean'), top10_overlap=('top10_overlap_ratio', 'mean')\n) if len(single_fold_comparison_final) else pd.DataFrame()\n\ninventory_roots = {\n    'reports': OUTPUT_ROOT / 'reports', 'predictions': OUTPUT_ROOT / 'predictions',\n    'xai_metrics': BASE_METRICS_DIR, 'xai_gallery': OUTPUT_ROOT / 'figures' / 'xai_gallery',\n    'quality_audit': OUTPUT_ROOT / 'figures' / 'xai_quality_audit',\n    'foldaware_figures': OUTPUT_ROOT / 'figures' / 'xai_foldaware',\n    'consistency_figures': OUTPUT_ROOT / 'figures' / 'xai_consistency',\n    'report_examples': OUTPUT_ROOT / 'figures' / 'xai_report_examples',\n    'base_overlays': OUTPUT_ROOT / 'xai' / 'overlays', 'foldaware_outputs': OUTPUT_ROOT / 'xai_foldaware',\n}\ninventory_final = pd.DataFrame([\n    {'area': key, 'path': str(path), 'files': sum(item.is_file() for item in path.rglob('*')) if path.exists() else 0}\n    for key, path in inventory_roots.items()\n])\n\nfinal_output_analysis_lines = [\n    '# Final Output Analysis',\n    f'- Baseline: {base_generation_final.image_id.nunique() if len(base_generation_final) else 0} images, {len(base_generation_final)} CAM records, success/fallback/failed={base_status_final[\"success\"]}/{base_status_final[\"fallback\"]}/{base_status_final[\"failed\"]}.',\n    f'- Baseline metrics: success={base_metric_ok_final}, failed={base_metric_fail_final}.',\n    f'- Fold-aware: {fold_generation_final.image_id.nunique() if len(fold_generation_final) else 0} images, {len(fold_generation_final)} CAM records, success/fallback/failed={fold_status_final[\"success\"]}/{fold_status_final[\"fallback\"]}/{fold_status_final[\"failed\"]}.',\n    f'- Fold-aware metrics: success={fold_metric_ok_final}, failed={fold_metric_fail_final}.',\n    f'- Quality audit: good={quality_count_final[\"good\"]}, manual-review={quality_count_final[\"manual\"]}, border-risk={quality_count_final[\"border\"]}, optic-disc-dominant={quality_count_final[\"optic\"]}.',\n    f'- Single-fold comparison records={len(single_fold_comparison_final)}; checkpoint-consistency successful pair records={len(consistency_ok_final)}.',\n    '- All counts describe the selected XAI subset and generated files, not global APTOS performance or clinical validation.',\n    '', '## Output inventory', inventory_final.to_string(index=False),\n    '', '## Quality per class', quality_class_final.to_string(index=False),\n    '', '## Single-fold vs fold-aware', comparison_summary_final.to_string(index=False),\n    '', '## Checkpoint consistency', consistency_summary_final.to_string(index=False),\n]\n(FINAL_REPORT_DIR / 'final_output_analysis.md').write_text('\\n'.join(final_output_analysis_lines))\ndisplay(inventory_final); display(quality_class_final); display(comparison_summary_final); display(consistency_summary_final)\nprint('\\n'.join(final_output_analysis_lines[:8]))\n","metadata":{},"outputs":[],"execution_count":null},{"id":"a6b5c0ff-1d64-4837-abaf-ecf9336be5f3","cell_type":"markdown","source":"## 21. Prediction Result Analysis\n\nPrediction summaries apply only to the selected XAI subset and are not global APTOS performance estimates.\n","metadata":{}},{"id":"14dd032c-d04c-41cb-87ec-a5fb35d811b7","cell_type":"code","source":"\n# Prediction Result Analysis: describe only the selected XAI subset.\ndef prepare_prediction_frame(frame, source_name):\n    frame = frame.copy()\n    required = ['image_id', 'true_label', 'true_class', 'predicted_label', 'predicted_class', 'confidence', 'correct']\n    if frame.empty:\n        return pd.DataFrame(columns=required + ['prediction_source'])\n    for name in required:\n        if name not in frame:\n            frame[name] = np.nan\n    frame = frame.drop_duplicates('image_id').copy()\n    frame['true_label'] = pd.to_numeric(frame.true_label, errors='coerce')\n    frame['predicted_label'] = pd.to_numeric(frame.predicted_label, errors='coerce')\n    label_map = dict(enumerate(REPORT_CLASS_NAMES))\n    frame['true_class'] = frame.true_class.fillna(frame.true_label.map(label_map))\n    frame['predicted_class'] = frame.predicted_class.fillna(frame.predicted_label.map(label_map))\n    frame['confidence'] = pd.to_numeric(frame.confidence, errors='coerce')\n    inferred = frame.true_label.eq(frame.predicted_label)\n    existing = bool_column(frame)\n    frame['correct'] = np.where(frame['correct'].isna(), inferred, existing)\n    frame['prediction_source'] = source_name\n    return frame[required + ['prediction_source']]\n\nif len(fold_generation_final) and {'true_label', 'predicted_label', 'confidence'}.issubset(fold_generation_final.columns):\n    prediction_subset_df = prepare_prediction_frame(fold_generation_final, 'fold_aware_generation_records')\n    prediction_visual_records_df = fold_generation_final.copy()\nelif (OUTPUT_ROOT / 'predictions' / 'xai_selected_samples_enhanced.csv').is_file():\n    prediction_subset_df = prepare_prediction_frame(read_report_csv(OUTPUT_ROOT / 'predictions' / 'xai_selected_samples_enhanced.csv'), 'xai_selected_samples_enhanced')\n    prediction_visual_records_df = base_generation_final.copy()\nelse:\n    prediction_subset_df = prepare_prediction_frame(read_report_csv(OUTPUT_ROOT / 'predictions' / 'selected_xai_samples.csv'), 'selected_xai_samples')\n    prediction_visual_records_df = base_generation_final.copy()\n\nprediction_subset_df['correct'] = prediction_subset_df.correct.astype(bool)\nprediction_subset_df['ordinal_distance'] = (prediction_subset_df.predicted_label - prediction_subset_df.true_label).abs()\nprediction_subset_df['confidence_band'] = pd.cut(prediction_subset_df.confidence, [-np.inf, .5, .8, np.inf], labels=['low', 'medium', 'high'])\nprediction_subset_df.to_csv(OUTPUT_ROOT / 'predictions' / 'prediction_subset_analysis_source.csv', index=False)\n\ntrue_counts = prediction_subset_df.true_class.value_counts().reindex(REPORT_CLASS_NAMES, fill_value=0)\npred_counts = prediction_subset_df.predicted_class.value_counts().reindex(REPORT_CLASS_NAMES, fill_value=0)\nsubset_accuracy = float(prediction_subset_df.correct.mean()) if len(prediction_subset_df) else np.nan\nconfidence_outcome = prediction_subset_df.groupby('correct', as_index=False).agg(samples=('image_id', 'count'), confidence_mean=('confidence', 'mean'), confidence_median=('confidence', 'median')) if len(prediction_subset_df) else pd.DataFrame()\nconfidence_class = prediction_subset_df.groupby('true_class', as_index=False).agg(samples=('image_id', 'count'), correct=('correct', 'sum'), confidence_mean=('confidence', 'mean'), subset_accuracy=('correct', 'mean')) if len(prediction_subset_df) else pd.DataFrame()\nconfusion_subset = pd.crosstab(prediction_subset_df.true_class, prediction_subset_df.predicted_class).reindex(index=REPORT_CLASS_NAMES, columns=REPORT_CLASS_NAMES, fill_value=0) if len(prediction_subset_df) else pd.DataFrame(0, index=REPORT_CLASS_NAMES, columns=REPORT_CLASS_NAMES)\n\nprediction_error_cases_df = prediction_subset_df[~prediction_subset_df.correct].copy()\nprediction_error_cases_df['error_pair'] = prediction_error_cases_df.true_class.astype(str) + ' -> ' + prediction_error_cases_df.predicted_class.astype(str)\nprediction_error_cases_df['ordinally_adjacent'] = prediction_error_cases_df.ordinal_distance.eq(1)\nprediction_error_cases_df['error_confidence_group'] = np.select(\n    [prediction_error_cases_df.confidence.ge(.8), prediction_error_cases_df.confidence.ge(.5)],\n    ['high_confidence_wrong', 'medium_confidence_wrong'], default='low_confidence_wrong'\n)\nerror_pair_counts = prediction_error_cases_df.error_pair.value_counts().rename_axis('error_pair').reset_index(name='count')\nprediction_error_cases_df.to_csv(OUTPUT_ROOT / 'predictions' / 'prediction_error_cases.csv', index=False)\n\nfig, ax = plt.subplots(figsize=(7, 6))\nimage = ax.imshow(confusion_subset.values, cmap='Blues')\nfig.colorbar(image, ax=ax, fraction=.046)\nax.set_xticks(range(5), REPORT_CLASS_NAMES, rotation=35, ha='right'); ax.set_yticks(range(5), REPORT_CLASS_NAMES)\nfor i in range(5):\n    for j in range(5): ax.text(j, i, str(int(confusion_subset.iloc[i, j])), ha='center', va='center')\nax.set_xlabel('Predicted class'); ax.set_ylabel('True class'); ax.set_title('Confusion matrix: selected XAI subset only')\nfig.tight_layout(); fig.savefig(PREDICTION_FIG_DIR / 'confusion_matrix_subset.png', dpi=220, bbox_inches='tight'); plt.show(); plt.close(fig)\n\nfig, ax = plt.subplots(figsize=(6, 4))\nboxes = [prediction_subset_df.loc[prediction_subset_df.correct, 'confidence'].dropna(), prediction_subset_df.loc[~prediction_subset_df.correct, 'confidence'].dropna()]\nif any(len(box) for box in boxes): ax.boxplot(boxes, labels=['Correct', 'Incorrect'], showmeans=True)\nax.set_ylim(0, 1.05); ax.set_ylabel('Prediction confidence'); ax.set_title('Confidence by prediction outcome')\nfig.tight_layout(); fig.savefig(PREDICTION_FIG_DIR / 'confidence_correct_vs_incorrect.png', dpi=220, bbox_inches='tight'); plt.show(); plt.close(fig)\n\nfig, ax = plt.subplots(figsize=(9, 4))\npositions = np.arange(len(REPORT_CLASS_NAMES))\nax.bar(positions-.19, true_counts.values, .38, label='True class'); ax.bar(positions+.19, pred_counts.values, .38, label='Predicted class')\nax.set_xticks(positions, REPORT_CLASS_NAMES, rotation=25, ha='right'); ax.set_ylabel('Selected subset count'); ax.legend(); ax.set_title('True vs predicted class distribution')\nfig.tight_layout(); fig.savefig(PREDICTION_FIG_DIR / 'class_distribution_true_vs_pred.png', dpi=220, bbox_inches='tight'); plt.show(); plt.close(fig)\n\nprediction_analysis_lines = [\n    '# Prediction Result Analysis',\n    f'- Source: {prediction_subset_df.prediction_source.iloc[0] if len(prediction_subset_df) else \"unavailable\"}.',\n    f'- Selected subset: n={len(prediction_subset_df)}, correct={int(prediction_subset_df.correct.sum()) if len(prediction_subset_df) else 0}, incorrect={int((~prediction_subset_df.correct).sum()) if len(prediction_subset_df) else 0}, descriptive subset accuracy={subset_accuracy:.3f}.' if np.isfinite(subset_accuracy) else '- No usable prediction records.',\n    f'- Ordinally adjacent errors: {int(prediction_error_cases_df.ordinally_adjacent.sum()) if len(prediction_error_cases_df) else 0}/{len(prediction_error_cases_df)}.',\n    '- This is a selected XAI subset, not a held-out global test result; it must not be used to claim overall accuracy.',\n    '', '## Confidence by outcome', confidence_outcome.to_string(index=False),\n    '', '## Confidence by true class', confidence_class.to_string(index=False),\n    '', '## Confusion matrix', confusion_subset.to_string(),\n    '', '## Frequent error pairs', error_pair_counts.to_string(index=False),\n    '', '## High-confidence correct', prediction_subset_df[prediction_subset_df.correct].sort_values('confidence', ascending=False).head(5).to_string(index=False),\n    '', '## High-confidence wrong', prediction_error_cases_df.sort_values('confidence', ascending=False).head(5).to_string(index=False),\n    '', '## Lower-confidence wrong', prediction_error_cases_df.sort_values('confidence').head(5).to_string(index=False),\n]\n(FINAL_REPORT_DIR / 'prediction_result_analysis.md').write_text('\\n'.join(prediction_analysis_lines))\ndisplay(confidence_outcome); display(confidence_class); display(error_pair_counts)\nprint('\\n'.join(prediction_analysis_lines[:5]))\n","metadata":{},"outputs":[],"execution_count":null},{"id":"3d06b5d3-286f-4f8e-8414-b9490b932321","cell_type":"markdown","source":"## 22. Prediction-XAI Joint Analysis\n\nThe analysis is descriptive: associations between prediction outcomes and model-attribution indicators are not causal or clinical claims.\n","metadata":{}},{"id":"901f3ea5-1a73-4fb4-9ca7-ed6b09a8d572","cell_type":"code","source":"\n# Prediction-XAI Joint Analysis: descriptive association within the selected subset.\njoint_quality_df = quality_score_final.copy()\npreferred_quality_mode = 'fold_aware' if 'fold_aware' in set(joint_quality_df.get('xai_mode', pd.Series(dtype=str)).dropna()) else 'single_fold_baseline'\nif len(joint_quality_df) and 'xai_mode' in joint_quality_df:\n    joint_quality_df = joint_quality_df[joint_quality_df.xai_mode.eq(preferred_quality_mode)].copy()\nif joint_quality_df.empty:\n    joint_quality_df = base_quality_final.copy()\nfor name in ['image_id', 'method', 'xai_quality_flag', 'xai_quality_score', 'important_drop', 'random_drop', 'retina_focus_ratio', 'background_focus_ratio', 'optic_disc_focus_ratio', 'lesion_proxy_overlap_top10', 'active_area_ratio']:\n    if name not in joint_quality_df: joint_quality_df[name] = np.nan\n\nconsistency_by_image = consistency_ok_final.groupby(['image_id', 'method'], as_index=False).agg(\n    consistency_pearson=('pearson_correlation', 'mean'), consistency_cosine=('cosine_similarity', 'mean'),\n    consistency_ssim=('ssim', 'mean'), consistency_top10_overlap=('top10_overlap_ratio', 'mean')\n) if len(consistency_ok_final) else pd.DataFrame(columns=['image_id', 'method'])\nprediction_merge = prediction_subset_df[['image_id', 'true_label', 'true_class', 'predicted_label', 'predicted_class', 'confidence', 'correct', 'ordinal_distance']].drop_duplicates('image_id')\njoint_prediction_xai_df = joint_quality_df.merge(prediction_merge, on='image_id', how='inner', suffixes=('', '_prediction'))\njoint_prediction_xai_df['correct'] = bool_column(joint_prediction_xai_df, 'correct_prediction') if 'correct_prediction' in joint_prediction_xai_df else bool_column(joint_prediction_xai_df)\njoint_prediction_xai_df = joint_prediction_xai_df.merge(consistency_by_image, on=['image_id', 'method'], how='left')\njoint_prediction_xai_df['needs_manual_review'] = joint_prediction_xai_df.xai_quality_flag.astype(str).str.contains('needs_manual_review', na=False)\njoint_prediction_xai_df['weak_faithfulness'] = joint_prediction_xai_df.xai_quality_flag.astype(str).str.contains('weak_faithfulness', na=False)\njoint_prediction_xai_df.to_csv(BASE_METRICS_DIR / 'prediction_xai_joint_analysis.csv', index=False)\n\ndef safe_corr(frame, x, y):\n    pair = frame[[x, y]].apply(pd.to_numeric, errors='coerce').dropna() if x in frame and y in frame else pd.DataFrame()\n    return float(pair[x].corr(pair[y])) if len(pair) >= 3 and pair[x].std() > 1e-12 and pair[y].std() > 1e-12 else np.nan\n\njoint_outcome = joint_prediction_xai_df.groupby(['method', 'correct'], as_index=False).agg(\n    records=('image_id', 'count'), confidence=('confidence', 'mean'), quality_score=('xai_quality_score', 'mean'),\n    important_drop=('important_drop', 'mean'), random_drop=('random_drop', 'mean'), retina_focus=('retina_focus_ratio', 'mean'),\n    manual_review_rate=('needs_manual_review', 'mean'), weak_faithfulness_rate=('weak_faithfulness', 'mean'),\n    consistency_pearson=('consistency_pearson', 'mean')\n) if len(joint_prediction_xai_df) else pd.DataFrame()\njoint_class = joint_prediction_xai_df.groupby(['true_class', 'method'], as_index=False).agg(\n    records=('image_id', 'count'), quality_score=('xai_quality_score', 'mean'), important_drop=('important_drop', 'mean'),\n    retina_focus=('retina_focus_ratio', 'mean'), manual_review_rate=('needs_manual_review', 'mean'),\n    consistency_pearson=('consistency_pearson', 'mean')\n) if len(joint_prediction_xai_df) else pd.DataFrame()\nconfidence_score_corr = safe_corr(joint_prediction_xai_df, 'confidence', 'xai_quality_score')\nconsistency_score_corr = safe_corr(joint_prediction_xai_df, 'consistency_pearson', 'xai_quality_score')\n\ndef outcome_box(column, title, filename, ylabel):\n    fig, ax = plt.subplots(figsize=(6, 4))\n    boxes = [pd.to_numeric(joint_prediction_xai_df.loc[joint_prediction_xai_df.correct, column], errors='coerce').dropna(),\n             pd.to_numeric(joint_prediction_xai_df.loc[~joint_prediction_xai_df.correct, column], errors='coerce').dropna()]\n    if any(len(box) for box in boxes): ax.boxplot(boxes, labels=['Correct', 'Incorrect'], showmeans=True)\n    ax.set_title(title); ax.set_ylabel(ylabel); fig.tight_layout(); fig.savefig(JOINT_FIG_DIR / filename, dpi=220, bbox_inches='tight'); plt.show(); plt.close(fig)\n\noutcome_box('xai_quality_score', 'Internal XAI quality score by prediction outcome', 'quality_score_correct_vs_incorrect.png', 'Internal quality score')\noutcome_box('important_drop', 'Important-region drop by prediction outcome', 'important_drop_correct_vs_incorrect.png', 'Confidence drop')\noutcome_box('retina_focus_ratio', 'Retina focus by prediction outcome', 'retina_focus_correct_vs_incorrect.png', 'Retina focus ratio')\n\nfig, ax = plt.subplots(figsize=(6, 5))\nfor correct, group in joint_prediction_xai_df.groupby('correct'):\n    ax.scatter(group.confidence, group.xai_quality_score, alpha=.75, label='Correct' if correct else 'Incorrect')\nax.set_xlabel('Prediction confidence'); ax.set_ylabel('Internal XAI quality score'); ax.set_title('Confidence vs XAI quality'); ax.legend()\nfig.tight_layout(); fig.savefig(JOINT_FIG_DIR / 'confidence_vs_xai_quality_score.png', dpi=220, bbox_inches='tight'); plt.show(); plt.close(fig)\n\nclass_score = joint_prediction_xai_df[joint_prediction_xai_df.method.eq('gradcam')].groupby('true_class', as_index=False).xai_quality_score.mean().set_index('true_class').reindex(REPORT_CLASS_NAMES).reset_index()\nfig, ax = plt.subplots(figsize=(8, 4)); ax.bar(class_score.true_class, class_score.xai_quality_score); ax.set_xticklabels(class_score.true_class, rotation=25, ha='right')\nax.set_ylabel('Mean internal XAI quality score'); ax.set_title('Class-wise XAI quality: Grad-CAM')\nfig.tight_layout(); fig.savefig(JOINT_FIG_DIR / 'classwise_xai_quality_score.png', dpi=220, bbox_inches='tight'); plt.show(); plt.close(fig)\n\nfig, ax = plt.subplots(figsize=(6, 5))\nfor correct, group in joint_prediction_xai_df.groupby('correct'):\n    ax.scatter(group.consistency_pearson, group.xai_quality_score, alpha=.75, label='Correct' if correct else 'Incorrect')\nax.set_xlabel('Mean cross-checkpoint Pearson similarity'); ax.set_ylabel('Internal XAI quality score'); ax.set_title('Consistency vs XAI quality'); ax.legend()\nfig.tight_layout(); fig.savefig(JOINT_FIG_DIR / 'consistency_vs_prediction_correctness.png', dpi=220, bbox_inches='tight'); plt.show(); plt.close(fig)\n\njoint_analysis_lines = [\n    '# Prediction-XAI Joint Analysis',\n    f'- Quality mode: {preferred_quality_mode}; image-method records={len(joint_prediction_xai_df)}.',\n    f'- Confidence vs quality-score correlation: {confidence_score_corr:.3f}.' if np.isfinite(confidence_score_corr) else '- Confidence vs quality-score correlation is unavailable or underpowered.',\n    f'- Consistency vs quality-score correlation: {consistency_score_corr:.3f}.' if np.isfinite(consistency_score_corr) else '- Consistency vs quality-score correlation is unavailable or underpowered.',\n    '- Associations are descriptive for a small selected subset. They do not establish causal relations, lesion localization, or clinical correctness.',\n    '', '## Correct vs incorrect', joint_outcome.to_string(index=False),\n    '', '## Per-class', joint_class.to_string(index=False),\n]\n(FINAL_REPORT_DIR / 'prediction_xai_joint_analysis.md').write_text('\\n'.join(joint_analysis_lines))\ndisplay(joint_outcome); display(joint_class)\nprint('\\n'.join(joint_analysis_lines[:5]))\n","metadata":{},"outputs":[],"execution_count":null},{"id":"a3d52272-f937-45dc-8654-cdfe621506a3","cell_type":"markdown","source":"## 23. Prediction Error and XAI Failure Review\n\nFailure cases are retained deliberately so that the report does not cherry-pick visually appealing heatmaps.\n","metadata":{}},{"id":"bf23166d-fb5f-4efe-b161-f51a88085075","cell_type":"code","source":"\n# Prediction Error and XAI Failure Review: retain failure cases for critical reporting.\njoint_primary = joint_prediction_xai_df[joint_prediction_xai_df.method.eq('gradcam')].drop_duplicates('image_id').copy() if len(joint_prediction_xai_df) else pd.DataFrame()\nreview_columns = ['image_id', 'xai_quality_flag', 'xai_quality_score', 'retina_focus_ratio', 'background_focus_ratio', 'optic_disc_focus_ratio', 'important_drop', 'random_drop', 'consistency_pearson']\nerror_xai_review_df = prediction_error_cases_df.merge(joint_primary[[name for name in review_columns if name in joint_primary]], on='image_id', how='left') if len(prediction_error_cases_df) and len(joint_primary) else prediction_error_cases_df.copy()\nfor name in review_columns:\n    if name not in error_xai_review_df: error_xai_review_df[name] = np.nan\nerror_xai_review_df['xai_review_category'] = np.select(\n    [error_xai_review_df.xai_quality_flag.astype(str).str.contains('needs_manual_review', na=False), error_xai_review_df.ordinally_adjacent],\n    ['prediction_and_xai_manual_review', 'ordinally_adjacent_prediction_error'], default='prediction_error_manual_review'\n)\n\ndef visual_record(image_id, method):\n    if len(prediction_visual_records_df) and 'method' in prediction_visual_records_df:\n        found = prediction_visual_records_df[(prediction_visual_records_df.image_id.astype(str).eq(str(image_id))) & prediction_visual_records_df.method.eq(method)]\n        if len(found): return found.iloc[0].to_dict()\n    found = base_generation_final[(base_generation_final.image_id.astype(str).eq(str(image_id))) & base_generation_final.method.eq(method)] if len(base_generation_final) else pd.DataFrame()\n    return found.iloc[0].to_dict() if len(found) else None\n\ndef review_overlay(image_id, method):\n    record = visual_record(image_id, method)\n    if record is None: return None\n    raw_path = Path(str(record.get('raw_heatmap_path', '')))\n    image_path = TRAIN_IMAGES_DIR / f'{image_id}.png'\n    if not raw_path.is_file() or not image_path.is_file(): return None\n    return overlay_heatmap(preprocess_image(image_path), np.load(raw_path))\n\ndef review_note(row):\n    note = f\"true={row['true_class']}; pred={row['predicted_class']}; confidence={row['confidence']:.3f}. \"\n    note += 'Adjacent ordinal error. ' if bool(row.get('ordinally_adjacent', False)) else 'Non-adjacent ordinal error or unavailable ordinal distance. '\n    flag = str(row.get('xai_quality_flag', 'not_available'))\n    if 'border_artifact_risk' in flag: note += 'Potential border/background emphasis; manual review.'\n    elif 'optic_disc_dominant' in flag: note += 'Estimated optic-disc emphasis; manual review.'\n    elif 'diffuse_heatmap' in flag: note += 'Diffuse attribution; focus is less specific.'\n    elif 'weak_faithfulness' in flag: note += 'Perturbation importance did not exceed the random baseline.'\n    else: note += 'No automatic artifact flag; prediction error remains a case for manual review.'\n    return note\n\ndef error_gallery(frame, title, output_path):\n    frame = frame.head(5).reset_index(drop=True)\n    if frame.empty:\n        fig, ax = plt.subplots(figsize=(8, 2)); ax.text(.5, .5, 'No cases available for this category.', ha='center'); ax.axis('off'); fig.savefig(output_path, dpi=180, bbox_inches='tight'); plt.close(fig); return\n    fig, axes = plt.subplots(len(frame), 4, figsize=(17, 4.1 * len(frame)), squeeze=False)\n    for index, row in frame.iterrows():\n        image_path = TRAIN_IMAGES_DIR / f\"{row['image_id']}.png\"\n        original = preprocess_image(image_path) if image_path.is_file() else None\n        panels = [(original, 'Original'), (review_overlay(row['image_id'], 'gradcam'), 'Grad-CAM'), (review_overlay(row['image_id'], 'gradcam_pp'), 'Grad-CAM++')]\n        for col, (image, label) in enumerate(panels):\n            if image is not None: axes[index, col].imshow(image)\n            axes[index, col].set_title(label); axes[index, col].axis('off')\n        axes[index, 3].text(.02, .98, review_note(row), va='top', wrap=True, fontsize=9); axes[index, 3].set_title('Notes'); axes[index, 3].axis('off')\n    fig.suptitle(title); fig.tight_layout(); fig.savefig(output_path, dpi=220, bbox_inches='tight'); plt.show(); plt.close(fig)\n\nhigh_wrong = error_xai_review_df[error_xai_review_df.confidence.ge(.8)].sort_values('confidence', ascending=False)\nlow_wrong = error_xai_review_df[error_xai_review_df.confidence.lt(.8)].sort_values('confidence')\nxai_failure_wrong = error_xai_review_df[error_xai_review_df.xai_quality_flag.astype(str).str.contains('needs_manual_review|weak_faithfulness|diffuse_heatmap|border_artifact_risk|optic_disc_dominant', regex=True, na=False)].sort_values('confidence', ascending=False)\nerror_gallery(high_wrong, 'High-confidence wrong predictions: XAI review', ERROR_REVIEW_DIR / 'high_confidence_wrong_predictions.png')\nerror_gallery(low_wrong, 'Low/medium-confidence wrong predictions: XAI review', ERROR_REVIEW_DIR / 'low_confidence_wrong_predictions.png')\nerror_gallery(xai_failure_wrong, 'Prediction errors with XAI manual-review flags', ERROR_REVIEW_DIR / 'xai_failure_error_cases.png')\n\nerror_review_lines = [\n    '# Prediction Error and XAI Failure Review',\n    f'- Wrong predictions={len(error_xai_review_df)}; high-confidence wrong={len(high_wrong)}; errors with XAI manual-review flags={len(xai_failure_wrong)}.',\n    f'- Adjacent ordinal errors={int(error_xai_review_df.ordinally_adjacent.sum()) if len(error_xai_review_df) else 0}/{len(error_xai_review_df)}.',\n    '- Notes characterize model output and attribution behaviour, not clinical lesions or the medical cause of an error.',\n    '', error_xai_review_df[['image_id', 'true_class', 'predicted_class', 'confidence', 'ordinal_distance', 'ordinally_adjacent', 'xai_quality_flag', 'xai_review_category']].to_string(index=False),\n]\n(FINAL_REPORT_DIR / 'prediction_error_xai_review.md').write_text('\\n'.join(error_review_lines))\ndisplay(error_xai_review_df)\nprint('\\n'.join(error_review_lines[:4]))\n","metadata":{},"outputs":[],"execution_count":null},{"id":"2a9abea7-2ea3-4bc5-9b71-a9d6bbe6649b","cell_type":"markdown","source":"## 24. Recommended Figures for Report\n\nSelections use transparent internal criteria, include cautions, and keep the optional prediction comparison disabled by default.\n","metadata":{}},{"id":"82c17fb2-e6a2-4f28-8944-beaf4434ccc8","cell_type":"code","source":"\n# Recommended Figures for Report and an optional no-retraining prediction-mode comparison.\nprimary_joint_for_report = joint_prediction_xai_df[joint_prediction_xai_df.method.eq('gradcam')].drop_duplicates('image_id').copy() if len(joint_prediction_xai_df) else pd.DataFrame()\nif len(primary_joint_for_report):\n    primary_joint_for_report['faithful'] = np.isfinite(pd.to_numeric(primary_joint_for_report.important_drop, errors='coerce')) & np.isfinite(pd.to_numeric(primary_joint_for_report.random_drop, errors='coerce')) & (primary_joint_for_report.important_drop > primary_joint_for_report.random_drop)\n    primary_joint_for_report['report_candidate'] = primary_joint_for_report.correct & primary_joint_for_report.xai_quality_flag.eq('good_candidate') & primary_joint_for_report.retina_focus_ratio.ge(.85) & primary_joint_for_report.background_focus_ratio.le(.15) & primary_joint_for_report.faithful\n    primary_joint_for_report['report_rank'] = primary_joint_for_report.xai_quality_score.fillna(-99) + primary_joint_for_report.confidence.fillna(0) + primary_joint_for_report.consistency_pearson.fillna(0)\n\ndef report_image(row, output_path, title):\n    image_path = TRAIN_IMAGES_DIR / f\"{row['image_id']}.png\"\n    original = preprocess_image(image_path) if image_path.is_file() else None\n    fig, axes = plt.subplots(1, 3, figsize=(13, 4.5))\n    for axis, (image, label) in zip(axes, [(original, 'Original'), (review_overlay(row['image_id'], 'gradcam'), 'Grad-CAM overlay'), (review_overlay(row['image_id'], 'gradcam_pp'), 'Grad-CAM++ overlay')]):\n        if image is not None: axis.imshow(image)\n        axis.set_title(label); axis.axis('off')\n    fig.suptitle(title, fontsize=10); fig.tight_layout(); fig.savefig(output_path, dpi=240, bbox_inches='tight'); plt.close(fig)\n\nrecommended_rows = []\nfor class_name in REPORT_CLASS_NAMES:\n    pool = primary_joint_for_report[primary_joint_for_report.true_class.eq(class_name)].copy() if len(primary_joint_for_report) else pd.DataFrame()\n    preferred = pool[pool.report_candidate].sort_values('report_rank', ascending=False)\n    fallback = pool[pool.correct].sort_values(['xai_quality_score', 'confidence'], ascending=False)\n    chosen = preferred if len(preferred) else fallback\n    if chosen.empty: continue\n    row = chosen.iloc[0]\n    figure_path = REPORT_SELECTED_DIR / f\"report_main_{class_name}_{row.image_id}.png\"\n    report_image(row, figure_path, f\"Recommended XAI example: {class_name} | {row.image_id}\")\n    reason = 'Correct prediction, high internal rank, retinal focus, and no automatic border/optic-disc dominant flag.'\n    if not bool(row.report_candidate): reason = 'Best available correct case for this class; at least one preferred score/faithfulness criterion was unavailable.'\n    recommended_rows.append({'figure_path': str(figure_path), 'image_id': row.image_id, 'true_class': row.true_class, 'predicted_class': row.predicted_class, 'confidence': row.confidence, 'method': 'gradcam_and_gradcam_pp', 'reason_for_selection': reason, 'caution_note': 'Internal report selection only. Heatmaps are not clinical lesion annotations.', 'figure_group': 'main_per_class'})\n\nfailure_report = error_xai_review_df.sort_values(['xai_quality_score', 'confidence'], ascending=[True, False]).head(3) if len(error_xai_review_df) else pd.DataFrame()\nfailure_report_path = REPORT_SELECTED_DIR / 'report_failure_cases.png'\nerror_gallery(failure_report, 'Recommended failure/manual-review cases', failure_report_path)\nfor row in failure_report.itertuples(index=False):\n    recommended_rows.append({'figure_path': str(failure_report_path), 'image_id': row.image_id, 'true_class': row.true_class, 'predicted_class': row.predicted_class, 'confidence': row.confidence, 'method': 'gradcam_and_gradcam_pp', 'reason_for_selection': 'Deliberately retained prediction error/manual-review case to avoid cherry-picking.', 'caution_note': 'Use as a limitation/failure case, not as clinical evidence.', 'figure_group': 'failure_cases'})\n\nfold_source = OUTPUT_ROOT / 'figures' / 'xai_foldaware' / 'xai_single_vs_foldaware_examples.png'\nfold_target = REPORT_SELECTED_DIR / 'report_foldaware_consistency_example.png'\nif fold_source.is_file():\n    shutil.copy2(fold_source, fold_target)\n    recommended_rows.append({'figure_path': str(fold_target), 'image_id': '', 'true_class': '', 'predicted_class': '', 'confidence': np.nan, 'method': 'single_fold_vs_fold_aware', 'reason_for_selection': 'Visualizes fold/checkpoint sensitivity without retraining.', 'caution_note': 'Similarity measures model stability, not clinical validation.', 'figure_group': 'foldaware_consistency'})\n\nrecommended_manifest = pd.DataFrame(recommended_rows, columns=['figure_path', 'image_id', 'true_class', 'predicted_class', 'confidence', 'method', 'reason_for_selection', 'caution_note', 'figure_group'])\nrecommended_manifest.to_csv(OUTPUT_ROOT / 'predictions' / 'recommended_figures_manifest.csv', index=False)\nrecommended_lines = [\n    '# Recommended Figures for Report',\n    f'- Main per-class figures={int((recommended_manifest.figure_group == \"main_per_class\").sum()) if len(recommended_manifest) else 0}; retained failure cases={int((recommended_manifest.figure_group == \"failure_cases\").sum()) if len(recommended_manifest) else 0}.',\n    '- Selections are criteria-based and include explicit caution notes. They do not turn model attribution into a diagnosis or lesion annotation.',\n    '', recommended_manifest.to_string(index=False),\n]\n(FINAL_REPORT_DIR / 'recommended_figures_for_report.md').write_text('\\n'.join(recommended_lines))\n\nmode_rows = []\nif RUN_PREDICTION_MODE_COMPARISON and len(prediction_subset_df):\n    for row in prediction_subset_df.itertuples(index=False):\n        image_path = TRAIN_IMAGES_DIR / f'{row.image_id}.png'\n        if not image_path.is_file(): continue\n        try:\n            image_rgb = preprocess_image(image_path)\n            probabilities = []\n            for fold in [0, 1, 2]:\n                fold_model, _ = load_model_for_fold(fold)\n                _, probs, _ = predict_rgb_with_model(fold_model, image_rgb)\n                probabilities.append(probs)\n            mean_probs = np.mean(probabilities, axis=0); predicted = int(np.argmax(mean_probs))\n            mode_rows.append({'image_id': row.image_id, 'true_label': row.true_label, 'true_class': row.true_class, 'foldaware_predicted_class': row.predicted_class, 'foldaware_confidence': row.confidence, 'c_threefold_average_predicted_class': REPORT_CLASS_NAMES[predicted], 'c_threefold_average_confidence': float(mean_probs[predicted]), 'foldaware_correct': row.correct, 'c_threefold_average_correct': predicted == int(row.true_label)})\n        except Exception as exc:\n            mode_rows.append({'image_id': row.image_id, 'mode_status': 'failed', 'mode_error': repr(exc)})\nif RUN_PREDICTION_MODE_COMPARISON:\n    mode_comparison = pd.DataFrame(mode_rows); mode_comparison.to_csv(OUTPUT_ROOT / 'predictions' / 'prediction_mode_comparison.csv', index=False)\n    valid = mode_comparison.dropna(subset=['foldaware_correct', 'c_threefold_average_correct'])\n    fig, ax = plt.subplots(figsize=(6, 4))\n    if len(valid): ax.bar(['Fold-aware C', 'C 3-fold average'], [valid.foldaware_correct.mean(), valid.c_threefold_average_correct.mean()])\n    ax.set_ylim(0, 1); ax.set_ylabel('Accuracy on selected subset'); ax.set_title('Optional prediction-mode comparison')\n    fig.tight_layout(); fig.savefig(PREDICTION_FIG_DIR / 'prediction_mode_comparison.png', dpi=220, bbox_inches='tight'); plt.close(fig)\n    mode_note = 'Executed on selected samples only; no threshold tuning or global-performance claim is allowed.'\nelse:\n    mode_note = 'Not executed because RUN_PREDICTION_MODE_COMPARISON=False.'\n(FINAL_REPORT_DIR / 'prediction_mode_comparison.md').write_text('# Optional Prediction Mode Comparison\\n\\n- ' + mode_note + '\\n')\n\nsummary_path = FINAL_REPORT_DIR / 'xai_final_summary.md'\nsummary_text = summary_path.read_text() if summary_path.is_file() else '# APTOS XAI Final Summary\\n'\nif '\\n## Final Output Analysis\\n' in summary_text:\n    summary_text = summary_text.split('\\n## Final Output Analysis\\n', 1)[0].rstrip() + '\\n'\nfinal_reporting_lines = [\n    '## Final Output Analysis', *final_output_analysis_lines[1:8],\n    '', '## Overall Prediction Result', *prediction_analysis_lines[1:5],\n    '', '## Prediction Error Analysis', f'- Adjacent ordinal errors={int(error_xai_review_df.ordinally_adjacent.sum()) if len(error_xai_review_df) else 0}/{len(error_xai_review_df)}. Error galleries retain high-confidence wrong and manual-review examples.',\n    '', '## XAI Visual Output Review', '- Technical CAM generation succeeded for the recorded outputs. Retina/proxy/quality indicators describe model attribution and require expert review for any clinical interpretation.',\n    '', '## Prediction-XAI Joint Analysis', *joint_analysis_lines[1:5],\n    '', '## Recommended Report Figures', *recommended_lines[:3],\n    '', '## Remaining Limitations', '- The selected XAI subset is small and purposive; no result here is a global performance estimate.', '- No lesion mask, expert annotation, or external validation is available. Proxy masks and internal quality scores are not clinical ground truth.', '- Fold-aware and consistency analysis improve methodology but remain model-stability evidence, not diagnostic validation.',\n    '', '## Next Research Recommendations', '- Add expert annotation or external data before stronger clinical interpretation.', '- Preserve balanced success/failure examples and per-class reporting.', '- Treat the optional prediction-mode comparison as descriptive until evaluated on proper OOF/external data.',\n]\nsummary_path.write_text(summary_text.rstrip() + '\\n\\n' + '\\n'.join(final_reporting_lines) + '\\n')\narchive_path = WORKING_ROOT / 'xai_outputs.zip'\nwith zipfile.ZipFile(archive_path, 'w', compression=zipfile.ZIP_DEFLATED) as archive:\n    for file_path in OUTPUT_ROOT.rglob('*'):\n        if file_path.is_file() and file_path.suffix not in {'.pth', '.zip'}:\n            archive.write(file_path, file_path.relative_to(OUTPUT_ROOT.parent))\nprint('Final reporting outputs:', FINAL_REPORT_DIR)\nprint('Updated archive:', archive_path)\ndisplay(recommended_manifest)\n","metadata":{},"outputs":[],"execution_count":null},{"id":"630cdaf8-5564-4bda-a2a7-612c56f22eb2","cell_type":"markdown","source":"## 25. Image Quality and XAI Anomaly Audit\n\nThis section audits saved images and heatmaps. It does not retrain, change the model input, alter CAM values, or crop the model input.\n","metadata":{}},{"id":"53067f1a-b1f6-491b-a8e9-19dc2539d1f1","cell_type":"code","source":"\n# Image Quality and XAI Anomaly Audit — audit only; model inference and CAM inputs remain unchanged.\nANOMALY_FIG_DIR = OUTPUT_ROOT / 'figures' / 'xai_anomaly_audit'\nANOMALY_FIG_DIR.mkdir(parents=True, exist_ok=True)\nAUDIT_METRICS_DIR = OUTPUT_ROOT / 'xai' / 'metrics'\nDISPLAY_CROP_FOR_AUDIT = False  # Display-only; never used for inference, CAM, or metric perturbation.\nIMAGE_QUALITY_AUDIT_PATH = AUDIT_METRICS_DIR / 'image_quality_audit.csv'\nXAI_ANOMALY_AUDIT_PATH = AUDIT_METRICS_DIR / 'xai_anomaly_audit.csv'\nFINAL_QUALITY_REVIEW_PATH = AUDIT_METRICS_DIR / 'xai_final_quality_review.csv'\n\ndef audit_read_csv(path):\n    try:\n        return pd.read_csv(path) if Path(path).is_file() else pd.DataFrame()\n    except Exception as exc:\n        print('Audit read skipped:', path, repr(exc))\n        return pd.DataFrame()\n\ndef audit_bool(frame, column='correct'):\n    return frame.get(column, pd.Series(False, index=frame.index)).astype(str).str.lower().isin(['true', '1', 'yes'])\n\ndef bbox_from_mask(mask):\n    rows, cols = np.where(mask)\n    if not len(rows):\n        return None\n    return int(rows.min()), int(rows.max()), int(cols.min()), int(cols.max())\n\ndef create_display_crop(image_rgb, retina_mask, padding_fraction=0.04):\n    # Optional display helper only. The 384x384 model input and original heatmap coordinates are untouched.\n    box = bbox_from_mask(retina_mask)\n    if box is None:\n        return image_rgb\n    top, bottom, left, right = box\n    pad = int(round(max(image_rgb.shape[:2]) * padding_fraction))\n    top, bottom = max(0, top - pad), min(image_rgb.shape[0], bottom + pad + 1)\n    left, right = max(0, left - pad), min(image_rgb.shape[1], right + pad + 1)\n    return image_rgb[top:bottom, left:right]\n\ndef edge_ring_mask(height, width, fraction=0.10):\n    band = max(1, int(round(min(height, width) * fraction)))\n    mask = np.zeros((height, width), dtype=bool)\n    mask[:band, :] = True; mask[-band:, :] = True; mask[:, :band] = True; mask[:, -band:] = True\n    return mask\n\ndef image_quality_metrics(image_rgb):\n    retina = create_retina_mask(image_rgb)\n    gray = cv2.cvtColor(image_rgb, cv2.COLOR_RGB2GRAY)\n    height, width = gray.shape\n    retina_area = int(retina.sum())\n    coverage = retina_area / float(height * width)\n    black_background = float((gray <= 15).mean())\n    values = gray[retina] if retina_area else gray.reshape(-1)\n    sharpness = float(cv2.Laplacian(gray, cv2.CV_64F)[retina].var()) if retina_area else float(cv2.Laplacian(gray, cv2.CV_64F).var())\n    brightness = float(values.mean()) if len(values) else np.nan\n    contrast = float(values.std()) if len(values) else np.nan\n    box = bbox_from_mask(retina)\n    touching_sides, min_margin_ratio = [], np.nan\n    if box is not None:\n        top, bottom, left, right = box\n        margins = {'top': top, 'bottom': height - 1 - bottom, 'left': left, 'right': width - 1 - right}\n        min_margin_ratio = min(margins.values()) / float(min(height, width))\n        touching_sides = [side for side, margin in margins.items() if margin <= 2]\n    edge = edge_ring_mask(height, width)\n    edge_retina = edge & retina\n    edge_extreme_ratio = float((((gray <= 20) | (gray >= 235)) & edge_retina).sum() / max(int(edge_retina.sum()), 1))\n    return {\n        'retina_mask': retina, 'retina_coverage_ratio': coverage, 'background_ratio': black_background,\n        'sharpness_score': sharpness, 'brightness_mean': brightness, 'contrast_score': contrast,\n        'retina_bbox_min_margin_ratio': min_margin_ratio, 'retina_touches_image_sides': '|'.join(touching_sides) if touching_sides else '',\n        'edge_extreme_ratio': edge_extreme_ratio,\n    }\n\nbase_records_audit = audit_read_csv(AUDIT_METRICS_DIR / 'xai_generation_records.csv')\nfold_records_audit = audit_read_csv(OUTPUT_ROOT / 'xai_foldaware' / 'metrics' / 'xai_foldaware_generation_records.csv')\nquality_score_audit = audit_read_csv(AUDIT_METRICS_DIR / 'xai_quality_score.csv')\nif len(fold_records_audit):\n    audit_prediction_source = fold_records_audit.drop_duplicates('image_id').copy()\nelif len(base_records_audit):\n    audit_prediction_source = base_records_audit.drop_duplicates('image_id').copy()\nelse:\n    audit_prediction_source = audit_read_csv(OUTPUT_ROOT / 'predictions' / 'xai_selected_samples_enhanced.csv')\n\nfor column in ['image_id', 'true_class', 'predicted_class', 'confidence', 'correct']:\n    if column not in audit_prediction_source:\n        audit_prediction_source[column] = np.nan\naudit_prediction_source['correct'] = audit_bool(audit_prediction_source)\naudit_prediction_source = audit_prediction_source.drop_duplicates('image_id')\n\n# First collect raw image values; then apply transparent subset-relative and absolute thresholds.\nraw_quality_rows = []\nfor row in audit_prediction_source.itertuples(index=False):\n    image_path = TRAIN_IMAGES_DIR / f'{row.image_id}.png'\n    if not image_path.is_file():\n        raw_quality_rows.append({'image_id': row.image_id, 'image_quality_status': 'failed', 'quality_error': f'missing_image:{image_path}'})\n        continue\n    try:\n        image_rgb = preprocess_image(image_path)\n        values = image_quality_metrics(image_rgb)\n        raw_quality_rows.append({\n            'image_id': row.image_id, 'true_class': row.true_class, 'predicted_class': row.predicted_class,\n            'confidence': row.confidence, 'correct': bool(row.correct), 'image_quality_status': 'success',\n            'quality_error': '', **{key: value for key, value in values.items() if key != 'retina_mask'}\n        })\n    except Exception as exc:\n        raw_quality_rows.append({'image_id': row.image_id, 'image_quality_status': 'failed', 'quality_error': repr(exc)})\n\nimage_quality_audit_df = pd.DataFrame(raw_quality_rows)\nsuccess_quality = image_quality_audit_df[image_quality_audit_df.image_quality_status.eq('success')].copy()\nsharpness_cutoff = max(8.0, float(success_quality.sharpness_score.quantile(.10))) if len(success_quality) else 8.0\ncontrast_cutoff = max(12.0, float(success_quality.contrast_score.quantile(.10))) if len(success_quality) else 12.0\nbrightness_low_cutoff, brightness_high_cutoff = 55.0, 205.0\ncoverage_cutoff, background_cutoff = .45, .50\n\ndef image_quality_flags(row):\n    flags = []\n    if row.retina_coverage_ratio < coverage_cutoff or row.background_ratio > background_cutoff:\n        flags.append('high_background_ratio')\n    if str(row.retina_touches_image_sides) or (np.isfinite(row.retina_bbox_min_margin_ratio) and row.retina_bbox_min_margin_ratio <= .005):\n        flags.append('cropping_or_fov_issue')\n    if row.sharpness_score < sharpness_cutoff:\n        flags.append('blurred_image')\n    if row.brightness_mean < brightness_low_cutoff:\n        flags.append('underexposed')\n    if row.brightness_mean > brightness_high_cutoff:\n        flags.append('overexposed')\n    if row.contrast_score < contrast_cutoff:\n        flags.append('low_contrast')\n    if row.edge_extreme_ratio > .35:\n        flags.append('edge_extreme_visual_content')\n    if not flags:\n        return 'good', ''\n    level = 'poor' if len(flags) >= 2 or 'cropping_or_fov_issue' in flags else 'moderate'\n    return level, '|'.join(flags)\n\nif len(success_quality):\n    applied = success_quality.apply(image_quality_flags, axis=1, result_type='expand')\n    image_quality_audit_df.loc[success_quality.index, 'overall_image_quality_level'] = applied[0]\n    image_quality_audit_df.loc[success_quality.index, 'overall_image_quality_flag'] = applied[1]\nimage_quality_audit_df.loc[image_quality_audit_df.image_quality_status.ne('success'), 'overall_image_quality_level'] = 'poor'\nimage_quality_audit_df.loc[image_quality_audit_df.image_quality_status.ne('success'), 'overall_image_quality_flag'] = 'audit_failed'\nimage_quality_audit_df['crop_issue_flag'] = image_quality_audit_df.overall_image_quality_flag.astype(str).str.contains('cropping_or_fov_issue', na=False)\nimage_quality_audit_df['blur_flag'] = image_quality_audit_df.overall_image_quality_flag.astype(str).str.contains('blurred_image', na=False)\nimage_quality_audit_df['exposure_flag'] = image_quality_audit_df.overall_image_quality_flag.astype(str).str.contains('underexposed|overexposed|low_contrast', regex=True, na=False)\nimage_quality_audit_df['edge_artifact_flag'] = image_quality_audit_df.overall_image_quality_flag.astype(str).str.contains('edge_extreme_visual_content', na=False)\nimage_quality_audit_df['audit_sharpness_cutoff'] = sharpness_cutoff\nimage_quality_audit_df['audit_contrast_cutoff'] = contrast_cutoff\nimage_quality_audit_df.to_csv(IMAGE_QUALITY_AUDIT_PATH, index=False)\n\nprint('Image-quality audit:', len(image_quality_audit_df), 'images | level counts:', image_quality_audit_df.overall_image_quality_level.value_counts(dropna=False).to_dict())\ndisplay(image_quality_audit_df)\n","metadata":{},"outputs":[],"execution_count":null},{"id":"8776156b-c3e5-46be-9796-52b9b88c653c","cell_type":"code","source":"\n# Heatmap anomaly audit from saved raw CAM files. No heatmap is changed or regenerated.\ndef heatmap_mass(heatmap, mask):\n    total = float(np.asarray(heatmap, dtype=np.float64).sum())\n    return np.nan if total <= 1e-12 else float(np.asarray(heatmap, dtype=np.float64)[mask].sum() / total)\n\ndef retina_zones(retina_mask):\n    retina_u8 = retina_mask.astype(np.uint8)\n    distance = cv2.distanceTransform(retina_u8, cv2.DIST_L2, 5)\n    maximum = float(distance.max())\n    if maximum <= 0:\n        return np.zeros_like(retina_mask, dtype=bool), np.zeros_like(retina_mask, dtype=bool)\n    peripheral = retina_mask & (distance <= maximum * .28)\n    central = retina_mask & (distance >= maximum * .50)\n    return peripheral, central\n\ndef top_heatmap_compactness(heatmap, retina_mask, top_fraction=.10):\n    allowed = np.flatnonzero(retina_mask.reshape(-1))\n    if not len(allowed):\n        return np.nan, np.nan\n    count = max(1, int(round(len(allowed) * top_fraction)))\n    flat = heatmap.reshape(-1)\n    selected = allowed[np.argpartition(flat[allowed], -count)[-count:]]\n    mask = np.zeros(flat.size, dtype=np.uint8); mask[selected] = 1; mask = mask.reshape(heatmap.shape)\n    box = bbox_from_mask(mask.astype(bool))\n    if box is None:\n        return np.nan, np.nan\n    top, bottom, left, right = box\n    bbox_area = max((bottom - top + 1) * (right - left + 1), 1)\n    compactness = float(mask.sum() / bbox_area)\n    return compactness, float(mask.sum() / mask.size)\n\ndef normalise_records(frame, mode):\n    frame = frame.copy()\n    if frame.empty:\n        return frame\n    frame['xai_mode'] = mode\n    for column in ['image_id', 'method', 'true_class', 'predicted_class', 'confidence', 'correct', 'raw_heatmap_path']:\n        if column not in frame:\n            frame[column] = np.nan\n    frame['correct'] = audit_bool(frame)\n    return frame\n\nbase_heatmap_records = normalise_records(base_records_audit, 'single_fold_baseline')\nfold_heatmap_records = normalise_records(fold_records_audit, 'fold_aware')\nall_heatmap_records = pd.concat([base_heatmap_records, fold_heatmap_records], ignore_index=True, sort=False)\nquality_by_image = image_quality_audit_df.set_index('image_id').to_dict('index') if len(image_quality_audit_df) else {}\nheatmap_anomaly_rows = []\nfor row in all_heatmap_records.itertuples(index=False):\n    try:\n        raw_path = Path(str(row.raw_heatmap_path))\n        image_path = TRAIN_IMAGES_DIR / f'{row.image_id}.png'\n        if not raw_path.is_file() or not image_path.is_file():\n            raise FileNotFoundError(f'raw_heatmap_or_image_missing:{raw_path}')\n        image_rgb = preprocess_image(image_path)\n        heatmap = np.load(raw_path).astype(np.float32)\n        retina = create_retina_mask(image_rgb)\n        edge = edge_ring_mask(*heatmap.shape)\n        peripheral, central = retina_zones(retina)\n        edge_ratio = heatmap_mass(heatmap, edge)\n        background_ratio = heatmap_mass(heatmap, ~retina)\n        peripheral_ratio = heatmap_mass(heatmap, peripheral)\n        central_ratio = heatmap_mass(heatmap, central)\n        retina_ratio = heatmap_mass(heatmap, retina)\n        compactness, top_active_ratio = top_heatmap_compactness(heatmap, retina)\n        active_ratio = float((heatmap >= .5).mean())\n        image_quality = quality_by_image.get(row.image_id, {})\n        reasons = []\n        if np.isfinite(edge_ratio) and edge_ratio > .35: reasons.append('edge_heatmap_dominant')\n        if np.isfinite(background_ratio) and background_ratio > .15: reasons.append('background_heatmap_risk')\n        if np.isfinite(peripheral_ratio) and np.isfinite(central_ratio) and peripheral_ratio > max(.55, central_ratio + .25): reasons.append('peripheral_focus_high')\n        if np.isfinite(compactness) and compactness < .18 and np.isfinite(edge_ratio) and edge_ratio > .25: reasons.append('edge_spike_heatmap')\n        if active_ratio > .15: reasons.append('diffuse_heatmap')\n        if np.isfinite(retina_ratio) and retina_ratio < .85: reasons.append('low_retina_overlap')\n        if image_quality.get('overall_image_quality_level') == 'poor': reasons.append('poor_image_quality')\n        if image_quality.get('edge_artifact_flag', False) and np.isfinite(edge_ratio) and edge_ratio > .25: reasons.append('edge_artifact_risk')\n        heatmap_anomaly_rows.append({\n            'image_id': row.image_id, 'method': row.method, 'xai_mode': row.xai_mode,\n            'true_class': row.true_class, 'predicted_class': row.predicted_class, 'confidence': row.confidence,\n            'correct': bool(row.correct), 'edge_heatmap_ratio': edge_ratio,\n            'background_heatmap_ratio': background_ratio, 'peripheral_focus_ratio': peripheral_ratio,\n            'central_focus_ratio': central_ratio, 'retina_focus_ratio': retina_ratio,\n            'active_area_ratio': active_ratio, 'heatmap_compactness_score': compactness,\n            'top10_active_area_ratio': top_active_ratio,\n            'image_quality_level': image_quality.get('overall_image_quality_level', 'unknown'),\n            'image_quality_flag': image_quality.get('overall_image_quality_flag', ''),\n            'suspicious_xai_flag': bool(reasons), 'anomaly_reason': '|'.join(reasons),\n            'audit_status': 'success', 'audit_error': '', 'raw_heatmap_path': str(raw_path),\n        })\n    except Exception as exc:\n        heatmap_anomaly_rows.append({\n            'image_id': row.image_id, 'method': row.method, 'xai_mode': row.xai_mode,\n            'true_class': row.true_class, 'predicted_class': row.predicted_class, 'confidence': row.confidence,\n            'correct': bool(row.correct), 'suspicious_xai_flag': True, 'anomaly_reason': 'audit_failed',\n            'audit_status': 'failed', 'audit_error': repr(exc), 'raw_heatmap_path': str(row.raw_heatmap_path),\n        })\n\nxai_anomaly_audit_df = pd.DataFrame(heatmap_anomaly_rows)\nxai_anomaly_audit_df.to_csv(XAI_ANOMALY_AUDIT_PATH, index=False)\nprint('Heatmap anomaly audit:', len(xai_anomaly_audit_df), 'records | suspicious:', int(xai_anomaly_audit_df.suspicious_xai_flag.sum()) if len(xai_anomaly_audit_df) else 0)\ndisplay(xai_anomaly_audit_df)\n","metadata":{},"outputs":[],"execution_count":null},{"id":"661ce5b0-b3c2-403b-98e2-fab8d5e44a83","cell_type":"code","source":"\n# Final quality review, audit galleries, filtered report recommendations, and final archive refresh.\nquality_score_for_review = quality_score_audit.copy()\nif len(quality_score_for_review):\n    for column in ['image_id', 'method', 'xai_mode', 'xai_quality_flag', 'xai_quality_score']:\n        if column not in quality_score_for_review: quality_score_for_review[column] = np.nan\n    quality_score_for_review = quality_score_for_review[['image_id', 'method', 'xai_mode', 'xai_quality_flag', 'xai_quality_score']].drop_duplicates(['image_id', 'method', 'xai_mode'])\n\nfinal_quality_review_df = xai_anomaly_audit_df.merge(\n    image_quality_audit_df[['image_id', 'overall_image_quality_level', 'overall_image_quality_flag', 'retina_coverage_ratio', 'background_ratio', 'sharpness_score', 'brightness_mean', 'contrast_score', 'crop_issue_flag', 'blur_flag', 'exposure_flag', 'edge_artifact_flag']],\n    on='image_id', how='left', suffixes=('', '_image')\n)\nfinal_quality_review_df = final_quality_review_df.merge(\n    quality_score_for_review, on=['image_id', 'method', 'xai_mode'], how='left', suffixes=('', '_quality')\n) if len(quality_score_for_review) else final_quality_review_df\nif 'xai_quality_flag' not in final_quality_review_df:\n    final_quality_review_df['xai_quality_flag'] = ''\nif 'xai_quality_score' not in final_quality_review_df:\n    final_quality_review_df['xai_quality_score'] = np.nan\nfinal_quality_review_df['image_quality_flag'] = final_quality_review_df.overall_image_quality_flag.fillna('')\nfinal_quality_review_df['xai_quality_flag'] = final_quality_review_df.xai_quality_flag.fillna('')\nfinal_quality_review_df['suspicious_xai_flag'] = final_quality_review_df.suspicious_xai_flag.fillna(True).astype(bool)\n\ndef recommendation_for_row(row):\n    correct = bool(row.get('correct', False))\n    bad_image = str(row.get('overall_image_quality_level', 'unknown')) == 'poor'\n    suspicious = bool(row.get('suspicious_xai_flag', True))\n    flags = str(row.get('xai_quality_flag', ''))\n    if not correct:\n        return 'failure_case_candidate'\n    if bad_image and suspicious:\n        return 'exclude_from_main_report'\n    if suspicious or 'needs_manual_review' in flags:\n        return 'manual_review_only'\n    if str(row.get('overall_image_quality_level', '')) == 'good' and 'good_candidate' in flags:\n        return 'main_report_candidate'\n    return 'supporting_example'\n\nfinal_quality_review_df['recommended_category'] = final_quality_review_df.apply(recommendation_for_row, axis=1)\nfinal_quality_review_df['recommended_for_report'] = final_quality_review_df.recommended_category.eq('main_report_candidate')\nfinal_quality_review_df.to_csv(FINAL_QUALITY_REVIEW_PATH, index=False)\n\n# Preserve the prior manifest and figures before replacing the main-report selection.\nold_manifest_path = OUTPUT_ROOT / 'predictions' / 'recommended_figures_manifest.csv'\npreserve_dir = REPORT_SELECTED_DIR / 'pre_image_quality_audit'\npreserve_dir.mkdir(parents=True, exist_ok=True)\nif old_manifest_path.is_file():\n    shutil.copy2(old_manifest_path, OUTPUT_ROOT / 'predictions' / 'recommended_figures_manifest_pre_image_quality_audit.csv')\nfor old_path in REPORT_SELECTED_DIR.glob('report_*.png'):\n    shutil.copy2(old_path, preserve_dir / old_path.name)\n\ndef audit_primary_rows(frame):\n    preferred = frame[frame.xai_mode.eq('fold_aware') & frame.method.eq('gradcam')].copy()\n    if preferred.empty:\n        preferred = frame[frame.method.eq('gradcam')].copy()\n    return preferred.drop_duplicates('image_id')\n\naudit_primary = audit_primary_rows(final_quality_review_df)\n\ndef audit_overlay(image_id, method, mode='fold_aware'):\n    options = final_quality_review_df[(final_quality_review_df.image_id.astype(str).eq(str(image_id))) & final_quality_review_df.method.eq(method) & final_quality_review_df.xai_mode.eq(mode)]\n    if options.empty:\n        options = final_quality_review_df[(final_quality_review_df.image_id.astype(str).eq(str(image_id))) & final_quality_review_df.method.eq(method)]\n    if options.empty:\n        return None\n    raw_path = Path(str(options.iloc[0].raw_heatmap_path))\n    image_path = TRAIN_IMAGES_DIR / f'{image_id}.png'\n    if not raw_path.is_file() or not image_path.is_file():\n        return None\n    return overlay_heatmap(preprocess_image(image_path), np.load(raw_path))\n\ndef audit_gallery(rows, title, output_path):\n    rows = rows.head(4).reset_index(drop=True)\n    if rows.empty:\n        fig, ax = plt.subplots(figsize=(8, 2)); ax.text(.5, .5, 'No records for this audit category.', ha='center', va='center'); ax.axis('off'); fig.savefig(output_path, dpi=180, bbox_inches='tight'); plt.close(fig); return\n    fig, axes = plt.subplots(len(rows), 5, figsize=(20, 4.0 * len(rows)), squeeze=False)\n    for index, row in rows.iterrows():\n        image_path = TRAIN_IMAGES_DIR / f\"{row['image_id']}.png\"\n        original = preprocess_image(image_path) if image_path.is_file() else None\n        retina = create_retina_mask(original) if original is not None else None\n        shown_original = create_display_crop(original, retina) if DISPLAY_CROP_FOR_AUDIT and original is not None else original\n        panels = [(shown_original, 'Original' + (' (display crop only)' if DISPLAY_CROP_FOR_AUDIT else '')), (retina, 'Retina mask'), (audit_overlay(row['image_id'], 'gradcam', row['xai_mode']), 'Grad-CAM'), (audit_overlay(row['image_id'], 'gradcam_pp', row['xai_mode']), 'Grad-CAM++')]\n        for col, (image, label) in enumerate(panels):\n            if image is not None: axes[index, col].imshow(image, cmap='gray' if np.asarray(image).ndim == 2 else None)\n            axes[index, col].set_title(label); axes[index, col].axis('off')\n        note = (f\"{row['image_id']}\\ntrue={row['true_class']} | pred={row['predicted_class']} | conf={row['confidence']:.3f}\\n\"\n                f\"image={row.get('overall_image_quality_level', 'unknown')} [{row.get('image_quality_flag', '')}]\\n\"\n                f\"xai={row.get('anomaly_reason', '') or 'no automatic anomaly'}\\nrecommend={row.get('recommended_category', '')}\")\n        axes[index, 4].text(.02, .98, note, va='top', wrap=True, fontsize=9); axes[index, 4].set_title('Audit notes'); axes[index, 4].axis('off')\n    fig.suptitle(title, fontsize=13); fig.tight_layout(); fig.savefig(output_path, dpi=220, bbox_inches='tight'); plt.show(); plt.close(fig)\n\naudit_gallery(audit_primary[audit_primary.suspicious_xai_flag], 'Suspicious XAI cases — audit only', ANOMALY_FIG_DIR / 'xai_anomaly_suspicious_cases.png')\naudit_gallery(audit_primary[audit_primary.anomaly_reason.astype(str).str.contains('edge_heatmap_dominant|edge_spike_heatmap|peripheral_focus_high', regex=True, na=False)], 'Edge/peripheral focus cases', ANOMALY_FIG_DIR / 'xai_anomaly_edge_focus_cases.png')\naudit_gallery(audit_primary[audit_primary.anomaly_reason.astype(str).str.contains('background_heatmap_risk', na=False)], 'Background-focus cases', ANOMALY_FIG_DIR / 'xai_anomaly_background_focus_cases.png')\naudit_gallery(audit_primary[audit_primary.overall_image_quality_level.eq('poor')], 'Low-quality image cases', ANOMALY_FIG_DIR / 'xai_anomaly_low_quality_images.png')\naudit_gallery(audit_primary[audit_primary.recommended_category.eq('main_report_candidate')], 'Clean main-report candidates', ANOMALY_FIG_DIR / 'xai_anomaly_clean_report_candidates.png')\n\n# Filter main-report figures; poor/suspicious examples may remain only as explicit failure/manual-review cases.\ndef save_clean_report_figure(row, output_path):\n    image_path = TRAIN_IMAGES_DIR / f\"{row['image_id']}.png\"\n    original = preprocess_image(image_path) if image_path.is_file() else None\n    fig, axes = plt.subplots(1, 3, figsize=(13, 4.5))\n    for axis, (image, label) in zip(axes, [(original, 'Original'), (audit_overlay(row['image_id'], 'gradcam', row['xai_mode']), 'Grad-CAM overlay'), (audit_overlay(row['image_id'], 'gradcam_pp', row['xai_mode']), 'Grad-CAM++ overlay')]):\n        if image is not None: axis.imshow(image)\n        axis.set_title(label); axis.axis('off')\n    fig.suptitle(f\"Clean report candidate: {row['true_class']} | {row['image_id']}\", fontsize=10)\n    fig.tight_layout(); fig.savefig(output_path, dpi=240, bbox_inches='tight'); plt.close(fig)\n\nnew_manifest_rows = []\nfor class_name in REPORT_CLASS_NAMES:\n    pool = audit_primary[(audit_primary.true_class.eq(class_name)) & audit_primary.recommended_category.eq('main_report_candidate')].copy()\n    if pool.empty:\n        pool = audit_primary[(audit_primary.true_class.eq(class_name)) & audit_primary.recommended_category.eq('supporting_example')].copy()\n    if pool.empty:\n        continue\n    pool['selection_rank'] = pool.xai_quality_score.fillna(-99) + pool.confidence.fillna(0) - pool.edge_heatmap_ratio.fillna(0) - pool.background_heatmap_ratio.fillna(0)\n    row = pool.sort_values('selection_rank', ascending=False).iloc[0]\n    figure_path = REPORT_SELECTED_DIR / f\"clean_main_{class_name}_{row.image_id}.png\"\n    save_clean_report_figure(row, figure_path)\n    category = 'main_report_candidate' if row.recommended_category == 'main_report_candidate' else 'supporting_example'\n    new_manifest_rows.append({\n        'figure_path': str(figure_path), 'image_id': row.image_id, 'true_class': row.true_class,\n        'predicted_class': row.predicted_class, 'confidence': row.confidence, 'method': 'gradcam_and_gradcam_pp',\n        'reason_for_selection': 'Filtered by image-quality and heatmap-anomaly audit; selected from the cleanest available correct candidate for its class.',\n        'caution_note': 'Attribution is model-derived. It is not a clinical lesion annotation or diagnostic proof.',\n        'recommended_category': category, 'image_quality_flag': row.image_quality_flag,\n        'xai_quality_flag': row.xai_quality_flag, 'suspicious_xai_flag': row.suspicious_xai_flag,\n    })\n\nfailure_pool = audit_primary[audit_primary.recommended_category.eq('failure_case_candidate') | audit_primary.recommended_category.eq('manual_review_only')].copy()\nfor row in failure_pool.sort_values(['suspicious_xai_flag', 'confidence'], ascending=[False, False]).head(3).itertuples(index=False):\n    new_manifest_rows.append({\n        'figure_path': str(ANOMALY_FIG_DIR / 'xai_anomaly_suspicious_cases.png'), 'image_id': row.image_id,\n        'true_class': row.true_class, 'predicted_class': row.predicted_class, 'confidence': row.confidence,\n        'method': 'gradcam_and_gradcam_pp', 'reason_for_selection': 'Retained deliberately as failure/manual-review evidence; not eligible for a main report example.',\n        'caution_note': 'Potential image/XAI anomaly; interpret only as a limitation or review case.',\n        'recommended_category': 'failure_case_candidate', 'image_quality_flag': row.image_quality_flag,\n        'xai_quality_flag': row.xai_quality_flag, 'suspicious_xai_flag': row.suspicious_xai_flag,\n    })\n\nfiltered_manifest_df = pd.DataFrame(new_manifest_rows)\nfiltered_manifest_df.to_csv(OUTPUT_ROOT / 'predictions' / 'recommended_figures_manifest.csv', index=False)\nrecommended_report_lines = [\n    '# Recommended Figures for Report — Image Quality Filtered',\n    f'- Main clean candidates={int((filtered_manifest_df.recommended_category == \"main_report_candidate\").sum()) if len(filtered_manifest_df) else 0}; supporting examples={int((filtered_manifest_df.recommended_category == \"supporting_example\").sum()) if len(filtered_manifest_df) else 0}; failure cases={int((filtered_manifest_df.recommended_category == \"failure_case_candidate\").sum()) if len(filtered_manifest_df) else 0}.',\n    '- The previous manifest and report figures were preserved under pre_image_quality_audit before this filtered manifest was written.',\n    '- Poor-quality or edge/background-suspicious images are not main-report candidates; they may be retained only for failure/manual-review discussion.',\n    '', filtered_manifest_df.to_string(index=False),\n]\n(FINAL_REPORT_DIR / 'recommended_figures_for_report.md').write_text('\\n'.join(recommended_report_lines))\n\nquality_level_counts = image_quality_audit_df.overall_image_quality_level.value_counts(dropna=False).to_dict()\nprimary_suspicious = int(audit_primary.suspicious_xai_flag.sum()) if len(audit_primary) else 0\nanomaly_counts = {\n    'edge_or_peripheral': int(audit_primary.anomaly_reason.astype(str).str.contains('edge_heatmap_dominant|edge_spike_heatmap|peripheral_focus_high', regex=True, na=False).sum()) if len(audit_primary) else 0,\n    'background': int(audit_primary.anomaly_reason.astype(str).str.contains('background_heatmap_risk', na=False).sum()) if len(audit_primary) else 0,\n    'poor_image': int(audit_primary.overall_image_quality_level.eq('poor').sum()) if len(audit_primary) else 0,\n}\nanomaly_report_lines = [\n    '# Image Quality and XAI Anomaly Audit Report',\n    f'- Images audited={len(image_quality_audit_df)}; quality levels={quality_level_counts}.',\n    f'- Heatmap records audited={len(xai_anomaly_audit_df)}; primary fold-aware Grad-CAM suspicious cases={primary_suspicious}.',\n    f'- Edge/peripheral-focus indications={anomaly_counts[\"edge_or_peripheral\"]}; background-focus indications={anomaly_counts[\"background\"]}; poor-image records={anomaly_counts[\"poor_image\"]}.',\n    f'- Main-report candidates after filtering={int((filtered_manifest_df.recommended_category == \"main_report_candidate\").sum()) if len(filtered_manifest_df) else 0}; failure/manual-review candidates={int((filtered_manifest_df.recommended_category == \"failure_case_candidate\").sum()) if len(filtered_manifest_df) else 0}.',\n    '- Edge/peripheral attribution is not inherently invalid in retinal imagery. It is flagged as a potential confounder requiring manual review, especially when combined with weak image quality or background emphasis.',\n    '- Automatic image-quality and anomaly thresholds are heuristics. They cannot determine whether an area is a lesion, artifact, or clinically meaningful structure without expert annotation.',\n    '', '## Image quality counts', image_quality_audit_df.overall_image_quality_flag.value_counts(dropna=False).to_string(),\n    '', '## Primary anomaly examples', audit_primary[['image_id', 'true_class', 'predicted_class', 'confidence', 'overall_image_quality_level', 'image_quality_flag', 'anomaly_reason', 'recommended_category']].to_string(index=False) if len(audit_primary) else 'No primary records.',\n]\n(FINAL_REPORT_DIR / 'xai_anomaly_audit_report.md').write_text('\\n'.join(anomaly_report_lines))\n\nsummary_path = FINAL_REPORT_DIR / 'xai_final_summary.md'\nsummary_text = summary_path.read_text() if summary_path.is_file() else '# APTOS XAI Final Summary\\n'\nif '\\n## Image Quality Audit\\n' in summary_text:\n    summary_text = summary_text.split('\\n## Image Quality Audit\\n', 1)[0].rstrip() + '\\n'\nimage_anomaly_summary_lines = [\n    '## Image Quality Audit',\n    f'- Audited {len(image_quality_audit_df)} selected XAI images. Quality levels: {quality_level_counts}.',\n    f'- Image-level flags identify potentially large background, crop/FOV, blur, exposure/contrast, or edge-extreme content; they do not modify model preprocessing.',\n    '', '## XAI Anomaly Audit',\n    f'- Audited {len(xai_anomaly_audit_df)} saved heatmap records. Primary suspicious cases={primary_suspicious}; edge/peripheral indications={anomaly_counts[\"edge_or_peripheral\"]}; background indications={anomaly_counts[\"background\"]}.',\n    '- An edge/peripheral hotspot can be a model-relevant retinal area, artifact, or other confounder. It is therefore routed to manual review, not clinically labelled.',\n    '', '## Clean Report Candidates',\n    f'- Filtered main candidates={int((filtered_manifest_df.recommended_category == \"main_report_candidate\").sum()) if len(filtered_manifest_df) else 0}. Main examples must have adequate image quality and no flagged edge/background anomaly.',\n    '', '## Suspicious and Failure Cases',\n    '- Images with poor quality plus edge/background-focused attribution are excluded from main examples and retained only as failure/manual-review cases when informative.',\n    '', '## Impact of Image Quality on XAI Interpretation',\n    '- Heatmaps are sensitive to the image content presented to the trained model. Poor framing, large dark borders, or peripheral spikes can make visual interpretation less reliable even when CAM generation is technically successful.',\n    '- No input crop, model checkpoint, preprocessing, heatmap, or colormap was altered by this audit.',\n]\nsummary_path.write_text(summary_text.rstrip() + '\\n\\n' + '\\n'.join(image_anomaly_summary_lines) + '\\n')\narchive_path = WORKING_ROOT / 'xai_outputs.zip'\nwith zipfile.ZipFile(archive_path, 'w', compression=zipfile.ZIP_DEFLATED) as archive:\n    for file_path in OUTPUT_ROOT.rglob('*'):\n        if file_path.is_file() and file_path.suffix not in {'.pth', '.zip'}:\n            archive.write(file_path, file_path.relative_to(OUTPUT_ROOT.parent))\nprint('Anomaly audit report:', FINAL_REPORT_DIR / 'xai_anomaly_audit_report.md')\nprint('Final quality review:', FINAL_QUALITY_REVIEW_PATH)\nprint('Archive refreshed:', archive_path)\ndisplay(filtered_manifest_df)\n","metadata":{},"outputs":[],"execution_count":null}]}