{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":113558,"databundleVersionId":14174843,"sourceType":"competition"}],"dockerImageVersionId":31154,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Recod.ai/LUC - Scientific Image Forgery Detection\n- Develop methods that can accurately detect and segment copy-move forgeries within biomedical research images.\n- Link: https://www.kaggle.com/competitions/recodai-luc-scientific-image-forgery-detection/","metadata":{}},{"cell_type":"markdown","source":"## Imports","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\nfrom torch.utils.data import Dataset, DataLoader\nimport cv2\nfrom glob import glob\nfrom sklearn.model_selection import train_test_split\nimport json\nfrom matplotlib import pyplot as plt\nimport random\nrandom.seed(42)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-31T08:15:53.701607Z","iopub.execute_input":"2025-10-31T08:15:53.701877Z","iopub.status.idle":"2025-10-31T08:15:56.236866Z","shell.execute_reply.started":"2025-10-31T08:15:53.701857Z","shell.execute_reply":"2025-10-31T08:15:56.235816Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Basic Config","metadata":{}},{"cell_type":"code","source":"# ==================== CONFIGURATION ====================\nIMG_SIZE = 256\nBATCH_SIZE = 8\nEPOCHS = 10\nLEARNING_RATE = 0.0001\nDEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\nDATA_PATH=\"/kaggle/input/recodai-luc-scientific-image-forgery-detection/\"\nprint(f\"Using device: {DEVICE}\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-31T08:15:56.238626Z","iopub.execute_input":"2025-10-31T08:15:56.239454Z","iopub.status.idle":"2025-10-31T08:15:56.302525Z","shell.execute_reply.started":"2025-10-31T08:15:56.239430Z","shell.execute_reply":"2025-10-31T08:15:56.301713Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Dataset Class - The Data Handler\n\n**What it does:**\n- Loads images and masks from disk\n- Resizes to 256×256 (faster training)\n- Normalizes pixels to 0-1 range","metadata":{}},{"cell_type":"code","source":"# ==================== DATASET ====================\nclass ForgeryDataset(Dataset):\n    def __init__(self, image_paths, mask_paths=None, is_authentic=None):\n        self.image_paths = image_paths\n        self.mask_paths = mask_paths if mask_paths is not None else [None] * len(image_paths)\n        self.is_authentic = is_authentic if is_authentic is not None else [False] * len(image_paths)\n    \n    def __len__(self):\n        return len(self.image_paths)\n    \n    def __getitem__(self, idx):\n        # Load image\n        img = cv2.imread(self.image_paths[idx])\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        img = cv2.resize(img, (IMG_SIZE, IMG_SIZE)) / 255.0\n        img = torch.from_numpy(img).float().permute(2, 0, 1)\n        \n        # Load or create mask\n        if self.is_authentic[idx]:\n            mask = np.zeros((IMG_SIZE, IMG_SIZE), dtype=np.float32)\n        else:\n            mask = np.load(self.mask_paths[idx])\n            if mask.ndim == 3:\n                mask = mask[0, :, :]  # FIX: Take first channel, not first column\n            mask = cv2.resize(mask.astype(np.uint8), (IMG_SIZE, IMG_SIZE))\n            mask = (mask > 0).astype(np.float32)\n        \n        mask = torch.from_numpy(mask).float().unsqueeze(0)\n        return img, mask\n\n# Load data\nforged_images = sorted(glob(DATA_PATH + 'train_images/forged/*.png'))\ntrain_masks = sorted(glob(DATA_PATH + 'train_masks/*.npy'))\nauthentic_images = sorted(glob(DATA_PATH + 'train_images/authentic/*.png'))\n\n# Match forged images with masks\nmatched_forged_images = []\nmatched_masks = []\nfor img_path in forged_images:\n    img_name = img_path.split('/')[-1].replace('.png', '')\n    mask_path = DATA_PATH + f'train_masks/{img_name}.npy'\n    if mask_path in train_masks:\n        matched_forged_images.append(img_path)\n        matched_masks.append(mask_path)\n\n# Combine datasets\nall_images = matched_forged_images + authentic_images\nall_masks = matched_masks + [None] * len(authentic_images)\nall_is_authentic = [False] * len(matched_forged_images) + [True] * len(authentic_images)\n\nprint(f\"Total images: {len(all_images)}\")\nprint(f\"  Forged: {len(matched_forged_images)}\")\nprint(f\"  Authentic: {len(authentic_images)}\")\n\n# Train/val split\ntrain_imgs, val_imgs, train_masks_split, val_masks_split, train_auth, val_auth = train_test_split(\n    all_images, all_masks, all_is_authentic, test_size=0.2, random_state=42, stratify=all_is_authentic\n)\n\ntrain_dataset = ForgeryDataset(train_imgs, train_masks_split, train_auth)\nval_dataset = ForgeryDataset(val_imgs, val_masks_split, val_auth)\n\ntrain_loader = DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=True)\nval_loader = DataLoader(val_dataset, batch_size=BATCH_SIZE, shuffle=False)\n\nprint(f\"\\nTrain: {len(train_dataset)} images ({sum(train_auth)} authentic, {len(train_auth) - sum(train_auth)} forged)\")\nprint(f\"Val: {len(val_dataset)} images ({sum(val_auth)} authentic, {len(val_auth) - sum(val_auth)} forged)\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-31T08:15:56.303271Z","iopub.execute_input":"2025-10-31T08:15:56.303574Z","iopub.status.idle":"2025-10-31T08:15:56.412846Z","shell.execute_reply.started":"2025-10-31T08:15:56.303548Z","shell.execute_reply":"2025-10-31T08:15:56.412132Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Visualization: Check Actual Masks\n\nLet's first visualize some forged training images with their ground truth masks to understand the data.","metadata":{}},{"cell_type":"code","source":"import os\ndef visualize_ground_truth(image_paths, mask_paths, is_authentic, num_authentic=1, num_forged=5):\n    \"\"\"Visualize authentic and forged images with their ground truth masks.\"\"\"\n    # Select authentic and forged images\n    authentic_indices = [i for i, auth in enumerate(is_authentic) if auth]\n    forged_indices = [i for i, auth in enumerate(is_authentic) if not auth]\n    \n    selected_authentic = np.random.choice(authentic_indices, min(num_authentic, len(authentic_indices)), replace=False)\n    selected_forged = np.random.choice(forged_indices, min(num_forged, len(forged_indices)), replace=False)\n    \n    selected_indices = list(selected_authentic) + list(selected_forged)\n    \n    fig, axes = plt.subplots(len(selected_indices), 2, figsize=(10, 5 * len(selected_indices)))\n    if len(selected_indices) == 1:\n        axes = axes.reshape(1, -1)\n    \n    for plot_idx, idx in enumerate(selected_indices):\n        img_path = image_paths[idx]\n        is_auth = is_authentic[idx]\n        \n        # Load image\n        img = cv2.imread(img_path)\n        img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        \n        # Load mask\n        if is_auth:\n            mask = np.zeros((img_rgb.shape[0], img_rgb.shape[1]), dtype=np.uint8)\n            label = \"AUTHENTIC\"\n        else:\n            mask_path = mask_paths[idx]\n            mask = np.load(mask_path)\n            if mask.ndim == 3:\n                mask = mask[0, :, :]  # FIX: Take first channel, not first column\n            mask = (mask > 0).astype(np.uint8)\n            label = \"FORGED\"\n        \n        # Calculate statistics\n        forged_pixels = mask.sum()\n        total_pixels = mask.size\n        forged_percent = (forged_pixels / total_pixels) * 100\n        \n        # Plot\n        axes[plot_idx, 0].imshow(img_rgb)\n        axes[plot_idx, 0].set_title(f\"{label}: {os.path.basename(img_path)}\", fontsize=10)\n        axes[plot_idx, 0].axis('off')\n        \n        axes[plot_idx, 1].imshow(mask, cmap='gray')\n        axes[plot_idx, 1].set_title(\n            f\"Ground Truth Mask\\n{forged_pixels:,} forged pixels ({forged_percent:.2f}%)\",\n            fontsize=10\n        )\n        axes[plot_idx, 1].axis('off')\n        \n        print(f\"Image {plot_idx+1} [{label}]: {forged_percent:.2f}% forged ({forged_pixels:,} pixels)\")\n    \n    plt.tight_layout()\n    plt.show()\n\nprint(\"Ground Truth Analysis:\")\nprint(\"=\"*60)\nvisualize_ground_truth(train_imgs, train_masks_split, train_auth, num_authentic=1, num_forged=5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-31T08:15:56.414663Z","iopub.execute_input":"2025-10-31T08:15:56.414877Z","iopub.status.idle":"2025-10-31T08:15:59.504836Z","shell.execute_reply.started":"2025-10-31T08:15:56.414861Z","shell.execute_reply":"2025-10-31T08:15:59.503937Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## SimpleCNN - Encoder-Decoder Structure\n\n**Encoder (Compress):**\n- 256×256×3 → 128×128×32 → 64×64×64 → 64×64×128\n- Extracts features: edges, textures, forgery patterns\n\n**Decoder (Expand):**\n- 64×64×128 → 128×128×64 → 256×256×32 → 256×256×1\n- Reconstructs spatial info, outputs pixel-wise predictions\n\n**Output:** Sigmoid gives 0-1 probability per pixel","metadata":{}},{"cell_type":"code","source":"# ==================== SIMPLE CNN MODEL ====================\nclass SimpleCNN(nn.Module):\n    def __init__(self):\n        super(SimpleCNN, self).__init__()\n        \n        # Encoder (downsampling)\n        self.enc1 = nn.Sequential(\n            nn.Conv2d(3, 32, 3, padding=1),\n            nn.ReLU(inplace=True),\n            nn.MaxPool2d(2)\n        )\n        self.enc2 = nn.Sequential(\n            nn.Conv2d(32, 64, 3, padding=1),\n            nn.ReLU(inplace=True),\n            nn.MaxPool2d(2)\n        )\n        self.enc3 = nn.Sequential(\n            nn.Conv2d(64, 128, 3, padding=1),\n            nn.ReLU(inplace=True)\n        )\n        \n        # Decoder (upsampling)\n        self.dec1 = nn.Sequential(\n            nn.ConvTranspose2d(128, 64, 3, stride=2, padding=1, output_padding=1),\n            nn.ReLU(inplace=True)\n        )\n        self.dec2 = nn.Sequential(\n            nn.ConvTranspose2d(64, 32, 3, stride=2, padding=1, output_padding=1),\n            nn.ReLU(inplace=True)\n        )\n        \n        # Output\n        self.out = nn.Conv2d(32, 1, 1)\n        self.sigmoid = nn.Sigmoid()\n    \n    def forward(self, x):\n        # Encoder\n        x1 = self.enc1(x)\n        x2 = self.enc2(x1)\n        x3 = self.enc3(x2)\n        \n        # Decoder\n        x = self.dec1(x3)\n        x = self.dec2(x)\n        \n        # Output\n        x = self.out(x)\n        x = self.sigmoid(x)\n        return x\n\nmodel = SimpleCNN().to(DEVICE)\nprint(model)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-31T08:15:59.505762Z","iopub.execute_input":"2025-10-31T08:15:59.506126Z","iopub.status.idle":"2025-10-31T08:15:59.731629Z","shell.execute_reply.started":"2025-10-31T08:15:59.506103Z","shell.execute_reply":"2025-10-31T08:15:59.730752Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Training - The Learning Process\n\n**Each iteration:**\n1. **Forward**: Images → Model → Predictions\n2. **Loss**: How wrong? (Binary Cross Entropy compares pixel-by-pixel)\n3. **Backward**: Calculate gradients (how to improve)\n4. **Update**: Adjust model weights\n\n**Validation**: Test on unseen data to check performance","metadata":{}},{"cell_type":"code","source":"# ==================== TRAINING SETUP ====================\ncriterion = nn.BCELoss()\noptimizer = optim.Adam(model.parameters(), lr=LEARNING_RATE)\n\n# ==================== TRAIN ====================\ndef train_epoch(model, loader, criterion, optimizer):\n    model.train()\n    total_loss = 0\n    \n    for images, masks in loader:\n        images, masks = images.to(DEVICE), masks.to(DEVICE)\n        \n        # Forward\n        outputs = model(images)\n        loss = criterion(outputs, masks)\n        \n        # Backward\n        optimizer.zero_grad()\n        loss.backward()\n        optimizer.step()\n        \n        total_loss += loss.item()\n    \n    return total_loss / len(loader)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-31T08:15:59.732618Z","iopub.execute_input":"2025-10-31T08:15:59.732851Z","iopub.status.idle":"2025-10-31T08:16:02.514935Z","shell.execute_reply.started":"2025-10-31T08:15:59.732832Z","shell.execute_reply":"2025-10-31T08:16:02.514342Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def validate(model, loader, criterion):\n    model.eval()\n    total_loss = 0\n    \n    with torch.no_grad():\n        for images, masks in loader:\n            images, masks = images.to(DEVICE), masks.to(DEVICE)\n            outputs = model(images)\n            loss = criterion(outputs, masks)\n            total_loss += loss.item()\n    \n    return total_loss / len(loader)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-31T08:16:02.515612Z","iopub.execute_input":"2025-10-31T08:16:02.515914Z","iopub.status.idle":"2025-10-31T08:16:02.521045Z","shell.execute_reply.started":"2025-10-31T08:16:02.515898Z","shell.execute_reply":"2025-10-31T08:16:02.520272Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(\"Starting training...\")\nfor epoch in range(EPOCHS):\n    train_loss = train_epoch(model, train_loader, criterion, optimizer)\n    val_loss = validate(model, val_loader, criterion)\n    print(f\"Epoch {epoch+1}/{EPOCHS} - Train Loss: {train_loss:.4f}, Val Loss: {val_loss:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-31T08:16:02.521951Z","iopub.execute_input":"2025-10-31T08:16:02.522320Z","iopub.status.idle":"2025-10-31T08:21:04.283248Z","shell.execute_reply.started":"2025-10-31T08:16:02.522299Z","shell.execute_reply":"2025-10-31T08:21:04.282500Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Calculate Competition F1 Score on Validation Data\n\nThis function generates predictions and calculates the exact competition score (oF1) on your validation ","metadata":{}},{"cell_type":"code","source":"import scipy.optimize\nimport os\nimport json\nimport numba\n\nclass ParticipantVisibleError(Exception):\n    pass\n\n\n@numba.jit(nopython=True)\ndef _rle_encode_jit(x: np.ndarray, fg_val: int = 1) -> list:\n    \"\"\"Numba-jitted RLE encoder.\"\"\"\n    dots = np.where(x.T.flatten() == fg_val)[0]\n    run_lengths = []\n    prev = -2\n    for b in dots:\n        if b > prev + 1:\n            run_lengths.extend((b + 1, 0))\n        run_lengths[-1] += 1\n        prev = b\n    return run_lengths\n\n\ndef rle_encode(masks: list, fg_val: int = 1) -> str:\n    \"\"\"\n    Adapted from contrails RLE https://www.kaggle.com/code/inversion/contrails-rle-submission\n    Args:\n        masks: list of numpy array of shape (height, width), 1 - mask, 0 - background\n    Returns: run length encodings as a string, with each RLE JSON-encoded and separated by a semicolon.\n    \"\"\"\n    return ';'.join([json.dumps(_rle_encode_jit(x, fg_val)) for x in masks])\n\n\n@numba.njit\ndef _rle_decode_jit(mask_rle: np.ndarray, height: int, width: int) -> np.ndarray:\n    \"\"\"\n    s: numpy array of run-length encoding pairs (start, length)\n    shape: (height, width) of array to return\n    Returns numpy array, 1 - mask, 0 - background\n    \"\"\"\n    if len(mask_rle) % 2 != 0:\n        # Numba requires raising a standard exception.\n        raise ValueError('One or more rows has an odd number of values.')\n\n    starts, lengths = mask_rle[0::2], mask_rle[1::2]\n    starts -= 1\n    ends = starts + lengths\n    for i in range(len(starts) - 1):\n        if ends[i] > starts[i + 1]:\n            raise ValueError('Pixels must not be overlapping.')\n    img = np.zeros(height * width, dtype=np.bool_)\n    for lo, hi in zip(starts, ends):\n        img[lo:hi] = 1\n    return img\n\n\ndef rle_decode(mask_rle: str, shape: tuple) -> np.ndarray:\n    \"\"\"\n    mask_rle: run-length as string formatted (start length)\n              empty predictions need to be encoded with '-'\n    shape: (height, width) of array to return\n    Returns numpy array, 1 - mask, 0 - background\n    \"\"\"\n\n    mask_rle = json.loads(mask_rle)\n    mask_rle = np.asarray(mask_rle, dtype=np.int32)\n    starts = mask_rle[0::2]\n    if sorted(starts) != list(starts):\n        raise ParticipantVisibleError('Submitted values must be in ascending order.')\n    try:\n        return _rle_decode_jit(mask_rle, shape[0], shape[1]).reshape(shape, order='F')\n    except ValueError as e:\n        raise ParticipantVisibleError(str(e)) from e\n\n\ndef calculate_f1_score(pred_mask: np.ndarray, gt_mask: np.ndarray):\n    pred_flat = pred_mask.flatten()\n    gt_flat = gt_mask.flatten()\n\n    tp = np.sum((pred_flat == 1) & (gt_flat == 1))\n    fp = np.sum((pred_flat == 1) & (gt_flat == 0))\n    fn = np.sum((pred_flat == 0) & (gt_flat == 1))\n\n    precision = tp / (tp + fp) if (tp + fp) > 0 else 0\n    recall = tp / (tp + fn) if (tp + fn) > 0 else 0\n\n    if (precision + recall) > 0:\n        return 2 * (precision * recall) / (precision + recall)\n    else:\n        return 0\n\n\ndef calculate_f1_matrix(pred_masks: list, gt_masks: list):\n    \"\"\"\n    Parameters:\n    pred_masks (np.ndarray):\n            First dimension is the number of predicted instances.\n            Each instance is a binary mask of shape (height, width).\n    gt_masks (np.ndarray):\n            First dimension is the number of ground truth instances.\n            Each instance is a binary mask of shape (height, width).\n    \"\"\"\n\n    num_instances_pred = len(pred_masks)\n    num_instances_gt = len(gt_masks)\n    f1_matrix = np.zeros((num_instances_pred, num_instances_gt))\n\n    # Calculate F1 scores for each pair of predicted and ground truth masks\n    for i in range(num_instances_pred):\n        for j in range(num_instances_gt):\n            pred_flat = pred_masks[i].flatten()\n            gt_flat = gt_masks[j].flatten()\n            f1_matrix[i, j] = calculate_f1_score(pred_mask=pred_flat, gt_mask=gt_flat)\n\n    if f1_matrix.shape[0] < len(gt_masks):\n        # Add a row of zeros to the matrix if the number of predicted instances is less than ground truth instances\n        f1_matrix = np.vstack((f1_matrix, np.zeros((len(gt_masks) - len(f1_matrix), num_instances_gt))))\n\n    return f1_matrix\n\n\ndef oF1_score(pred_masks: list, gt_masks: list):\n    \"\"\"\n    Calculate the optimal F1 score for a set of predicted masks against\n    ground truth masks which considers the optimal F1 score matching.\n    This function uses the Hungarian algorithm to find the optimal assignment\n    of predicted masks to ground truth masks based on the F1 score matrix.\n    If the number of predicted masks is less than the number of ground truth masks,\n    it will add a row of zeros to the F1 score matrix to ensure that the dimensions match.\n\n    Parameters:\n    pred_masks (list of np.ndarray): List of predicted binary masks.\n    gt_masks (np.ndarray): Array of ground truth binary masks.\n    Returns:\n    float: Optimal F1 score.\n    \"\"\"\n    f1_matrix = calculate_f1_matrix(pred_masks, gt_masks)\n\n    # Find the best matching between predicted and ground truth masks\n    row_ind, col_ind = scipy.optimize.linear_sum_assignment(-f1_matrix)\n    # The linear_sum_assignment discards excess predictions so we need a separate penalty.\n    excess_predictions_penalty = len(gt_masks) / max(len(pred_masks), len(gt_masks))\n    return np.mean(f1_matrix[row_ind, col_ind]) * excess_predictions_penalty\n\n\ndef evaluate_single_image(label_rles: str, prediction_rles: str, shape_str: str) -> float:\n    shape = json.loads(shape_str)\n    label_rles = [rle_decode(x, shape=shape) for x in label_rles.split(';')]\n    prediction_rles = [rle_decode(x, shape=shape) for x in prediction_rles.split(';')]\n    return oF1_score(prediction_rles, label_rles)\n\n\ndef score(solution: pd.DataFrame, submission: pd.DataFrame, row_id_column_name: str) -> float:\n    \"\"\"\n    Args:\n        solution (pd.DataFrame): The ground truth DataFrame.\n        submission (pd.DataFrame): The submission DataFrame.\n        row_id_column_name (str): The name of the column containing row IDs.\n    Returns:\n        float\n    \"\"\"\n    df = solution.copy()\n    df = df.rename(columns={'annotation': 'label'})\n\n    df['prediction'] = submission['annotation']\n    # Check for correct 'authentic' label\n    authentic_indices = (df['label'] == 'authentic') | (df['prediction'] == 'authentic')\n    df['image_score'] = ((df['label'] == df['prediction']) & authentic_indices).astype(float)\n\n    df.loc[~authentic_indices, 'image_score'] = df.loc[~authentic_indices].apply(\n        lambda row: evaluate_single_image(row['label'], row['prediction'], row['shape']), axis=1\n    )\n    return float(np.mean(df['image_score']))\n\nprint(\"Competition scoring functions loaded successfully!\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-31T08:23:54.539929Z","iopub.execute_input":"2025-10-31T08:23:54.540549Z","iopub.status.idle":"2025-10-31T08:23:55.669146Z","shell.execute_reply.started":"2025-10-31T08:23:54.540520Z","shell.execute_reply":"2025-10-31T08:23:55.668223Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def calculate_competition_score_with_split(model, image_paths, mask_paths, is_authentic, device=DEVICE, img_size=IMG_SIZE):\n    \"\"\"\n    Calculate competition F1 score on a dataset (train or val) with split by class.\n    \n    Args:\n        model: Trained PyTorch model\n        image_paths: List of image file paths\n        mask_paths: List of mask file paths (None for authentic images)\n        is_authentic: List of booleans indicating if image is authentic\n        device: Device to run inference on\n        img_size: Size to resize images for inference\n    \n    Returns:\n        dict: Dictionary containing overall score and per-class scores\n    \"\"\"\n    model.eval()\n    \n    # Prepare ground truth and predictions\n    solution_data = []\n    submission_data = []\n    \n    # Track authentic and forged separately\n    authentic_correct = 0\n    authentic_total = 0\n    forged_solution_data = []\n    forged_submission_data = []\n    \n    print(f\"Generating predictions for {len(image_paths)} images...\")\n    \n    with torch.no_grad():\n        for idx, img_path in enumerate(image_paths):\n            # Load image\n            img = cv2.imread(img_path)\n            img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n            original_shape = img_rgb.shape[:2]\n            \n            # Get image ID\n            image_id = os.path.basename(img_path).replace('.png', '')\n            \n            # Prepare ground truth\n            if is_authentic[idx]:\n                # Authentic image\n                gt_annotation = \"authentic\"\n                gt_shape = \"authentic\"\n            else:\n                # Forged image - load mask and encode to RLE\n                gt_mask = np.load(mask_paths[idx])\n                if gt_mask.ndim == 3:\n                    gt_mask = gt_mask[:, :, 0]\n                gt_mask = (gt_mask > 0).astype(np.uint8)\n                \n                # RLE encode ground truth\n                gt_annotation = rle_encode([gt_mask], fg_val=1)\n                gt_shape = json.dumps(list(original_shape))\n            \n            solution_data.append({\n                'case_id': image_id,\n                'annotation': gt_annotation,\n                'shape': gt_shape\n            })\n            \n            # Generate prediction\n            img_resized = cv2.resize(img_rgb, (img_size, img_size)) / 255.0\n            img_tensor = torch.from_numpy(img_resized).float().permute(2, 0, 1).unsqueeze(0)\n            img_tensor = img_tensor.to(device)\n            \n            # Predict\n            pred = model(img_tensor).cpu().numpy()[0, 0]\n            \n            # Resize back to original\n            pred = cv2.resize(pred, (original_shape[1], original_shape[0]))\n            pred_binary = (pred > 0.5).astype(np.uint8)\n            \n            # Check if authentic or forged\n            if pred_binary.sum() == 0:\n                pred_annotation = \"authentic\"\n            else:\n                # RLE encode prediction\n                pred_annotation = rle_encode([pred_binary], fg_val=1)\n            \n            submission_data.append({\n                'case_id': image_id,\n                'annotation': pred_annotation\n            })\n            \n            # Track by class\n            if is_authentic[idx]:\n                authentic_total += 1\n                if pred_annotation == \"authentic\":\n                    authentic_correct += 1\n            else:\n                # Store forged images separately for oF1 calculation\n                forged_solution_data.append({\n                    'case_id': image_id,\n                    'annotation': gt_annotation,\n                    'shape': gt_shape\n                })\n                forged_submission_data.append({\n                    'case_id': image_id,\n                    'annotation': pred_annotation\n                })\n    \n    # Create dataframes\n    solution_df = pd.DataFrame(solution_data)\n    submission_df = pd.DataFrame(submission_data)\n    \n    # Calculate overall score\n    overall_score = score(solution_df, submission_df, row_id_column_name='case_id')\n    \n    # Calculate authentic F1 (accuracy for authentic class)\n    authentic_f1 = authentic_correct / authentic_total if authentic_total > 0 else 0.0\n    \n    # Calculate forged oF1 score\n    if len(forged_solution_data) > 0:\n        forged_solution_df = pd.DataFrame(forged_solution_data)\n        forged_submission_df = pd.DataFrame(forged_submission_data)\n        forged_f1 = score(forged_solution_df, forged_submission_df, row_id_column_name='case_id')\n    else:\n        forged_f1 = 0.0\n    \n    return {\n        'overall': overall_score,\n        'authentic_f1': authentic_f1,\n        'forged_f1': forged_f1,\n        'authentic_count': authentic_total,\n        'forged_count': len(forged_solution_data)\n    }\n\nprint(\"Competition scoring validation function with class split ready!\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-31T08:23:59.136434Z","iopub.execute_input":"2025-10-31T08:23:59.136769Z","iopub.status.idle":"2025-10-31T08:23:59.149200Z","shell.execute_reply.started":"2025-10-31T08:23:59.136749Z","shell.execute_reply":"2025-10-31T08:23:59.148433Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Calculate competition F1 score on validation data\nprint(\"\\n\" + \"=\"*50)\nprint(\"CALCULATING COMPETITION F1 SCORE ON VALIDATION DATA\")\nprint(\"=\"*50)\n\nval_results = calculate_competition_score_with_split(\n    model=model,\n    image_paths=val_imgs,\n    mask_paths=val_masks_split,\n    is_authentic=val_auth,\n    device=DEVICE,\n    img_size=IMG_SIZE\n)\n\nprint(f\"\\n{'='*50}\")\nprint(f\"VALIDATION RESULTS\")\nprint(f\"{'='*50}\")\nprint(f\"Overall F1 Score:     {val_results['overall']:.4f}\")\nprint(f\"\\nPer-Class Performance:\")\nprint(f\"  Authentic F1:       {val_results['authentic_f1']:.4f}  ({val_results['authentic_count']} images)\")\nprint(f\"  Forged (oF1):       {val_results['forged_f1']:.4f}  ({val_results['forged_count']} images)\")\nprint(f\"{'='*50}\\n\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-31T08:24:07.148870Z","iopub.execute_input":"2025-10-31T08:24:07.149436Z","iopub.status.idle":"2025-10-31T08:24:54.509958Z","shell.execute_reply.started":"2025-10-31T08:24:07.149410Z","shell.execute_reply":"2025-10-31T08:24:54.508990Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Making Predictions & Submission\n\n**Steps:**\n1. Load test image → Preprocess (same as training)\n2. Model predicts → Get probability mask\n3. Threshold at 0.5 → Binary mask (0 or 1)\n4. Resize to original dimensions\n5. **Check mask:**\n   - If `sum == 0` (no forged pixels) → Label as `\"authentic\"`\n   - If `sum > 0` (has forged pixels) → RLE encode mask\n\n**Output:**\n- Authentic images: `\"authentic\"` string\n- Forged images: RLE compressed mask\n","metadata":{}},{"cell_type":"code","source":"# ==================== PREDICTION (AUTHENTIC vs RLE) ====================\n# Using the official competition RLE encoder defined earlier\n\nmodel.eval()\ntest_image_paths = sorted(glob(DATA_PATH+'test_images/*.png'))\ntest_predictions = {}\n\nprint(f\"Predicting {len(test_image_paths)} test images...\")\nwith torch.no_grad():\n    for img_path in test_image_paths:\n        # Load and preprocess\n        img = cv2.imread(img_path)\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        original_shape = img.shape[:2]\n        \n        img_resized = cv2.resize(img, (IMG_SIZE, IMG_SIZE)) / 255.0\n        img_tensor = torch.from_numpy(img_resized).float().permute(2, 0, 1).unsqueeze(0)\n        img_tensor = img_tensor.to(DEVICE)\n        \n        # Predict\n        pred = model(img_tensor).cpu().numpy()[0, 0]\n        \n        # Resize back to original\n        pred = cv2.resize(pred, (original_shape[1], original_shape[0]))\n        pred_binary = (pred > 0.5).astype(np.uint8)\n        \n        # Get image_id\n        image_id = os.path.splitext(os.path.basename(img_path))[0]\n        \n        # Check if authentic or forged\n        if pred_binary.sum() == 0:\n            test_predictions[image_id] = \"authentic\"\n        else:\n            # Use official competition RLE encoder\n            rle = rle_encode([pred_binary], fg_val=1)\n            test_predictions[image_id] = rle\n\nprint(f\"Processed {len(test_predictions)} test images\")\n\n# ==================== CREATE SUBMISSION ====================\nsubmission_data = [{'case_id': k, 'annotation': v} for k, v in test_predictions.items()]\nsubmission_df = pd.DataFrame(submission_data)\nsubmission_df.to_csv('submission.csv', index=False)\n\nprint(\"Submission created!\")\nprint(f\"\\nSample predictions:\")\nprint(submission_df.head(10))\nprint(f\"\\nAuthentic images: {(submission_df['annotation'] == 'authentic').sum()}\")\nprint(f\"Forged images: {(submission_df['annotation'] != 'authentic').sum()}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-31T08:26:58.848054Z","iopub.execute_input":"2025-10-31T08:26:58.848858Z","iopub.status.idle":"2025-10-31T08:26:58.903951Z","shell.execute_reply.started":"2025-10-31T08:26:58.848818Z","shell.execute_reply":"2025-10-31T08:26:58.903000Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"submission_df[[\"case_id\",\"annotation\"]].head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-31T08:25:22.720317Z","iopub.execute_input":"2025-10-31T08:25:22.720648Z","iopub.status.idle":"2025-10-31T08:25:22.737283Z","shell.execute_reply.started":"2025-10-31T08:25:22.720628Z","shell.execute_reply":"2025-10-31T08:25:22.736508Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Summary\n\n**The score does not improve beyond 0.303 significantly. This may be due to the model’s limited ability to accurately predict forged pixels. Since the number of non-forged pixels is significantly higher than that of forged pixels, standard loss functions tend to bias toward the majority class, effectively ignoring the forged regions to minimize the overall loss. Hence, the model predicts the portion of the authetic samples from test-set accurately and hence a score of 0.303.**\n\n1. **Problem Identified:** Baseline model has 0.0000 F1 in CV on forged images due to:\n   - Severe pixel-level class imbalance (majority of the pixels are non-forged)\n   - BCE loss may not be suitable for imbalanced data\n   - Too simple CNN architecture \n\n2. **Solutions to be explored:**\n   - Better loss fucntion (handles imbalance)\n   - Better, Deeper architecture (Ex: U-Net)","metadata":{}}]}