{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":91498,"databundleVersionId":11655853,"sourceType":"competition"},{"sourceId":7884485,"sourceType":"datasetVersion","datasetId":4628051},{"sourceId":11217117,"sourceType":"datasetVersion","datasetId":6988459},{"sourceId":4534,"sourceType":"modelInstanceVersion","modelInstanceId":3326,"modelId":986},{"sourceId":17191,"sourceType":"modelInstanceVersion","modelInstanceId":14317,"modelId":21716},{"sourceId":17555,"sourceType":"modelInstanceVersion","modelInstanceId":14611,"modelId":22086},{"sourceId":332398,"sourceType":"modelInstanceVersion","isSourceIdPinned":true,"modelInstanceId":278629,"modelId":299532}],"dockerImageVersionId":31012,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Image Matching Challenge 2025: Beginner-Friendly 3D Reconstruction\n\n<a href=\"https://www.kaggle.com/competitions/image-matching-challenge-2025\" target=\"_blank\">\n  <img src=\"https://img.shields.io/badge/Kaggle-Competition-blue?style=for-the-badge&logo=kaggle\" alt=\"Kaggle Competition\">\n</a>\n\nThe **Image Matching Challenge 2025** is an exciting computer vision competition where we take a collection of images and turn them into a 3D model of a scene. Imagine you have a bunch of photos taken from different angles around a statue or a building—our job is to figure out how these images relate to each other, group them into scenes, and compute the exact position and orientation of each camera that took them. This process, called **Structure from Motion (SfM)**, is like solving a 3D puzzle! \n\nIn this script, we'll:\n1. Find similar images to pair them up efficiently\n2. Detect key points (like corners or distinctive features) in each image\n3. Match these points between image pairs\n4. Verify matches to ensure accuracy\n5. Build a 3D model using COLMAP\n6. Create a submission file with camera poses\n\n\n\n### Models We'll Use \n- **DINOv2**: A vision transformer that creates \"global descriptors\" to measure image similarity. Think of it as a super-smart librarian who can tell which photos look alike.\n- **DISK**: Detects robust keypoints and descriptors in images, even under tough conditions like low light or weird angles.\n- **SuperGlue**: A neural network that matches keypoints between images with incredible precision, like a master matchmaker for image features.\n- **COLMAP**: The industry-standard tool for turning matched keypoints into a 3D model.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"code","source":"# Install dependencies and copy model weights to run the notebook without internet access when submitting to the competition.\n\n!pip install --no-index /kaggle/input/imc2024-packages-lightglue-rerun-kornia/* --no-deps\n!mkdir -p /root/.cache/torch/hub/checkpoints\n!cp /kaggle/input/aliked/pytorch/aliked-n16/1/aliked-n16.pth /root/.cache/torch/hub/checkpoints/\n!cp /kaggle/input/lightglue/pytorch/aliked/1/aliked_lightglue.pth /root/.cache/torch/hub/checkpoints/\n!cp /kaggle/input/lightglue/pytorch/aliked/1/aliked_lightglue.pth /root/.cache/torch/hub/checkpoints/aliked_lightglue_v0-1_arxiv-pth","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-01T08:55:23.780908Z","iopub.execute_input":"2025-04-01T08:55:23.781261Z","iopub.status.idle":"2025-04-01T08:55:28.928298Z","shell.execute_reply.started":"2025-04-01T08:55:23.781228Z","shell.execute_reply":"2025-04-01T08:55:28.926942Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import sys\nimport os\nfrom tqdm import tqdm\nfrom time import time, sleep\nimport gc\nimport numpy as np\nimport h5py\nimport dataclasses\nimport pandas as pd\nfrom IPython.display import clear_output\nfrom collections import defaultdict\nfrom copy import deepcopy\nfrom PIL import Image\n\nimport cv2\nimport torch\nimport torch.nn.functional as F\nimport kornia as K\nimport kornia.feature as KF\n\nimport torch\nfrom lightglue import match_pair\nfrom lightglue import ALIKED, LightGlue\nfrom lightglue.utils import load_image, rbd\nfrom transformers import AutoImageProcessor, AutoModel\n\nimport pycolmap\nsys.path.append('/kaggle/input/imc25-utils')\nfrom database import *\nfrom h5_to_db import *\nimport metric","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-01T08:55:33.353942Z","iopub.execute_input":"2025-04-01T08:55:33.354302Z","iopub.status.idle":"2025-04-01T08:55:54.828282Z","shell.execute_reply.started":"2025-04-01T08:55:33.354276Z","shell.execute_reply":"2025-04-01T08:55:54.827276Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"device = K.utils.get_cuda_device_if_available(0)\nprint(f'{device=}')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-01T08:55:54.829473Z","iopub.execute_input":"2025-04-01T08:55:54.830005Z","iopub.status.idle":"2025-04-01T08:55:54.892418Z","shell.execute_reply.started":"2025-04-01T08:55:54.82998Z","shell.execute_reply":"2025-04-01T08:55:54.891625Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Helper Functions\nThese functions do the heavy lifting. Let's break them down:\n\n### `load_torch_image`\nLoads an image and converts it to a PyTorch tensor for processing.\n\n### `get_global_desc`\nUses **DINOv2** to compute a global descriptor for each image. DINOv2 is a vision transformer trained on massive datasets, producing a compact \"fingerprint\" that captures the image's overall content. We use these to find similar images.\n\n### `get_image_pairs`\nFinds pairs of similar images by comparing DINOv2 descriptors. If two images have similar descriptors, they're likely from the same scene. We use a similarity threshold (0.5) and ensure at least 20 pairs per image.\n\n### `detect_disk`\nUses **DISK** to find keypoints (distinctive points like corners) and their descriptors. DISK is robust to changes in lighting and viewpoint, making it great for outdoor scenes. We extract up to 5000 keypoints per image.\n\n### `match_superglue`\nMatches keypoints between image pairs using **SuperGlue**. SuperGlue uses a graph neural network to learn which keypoints correspond, even with perspective changes. It's more accurate than LightGlue from the original notebook.\n\n### `geometric_verification`\nChecks matches using OpenCV's RANSAC to estimate the fundamental matrix, ensuring only geometrically consistent matches are kept. This step reduces errors.\n\n### `import_into_colmap`\nImports keypoints and matches into a COLMAP database for 3D reconstruction.\n\"\"\"","metadata":{}},{"cell_type":"code","source":"def load_torch_image(fname, device):\n    img = K.io.load_image(fname, K.io.ImageLoadType.RGB32, device=device)[None, ...]\n    return img\n\ndef get_global_desc(fnames, device):\n    processor = AutoImageProcessor.from_pretrained('/kaggle/input/dinov2/pytorch/base/1')\n    model = AutoModel.from_pretrained('/kaggle/input/dinov2/pytorch/base/1').eval().to(device)\n    descs = []\n    for img_path in tqdm(fnames, desc='Computing global descriptors'):\n        img = load_torch_image(img_path, device)\n        with torch.no_grad():\n            inputs = processor(images=img, return_tensors=\"pt\", do_rescale=False).to(device)\n            outputs = model(**inputs)\n            desc = F.normalize(outputs.last_hidden_state[:,1:].max(dim=1)[0], dim=1)\n        descs.append(desc.detach().cpu())\n    return torch.cat(descs, dim=0)\n\ndef get_image_pairs(fnames, sim_th=0.5, min_pairs=20, exhaustive_if_less=20):\n    if len(fnames) <= exhaustive_if_less:\n        return [(i, j) for i in range(len(fnames)) for j in range(i+1, len(fnames))]\n    descs = get_global_desc(fnames, device)\n    dists = torch.cdist(descs, descs).cpu().numpy()\n    pairs = []\n    for i in range(len(fnames)):\n        matches = np.where(dists[i] < sim_th)[0]\n        if len(matches) < min_pairs:\n            matches = np.argsort(dists[i])[:min_pairs]\n        for j in matches:\n            if i < j:\n                pairs.append((i, j))\n    return sorted(list(set(pairs)))\n\ndef detect_disk(fnames, feature_dir, num_features=5000):\n    os.makedirs(feature_dir, exist_ok=True)\n    disk = KF.DISK(n=num_features, score_threshold=0.1).eval().to(device)\n    with h5py.File(f'{feature_dir}/keypoints.h5', 'w') as f_kp, \\\n         h5py.File(f'{feature_dir}/descriptors.h5', 'w') as f_desc:\n        for img_path in tqdm(fnames, desc='Detecting keypoints with DISK'):\n            img = load_torch_image(img_path, device)\n            with torch.no_grad():\n                features = disk(img)\n                kpts = features.keypoints.cpu().numpy()\n                descs = features.descriptors.cpu().numpy()\n            key = os.path.basename(img_path)\n            f_kp[key] = kpts\n            f_desc[key] = descs\n\ndef match_superglue(fnames, pairs, feature_dir):\n    config = {\n        'weights': 'outdoor',\n        'sinkhorn_iterations': 100,\n        'match_threshold': 0.2,\n    }\n    matcher = SuperGlue(config).eval().to(device)\n    with h5py.File(f'{feature_dir}/keypoints.h5', 'r') as f_kp, \\\n         h5py.File(f'{feature_dir}/descriptors.h5', 'r') as f_desc, \\\n         h5py.File(f'{feature_dir}/matches.h5', 'w') as f_match:\n        for i, j in tqdm(pairs, desc='Matching with SuperGlue'):\n            fname1, fname2 = fnames[i], fnames[j]\n            key1, key2 = os.path.basename(fname1), os.path.basename(fname2)\n            kp1 = torch.from_numpy(f_kp[key1][...]).to(device)\n            desc1 = torch.from_numpy(f_kp[key1][...]).to(device)\n            kp2 = torch.from_numpy(f_kp[key2][...]).to(device)\n            desc2 = torch.from_numpy(f_kp[key2][...]).to(device)\n            data = {\n                'keypoints0': kp1[None],\n                'descriptors0': desc1[None],\n                'keypoints1': kp2[None],\n                'descriptors1': desc2[None]\n            }\n            with torch.no_grad():\n                pred = matcher(data)\n                matches = pred['matches0'][0].cpu().numpy()\n                valid = matches > -1\n                matches = np.stack([np.where(valid)[0], matches[valid]], axis=1)\n            if len(matches) >= 15:\n                group = f_match.require_group(key1)\n                group.create_dataset(key2, data=matches)\n\ndef geometric_verification(fnames, pairs, feature_dir):\n    with h5py.File(f'{feature_dir}/keypoints.h5', 'r') as f_kp, \\\n         h5py.File(f'{feature_dir}/matches.h5', 'r+') as f_match:\n        for i, j in tqdm(pairs, desc='Geometric verification'):\n            fname1, fname2 = fnames[i], fnames[j]\n            key1, key2 = os.path.basename(fname1), os.path.basename(fname2)\n            if key1 in f_match and key2 in f_match[key1]:\n                kp1 = f_kp[key1][...]\n                kp2 = f_kp[key2][...]\n                matches = f_match[key1][key2][...]\n                if len(matches) >= 8:\n                    _, inliers = cv2.findFundamentalMat(\n                        kp1[matches[:, 0]],\n                        kp2[matches[:, 1]],\n                        cv2.FM_RANSAC,\n                        1.0,\n                        0.999\n                    )\n                    if inliers is not None:\n                        valid = inliers.ravel() > 0\n                        if np.sum(valid) >= 15:\n                            f_match[key1][key2][...] = matches[valid]\n                        else:\n                            del f_match[key1][key2]\n\ndef import_into_colmap(img_dir, feature_dir, database_path):\n    db = COLMAPDatabase.connect(database_path)\n    db.create_tables()\n    fname_to_id = add_keypoints(db, feature_dir, img_dir, '', 'simple-pinhole', False)\n    add_matches(db, feature_dir, fname_to_id)\n    db.commit()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-01T08:56:00.444399Z","iopub.execute_input":"2025-04-01T08:56:00.444721Z","iopub.status.idle":"2025-04-01T08:56:00.462604Z","shell.execute_reply.started":"2025-04-01T08:56:00.444692Z","shell.execute_reply":"2025-04-01T08:56:00.461455Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import dataclasses\n\n@dataclasses.dataclass\nclass Prediction:\n    image_id: str\n    dataset: str\n    filename: str\n    cluster_index: int = None\n    rotation: np.ndarray = None\n    translation: np.ndarray = None\n\nis_train = False\ndata_dir = '/kaggle/input/image-matching-challenge-2025'\nwork_dir = '/kaggle/working/result'\nos.makedirs(work_dir, exist_ok=True)\n\nsubmission_csv = os.path.join(data_dir, 'train_labels.csv' if is_train else 'sample_submission.csv')\nsamples = {}\nfor _, row in pd.read_csv(submission_csv).iterrows():\n    if row.dataset not in samples:\n        samples[row.dataset] = []\n    samples[row.dataset].append(Prediction(\n        image_id=row.image_id if not is_train else None,\n        dataset=row.dataset,\n        filename=row.image\n    ))\n\nprint('Datasets loaded:')\nfor dataset in samples:\n    print(f'  - {dataset}: {len(samples[dataset])} images')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-01T08:56:08.364082Z","iopub.execute_input":"2025-04-01T08:56:08.364428Z","iopub.status.idle":"2025-04-01T08:56:08.497281Z","shell.execute_reply.started":"2025-04-01T08:56:08.3644Z","shell.execute_reply":"2025-04-01T08:56:08.49626Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Main Processing Loop\n\nThis is where the magic happens! For each dataset:\n1. We find similar image pairs using DINOv2.\n2. Extract keypoints with DISK.\n3. Match keypoints with SuperGlue.\n4. Verify matches geometrically.\n5. Build a 3D model with COLMAP.\n6. Store the results.\n\nWe track timing for each step and handle errors gracefully to ensure the script completes even if one dataset fails.","metadata":{}},{"cell_type":"code","source":"from time import time\n\ntimings = {\n    'shortlisting': [],\n    'feature_detection': [],\n    'feature_matching': [],\n    'geometric_verification': [],\n    'reconstruction': []\n}\nresults = []\n\nprint('Starting processing...')\nfor dataset, predictions in samples.items():\n    print(f'\\nProcessing dataset: {dataset}')\n    images_dir = os.path.join(data_dir, 'train' if is_train else 'test', dataset)\n    images = [os.path.join(images_dir, p.filename) for p in predictions]\n    \n    feature_dir = os.path.join(work_dir, 'featureout', dataset)\n    database_path = os.path.join(feature_dir, 'colmap.db')\n    output_path = os.path.join(feature_dir, 'colmap_rec')\n    \n    try:\n        # Find similar image pairs\n        t = time()\n        pairs = get_image_pairs(images, sim_th=0.5, min_pairs=20)\n        timings['shortlisting'].append(time() - t)\n        print(f'  - Shortlisted {len(pairs)} pairs in {time()-t:.2f}s')\n        \n        # Detect features\n        t = time()\n        detect_disk(images, feature_dir)\n        timings['feature_detection'].append(time() - t)\n        print(f'  - Detected features in {time()-t:.2f}s')\n        \n        # Match features\n        t = time()\n        match_superglue(images, pairs, feature_dir)\n        timings['feature_matching'].append(time() - t)\n        print(f'  - Matched features in {time()-t:.2f}s')\n        \n        # Geometric verification\n        t = time()\n        geometric_verification(images, pairs, feature_dir)\n        timings['geometric_verification'].append(time() - t)\n        print(f'  - Verified matches in {time()-t:.2f}s')\n        \n        # Create COLMAP database\n        if os.path.exists(database_path):\n            os.remove(database_path)\n        import_into_colmap(images_dir, feature_dir, database_path)\n        \n        # Run reconstruction\n        t = time()\n        pycolmap.match_exhaustive(database_path)\n        mapper_options = pycolmap.IncrementalPipelineOptions()\n        mapper_options.min_model_size = 3\n        mapper_options.max_num_models = 25\n        os.makedirs(output_path, exist_ok=True)\n        maps = pycolmap.incremental_mapping(database_path, images_dir, output_path, mapper_options)\n        timings['reconstruction'].append(time() - t)\n        \n        # Store results\n        registered = 0\n        filename_to_index = {p.filename: idx for idx, p in enumerate(predictions)}\n        for map_idx, cur_map in maps.items():\n            for _, image in cur_map.images.items():\n                idx = filename_to_index[image.name]\n                predictions[idx].cluster_index = map_idx\n                predictions[idx].rotation = image.cam_from_world.rotation.matrix()\n                predictions[idx].translation = image.cam_from_world.translation\n                registered += 1\n        \n        result = f'Dataset \"{dataset}\": Registered {registered}/{len(images)} images in {len(maps)} clusters'\n        results.append(result)\n        print(result)\n    except Exception as e:\n        result = f'Dataset \"{dataset}\": Failed - {str(e)}'\n        results.append(result)\n        print(result)\n\nprint('\\nSummary:')\nfor result in results:\n    print(result)\nprint('\\nTimings:')\nfor k, v in timings.items():\n    print(f'  - {k}: {sum(v):.2f}s')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-01T08:57:26.547079Z","iopub.execute_input":"2025-04-01T08:57:26.547416Z","iopub.status.idle":"2025-04-01T08:57:43.051639Z","shell.execute_reply.started":"2025-04-01T08:57:26.547389Z","shell.execute_reply":"2025-04-01T08:57:43.050766Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Create Submission File\n\nWe create a `submission.csv` file with the required format, including image IDs, datasets, scenes (clusters), and camera poses (rotation matrix and translation vector). If a camera pose isn't computed, we use 'nan' values.","metadata":{}},{"cell_type":"code","source":"def array_to_str(array):\n    return ';'.join([f'{x:.9f}' for x in array])\n\ndef none_to_str(n):\n    return ';'.join(['nan'] * n)\n\nsubmission_file = '/kaggle/working/submission.csv'\nwith open(submission_file, 'w') as f:\n    header = 'image_id,dataset,scene,image,rotation_matrix,translation_vector\\n' if not is_train else \\\n             'dataset,scene,image,rotation_matrix,translation_vector\\n'\n    f.write(header)\n    \n    for dataset in samples:\n        for pred in samples[dataset]:\n            cluster = 'outliers' if pred.cluster_index is None else f'cluster{pred.cluster_index}'\n            rot = none_to_str(9) if pred.rotation is None else array_to_str(pred.rotation.flatten())\n            trans = none_to_str(3) if pred.translation is None else array_to_str(pred.translation)\n            \n            if is_train:\n                f.write(f'{pred.dataset},{cluster},{pred.filename},{rot},{trans}\\n')\n            else:\n                f.write(f'{pred.image_id},{pred.dataset},{cluster},{pred.filename},{rot},{trans}\\n')\n\nprint(f'📄 Submission file created: {submission_file}')\n!head {submission_file}","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-01T08:57:47.184297Z","iopub.execute_input":"2025-04-01T08:57:47.184637Z","iopub.status.idle":"2025-04-01T08:57:47.354686Z","shell.execute_reply.started":"2025-04-01T08:57:47.184609Z","shell.execute_reply":"2025-04-01T08:57:47.35361Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Evaluate (Training Only)","metadata":{}},{"cell_type":"code","source":"if is_train:\n    t = time()\n    final_score, dataset_scores = metric.score(\n        gt_csv='/kaggle/input/image-matching-challenge-2025/train_labels.csv',\n        user_csv=submission_file,\n        thresholds_csv='/kaggle/input/image-matching-challenge-2025/train_thresholds.csv',\n        mask_csv=None,\n        inl_cf=0,\n        strict_cf=-1,\n        verbose=True\n    )\n    print(f'Evaluation score: {final_score:.3f}')\n    print(f'Evaluation time: {time()-t:.2f}s')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-30T14:33:54.21148Z","iopub.execute_input":"2025-03-30T14:33:54.211852Z","iopub.status.idle":"2025-03-30T14:35:48.181353Z","shell.execute_reply.started":"2025-03-30T14:33:54.211787Z","shell.execute_reply":"2025-03-30T14:35:48.18056Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}