{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.13"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":71885,"databundleVersionId":8143495,"sourceType":"competition"},{"sourceId":17198,"sourceType":"modelInstanceVersion","modelInstanceId":14323,"modelId":21716}],"dockerImageVersionId":30664,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false},"papermill":{"default_parameters":{},"duration":35.174539,"end_time":"2025-09-28T07:26:07.710845","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2025-09-28T07:25:32.536306","version":"2.5.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"","metadata":{"papermill":{"duration":0.003175,"end_time":"2025-09-28T07:25:35.701276","exception":false,"start_time":"2025-09-28T07:25:35.698101","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# **Image Matching Pipeline using SIFT + LightGlue**\n\n","metadata":{"papermill":{"duration":0.00249,"end_time":"2025-09-28T07:25:35.706588","exception":false,"start_time":"2025-09-28T07:25:35.704098","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"Here’s a detailed explanation of your **image matching pipeline** using SIFT + LightGlue in PyTorch/Kornia:\n\n---\n\n### **1️⃣ Imports and Setup**\n\n```python\nimport os, io, requests, cv2, torch, kornia as K, kornia.feature as KF\nfrom kornia_moons.feature import laf_from_opencv_SIFT_kpts\nfrom kornia_moons.viz import draw_LAF_matches\nimport matplotlib.pyplot as plt\n```\n\n* **OpenCV (cv2)**: for SIFT keypoint detection.\n* **Kornia / Kornia-Moons**: differentiable computer vision in PyTorch. Includes LightGlue matcher, LAF utilities, and visualization helpers.\n* **Torch**: core tensor operations, device handling.\n* **Matplotlib**: visualize matches.\n\n---\n\n### **2️⃣ Device Selection**\n\n```python\ndevice = K.utils.get_cuda_or_mps_device_if_available()\nprint(device)\n```\n\n* Chooses GPU if available (CUDA or Apple MPS).\n* Speeds up descriptor matching and LightGlue computations.\n\n---\n\n### **3️⃣ LightGlue Matcher and SIFT Initialization**\n\n```python\nlg_matcher = KF.LightGlueMatcher(\"sift\").eval().to(device)\nnum_features = 4096\nsift = cv2.SIFT_create(num_features)\n```\n\n* **LightGlueMatcher(\"sift\")**: pretrained model for SIFT descriptors.\n* **SIFT_create(4096)**: detects up to 4096 features per image.\n\n---\n\n### **4️⃣ Image Loading**\n\n```python\nimg1 = cv2.cvtColor(cv2.imread(path_a), cv2.COLOR_BGR2RGB)\nimg2 = cv2.cvtColor(cv2.imread(path_b), cv2.COLOR_BGR2RGB)\nhw1 = torch.tensor(img1.shape[:2], device=device)\nhw2 = torch.tensor(img2.shape[:2], device=device)\n```\n\n* Reads and converts OpenCV BGR → RGB.\n* Stores height/width for LightGlue scaling.\n\n---\n\n### **5️⃣ Descriptor Normalization: RootSIFT**\n\n```python\ndef sift_to_rootsift(x: torch.Tensor, eps=1e-6) -> torch.Tensor:\n    x = torch.nn.functional.normalize(x, p=1, dim=-1, eps=eps)\n    x.clip_(min=eps).sqrt_()\n    return torch.nn.functional.normalize(x, p=2, dim=-1, eps=eps)\n```\n\n* **RootSIFT** improves SIFT matching by:\n\n  1. L1-normalizing the descriptor.\n  2. Taking square root (Hellinger kernel).\n  3. L2-normalizing again.\n\n---\n\n### **6️⃣ Detect Keypoints and Compute Descriptors**\n\n```python\nwith torch.inference_mode():\n    kpts1, descs1 = sift.detectAndCompute(img1, None)\n    kpts2, descs2 = sift.detectAndCompute(img2, None)\n```\n\n* Uses OpenCV SIFT to extract **keypoints** and **descriptors**.\n\n```python\nlafs1 = laf_from_opencv_SIFT_kpts(kpts1, device)\nlafs2 = laf_from_opencv_SIFT_kpts(kpts2, device)\ndescs1 = sift_to_rootsift(torch.from_numpy(descs1)).to(device)\ndescs2 = sift_to_rootsift(torch.from_numpy(descs2)).to(device)\n```\n\n* Converts OpenCV keypoints → **Local Affine Frames (LAFs)** compatible with Kornia.\n* Converts descriptors to PyTorch and applies **RootSIFT**.\n\n---\n\n### **7️⃣ LightGlue Matching**\n\n```python\ndists, idxs = lg_matcher(descs1, descs2, lafs1, lafs2, hw1=hw1, hw2=hw2)\n```\n\n* LightGlue produces **tentative matches** between two sets of descriptors using attention-based matching.\n* `idxs` contains matched indices: `(idx_img1, idx_img2)`.\n\n```python\nprint(f\"{idxs.shape[0]} tentative matches with SIFT-LightGlue\")\n```\n\n---\n\n### **8️⃣ Extract Matched Keypoints**\n\n```python\ndef get_matching_keypoints(kp1, kp2, idxs):\n    mkpts1 = kp1[idxs[:, 0]]\n    mkpts2 = kp2[idxs[:, 1]]\n    return mkpts1, mkpts2\n\nmkpts1, mkpts2 = get_matching_keypoints(KF.get_laf_center(lafs1)[0], KF.get_laf_center(lafs2)[0], idxs.detach().cpu())\n```\n\n* Converts LAFs → **keypoint centers**.\n* Retrieves matched coordinates for both images.\n\n---\n\n### **9️⃣ Estimate Fundamental Matrix + Inliers**\n\n```python\nFm, inliers = cv2.findFundamentalMat(\n    mkpts1.detach().cpu().numpy(), mkpts2.detach().cpu().numpy(),\n    cv2.USAC_MAGSAC, 1.5, 0.999, 100000\n)\ninliers = inliers > 0\n```\n\n* **RANSAC-based fundamental matrix estimation**.\n* Filters out outliers to keep only **geometrically consistent matches**.\n\n---\n\n### **🔟 Visualize Matches**\n\n```python\ndraw_LAF_matches(\n    lafs1, lafs2,\n    idxs.cpu().numpy(),\n    img1, img2,\n    inliers,\n    draw_dict={\n        \"inlier_color\": (0.2, 1, 0.2),\n        \"tentative_color\": (1, 1, 0.2, 0.3),\n        \"feature_color\": None,\n        \"vertical\": False\n    },\n)\nplt.title(f\"{inliers.sum()} inliers with SIFT-LightGlue\")\n```\n\n* Draws tentative matches (yellow) and **inliers** (green) between images.\n* LAFs allow visualization of **keypoint affine frames**, not just points.\n\n---\n\n### ✅ **Pipeline Summary**\n\n1. Load images and initialize SIFT.\n2. Detect keypoints & compute SIFT descriptors.\n3. Convert keypoints → LAFs (affine frames).\n4. Convert descriptors → RootSIFT + PyTorch tensors.\n5. Run **LightGlue** to get tentative matches.\n6. Extract keypoint coordinates from LAFs.\n7. Compute **fundamental matrix** to filter inliers.\n8. Visualize matches with LAFs and inlier overlay.\n\nThis approach combines **classic SIFT features** with **modern LightGlue attention-based matching**, giving robust matches even under scale, rotation, and lighting changes.\n\n---\n\nIf you want, I can also **draw a step-by-step diagram** showing this pipeline visually from images → SIFT → LightGlue → inliers. It helps a lot for presentations. Do you want me to do that?\n","metadata":{"papermill":{"duration":0.002288,"end_time":"2025-09-28T07:25:35.711371","exception":false,"start_time":"2025-09-28T07:25:35.709083","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Using LightGlue via kornia requires kornia>=0.7.2\n!pip install kornia==0.7.2\n!pip install kornia_moons --no-deps","metadata":{"execution":{"iopub.execute_input":"2025-09-28T07:25:35.717811Z","iopub.status.busy":"2025-09-28T07:25:35.717423Z","iopub.status.idle":"2025-09-28T07:25:51.916472Z","shell.execute_reply":"2025-09-28T07:25:51.915461Z"},"papermill":{"duration":16.20516,"end_time":"2025-09-28T07:25:51.918975","exception":false,"start_time":"2025-09-28T07:25:35.713815","status":"completed"},"tags":[],"_kg_hide-output":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"!mkdir -p .cache/torch/hub/checkpoints\n!cp /kaggle/input/lightglue/pytorch/sift/1/sift_lightglue.pth .cache/torch/hub/checkpoints/","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.execute_input":"2025-09-28T07:25:51.928068Z","iopub.status.busy":"2025-09-28T07:25:51.927702Z","iopub.status.idle":"2025-09-28T07:25:54.896313Z","shell.execute_reply":"2025-09-28T07:25:54.894954Z"},"papermill":{"duration":2.975998,"end_time":"2025-09-28T07:25:54.898824","exception":false,"start_time":"2025-09-28T07:25:51.922826","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import os\nimport io\nimport requests\nimport cv2\nimport kornia as K\nimport kornia.feature as KF\nimport matplotlib.pyplot as plt\nimport torch\nfrom kornia_moons.feature import laf_from_opencv_SIFT_kpts\nfrom kornia_moons.viz import draw_LAF_matches\nimport numpy as np","metadata":{"execution":{"iopub.execute_input":"2025-09-28T07:25:54.907837Z","iopub.status.busy":"2025-09-28T07:25:54.907446Z","iopub.status.idle":"2025-09-28T07:26:01.878851Z","shell.execute_reply":"2025-09-28T07:26:01.877865Z"},"papermill":{"duration":6.978696,"end_time":"2025-09-28T07:26:01.881216","exception":false,"start_time":"2025-09-28T07:25:54.902520","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"cell_type":"code","source":"path_a = \"/kaggle/input/image-matching-challenge-2024/train/church/images/00017.png\"\npath_b = \"/kaggle/input/image-matching-challenge-2024/train/church/images/00020.png\"","metadata":{"execution":{"iopub.execute_input":"2025-09-28T07:26:01.890355Z","iopub.status.busy":"2025-09-28T07:26:01.889897Z","iopub.status.idle":"2025-09-28T07:26:01.894435Z","shell.execute_reply":"2025-09-28T07:26:01.893613Z"},"papermill":{"duration":0.011445,"end_time":"2025-09-28T07:26:01.896535","exception":false,"start_time":"2025-09-28T07:26:01.885090","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def sift_to_rootsift(x: torch.Tensor, eps=1e-6) -> torch.Tensor:\n    x = torch.nn.functional.normalize(x, p=1, dim=-1, eps=eps)\n    x.clip_(min=eps).sqrt_()\n    return torch.nn.functional.normalize(x, p=2, dim=-1, eps=eps)\n\ndevice = K.utils.get_cuda_or_mps_device_if_available()\nprint(device)\n\n\nlg_matcher = KF.LightGlueMatcher(\"sift\").eval().to(device)\n\nnum_features = 4096\nsift = cv2.SIFT_create(num_features)\n\nimg1 = cv2.cvtColor(cv2.imread(path_a), cv2.COLOR_BGR2RGB)\nimg2 = cv2.cvtColor(cv2.imread(path_b), cv2.COLOR_BGR2RGB)\n\nhw1 = torch.tensor(img1.shape[:2], device=device)\nhw2 = torch.tensor(img2.shape[:2], device=device)\n\n\nwith torch.inference_mode():\n    kpts1, descs1 = sift.detectAndCompute(img1, None)\n    kpts2, descs2 = sift.detectAndCompute(img2, None)\n    lafs1 = laf_from_opencv_SIFT_kpts(kpts1, device)\n    lafs2 = laf_from_opencv_SIFT_kpts(kpts2, device)\n    descs1 = sift_to_rootsift(torch.from_numpy(descs1)).to(device)\n    descs2 = sift_to_rootsift(torch.from_numpy(descs2)).to(device)\n    dists, idxs = lg_matcher(descs1, descs2, lafs1, lafs2, hw1=hw1, hw2=hw2)\n\nprint(f\"{idxs.shape[0]} tentative matches with SIFT-LightGlue\")\n","metadata":{"execution":{"iopub.execute_input":"2025-09-28T07:26:01.905322Z","iopub.status.busy":"2025-09-28T07:26:01.904589Z","iopub.status.idle":"2025-09-28T07:26:04.280019Z","shell.execute_reply":"2025-09-28T07:26:04.278825Z"},"papermill":{"duration":2.381773,"end_time":"2025-09-28T07:26:04.281958","exception":false,"start_time":"2025-09-28T07:26:01.900185","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from PIL import Image\nimg_a = Image.open(path_a)\nimg_b = Image.open(path_b)\ndisplay(img_a)\ndisplay(img_b)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def get_matching_keypoints(kp1, kp2, idxs):\n    mkpts1 = kp1[idxs[:, 0]]\n    mkpts2 = kp2[idxs[:, 1]]\n    return mkpts1, mkpts2\n\n\nmkpts1, mkpts2 = get_matching_keypoints(KF.get_laf_center(lafs1)[0], KF.get_laf_center(lafs2)[0], idxs.detach().cpu())\n\nFm, inliers = cv2.findFundamentalMat(\n    mkpts1.detach().cpu().numpy(), mkpts2.detach().cpu().numpy(), cv2.USAC_MAGSAC, 1.5, 0.999, 100000\n)\ninliers = inliers > 0\n\ndraw_LAF_matches(\n    lafs1, \n    lafs2,\n    idxs.cpu().numpy(),\n    img1,\n    img2,\n    inliers,\n    draw_dict={\"inlier_color\": (0.2, 1, 0.2), \n               \"tentative_color\": (1, 1, 0.2, 0.3), \n               \"feature_color\": None, \n               \"vertical\": True},\n)\nplt.title(f\"{inliers.sum()} inliers with SIFT-LightGlue\")\n","metadata":{"execution":{"iopub.execute_input":"2025-09-28T07:26:04.292113Z","iopub.status.busy":"2025-09-28T07:26:04.291722Z","iopub.status.idle":"2025-09-28T07:26:06.437413Z","shell.execute_reply":"2025-09-28T07:26:06.436388Z"},"papermill":{"duration":2.159319,"end_time":"2025-09-28T07:26:06.445441","exception":false,"start_time":"2025-09-28T07:26:04.286122","status":"completed"},"tags":[]},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"    Inliers\n    Correct correspondences (shown in green)\n    Reliable correspondences that are geometrically consistent\n\n    Outliers\n    Incorrect correspondences (remain yellow)\n    Incorrect correspondences that are geometrically inconsistent","metadata":{}}]}