{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":71885,"databundleVersionId":8143495,"sourceType":"competition"}],"dockerImageVersionId":30733,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Basic of Kornia for Image Matching","metadata":{}},{"cell_type":"markdown","source":"## Image matching example with DISK local features","metadata":{}},{"cell_type":"markdown","source":"Kornia is a library for computer vision based on PyTorch. kornia provides functions such as image transformation, filtering, geometric transformation, image warping, and feature extraction. This library is useful for image processing tasks in combination with deep learning models.","metadata":{}},{"cell_type":"markdown","source":"https://github.com/kornia/kornia\n\nhttps://kornia.github.io/tutorials/nbs/image_matching_disk.html","metadata":{}},{"cell_type":"code","source":"!pip install kornia\n!pip install kornia-rs\n!pip install kornia_moons --no-deps\n!pip install opencv-python --upgrade","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-06-10T13:44:03.514287Z","iopub.execute_input":"2024-06-10T13:44:03.515316Z","iopub.status.idle":"2024-06-10T13:44:42.548095Z","shell.execute_reply.started":"2024-06-10T13:44:03.515280Z","shell.execute_reply":"2024-06-10T13:44:42.546922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import cv2\nimport kornia as K\nimport kornia.feature as KF\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport torch\nfrom kornia.feature.adalam import AdalamFilter\nfrom kornia_moons.viz import *\n\ndevice = K.utils.get_cuda_or_mps_device_if_available()\nprint(device)","metadata":{"execution":{"iopub.status.busy":"2024-06-10T13:44:42.550117Z","iopub.execute_input":"2024-06-10T13:44:42.550477Z","iopub.status.idle":"2024-06-10T13:44:42.557276Z","shell.execute_reply.started":"2024-06-10T13:44:42.550448Z","shell.execute_reply":"2024-06-10T13:44:42.556281Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The process involves detecting and matching feature points in images.\n\n1. Loading images: Loading two images from files.\n2. Adjusting settings: Customizing AdaLAM settings.\n3. Preparing the feature detector: Using a feature detector called DISK, we set the number of feature points.\n4. Getting the image size:\n5. Extracting feature points and descriptors: Extracting feature points and descriptors from images.\n6. Matching feature points: Matching feature points is performed using AdaLAM.\n7. Outputting the number of matches:","metadata":{}},{"cell_type":"code","source":"# %%capture\nfname1 = \"/kaggle/input/image-matching-challenge-2024/train/church/images/00001.png\"\nfname2 = \"/kaggle/input/image-matching-challenge-2024/train/church/images/00002.png\"\n\nadalam_config = KF.adalam.get_adalam_default_config()\n# adalam_config['orientation_difference_threshold'] = None\n# adalam_config['scale_rate_threshold'] = None\nadalam_config[\"force_seed_mnn\"] = False\nadalam_config[\"search_expansion\"] = 16\nadalam_config[\"ransac_iters\"] = 256\n\n\nimg1 = K.io.load_image(fname1, K.io.ImageLoadType.RGB32, device=device)[None, ...]\nimg2 = K.io.load_image(fname2, K.io.ImageLoadType.RGB32, device=device)[None, ...]\n\nnum_features = 512\ndisk = KF.DISK.from_pretrained(\"depth\").to(device)\n\nhw1 = torch.tensor(img1.shape[2:], device=device)\nhw2 = torch.tensor(img2.shape[2:], device=device)\n\nmatch_with_adalam = True\n\nwith torch.inference_mode():\n    inp = torch.cat([img1, img2], dim=0)\n    features1, features2 = disk(inp, num_features, pad_if_not_divisible=True)\n    kps1, descs1 = features1.keypoints, features1.descriptors\n    kps2, descs2 = features2.keypoints, features2.descriptors\n    if match_with_adalam:\n        lafs1 = KF.laf_from_center_scale_ori(kps1[None], 96 * torch.ones(1, len(kps1), 1, 1, device=device))\n        lafs2 = KF.laf_from_center_scale_ori(kps2[None], 96 * torch.ones(1, len(kps2), 1, 1, device=device))\n\n        dists, idxs = KF.match_adalam(descs1, descs2, lafs1, lafs2, hw1=hw1, hw2=hw2, config=adalam_config)\n    else:\n        dists, idxs = KF.match_smnn(descs1, descs2, 0.98)\n\n\nprint(f\"{idxs.shape[0]} tentative matches with DISK AdaLAM\")","metadata":{"execution":{"iopub.status.busy":"2024-06-10T13:44:42.558459Z","iopub.execute_input":"2024-06-10T13:44:42.558731Z","iopub.status.idle":"2024-06-10T13:44:43.061965Z","shell.execute_reply.started":"2024-06-10T13:44:42.558708Z","shell.execute_reply":"2024-06-10T13:44:43.061030Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This code extracts inliers from matched feature points and calculates the fundamental matrix.\n\n1. Get matched feature points: The function get_matching_keypoints extracts matched feature points. kp1 and kp2 are feature points in image 1 and image 2 respectively, and idxs is the index of matching. mkpts1 and mkpts2 store matched feature points.\n\n2. Calculate fundamental matrix: Use cv2.findFundamentalMat function to calculate fundamental matrix Fm and get inliers as inliers. Here, MAGSAC algorithm is used. The parameters are as follows:\n\n    * mkpts1 and mkpts2 are coordinates of matched feature points (converted to NumPy array).\n    * cv2.USAC_MAGSAC specifies MAGSAC algorithm.\n    * 1.0 is the reprojection error threshold.\n    * 0.999 is the confidence level.\n    * 100000 is the maximum number of attempts.\n    \n\n3. Output the number of inliers: Convert the inliers array to a boolean and output the number of inliers. An inlier is a correct match that matches the fundamental matrix.","metadata":{}},{"cell_type":"code","source":"def get_matching_keypoints(kp1, kp2, idxs):\n    mkpts1 = kp1[idxs[:, 0]]\n    mkpts2 = kp2[idxs[:, 1]]\n    return mkpts1, mkpts2\n\n\nmkpts1, mkpts2 = get_matching_keypoints(kps1, kps2, idxs)\n\nFm, inliers = cv2.findFundamentalMat(\n    mkpts1.detach().cpu().numpy(), mkpts2.detach().cpu().numpy(), cv2.USAC_MAGSAC, 1.0, 0.999, 100000\n)\ninliers = inliers > 0\nprint(f\"{inliers.sum()} inliers with DISK\")","metadata":{"execution":{"iopub.status.busy":"2024-06-10T13:44:43.063780Z","iopub.execute_input":"2024-06-10T13:44:43.064052Z","iopub.status.idle":"2024-06-10T13:44:43.076946Z","shell.execute_reply.started":"2024-06-10T13:44:43.064029Z","shell.execute_reply":"2024-06-10T13:44:43.075996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"When we run the draw_LAF_matches function, we get the following result:\nWe see an image with the feature point matches visualized.\nInliers are shown in green, tentative matches are shown in semi-transparent yellow.","metadata":{}},{"cell_type":"code","source":"draw_LAF_matches(\n    KF.laf_from_center_scale_ori(kps1[None].cpu()),\n    KF.laf_from_center_scale_ori(kps2[None].cpu()),\n    idxs.cpu(),\n    K.tensor_to_image(img1.cpu()),\n    K.tensor_to_image(img2.cpu()),\n    inliers,\n    draw_dict={\"inlier_color\": (0.2, 1, 0.2), \"tentative_color\": (1, 1, 0.2, 0.3), \"feature_color\": None, \"vertical\": False},\n)","metadata":{"execution":{"iopub.status.busy":"2024-06-10T13:44:43.077921Z","iopub.execute_input":"2024-06-10T13:44:43.078191Z","iopub.status.idle":"2024-06-10T13:44:45.528648Z","shell.execute_reply.started":"2024-06-10T13:44:43.078168Z","shell.execute_reply":"2024-06-10T13:44:45.527504Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}