{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# WHALES MATCHING USING LoFTR and KORNIA\n\nWhat is LoFTR?\n\nLoFTR stands for Detector-Free Local Feature Matching with Transformers. It is detailed described in [LoFTR: Detector-Free Local Feature Matching with Transformers](https://arxiv.org/pdf/2104.00680.pdf)(1). \n\nSource[1]: \nNovel method for local image feature matching. Instead of performing image feature detection, description, and matching sequentially, we propose to first establish pixel-wise dense matches at a coarse level and later refine the good matches at a fine level. In contrast\nto dense methods that use a cost volume to search correspondences, we use self and cross attention layers in Transformer to obtain feature descriptors that are conditioned on both images. The global receptive field provided by Transformer enables our method to produce dense matches in low-texture areas, where feature detectors usually struggle to produce repeatable interest points. The experiments on indoor and outdoor datasets show that LoFTR outperforms state-of-the-art methods by a large margin. LoFTR also ranks first on two public benchmarks of visual localization among the published methods. Code is available at project page: https://zju3dv.github.io/loftr/\n\n\n
\n\n\nIn this notebook I will show you how use LoFTR using Kornia for matching whales features. Kornia is a differentiable library that allows classical computer vision to be integrated into deep learning models.\n\nIt consists of a set of routines and differentiable modules to solve generic computer vision problems. At its core, the package uses PyTorch as its main backend both for efficiency and to take advantage of the reverse-mode auto-differentiation to define and compute the gradient of complex functions.\n\nWhy Kornia ?\nWith Kornia we fill the gap between classical and deep computer vision that implements standard and advanced vision algorithms for AI:\n\n* Computer Vision: Kornia fills the gap between Classical and Deep computer Vision.\n* Differentiable: We leverage the Computer Vision 2.0 paradigm.\n* Open Source: Our libraries and initiatives are always according to the community needs.\n* PyTorch: At our core we use PyTorch and its Autograd engine for its efficiency.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"
","metadata":{}},{"cell_type":"markdown","source":"Update (2022.03.06)\n\n**Today I did SURF and LoFTR comparision .... Two observations:**\n* SURF and SIFT are not in Kaggle distribution - algorithm is patented and is excluded in this configuration \n* In my opinion SURF works worse then LoFTR on this dataset - it was difficult to find a match above 75% even on very similar fins (belonging to the same individual)\n\nSee comparision:\n
\n
","metadata":{}},{"cell_type":"code","source":"%%capture \n\n!pip install git+https://github.com/kornia/kornia\n!pip install kornia_moons","metadata":{"execution":{"iopub.status.busy":"2022-03-07T22:23:01.507596Z","iopub.execute_input":"2022-03-07T22:23:01.507957Z","iopub.status.idle":"2022-03-07T22:23:50.490177Z","shell.execute_reply.started":"2022-03-07T22:23:01.507871Z","shell.execute_reply":"2022-03-07T22:23:50.488912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import cv2\nimport kornia as K\nimport kornia.feature as KF\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport torch\n\nfrom kornia_moons.feature import *\nfrom PIL import Image","metadata":{"execution":{"iopub.status.busy":"2022-03-07T22:23:50.49259Z","iopub.execute_input":"2022-03-07T22:23:50.492826Z","iopub.status.idle":"2022-03-07T22:23:52.275551Z","shell.execute_reply.started":"2022-03-07T22:23:50.492799Z","shell.execute_reply":"2022-03-07T22:23:52.274471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For this demo I will use cropped images from dataset provided by @ [cropped&resized(512x512) dataset using detic](https://www.kaggle.com/c/happy-whale-and-dolphin/discussion/305503). Thank you for contributing in this competition.","metadata":{}},{"cell_type":"code","source":"im_path = '../input/whale2-cropped-dataset/cropped_train_images/cropped_train_images/' \nwh_dataset = pd.read_csv(\"../input/happy-whale-and-dolphin/train.csv\")\nwh_dataset.species.value_counts().head(8)","metadata":{"execution":{"iopub.status.busy":"2022-03-07T22:23:52.296Z","iopub.execute_input":"2022-03-07T22:23:52.296451Z","iopub.status.idle":"2022-03-07T22:23:52.421888Z","shell.execute_reply.started":"2022-03-07T22:23:52.296407Z","shell.execute_reply":"2022-03-07T22:23:52.420941Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# I can just take one flase killer whale to watch them \nwh_dataset.query(\"species=='spinner_dolphin'\").individual_id.value_counts().head(5)","metadata":{"execution":{"iopub.status.busy":"2022-03-07T22:23:52.423517Z","iopub.execute_input":"2022-03-07T22:23:52.424107Z","iopub.status.idle":"2022-03-07T22:23:52.445839Z","shell.execute_reply.started":"2022-03-07T22:23:52.424062Z","shell.execute_reply":"2022-03-07T22:23:52.445013Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_whales(im_path):\n \n fig, axes = plt.subplots(3, 3, figsize=(20,20))\n \n for idx, img in enumerate(img_to_draw):\n i = idx % 3 \n j = idx // 3 \n image = Image.open(im_path + img)\n iar_shp = np.array(image).shape\n axes[i, j].imshow(image)\n axes[i, j].set_title(img)\n\n plt.subplots_adjust(wspace=0, hspace=.2)\n plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-07T22:23:52.44899Z","iopub.execute_input":"2022-03-07T22:23:52.449814Z","iopub.status.idle":"2022-03-07T22:23:52.456488Z","shell.execute_reply.started":"2022-03-07T22:23:52.449773Z","shell.execute_reply":"2022-03-07T22:23:52.455676Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img_to_draw = [file for file in wh_dataset.query(\"individual_id == '9ab8c57f10bc'\").sample(9).image]\n\nplot_whales(im_path)","metadata":{"execution":{"iopub.status.busy":"2022-03-07T22:23:52.4578Z","iopub.execute_input":"2022-03-07T22:23:52.458814Z","iopub.status.idle":"2022-03-07T22:23:54.802106Z","shell.execute_reply.started":"2022-03-07T22:23:52.458767Z","shell.execute_reply":"2022-03-07T22:23:54.801488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# This is demo only - I removed background only for whale ID = 9ab8c57f10bc and put images in separate dataset. \n# There is no model and solution provided yet - still working on improving background separation.\n\nplot_whales('../input/whalebackground/')","metadata":{"execution":{"iopub.status.busy":"2022-03-07T22:23:54.803121Z","iopub.execute_input":"2022-03-07T22:23:54.803491Z","iopub.status.idle":"2022-03-07T22:23:56.511453Z","shell.execute_reply.started":"2022-03-07T22:23:54.803459Z","shell.execute_reply":"2022-03-07T22:23:56.510542Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_torch_image(fname):\n img = K.image_to_tensor(cv2.imread(fname), False).float() /255.\n img = K.color.bgr_to_rgb(img)\n return img","metadata":{"execution":{"iopub.status.busy":"2022-03-07T22:23:56.512706Z","iopub.execute_input":"2022-03-07T22:23:56.513553Z","iopub.status.idle":"2022-03-07T22:23:56.519349Z","shell.execute_reply.started":"2022-03-07T22:23:56.513512Z","shell.execute_reply":"2022-03-07T22:23:56.518349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In this experiment we use SoTA feature extractor LoFTR however Kornia gives you wide range of features extractor and they utylize GPU (Pytorch): https://kornia.readthedocs.io/en/latest/feature.html","metadata":{}},{"cell_type":"code","source":"def match_and_draw(im_path, img_in1, img_in2):\n img1 = load_torch_image(im_path + img_in1)\n img2 = load_torch_image(im_path + img_in2)\n matcher = KF.LoFTR(pretrained='outdoor')\n \n input_dict = {\"image0\": K.color.rgb_to_grayscale(img1), \n \"image1\": K.color.rgb_to_grayscale(img2)}\n \n with torch.no_grad():\n correspondences = matcher(input_dict)\n \n mkpts0 = correspondences['keypoints0'].cpu().numpy()\n mkpts1 = correspondences['keypoints1'].cpu().numpy()\n H, inliers = cv2.findFundamentalMat(mkpts0, mkpts1, cv2.USAC_MAGSAC, 0.5, 0.999, 100000)\n inliers = inliers > 0\n \n draw_LAF_matches(\n KF.laf_from_center_scale_ori(torch.from_numpy(mkpts0).view(1,-1, 2),\n torch.ones(mkpts0.shape[0]).view(1,-1, 1, 1),\n torch.ones(mkpts0.shape[0]).view(1,-1, 1)),\n\n KF.laf_from_center_scale_ori(torch.from_numpy(mkpts1).view(1,-1, 2),\n torch.ones(mkpts1.shape[0]).view(1,-1, 1, 1),\n torch.ones(mkpts1.shape[0]).view(1,-1, 1)),\n torch.arange(mkpts0.shape[0]).view(-1,1).repeat(1,2),\n K.tensor_to_image(img1),\n K.tensor_to_image(img2),\n inliers,\n draw_dict={'inlier_color': (0.2, 1, 0.2),\n 'tentative_color': None, \n 'feature_color': (0.2, 0.5, 1), 'vertical': False})\n return correspondences","metadata":{"execution":{"iopub.status.busy":"2022-03-07T22:35:11.41654Z","iopub.execute_input":"2022-03-07T22:35:11.416888Z","iopub.status.idle":"2022-03-07T22:35:11.430402Z","shell.execute_reply.started":"2022-03-07T22:35:11.416852Z","shell.execute_reply":"2022-03-07T22:35:11.429768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img_to_draw = [file for file in wh_dataset.query(\"individual_id == '9ab8c57f10bc'\").image]\nrandom_samples = np.random.randint(len(img_to_draw), size=(2, 4))\n\nfor i in range(random_samples.shape[1]):\n whale_1 = img_to_draw[random_samples[0][i]]\n whale_2 = img_to_draw[random_samples[1][i]]\n print(f'Matching: {whale_1} to {whale_2}')\n correspondences = match_and_draw(im_path, whale_1, whale_2)","metadata":{"execution":{"iopub.status.busy":"2022-03-07T22:35:13.836052Z","iopub.execute_input":"2022-03-07T22:35:13.836409Z","iopub.status.idle":"2022-03-07T22:35:53.57675Z","shell.execute_reply.started":"2022-03-07T22:35:13.836372Z","shell.execute_reply":"2022-03-07T22:35:53.574515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's look on the structure of correcpondences\n\nfor k,v in correspondences.items():\n print (k)","metadata":{"execution":{"iopub.status.busy":"2022-03-07T22:24:40.742333Z","iopub.execute_input":"2022-03-07T22:24:40.742618Z","iopub.status.idle":"2022-03-07T22:24:40.748555Z","shell.execute_reply.started":"2022-03-07T22:24:40.742586Z","shell.execute_reply":"2022-03-07T22:24:40.747719Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Keypoint coordinates for last prediction - only for showing structure \nprint(\"Coordinate for each matching feature - X and Y\")\nprint(correspondences['keypoints0'].cpu().numpy().T)","metadata":{"execution":{"iopub.status.busy":"2022-03-07T22:24:40.750003Z","iopub.execute_input":"2022-03-07T22:24:40.750265Z","iopub.status.idle":"2022-03-07T22:24:40.767219Z","shell.execute_reply.started":"2022-03-07T22:24:40.750236Z","shell.execute_reply":"2022-03-07T22:24:40.766255Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Keypoint confidence\n# Blue one - low confidence, green one confidence over threshold\nprint(\"Scores for each feature:\")\nprint(correspondences['confidence'].cpu().numpy())","metadata":{"execution":{"iopub.status.busy":"2022-03-07T22:24:40.768429Z","iopub.execute_input":"2022-03-07T22:24:40.76911Z","iopub.status.idle":"2022-03-07T22:24:40.78573Z","shell.execute_reply.started":"2022-03-07T22:24:40.769063Z","shell.execute_reply":"2022-03-07T22:24:40.784822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This is experimental code - demo only to show you one way of dealing with problem. Hope you like it. \nFor sure I will investigate this way. It looks promisting.","metadata":{}},{"cell_type":"code","source":"# This is demo only - I removed background only for whale ID = 9ab8c57f10bc and put images in separate dataset. \n# There is no model and solution provided yet - still working on improving background separation.\n\nfor i in range(random_samples.shape[1]):\n whale_1 = img_to_draw[random_samples[0][i]]\n whale_2 = img_to_draw[random_samples[1][i]]\n print(f'Matching: {whale_1} to {whale_2}')\n match_and_draw('../input/whalebackground/', whale_1, whale_2)","metadata":{"execution":{"iopub.status.busy":"2022-03-07T22:24:40.787077Z","iopub.execute_input":"2022-03-07T22:24:40.787404Z","iopub.status.idle":"2022-03-07T22:24:56.155663Z","shell.execute_reply.started":"2022-03-07T22:24:40.787369Z","shell.execute_reply":"2022-03-07T22:24:56.154752Z"},"trusted":true},"execution_count":null,"outputs":[]}]}