{"cells":[{"metadata":{},"cell_type":"markdown","source":"*This Kernel can be used to evaluate the generated descriptors for Landmark Retrieval 2020*","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Landmark Image Retrieval\nThe goal of the Landmark Recognition challenge is to recognize a landmark presented in a query image, while the goal of Landmark Retrieval 2019 is to find all images showing that landmark.\n![](https://storage.googleapis.com/groundai-web-prod/media%2Fusers%2Fuser_14%2Fproject_374127%2Fimages%2Fpictures%2Frec_demo.jpg)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Deep Local and Global Image Features\nIn this kernel I will use pretrained DELF model for landmark retrieval. DELF project presents code for deep local and global image feature methods, which are particularly useful for the computer vision tasks of instance-level recognition and retrieval. These were introduced in the DELF, Detect-to-Retrieve, DELG.  <br>\n**Acknowledgment:** In the following link, you can find the project source code, installation guidlines and pretrained models by **@andre faraujo**: <br>\nhttps://github.com/tensorflow/models/tree/master/research/delf\n\n**Please upvote if you find this kernel useful**\n\n![](https://www.i-programmer.info/images/stories/News/2018/march/A/delfpipeline.JPG)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Acknowledgement:\nSource code published on [Colab](https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/tf_hub_delf_module.ipynb) by Tensorflow developers. In this kernel, I am elaborating more on the sub modules\n<br>Installing needed Packages","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"!pip install -q scikit-image\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom absl import logging\n\nimport matplotlib.pyplot as plt\nfrom PIL import Image, ImageOps\nfrom scipy.spatial import cKDTree\nfrom skimage.feature import plot_matches\nfrom skimage.measure import ransac\nfrom skimage.transform import AffineTransform\nfrom six import BytesIO\n\nimport tensorflow as tf\nimport tensorflow_hub as hub\nfrom six.moves.urllib.request import urlopen","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# The data\nWe will use a set of images on landmarks in different viewports. Then, we specify the URLs of two images we would like to process with DELF in order to match and compare them.","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# The tutorial is based on the DElf Hub tutorial from Tensorflow\n# from: https://commons.wikimedia.org/wiki/File:Bridge_of_Sighs,_Oxford.jpg\n# by: N.H. Fischer\nIMAGE_1_URL = 'https://upload.wikimedia.org/wikipedia/commons/2/28/Bridge_of_Sighs%2C_Oxford.jpg'\n# from https://commons.wikimedia.org/wiki/File:The_Bridge_of_Sighs_and_Sheldonian_Theatre,_Oxford.jpg\n# by: Matthew Hoser\nIMAGE_2_URL = 'https://upload.wikimedia.org/wikipedia/commons/c/c3/The_Bridge_of_Sighs_and_Sheldonian_Theatre%2C_Oxford.jpg'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def download_and_resize(name, url, new_width=256, new_height=256):\n  path = tf.keras.utils.get_file(url.split('/')[-1], url)\n  image = Image.open(path)\n  image = ImageOps.fit(image, (new_width, new_height), Image.ANTIALIAS)\n  return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image1 = download_and_resize('image_1.jpg', IMAGE_1_URL)\nimage2 = download_and_resize('image_2.jpg', IMAGE_2_URL)\n\nplt.subplot(1,2,1)\nplt.imshow(image1)\nplt.subplot(1,2,2)\nplt.imshow(image2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# The pre-trained DELF(Deep Local Feature) module:\nThis module is available on TensorFlow Hub can be used for image retrieval as a drop-in replacement for other keypoint detectors and descriptors. It describes each noteworthy point in a given image with multidimensional vectors known as feature descriptor.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# You can replace this line to use your custome model\ndelf = hub.load('https://tfhub.dev/google/delf/1').signatures['default']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Apply the DELF module to the data\nThe DELF module takes an image as input and represnt the image in a reduced dimentional space asfeature descriptor. The following cell contains the core of this colab's logic. The DELF Image retrieval system can be decomposed into four main blocks:\n\n* Dense localized feature extraction,\n* Keypoint selection,\n* Dimensionality reduction,\n* Indexing and retrieval.\n\nThe model will return the descriptors and feature locations. This can be used now to compare images using similarity-based methods (nearest-neighbor matches using a KD tree)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def run_delf(image):\n  np_image = np.array(image)\n  float_image = tf.image.convert_image_dtype(np_image, tf.float32)\n\n  return delf(\n      image=float_image,\n      score_threshold=tf.constant(100.0),\n      image_scales=tf.constant([0.25, 0.3536, 0.5, 0.7071, 1.0, 1.4142, 2.0]),\n      max_feature_num=tf.constant(1000))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"result1 = run_delf(image1)\nresult2 = run_delf(image2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Match Images at Runtime\nAt runtime, the query image was first resized and cropped to 256x256 resolution followed by the DELF module computing its descriptors and locations. Then we query the KD-tree to find K nearest neighbors for each descriptor of the query image. Next, aggregate all the matches per database image. Finally, we perform geometric verification using RANSAC and employ the number of inliers as the score for retrieved images.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# RANSAC for geometric verification\n\nRANSAC for geometric verification can be used to estimate geometric transformations. We want to make sure all matches are consistent with a global geometric transformation; however, there are many incorrect matches. Take the following graph for example, without the geometric verification there are many inconsistent matches while after applying RANSAC, we can estimate the geometric transformation and the set of consistent matches simultaneously.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def match_images(image1, image2, result1, result2):\n  distance_threshold = 0.8\n\n  # Read features.\n  num_features_1 = result1['locations'].shape[0]\n  print(\"Loaded image 1's %d features\" % num_features_1)\n  \n  num_features_2 = result2['locations'].shape[0]\n  print(\"Loaded image 2's %d features\" % num_features_2)\n\n  # Find nearest-neighbor matches using a KD tree.\n  d1_tree = cKDTree(result1['descriptors'])\n  _, indices = d1_tree.query(\n      result2['descriptors'],\n      distance_upper_bound=distance_threshold)\n  \n  # Select feature locations for putative matches.\n  locations_2_to_use = np.array([\n      result2['locations'][i,]\n      for i in range(num_features_2)\n      if indices[i] != num_features_1\n  ])\n  locations_1_to_use = np.array([\n      result1['locations'][indices[i],]\n      for i in range(num_features_2)\n      if indices[i] != num_features_1\n  ])\n    # Perform geometric verification using RANSAC.\n  _, inliers = ransac(\n      (locations_1_to_use, locations_2_to_use),\n      AffineTransform,\n      min_samples=3,\n      residual_threshold=20,\n      max_trials=1000)\n\n  print('Found %d inliers' % sum(inliers))\n\n  # Visualize correspondences.\n  _, ax = plt.subplots(figsize=(20, 20))\n  inlier_idxs = np.nonzero(inliers)[0]\n  plot_matches(\n      ax,\n      image1,\n      image2,\n      locations_1_to_use,\n      locations_2_to_use,\n      np.column_stack((inlier_idxs, inlier_idxs)),\n      matches_color='b')\n  ax.axis('off')\n  ax.set_title('DELF correspondences')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"match_images(image1, image2, result1, result2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **Please upvote if you find this kernel useful**","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}