{"cells":[{"metadata":{},"cell_type":"markdown","source":"\n![Acropolis-Partheon](https://drive.google.com/uc?export=view&id=1G8hJ0KK5i0wNaDZYoiTSJskxaIe2Lt-B)\n\nIn this notebook we use a module that packages the DELF neural network for processng images and identifying keypoints and descriptors \n\n**What is DELF??**\n\nDEep Local Feature module can be used for image retrievals. Each noteworthy point in the image is described using a 40-dimensional vector (*known as a feature vector*)\n\n\nThe model in TF-Hub is trained on GLDv1 \n\n\n[Link to the T-Hub Colab notebook](https://www.tensorflow.org/hub/tutorials/tf_hub_delf_module)","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# Importing basic libraries\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Importing libraries required for the matching operation\n\nfrom absl import logging\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nfrom PIL import Image, ImageOps\nfrom scipy.spatial import cKDTree\nfrom skimage.feature import plot_matches\nfrom skimage.measure import ransac\nfrom skimage.transform import AffineTransform\nfrom six import BytesIO\n\nimport tensorflow as tf\n\nimport tensorflow_hub as hub\nfrom six.moves.urllib.request import urlopen","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Replace the value of `images` with any of the other possible options in the next code block","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\nimages = \"Acropolis\" \nif images == \"Bridge of Sighs\":\n  # from: https://commons.wikimedia.org/wiki/File:Bridge_of_Sighs,_Oxford.jpg\n  # by: N.H. Fischer\n  IMAGE_1_URL = 'https://upload.wikimedia.org/wikipedia/commons/2/28/Bridge_of_Sighs%2C_Oxford.jpg'\n  # from https://commons.wikimedia.org/wiki/File:The_Bridge_of_Sighs_and_Sheldonian_Theatre,_Oxford.jpg\n  # by: Matthew Hoser\n  IMAGE_2_URL = 'https://upload.wikimedia.org/wikipedia/commons/c/c3/The_Bridge_of_Sighs_and_Sheldonian_Theatre%2C_Oxford.jpg'\nelif images == \"Golden Gate\":\n  IMAGE_1_URL = 'https://upload.wikimedia.org/wikipedia/commons/1/1e/Golden_gate2.jpg'\n  IMAGE_2_URL = 'https://upload.wikimedia.org/wikipedia/commons/3/3e/GoldenGateBridge.jpg'\nelif images == \"Acropolis\":\n  IMAGE_1_URL = 'https://upload.wikimedia.org/wikipedia/commons/c/ce/2006_01_21_Ath%C3%A8nes_Parth%C3%A9non.JPG'\n  IMAGE_2_URL = 'https://upload.wikimedia.org/wikipedia/commons/5/5c/ACROPOLIS_1969_-_panoramio_-_jean_melis.jpg'\nelse:\n  IMAGE_1_URL = 'https://upload.wikimedia.org/wikipedia/commons/d/d8/Eiffel_Tower%2C_November_15%2C_2011.jpg'\n  IMAGE_2_URL = 'https://upload.wikimedia.org/wikipedia/commons/a/a8/Eiffel_Tower_from_immediately_beside_it%2C_Paris_May_2008.jpg'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The following function downloads the image dfrom the given link `url` and resizes (*standardizes*) it to a square of 256x256","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def download_and_resize(name, url, new_width=256, new_height=256):\n  path = tf.keras.utils.get_file(url.split('/')[-1], url)\n  image = Image.open(path)\n  image = ImageOps.fit(image, (new_width, new_height), Image.ANTIALIAS)\n  return image","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets download the image of Acropolis-Paratheon and then plot it using `matplotlib`","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"image1 = download_and_resize('image_1.jpg', IMAGE_1_URL)\nimage2 = download_and_resize('image_2.jpg', IMAGE_2_URL)\n\nplt.subplot(1,2,1)\nplt.imshow(image1)\nplt.subplot(1,2,2)\nplt.imshow(image2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Load the DELF neural model from TensorFlow-Hub**\n\nNext create a function to return the features","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"delf = hub.load('https://tfhub.dev/google/delf/1').signatures['default']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def run_delf(image):\n  np_image = np.array(image)\n  float_image = tf.image.convert_image_dtype(np_image, tf.float32)\n\n  return delf(\n      image=float_image,\n      score_threshold=tf.constant(100.0),\n      image_scales=tf.constant([0.25, 0.3536, 0.5, 0.7071, 1.0, 1.4142, 2.0]),\n      max_feature_num=tf.constant(1000))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Calculate the DELF features of both the images","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"result1 = run_delf(image1)\nresult2 = run_delf(image2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here's the **meat** of the operation where the extracted features are mapped","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef match_images(image1, image2, result1, result2 ):\n  distance_threshold = 0.8\n\n  # Read features.\n  num_features_1 = result1['locations'].shape[0]\n  print(\"Loaded image 1's %d features\" % num_features_1)\n  \n  num_features_2 = result2['locations'].shape[0]\n  print(\"Loaded image 2's %d features\" % num_features_2)\n\n  # Find nearest-neighbor matches using a KD tree.\n  d1_tree = cKDTree(result1['descriptors'])\n  _, indices = d1_tree.query(\n      result2['descriptors'],\n      distance_upper_bound=distance_threshold)\n\n  # Select feature locations for putative matches.\n  locations_2_to_use = np.array([\n      result2['locations'][i,]\n      for i in range(num_features_2)\n      if indices[i] != num_features_1\n  ])\n  locations_1_to_use = np.array([\n      result1['locations'][indices[i],]\n      for i in range(num_features_2)\n      if indices[i] != num_features_1\n  ])\n\n  # Perform geometric verification using RANSAC.\n  _, inliers = ransac(\n      (locations_1_to_use, locations_2_to_use),\n      AffineTransform,\n      min_samples=3,\n      residual_threshold=20,\n      max_trials=1000)\n\n  print('Found %d inliers' % sum(inliers))\n\n  # Visualize correspondences.\n  _, ax = plt.subplots()\n  inlier_idxs = np.nonzero(inliers)[0]\n  plot_matches(\n      ax,\n      image1,\n      image2,\n      locations_1_to_use,\n      locations_2_to_use,\n      np.column_stack((inlier_idxs, inlier_idxs)),\n      matches_color='b')\n  ax.axis('off')\n  ax.set_title('DELF correspondences')\n\n  return inliers\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"match_inliers = match_images(image1, image2, result1, result2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sum(match_inliers)/len(match_inliers)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* We can now create a pipeline where each candidate image in the index is run against the test image \n* Once the scoring is done, then the best candidate with the highest number of matched inliers can be selected\n\nLet's check what we can do with another example from the Oxford Buildings dataset\n\nWe will take some sample `IMAGES` and then calculat the inliers for a few pairs and then check which one provides the best match","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"IMAGES = ['all_souls_000002.jpg', 'all_souls_000005.jpg', 'all_souls_000001.jpg', 'bodleian_000396.jpg', \n          'bodleian_000395.jpg', 'christ_church_000061.jpg', 'christ_church_000081.jpg']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import tarfile\n\ndef download_and_resize_local(fname, zipfile, new_width=256, new_height=256):\n    zipFile = tarfile.open(zipfile)\n    zipFile.extract(fname, '.')\n    image = Image.open(fname)\n    image = ImageOps.fit(image, (new_width, new_height), Image.ANTIALIAS)\n    return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef match_images_2(image1, image2, result1, result2, fnamelist ):\n  distance_threshold = 0.8\n\n  # Read features.\n  num_features_1 = result1['locations'].shape[0]\n  #print(\"Loaded image 1's %d features\" % num_features_1)\n  \n  num_features_2 = result2['locations'].shape[0]\n  #print(\"Loaded image 2's %d features\" % num_features_2)\n\n  # Find nearest-neighbor matches using a KD tree.\n  d1_tree = cKDTree(result1['descriptors'])\n  _, indices = d1_tree.query(\n      result2['descriptors'],\n      distance_upper_bound=distance_threshold)\n\n  # Select feature locations for putative matches.\n  locations_2_to_use = np.array([\n      result2['locations'][i,]\n      for i in range(num_features_2)\n      if indices[i] != num_features_1\n  ])\n  locations_1_to_use = np.array([\n      result1['locations'][indices[i],]\n      for i in range(num_features_2)\n      if indices[i] != num_features_1\n  ])\n\n  # Perform geometric verification using RANSAC.\n  _, inliers = ransac(\n      (locations_1_to_use, locations_2_to_use),\n      AffineTransform,\n      min_samples=3,\n      residual_threshold=20,\n      max_trials=1000)\n\n  #print('Found %d inliers' % sum(inliers))\n\n  # Visualize correspondences.\n  _, ax = plt.subplots()\n  inlier_idxs = np.nonzero(inliers)[0]\n  plot_matches(\n      ax,\n      image1,\n      image2,\n      locations_1_to_use,\n      locations_2_to_use,\n      np.column_stack((inlier_idxs, inlier_idxs)),\n      matches_color='b')\n  ax.axis('off')\n  ax.set_title(fnamelist[0] + \" ||| \" + fnamelist[1])\n\n  return inliers\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from tqdm import tqdm\n\nzippath = '/kaggle/input/oxbuildings/oxbuild_images.tgz'\nimage1 = download_and_resize_local(IMAGES[0], zippath )\nprint(\"Retrieving for sample image: \", IMAGES[0])\n\ninlier_list = list()\n\nfor idx in tqdm(range(1,len(IMAGES))):\n    #print(\"Matching with image: \", IMAGES[idx])\n    image2 = download_and_resize_local(IMAGES[idx], zippath)\n    result1 = run_delf(image1)\n    result2 = run_delf(image2)\n\n    match_inliers = match_images_2(image1, image2, result1, result2, [IMAGES[0], IMAGES[idx]])\n    inlier_list.append({'image': IMAGES[idx], 'inlier': sum(match_inliers)})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"inlierdf = pd.DataFrame(inlier_list).sort_values(by='inlier', ascending=False)\ninlierdf","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The good stuff first! \n\n1. The system matched with a night pic of the same church \n\n\nsome drawbacks - \n\n1. The neural network got confused with the towers(christ church & all souls)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}