{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":49349,"databundleVersionId":5447706,"sourceType":"competition"},{"sourceId":3846,"sourceType":"modelInstanceVersion","modelInstanceId":2747}],"dockerImageVersionId":30461,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\nIn this note book, I try implementing part of the solution gaining the [2nd position at IMC2023](https://www.kaggle.com/competitions/image-matching-challenge-2023/discussion/416873). The authors had used this [solution](https://github.com/ternaus/check_orientation) to correct rotated images before using it in the rest of the pipeline. From the authors' description  \n> We use it w/o any threshold and look at the number of rotations that we need to apply to an image. After applying rotation, the score for cyprus scene jumped up significantly from ~0.02 up to ~0.55. RotNet implementation did not work out for us.\n\nThis notebook allows user to explore the dataset of choice. Then run the rotation correction and visualize the outputs.","metadata":{}},{"cell_type":"code","source":"!pip install mediapy","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-04-27T13:37:14.997725Z","iopub.execute_input":"2024-04-27T13:37:14.998361Z","iopub.status.idle":"2024-04-27T13:37:28.693781Z","shell.execute_reply.started":"2024-04-27T13:37:14.998328Z","shell.execute_reply":"2024-04-27T13:37:28.692652Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get some utility functions from https://github.com/cvg/Hierarchical-Localization/\n%cd /kaggle/working/\n!rm -rf /kaggle/working/Hierarchical-Localization\n!git clone --quiet --recursive https://github.com/cvg/Hierarchical-Localization/\n%cd /kaggle/working/Hierarchical-Localization\n!pip install -e .\n\n# from hloc import extract_features, match_features, reconstruction, visualization, pairs_from_exhaustive\n# from hloc.visualization import plot_images, read_image\nfrom hloc.utils import viz_3d\n%cd /kaggle/working/","metadata":{"execution":{"iopub.status.busy":"2024-04-27T13:37:28.695821Z","iopub.execute_input":"2024-04-27T13:37:28.696115Z","iopub.status.idle":"2024-04-27T13:37:55.410894Z","shell.execute_reply.started":"2024-04-27T13:37:28.696086Z","shell.execute_reply":"2024-04-27T13:37:55.409557Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nimport pycolmap\nimport numpy as np\nimport mediapy as media\nimport cv2\nfrom glob import glob\nfrom pathlib import Path\nfrom time import time\nfrom pprint import pprint","metadata":{"execution":{"iopub.status.busy":"2024-04-27T13:37:55.412642Z","iopub.execute_input":"2024-04-27T13:37:55.413002Z","iopub.status.idle":"2024-04-27T13:37:55.606290Z","shell.execute_reply.started":"2024-04-27T13:37:55.412970Z","shell.execute_reply":"2024-04-27T13:37:55.605080Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"imagesDirList = glob('/kaggle/input/image-matching-challenge-2023/train/*/*/images')\npprint(imagesDirList)","metadata":{"execution":{"iopub.status.busy":"2024-04-27T13:37:55.608826Z","iopub.execute_input":"2024-04-27T13:37:55.609252Z","iopub.status.idle":"2024-04-27T13:37:55.753467Z","shell.execute_reply.started":"2024-04-27T13:37:55.609208Z","shell.execute_reply":"2024-04-27T13:37:55.752259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Visualizing the images","metadata":{}},{"cell_type":"markdown","source":"In this section, I iterate through each of the directories of images and display them. I also counted the number of images that needed rotation correction. Looks like only \"heritage/dioscuri\" and \"heritage/cyprus\" dataset have a lot of rotated images. Please change the `dirIdx` in the following code cell to view the data in that directory.","metadata":{}},{"cell_type":"markdown","source":"Rotated images observed per directory  \n0 - /kaggle/input/image-matching-challenge-2023/train/heritage/dioscuri/images - 79 images 90 degrees rotated.  \n1 - /kaggle/input/image-matching-challenge-2023/train/heritage/cyprus/images - 18 images 90 degrees rotated.  \n2 - /kaggle/input/image-matching-challenge-2023/train/heritage/wall/images - 0  \n3 - /kaggle/input/image-matching-challenge-2023/train/phototourism/british_museum/images - 0  \n4 - /kaggle/input/image-matching-challenge-2023/train/phototourism/piazza_san_marco/images - 0  \n5 - /kaggle/input/image-matching-challenge-2023/train/phototourism/trevi_fountain/images - 0  \n6 - /kaggle/input/image-matching-challenge-2023/train/phototourism/st_pauls_cathedral/images = 0  \n7 - /kaggle/input/image-matching-challenge-2023/train/phototourism/colosseum_exterior/images = 0  \n8 - /kaggle/input/image-matching-challenge-2023/train/phototourism/buckingham_palace/images = 0  \n9 - /kaggle/input/image-matching-challenge-2023/train/phototourism/temple_nara_japan/images = 0  \n10 - /kaggle/input/image-matching-challenge-2023/train/phototourism/sagrada_familia/images = 0  \n11 - /kaggle/input/image-matching-challenge-2023/train/phototourism/grand_place_brussels/images = 0  \n12 - /kaggle/input/image-matching-challenge-2023/train/phototourism/pantheon_exterior/images = 2 images about 45 degrees rotated  \n13 - /kaggle/input/image-matching-challenge-2023/train/phototourism/notre_dame_front_facade/images - 4 images about 45 degrees rotated.  \n14 - /kaggle/input/image-matching-challenge-2023/train/phototourism/st_peters_square/images - 0  \n15 - /kaggle/input/image-matching-challenge-2023/train/phototourism/sacre_coeur/images - 0  \n16 - /kaggle/input/image-matching-challenge-2023/train/phototourism/taj_mahal/images - 0  \n17 - /kaggle/input/image-matching-challenge-2023/train/phototourism/lincoln_memorial_statue/images - 0  \n18 - /kaggle/input/image-matching-challenge-2023/train/phototourism/brandenburg_gate/images - 0  \n19 - /kaggle/input/image-matching-challenge-2023/train/urban/kyiv-puppet-theater/images - 1 image is around 45 degrees tilted  \n20 - /kaggle/input/image-matching-challenge-2023/train/haiper/fountain/images - 0  \n21 - /kaggle/input/image-matching-challenge-2023/train/haiper/bike/images - 0  \n22 - /kaggle/input/image-matching-challenge-2023/train/haiper/chairs/images - 0  \n","metadata":{}},{"cell_type":"code","source":"dirIdx = 0\nsrc = imagesDirList[dirIdx]\nimages = [cv2.cvtColor(cv2.imread(im), cv2.COLOR_BGR2RGB) for im in glob(f'{src}/*')]\nprint(f'Showing {len(images)} image from {src}')\nmedia.show_images(images, height=100, columns=10)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Rotation Correction\nIn this section, we correct the rotation of the images.","metadata":{}},{"cell_type":"markdown","source":"## Create a Torch Dataset\nIn order to pass that data as input to the NN, we needed to convert to tensor and apply some transforms. I decided create a torch dataset that will iterate through the images in a given folder.","metadata":{}},{"cell_type":"code","source":"import torch\nfrom torch.utils.data import Dataset\nfrom torchvision.io import read_image\n\nclass IMCDataset(Dataset):\n    def __init__(self, imgDir, transform=None):\n        self.imgDir = imgDir\n        self.transform = transform\n        self.imgPathList = [imgPath for imgPath in glob(f'{imgDir}/*')]\n\n    def __len__(self):\n        return len(self.imgPathList)\n\n    def __getitem__(self, idx):\n        imgPath = self.imgPathList[idx]\n        image = read_image(imgPath)\n        if self.transform:\n            image = self.transform(image)\n        return image","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The following transform is applied to every image read from the directory. The transform resizes and normalizes the images as required by ResNet50.","metadata":{}},{"cell_type":"code","source":"from torchvision import transforms as T\ntransform = T.Compose([\n    T.Resize((224, 224)),\n    T.ConvertImageDtype(torch.float),\n    T.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225))\n])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"heritageDioscuriDataset = IMCDataset(imagesDirList[dirIdx], transform=transform)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Rotation Correction","metadata":{}},{"cell_type":"code","source":"!pip install check_orientation  > /dev/null\nfrom check_orientation.pre_trained_models import create_model\nimport torch","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = create_model(\"swsl_resnext50_32x4d\")\nmodel.eval();","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"First, let's try the model on only one image.","metadata":{}},{"cell_type":"code","source":"imgIdx = 1\nmedia.show_image(images[imgIdx])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with torch.no_grad():\n  prediction = model(heritageDioscuriDataset[imgIdx][None, ...]).numpy()\n\nprint(f'Prediction: {prediction}')\nprint(f\"Prediction argmax: {prediction[0].argmax()}\")\n\npredictedRotationDeg = prediction[0].argmax() * 90\nprint(f\"Predicted rotation is {predictedRotationDeg} degrees.\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The `prediction` is an array with probabilities distributed among 4 possible options. These are 0, 90, 180, and 270 degrees of rotation. To rotate the images using `cv2.rotate`, we need the rotation code. Below, maps the predicted rotation (in degrees) to the cv2 rotation code.","metadata":{}},{"cell_type":"code","source":"predictionToRotationCodeMap = {\n    90: cv2.ROTATE_90_CLOCKWISE,\n    180: cv2.ROTATE_180,\n    270: cv2.ROTATE_90_COUNTERCLOCKWISE\n}","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"cv2 rotation code is: {predictionToRotationCodeMap[predictedRotationDeg]}\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's rotate the image using `cv2.rotate` and display the output.","metadata":{}},{"cell_type":"code","source":"if predictedRotationDeg == 0:\n    rotatedImage = images[imgIdx]\nelse:\n    rotatedImage = cv2.rotate(images[imgIdx], predictionToRotationCodeMap[predictedRotationDeg])\n    \nmedia.show_image(rotatedImage)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next let's process all images of this dataset using the model. The following code iterates through the dataset and predicts the rotation of image. Then `cv2.rotate` is used to rotate the images and store them in the list `rotatedImages`.","metadata":{}},{"cell_type":"code","source":"rotatedImages = []\nnumberOfImagesRotated = 0\nfor imgIdx, image in enumerate(images):\n    with torch.no_grad():\n        prediction = model(heritageDioscuriDataset[imgIdx][None, ...]).numpy()[0]\n    predictedRotationDeg = prediction.argmax() * 90\n    if predictedRotationDeg == 0:\n        rotatedImages.append(image.copy())\n    else:\n        rotatedImages.append(cv2.rotate(image, predictionToRotationCodeMap[predictedRotationDeg]))\n        numberOfImagesRotated += 1\nprint(f\"Total {numberOfImagesRotated} images corrected.\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The rotated images are displayed below. Looks like some zoomed in images of structures where no part of sky or ground is clearly visible got rotated too.\n\nIn the subplots, original images are shown on column 0 and 2. The processed images shown on the columns 1 and 3.","metadata":{}},{"cell_type":"code","source":"from matplotlib import pyplot as plt\n\nfor imgIdx in range(0, len(images), 2):\n    fig, axs = plt.subplots(1, 4, figsize=(10,10))\n    axs[0].imshow(images[imgIdx])\n    axs[1].imshow(rotatedImages[imgIdx])\n    axs[2].imshow(images[imgIdx + 1])\n    axs[3].imshow(rotatedImages[imgIdx + 1])\n    \n    axs[0].set_axis_off()\n    axs[1].set_axis_off()\n    axs[2].set_axis_off()\n    axs[3].set_axis_off()\n    \n    axs[0].set_title(f\"Image id: {imgIdx}\")\n    axs[1].set_title(f\"Rotated id: {imgIdx}\")\n    axs[2].set_title(f\"Image id: {imgIdx + 1}\")\n    axs[3].set_title(f\"Rotated id: {imgIdx + 1}\")\n    ","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}