{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":71885,"databundleVersionId":8015523,"sourceType":"competition"}],"dockerImageVersionId":30673,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# <center style=\"font-family: consolas; font-size: 32px; font-weight: bold;\"> 📸 Image Matching Challenge - 📊 Exploratory Data Analysis</center>\n<p><center style=\"color:#949494; font-family: consolas; font-size: 20px;\">Reconstruct 3D scenes from 2D images over six different domains</center></p>\n\n***\n\n**IMPORTANT LINKS**\n\n- [CVPR 2024 Workshop Page](https://image-matching-workshop.github.io/)\n- [2023 Edition of the competition](https://www.kaggle.com/competitions/image-matching-challenge-2023): Reconstruct 3D scenes from 2D images\n- [2022 Edition of the competition](https://www.kaggle.com/competitions/image-matching-challenge-2022): Register two images from different viewpoints\n***\n\n\nThe objectives of this competition are to:\n- Construct precise 3D maps using sets of images in diverse scenarios and environments by developing **a model to generate accurate spatial representations, regardless of the source domain** \n    - The process of reconstructing a 3D model of an environment from a collection of images is called **Structure from Motion (SfM)**. \n    - These images are often captured by trained operators or with additional sensor data. \n    - This ensures homogeneous, high-quality data. \n- Explore various image sources in more realistic and applicable scenarios: images can be taken from drones 🤖, amidst dense forests 🌲, during nighttime 🌙\n    - It’s much more difficult to build 3D models from assorted images, the real-world examples that the organizers put together for this competition.\n \n\nFor this, the organizers have designated 6 categories of images with distinct challenges:\n- 🏛️ **Phototourism and historical preservation**: different viewpoints, sensor types, time of day/year, and occlusions. Ancient historical sites add a unique set of challenges\n\n- ☀️ **Night vs day and temporal changes**: combination of day and night photographs, including poor lighting, or photographs taken months or years apart, in different weather\n\n- ✈️ **Aerial and mixed aerial-ground**: images from drones, featuring arbitrary in-plane rotations, matched against similar images and also images taken from the ground\n\n- ♻️ **Repeated structures**: symmetrical objects require details to disambiguate perspective\n\n- 🌲 **Natural environments**: highly non-regular structures such as trees and foliage\n\n- 🪞 **Transparencies and reflections**: objects like glassware are lacking in texture and create reflections and specularities which pose a different set of problems\n\n\nInspired by [this notebook](https://www.kaggle.com/code/leonidkulyk/eda-imc-3d-plots-interactive-vis#notebook-container) from previous year's edition of the competition.","metadata":{}},{"cell_type":"markdown","source":"<a id=\"top\"></a>\n# Table of Contents\n\n* [0. Install & Import dependencies](#0)\n* [1. Dataset Overview](#1)\n* [2. Exploring each dataset](#2)\n    * [2.1 Church](#church)\n    * [2.2 Dioscuri](#dioscuri)\n    * [2.3 Lizard](#lizard)\n    * [2.4 Temple](#temple)\n    * [2.5 Pond](#pond)\n    * [2.6 Glass Cup](#cup)\n    * [2.7 Glass Cylinder](#cylinder)","metadata":{}},{"cell_type":"markdown","source":"<a id=\"0\"></a>\n# Install & Import dependencies","metadata":{"_kg_hide-input":false}},{"cell_type":"code","source":"!pip install -q mediapy","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-03-25T20:26:53.147918Z","iopub.execute_input":"2024-03-25T20:26:53.148692Z","iopub.status.idle":"2024-03-25T20:27:07.784926Z","shell.execute_reply.started":"2024-03-25T20:26:53.148657Z","shell.execute_reply":"2024-03-25T20:27:07.783366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%cd /kaggle/working/\n!rm -rf /kaggle/working/Hierarchical-Localization\n!git clone --quiet --recursive https://github.com/cvg/Hierarchical-Localization/\n%cd /kaggle/working/Hierarchical-Localization\n!pip install -e .\n\nfrom hloc import extract_features, match_features, reconstruction, visualization, pairs_from_exhaustive\nfrom hloc.visualization import plot_images, read_image\nfrom hloc.utils import viz_3d\n\n%cd /kaggle/working/","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-03-25T20:27:07.787614Z","iopub.execute_input":"2024-03-25T20:27:07.788003Z","iopub.status.idle":"2024-03-25T20:28:06.393693Z","shell.execute_reply.started":"2024-03-25T20:27:07.787966Z","shell.execute_reply":"2024-03-25T20:28:06.392343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from pathlib import Path\n\nimport cv2\nimport mediapy\nimport pandas as pd\nimport plotly.express as px\nimport pycolmap\n","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-03-25T20:28:06.395759Z","iopub.execute_input":"2024-03-25T20:28:06.396496Z","iopub.status.idle":"2024-03-25T20:28:06.404390Z","shell.execute_reply.started":"2024-03-25T20:28:06.396446Z","shell.execute_reply":"2024-03-25T20:28:06.403218Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"1\"></a>\n# Dataset Overview","metadata":{}},{"cell_type":"markdown","source":"- `[train/test]/*/*/images`: A batch of images all taken near the same location. Some of training datasets may also contain a folder named images_full with additional images. The published test folder comprises a subset of the church scene from train and is provided solely for example purposes. The training data usually has a sequential capture ordering and significant image-to-image content overlap while the test set has limited image-to-image overlap and the image ordering is randomized.\n\n- `train/*/*/smf`: A 3-D reconstruction for this batch of images, which can be opened with colmap, the 3-D structure-from-motion library bundled with this competition.\n\n- `train/*/*/LICENSE.txt`: The license for this dataset.\n\n- `train/train_labels.csv`: A list of images in these datasets, with ground truths.","metadata":{}},{"cell_type":"markdown","source":"### 1️⃣ Lets inspect `train/train_labels.csv`","metadata":{}},{"cell_type":"markdown","source":"- `dataset`: The unique identifier for the dataset.\n- `scene`: The unique identifier for the scene.\n- `image_path`: The image filename, including the path.\n- `rotation_matrix`: The first target column. A 3x3 matrix, flattened into a vector in row-major convention, with values separated by `;`.\n- `translation_vector`: The second target column. A 3-D dimensional vector, with values separated by ;.","metadata":{}},{"cell_type":"code","source":"train_labels = pd.read_csv(\"/kaggle/input/image-matching-challenge-2024/train/train_labels.csv\")\ntrain_labels","metadata":{"execution":{"iopub.status.busy":"2024-03-25T20:28:06.405834Z","iopub.execute_input":"2024-03-25T20:28:06.406125Z","iopub.status.idle":"2024-03-25T20:28:06.452817Z","shell.execute_reply.started":"2024-03-25T20:28:06.406101Z","shell.execute_reply":"2024-03-25T20:28:06.451685Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2️⃣ What is the relationship between datasets and scenes?","metadata":{}},{"cell_type":"code","source":"train_labels.groupby(\"dataset\")[\"scene\"].nunique()","metadata":{"execution":{"iopub.status.busy":"2024-03-25T20:28:06.456751Z","iopub.execute_input":"2024-03-25T20:28:06.457171Z","iopub.status.idle":"2024-03-25T20:28:06.468536Z","shell.execute_reply.started":"2024-03-25T20:28:06.457139Z","shell.execute_reply":"2024-03-25T20:28:06.467244Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If we take a look at the datasets and scenes, we can observe that **there is only one scene per dataset** so we can disregard that for training","metadata":{}},{"cell_type":"markdown","source":"### 3️⃣ What is the distribution of the datasets?","metadata":{}},{"cell_type":"code","source":"dataset_counts = train_labels[\"dataset\"].value_counts()\n\nfig = px.pie(values=dataset_counts.values, names=dataset_counts.index)\nfig.update_traces(textposition='inside', textfont_size=14)\nfig.update_layout(\n    title={\n        'text': \"Pie distribution of dataset images\",\n        'y':0.95,\n        'x':0.5,\n        'xanchor': 'center',\n        'yanchor': 'top'\n    },\n    legend_title_text='Dataset names:'\n)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2024-03-25T20:28:06.470194Z","iopub.execute_input":"2024-03-25T20:28:06.470534Z","iopub.status.idle":"2024-03-25T20:28:06.544455Z","shell.execute_reply.started":"2024-03-25T20:28:06.470507Z","shell.execute_reply":"2024-03-25T20:28:06.543405Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4️⃣ Let's take a closer look at the categories in the train dataset","metadata":{}},{"cell_type":"code","source":"train_categories = pd.read_csv(\"/kaggle/input/image-matching-challenge-2024/train/categories.csv\")\n\n# From comma separated list of categories for each dataset\n# To one dataset & category per row\ntrain_categories[\"category\"] = train_categories[\"categories\"].str.split(\";\")\ntrain_categories = train_categories.explode(\"category\")","metadata":{"execution":{"iopub.status.busy":"2024-03-25T20:28:06.545930Z","iopub.execute_input":"2024-03-25T20:28:06.546378Z","iopub.status.idle":"2024-03-25T20:28:06.559408Z","shell.execute_reply.started":"2024-03-25T20:28:06.546323Z","shell.execute_reply":"2024-03-25T20:28:06.558532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 5️⃣ How are the categories distributed?","metadata":{}},{"cell_type":"code","source":"category_counts = train_categories[\"category\"].value_counts()\n\nfig = px.pie(values=dataset_counts.values, names=category_counts.index)\nfig.update_traces(textposition='inside', textfont_size=14)\nfig.update_layout(\n    title={\n        'text': \"Pie distribution of categories of datasets\",\n        'y':0.95,\n        'x':0.5,\n        'xanchor': 'center',\n        'yanchor': 'top'\n    },\n    legend_title_text='Category names:'\n)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2024-03-25T20:28:06.560722Z","iopub.execute_input":"2024-03-25T20:28:06.561851Z","iopub.status.idle":"2024-03-25T20:28:06.632534Z","shell.execute_reply.started":"2024-03-25T20:28:06.561814Z","shell.execute_reply":"2024-03-25T20:28:06.631436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 6️⃣ What is the relationship between a scene  and a category? \n\n❗ Remember that scenes and datasets  are related one-to-one!","metadata":{}},{"cell_type":"code","source":"fig = px.sunburst(train_categories, path=['scene', 'category'])\nfig.update_layout(\n    title={\n        'text': \"Scene and category relation\",\n        'y':0.95,\n        'x':0.5,\n        'xanchor': 'center',\n        'yanchor': 'top'\n    }\n)\n\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2024-03-25T20:28:06.634199Z","iopub.execute_input":"2024-03-25T20:28:06.634947Z","iopub.status.idle":"2024-03-25T20:28:06.726213Z","shell.execute_reply.started":"2024-03-25T20:28:06.634903Z","shell.execute_reply":"2024-03-25T20:28:06.725041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"2\"></a>\n# Exploring each dataset","metadata":{}},{"cell_type":"code","source":"def explore(split: str, dataset: str, plot_image_limit: int = 12) -> None:\n    path = Path(\"/kaggle/input/image-matching-challenge-2024\") / split / dataset\n    images_path = path / \"images\"\n    smf_path = path / \"smf\"    \n\n    images = [cv2.cvtColor(cv2.imread(str(p)), cv2.COLOR_BGR2RGB) for p in list(images_path.glob(\"*\"))[:plot_image_limit]]\n    mediapy.show_images(images, height=300, columns=3)\n    \n    if split != \"test\":\n        rec_gt = pycolmap.Reconstruction(smf_path)\n\n        fig = viz_3d.init_figure()\n        viz_3d.plot_reconstruction(fig, rec_gt, cameras=False, color='rgba(227,168,30,0.5)', name=\"Ground Truth\", cs=5)\n        fig.show()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-03-25T20:28:52.758234Z","iopub.execute_input":"2024-03-25T20:28:52.758647Z","iopub.status.idle":"2024-03-25T20:28:52.767850Z","shell.execute_reply.started":"2024-03-25T20:28:52.758618Z","shell.execute_reply":"2024-03-25T20:28:52.766689Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"church\"></a>\n## 1️⃣ Church","metadata":{}},{"cell_type":"markdown","source":"The *Church of the Most Sacred Heart of Our Lord* is a Roman Catholic church located in  Jiřího z Poděbrad Square in Prague's Vinohrady district.\n\n- This was one of three new buildings constructed in 1929 in Prague, inspired by the 1000th anniversary of the death of St. Wenceslas.\n- It is considered one of the most significant Czech religious constructions of the 20th century.}\n- During World War II, the six bells from the tower were melted down for arms production, and in 1992, two copies were returned. Since 2010, the church has been ranked among national cultural monuments. [Click here for more information](https://en.wikipedia.org/wiki/Church_of_the_Most_Sacred_Heart_of_Our_Lord).\n\n[Explore the location on Google Maps!](https://www.google.com/maps/place/Kostel+Nejsv%C4%9Bt%C4%9Bj%C5%A1%C3%ADho+Srdce+P%C3%A1n%C4%9B/@50.0780029,14.4477159,17z/data=!3m1!4b1!4m6!3m5!1s0x470b949cd90326df:0xdf5dfb58f652dbac!8m2!3d50.0779995!4d14.4502908!16s%2Fm%2F0t524gl?authuser=0&entry=ttu)","metadata":{}},{"cell_type":"code","source":"explore(split=\"train\", dataset=\"church\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"explore(split=\"test\", dataset=\"church\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"dioscuri\"></a>\n## 2️⃣ Dioscuri","metadata":{}},{"cell_type":"markdown","source":"The *Temple of the Dioscuri* is a beautiful temple located in Agrigento, Italy.\n❗ This temple was also featured in the [2023 edition of the IMC competition](https://www.kaggle.com/competitions/image-matching-challenge-2023).\n\n- It was built in the middle of the 5th century BCE. The preserved four columns prove that they were made in the Doric order. The building had six columns on both sides; of the other two, thirteen.\n- Dioscuri were twins of divine origin who were worshipped in ancient Greece and Rome. According to Greek mythology, they took part in the Argonaut’s expedition, and after their death, Zeus (Jupiter) placed them in the sky as a constellation of Twins.\n- The temple is located in the so-called The Valley of the Temples in the central part of Sicily, in Agrigento (Roman Agrigentum, Greek Akragas).\n\n[Explore the location on Google Maps!](https://www.google.com/maps/place/Temple+of+the+Dioscuri/@37.2913186,13.5815322,15z/data=!4m6!3m5!1s0x13108230e21d2e2f:0x9f9aa044be1dff05!8m2!3d37.2913186!4d13.5815322!16s%2Fg%2F1234zsmq?hl=en&entry=ttu)","metadata":{}},{"cell_type":"code","source":"explore(split=\"train\", dataset=\"dioscuri\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"lizard\"></a>\n## 3️⃣ Lizard","metadata":{}},{"cell_type":"markdown","source":"This lizard appears to be in the *Rangherka & Herold Park*, situated in the heart of Vršovice district in Prague, Czech Republic.  \n- The park is named after the Italian businessman Rangheri, who planted mulberry orchards here and founded Prague’s silk industry.\n- The Chateau in the park was also named after him. Nowadays there is a retirement home and a ceremonial hall.\n- This cute lizard is wandering around the Herold orchards (Sluneční hodiny). \n\n[Explore the location on Google Maps!](https://www.google.com/maps/place/Slune%C4%8Dn%C3%AD+hodiny/@50.069777,14.4531037,19.42z/data=!4m15!1m8!3m7!1s0x470b937e6803b4ed:0xb9e4c6c93639025c!2sHerold+orchards!8m2!3d50.0702835!4d14.4530186!10e5!16s%2Fg%2F122kj785!3m5!1s0x470b9300e8f5e8cd:0x82fd9a75524f3758!8m2!3d50.0697772!4d14.4531745!16s%2Fg%2F11lgky0py0?authuser=0&entry=ttu)","metadata":{}},{"cell_type":"code","source":"explore(split=\"train\", dataset=\"lizard\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"explore(split=\"test\", dataset=\"lizard\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"temple\"></a>\n## 4️⃣ Temple","metadata":{}},{"cell_type":"markdown","source":"The *Temple of Baal-Shamin* is an ancient temple located in Palmyra, Syria. \n\n- It was dedicated to the Canaanite sky deity Baalshamin, dating back to the late 2nd century BC. - Its altar was built in 115 AD, and the temple was substantially rebuilt in 131 AD.\n- It was one of the most complete ancient structures in Palmyra.[4] In 1980, UNESCO designated the temple as a World Heritage Site.\n- In 2015, the Islamic State of Iraq and the Levant demolished the Temple of Baalshamin after capturing Palmyra during the Syrian Civil War. [Click here for more information](https://en.wikipedia.org/wiki/Temple_of_Bel).\n\n[Explore the location on Google Maps!](https://www.google.com/maps/place/Temple+of+Bel-Shamin/@34.5533038,38.2673928,17z/data=!3m1!4b1!4m6!3m5!1s0x153c75a6b303eb01:0x1eba14a148d2bb48!8m2!3d34.5532994!4d38.2699677!16s%2Fg%2F11bw223lsn?authuser=0&entry=ttu)","metadata":{}},{"cell_type":"code","source":"explore(split=\"train\", dataset=\"multi-temporal-temple-baalshamin\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"pond\"></a>\n## 5️⃣ Pond","metadata":{}},{"cell_type":"markdown","source":"This appears to be a small pond or lake in *Grebovka Park and the Havlíček Gardens* in Prague, Czech Republic.\n\n- Grebovka Park is a large park (nearly 11 acres) which was established in 1871-1888 as a part of the Grobov Villa. \n- The park includes a beautiful pavillion, originally used for indoor games, including billiard, balling archery or chess.\n- Moreover, there is a vinyeard, a villa, a wine cellar, as well as various buildings where many events are held\n\n[Explore the location on Google Maps!](https://www.google.com/maps/place/Jez%C3%ADrko+Grebovka/@50.0695595,14.4400903,17z/data=!3m1!4b1!4m6!3m5!1s0x470b959427db8961:0xda0473ddae926d35!8m2!3d50.0695561!4d14.4426652!16s%2Fg%2F11sjzb3hg3?authuser=0&entry=ttu)\n\n⚠️ The 3D plot might be not work here (work in progress!).","metadata":{}},{"cell_type":"code","source":"explore(split=\"train\", dataset=\"pond\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"cup\"></a>\n## 6️⃣ Glass Cup","metadata":{}},{"cell_type":"markdown","source":"Not much to say here, this appears to be the upper half of a green-ish transparent glass cup.","metadata":{}},{"cell_type":"code","source":"explore(split=\"train\", dataset=\"transp_obj_glass_cup\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"cylinder\"></a>\n## 7️⃣ Glass Cylinder","metadata":{}},{"cell_type":"markdown","source":"Again not much here. It looks like a white-ish transparent glass cylinder, similar to a test tube like the ones that could be found in a chemical testing laboratory.","metadata":{}},{"cell_type":"code","source":"explore(split=\"train\", dataset=\"transp_obj_glass_cylinder\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Work in progress!","metadata":{}},{"cell_type":"markdown","source":"❤️ Thank you for taking the time to read through my notebook. I hope you found it interesting and informative. If you have any feedback or suggestions for improvement, please don't hesitate to let me know in the comments.\n\n🚀 If you liked this notebook, please consider upvoting it so that others can discover it too. Your support means a lot to me, and it helps to motivate me to create more notebooks like this one in future.","metadata":{}}]}