","metadata":{}},{"cell_type":"markdown","source":"# **Introduction**\n\nThe Image Matching Challenge 2023 competition advances 3D map reconstruction using multiple perspectives, aiming to reconstruct entire scenes. It improves world mapping with diverse data sources, including user-uploaded images on platforms like Google Maps. By combining photos from different individuals, a more immersive three-dimensional view is created. Machine learning extracts insights from unstructured online image collections.\n\nReconstructing 3D models from diverse images, encompassing various viewpoints and conditions, is the challenge. Google, a mapping leader, employs similar techniques in services like Google Maps. Collaborating with Haiper and Kaggle, this competition accelerates research and leverages publicly available data.\n\nParticipating in this competition contributes to accurate 3D models with impacts in photography, cultural heritage preservation, and Google services.","metadata":{}},{"cell_type":"markdown","source":"# **Import Modules**","metadata":{}},{"cell_type":"code","source":"import os\nimport pandas as pd\nimport numpy as np\nimport cv2\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\n%matplotlib inline\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","metadata":{"execution":{"iopub.status.busy":"2023-05-25T02:11:21.992328Z","iopub.execute_input":"2023-05-25T02:11:21.992829Z","iopub.status.idle":"2023-05-25T02:11:22.257011Z","shell.execute_reply.started":"2023-05-25T02:11:21.992790Z","shell.execute_reply":"2023-05-25T02:11:22.255725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%capture\n!pip install opencv-contrib-python","metadata":{"execution":{"iopub.status.busy":"2023-05-25T02:11:29.222981Z","iopub.execute_input":"2023-05-25T02:11:29.223439Z","iopub.status.idle":"2023-05-25T02:11:43.951338Z","shell.execute_reply.started":"2023-05-25T02:11:29.223405Z","shell.execute_reply":"2023-05-25T02:11:43.949658Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Load the dataset**","metadata":{}},{"cell_type":"code","source":"train_labels = pd.read_csv('/kaggle/input/image-matching-challenge-2023/train/train_labels.csv')\ndisplay(train_labels[0:4])","metadata":{"execution":{"iopub.status.busy":"2023-05-25T02:12:50.156885Z","iopub.execute_input":"2023-05-25T02:12:50.157924Z","iopub.status.idle":"2023-05-25T02:12:50.217994Z","shell.execute_reply.started":"2023-05-25T02:12:50.157875Z","shell.execute_reply":"2023-05-25T02:12:50.216835Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission_data = pd.read_csv('/kaggle/input/image-matching-challenge-2023/sample_submission.csv')\nsubmission_data.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-25T02:12:55.508761Z","iopub.execute_input":"2023-05-25T02:12:55.509199Z","iopub.status.idle":"2023-05-25T02:12:55.534095Z","shell.execute_reply.started":"2023-05-25T02:12:55.509165Z","shell.execute_reply":"2023-05-25T02:12:55.532933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Process the dataset**","metadata":{}},{"cell_type":"code","source":"data = train_labels[train_labels['dataset'] =='heritage'][train_labels['scene']== 'dioscuri']\ndisplay(data.columns.tolist())","metadata":{"execution":{"iopub.status.busy":"2023-05-25T02:12:59.805151Z","iopub.execute_input":"2023-05-25T02:12:59.805536Z","iopub.status.idle":"2023-05-25T02:12:59.821323Z","shell.execute_reply.started":"2023-05-25T02:12:59.805509Z","shell.execute_reply":"2023-05-25T02:12:59.820099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"paths = []\nfiles = []\nfor dirname, _, filenames in os.walk('/kaggle/input/image-matching-challenge-2023/train/heritage/dioscuri/images'):\n for filename in filenames:\n paths+= [(os.path.join(dirname, filename))]\n files+= [filename]\n\nfig, axs = plt.subplots(5,6, figsize=(12,12))\nfor i, ax in enumerate(axs.flat):\n if i < len(paths):\n img = mpimg.imread(paths[i])\n ax.imshow(img)\n ax.axis('off')\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-25T02:13:05.549304Z","iopub.execute_input":"2023-05-25T02:13:05.549704Z","iopub.status.idle":"2023-05-25T02:13:13.042528Z","shell.execute_reply.started":"2023-05-25T02:13:05.549675Z","shell.execute_reply":"2023-05-25T02:13:13.041324Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Process the dataset\nrotation_matrices = []\ntranslation_vectors = []\n\nfor i in range(len(data)):\n # Extract rotation matrix and translation vector\n rotation_matrix = np.array([float(val) for val in data.iloc[i]['rotation_matrix'].split(';')])\n translation_vector = np.array([float(val) for val in data.iloc[i]['translation_vector'].split(';')])\n \n # Append to the lists\n rotation_matrices.append(rotation_matrix)\n translation_vectors.append(translation_vector)\n\n# Perform SfM reconstruction\n# ... (Perform the desired SfM steps using the rotation_matrices and translation_vectors)\n\n# Print the reconstructed 3D model or perform further analysis\n# ... (Print or analyze the reconstructed 3D model)\n\n# Print the rotation matrices and translation vectors\nfor i in range(len(rotation_matrices)):\n print(\"Rotation Matrix {}:\".format(i))\n print(rotation_matrices[i])\n print(\"Translation Vector {}:\".format(i))\n print(translation_vectors[i])\n print()\n\n# Calculate the number of images and scenes in the dataset\nnum_images = len(data)\nnum_scenes = len(data['scene'].unique())\nprint(\"Number of Images:\", num_images)\nprint(\"Number of Scenes:\", num_scenes)\n","metadata":{"_kg_hide-output":false,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-05-25T02:13:20.621344Z","iopub.execute_input":"2023-05-25T02:13:20.621843Z","iopub.status.idle":"2023-05-25T02:13:20.725304Z","shell.execute_reply.started":"2023-05-25T02:13:20.621807Z","shell.execute_reply":"2023-05-25T02:13:20.724065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_labels.info()","metadata":{"execution":{"iopub.status.busy":"2023-05-25T02:14:31.606279Z","iopub.execute_input":"2023-05-25T02:14:31.607568Z","iopub.status.idle":"2023-05-25T02:14:31.627204Z","shell.execute_reply.started":"2023-05-25T02:14:31.607515Z","shell.execute_reply":"2023-05-25T02:14:31.626311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(data.shape)","metadata":{"execution":{"iopub.status.busy":"2023-05-25T02:14:35.890553Z","iopub.execute_input":"2023-05-25T02:14:35.890950Z","iopub.status.idle":"2023-05-25T02:14:35.897326Z","shell.execute_reply.started":"2023-05-25T02:14:35.890916Z","shell.execute_reply":"2023-05-25T02:14:35.895925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dir1 = '/kaggle/input/image-matching-challenge-2023/train/'\nprint(data.iloc[0,2]) # image path\nprint(data.iloc[0,3]) # 'rotation_matrix'\nprint(data.iloc[0,4]) # 'translation_vector'\n\npath1 = dir1 + data.iloc[0, 2]\nimg = mpimg.imread(path1)\nplt.imshow(img)\nplt.axis('off')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-25T02:14:39.525579Z","iopub.execute_input":"2023-05-25T02:14:39.526013Z","iopub.status.idle":"2023-05-25T02:14:39.905163Z","shell.execute_reply.started":"2023-05-25T02:14:39.525981Z","shell.execute_reply":"2023-05-25T02:14:39.904014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load the image\nimage1 = cv2.imread('/kaggle/input/image-matching-challenge-2023/train/heritage/dioscuri/images/archive_0057.png')\n# Convert the training image to RGB\ntraining_image = cv2.cvtColor(image1, cv2.COLOR_BGR2RGB)\n\n# Convert the training image to gray scale\ntraining_gray = cv2.cvtColor(training_image, cv2.COLOR_RGB2GRAY)\n\n# Create test image by adding Scale Invariance and Rotational Invariance\ntest_image = cv2.pyrDown(training_image)\ntest_image = cv2.pyrDown(test_image)\nnum_rows, num_cols = test_image.shape[:2]\n\nrotation_matrix = cv2.getRotationMatrix2D((num_cols/2, num_rows/2), 30, 1)\ntest_image = cv2.warpAffine(test_image, rotation_matrix, (num_cols, num_rows))\n\ntest_gray = cv2.cvtColor(test_image, cv2.COLOR_RGB2GRAY)\n\n# Display traning image and testing image\nfx, plots = plt.subplots(1, 2, figsize=(20,10))\n\nplots[0].set_title(\"Training Image\")\nplots[0].imshow(training_image)\n\nplots[1].set_title(\"Testing Image\")\nplots[1].imshow(test_image)","metadata":{"execution":{"iopub.status.busy":"2023-05-25T02:14:45.685148Z","iopub.execute_input":"2023-05-25T02:14:45.685846Z","iopub.status.idle":"2023-05-25T02:14:46.773772Z","shell.execute_reply.started":"2023-05-25T02:14:45.685809Z","shell.execute_reply":"2023-05-25T02:14:46.772519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Detect keypoints and Create Descriptor**\n\nIn computer vision and image processing, the process of detecting keypoints and creating descriptors is a fundamental step in many feature-based algorithms for tasks such as image matching, object recognition, and tracking.\n\nHere's an explanation of the concepts:\n\n* **Keypoint Detection:** Keypoint detection involves identifying distinctive and informative points or regions in an image that can be used to describe its content. These keypoints are locations in the image that are robust to changes in scale, rotation, illumination, and viewpoint. Keypoints are typically detected based on certain characteristics, such as corners, blobs, or other salient image structures. Various algorithms exist for keypoint detection, such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), and ORB (Oriented FAST and Rotated BRIEF).\n\n* **Descriptor Creation:** Once keypoints are detected, descriptors are computed to capture the local image information around each keypoint. Descriptors encode the appearance or characteristics of the image patch surrounding the keypoint. They provide a compact and distinctive representation of the local image region, which can be used for matching keypoints across different images or for other tasks like object recognition. Descriptors are typically vectors or feature descriptors, where each element represents a certain attribute or property of the image patch. Common descriptor algorithms include SIFT, SURF, ORB, and BRIEF (Binary Robust Independent Elementary Features).\n\nThe combination of keypoints and descriptors allows for efficient and robust image matching, as keypoints provide the locations of distinctive image features, while descriptors encode the information necessary to compare and match those features across different images. These keypoints and descriptors can be used in algorithms like feature matching, object detection, image registration, and more.\n\nIt's important to note that different algorithms may have varying approaches and characteristics for keypoint detection and descriptor creation. Each algorithm has its own strengths, limitations, and computational considerations, making it suitable for different applications and scenarios.","metadata":{}},{"cell_type":"code","source":"sift = cv2.SIFT_create()\n\ntrain_keypoints, train_descriptor = sift.detectAndCompute(training_gray, None)\ntest_keypoints, test_descriptor = sift.detectAndCompute(test_gray, None)\n\nkeypoints_without_size = np.copy(training_image)\nkeypoints_with_size = np.copy(training_image)\n\ncv2.drawKeypoints(training_image, train_keypoints, keypoints_without_size, color = (0, 255, 0))\n\ncv2.drawKeypoints(training_image, train_keypoints, keypoints_with_size, flags = cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS)\n\n# Display image with and without keypoints size\nfx, plots = plt.subplots(1, 2, figsize=(20,10))\n\nplots[0].set_title(\"Train keypoints With Size\")\nplots[0].imshow(keypoints_with_size, cmap='gray')\n\nplots[1].set_title(\"Train keypoints Without Size\")\nplots[1].imshow(keypoints_without_size, cmap='gray')\n\n# Print the number of keypoints detected in the training image\nprint(\"Number of Keypoints Detected In The Training Image: \", len(train_keypoints))\n\n# Print the number of keypoints detected in the query image\nprint(\"Number of Keypoints Detected In The Query Image: \", len(test_keypoints))","metadata":{"execution":{"iopub.status.busy":"2023-05-25T02:14:56.994907Z","iopub.execute_input":"2023-05-25T02:14:56.995329Z","iopub.status.idle":"2023-05-25T02:14:58.736020Z","shell.execute_reply.started":"2023-05-25T02:14:56.995298Z","shell.execute_reply":"2023-05-25T02:14:58.734229Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Matching Keypoints**\n\nMatching keypoints refers to the process of finding corresponding keypoints between two images. The matching process is performed using the Brute-Force Matcher algorithm.\n\n* **Create a Brute Force Matcher:** The code initializes a Brute Force Matcher object (`bf`) using `cv2.BFMatcher`. The Brute-Force Matcher compares each descriptor from the training image with all descriptors from the test image.\n\n* **Perform Matching:** The `bf.match()` function is used to perform the actual matching. It takes the descriptors of the training image (`train_descriptor`) and the test image (`test_descriptor`) as inputs and returns a list of matches.\n\n* **Sorting Matches:** The matches obtained from the previous step are sorted based on the distance between the descriptors. Matches with shorter distances are considered better matches. The `sorted()` function is used to sort the matches list based on the `distance` attribute of each match.\n\n* **Visualize Matches:** The code then draws the best matching points on a new image (`result`) using `cv2.drawMatches()`. It takes the training image (`training_image`), the keypoints of the training image (`train_keypoints`), the test image (`test_gray`), the keypoints of the test image (`test_keypoints`), and the sorted matches as inputs.\n\n* **Display Matching Points:** The resulting image (`result`) with the best matching points is displayed using `plt.imshow()`.\n\n* **Print Total Number of Matches:** Finally, to prints the total number of matching keypoints between the training and test images using len(`matches`).\n\nThe matching process allows you to establish correspondences between keypoints in different images, which is useful for tasks such as image alignment, object recognition, and image stitching. By identifying matching keypoints, you can determine the similarity or overlap between images and extract valuable information for further analysis or applications.","metadata":{}},{"cell_type":"code","source":"# Create a Brute Force Matcher object.\nbf = cv2.BFMatcher(cv2.NORM_L1, crossCheck = False)\n\n# Perform the matching between the SIFT descriptors of the training image and the test image\nmatches = bf.match(train_descriptor, test_descriptor)\n\n# The matches with shorter distance are the ones we want.\nmatches = sorted(matches, key = lambda x : x.distance)\n\nresult = cv2.drawMatches(training_image, train_keypoints, test_gray, test_keypoints, matches, test_gray, flags = 2)\n\n# Display the best matching points\nplt.rcParams['figure.figsize'] = [14.0, 7.0]\nplt.title('Best Matching Points')\nplt.imshow(result)\nplt.show()\n\n# Print total number of matching points between the training and query images\nprint(\"\\nNumber of Matching Keypoints Between The Training and Query Images: \", len(matches))","metadata":{"execution":{"iopub.status.busy":"2023-05-25T02:15:06.415135Z","iopub.execute_input":"2023-05-25T02:15:06.415579Z","iopub.status.idle":"2023-05-25T02:15:07.123377Z","shell.execute_reply.started":"2023-05-25T02:15:06.415544Z","shell.execute_reply":"2023-05-25T02:15:07.122146Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"matrixs = data.iloc[0,3].split(';')\nM = []\nfor m in matrixs:\n M += [float(m)]\n\nrotation_matrix = np.array(M).reshape(3,3)\nprint(rotation_matrix)\n","metadata":{"execution":{"iopub.status.busy":"2023-05-25T02:15:18.063683Z","iopub.execute_input":"2023-05-25T02:15:18.064767Z","iopub.status.idle":"2023-05-25T02:15:18.073189Z","shell.execute_reply.started":"2023-05-25T02:15:18.064716Z","shell.execute_reply":"2023-05-25T02:15:18.071696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import math\n\nR = rotation_matrix\ntheta_x = math.atan2(R[2,1],R[2,2])\ntheta_y = math.asin(-R[2,0])\ntheta_z = math.atan2(R[1,0], R[0,0])\nprint(theta_x,theta_y,theta_z)","metadata":{"execution":{"iopub.status.busy":"2023-05-25T02:15:24.099481Z","iopub.execute_input":"2023-05-25T02:15:24.100635Z","iopub.status.idle":"2023-05-25T02:15:24.107172Z","shell.execute_reply.started":"2023-05-25T02:15:24.100595Z","shell.execute_reply":"2023-05-25T02:15:24.106336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"matrix = rotation_matrix\nis_square = matrix.shape[0] == matrix.shape[1]\nis_orthogonal = np.allclose(np.dot(matrix, matrix.T), np.eye(3)) and np.allclose(np.linalg.norm(matrix, axis=0), 1)\nis_determinant_one = np.isclose(np.linalg.det(matrix), 1)\nis_rotation_matrix = is_square and is_orthogonal and is_determinant_one\nprint(\"is_rotation_matrix:\", is_rotation_matrix)\n","metadata":{"execution":{"iopub.status.busy":"2023-05-25T02:15:28.121041Z","iopub.execute_input":"2023-05-25T02:15:28.121447Z","iopub.status.idle":"2023-05-25T02:15:28.131352Z","shell.execute_reply.started":"2023-05-25T02:15:28.121416Z","shell.execute_reply":"2023-05-25T02:15:28.130038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vectors = data.iloc[0,4].split(';')\nV = []\nfor v in vectors:\n V += [float(v)]\n\ntranslation_vector = np.array(V)\nprint(translation_vector)\n","metadata":{"execution":{"iopub.status.busy":"2023-05-25T02:15:32.845044Z","iopub.execute_input":"2023-05-25T02:15:32.845455Z","iopub.status.idle":"2023-05-25T02:15:32.853750Z","shell.execute_reply.started":"2023-05-25T02:15:32.845422Z","shell.execute_reply":"2023-05-25T02:15:32.852257Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Create the thresholds for rotation and translation:","metadata":{}},{"cell_type":"code","source":"thresholds_r = np.linspace(1, 10, 10) # In degrees\nthresholds_t = np.geomspace(0.2, 5, 10) # In meters\n","metadata":{"execution":{"iopub.status.busy":"2023-05-25T02:15:36.804176Z","iopub.execute_input":"2023-05-25T02:15:36.804585Z","iopub.status.idle":"2023-05-25T02:15:36.810310Z","shell.execute_reply.started":"2023-05-25T02:15:36.804552Z","shell.execute_reply":"2023-05-25T02:15:36.809467Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Calculate the accuracy for each pair of poses:","metadata":{}},{"cell_type":"code","source":"# Calculate accuracy for each pair of poses\naccurate_samples = []\n\nfor i in range(len(rotation_matrices)):\n for j in range(i + 1, len(rotation_matrices)):\n # Get the rotation matrices for pair (i, j)\n rotation_matrix_i = rotation_matrices[i]\n rotation_matrix_j = rotation_matrices[j]\n \n # Check if the rotation matrices are valid\n if rotation_matrix_i.shape != (3, 3) or rotation_matrix_j.shape != (3, 3):\n print(\"Invalid rotation matrix shape for pair ({} , {})\".format(i, j))\n continue\n \n is_square_i = rotation_matrix_i.shape[0] == rotation_matrix_i.shape[1]\n is_orthogonal_i = np.allclose(np.dot(rotation_matrix_i, rotation_matrix_i.T), np.eye(3)) and np.allclose(np.linalg.norm(rotation_matrix_i, axis=0), 1)\n is_determinant_one_i = np.isclose(np.linalg.det(rotation_matrix_i), 1)\n is_rotation_matrix_i = is_square_i and is_orthogonal_i and is_determinant_one_i\n \n is_square_j = rotation_matrix_j.shape[0] == rotation_matrix_j.shape[1]\n is_orthogonal_j = np.allclose(np.dot(rotation_matrix_j, rotation_matrix_j.T), np.eye(3)) and np.allclose(np.linalg.norm(rotation_matrix_j, axis=0), 1)\n is_determinant_one_j = np.isclose(np.linalg.det(rotation_matrix_j), 1)\n is_rotation_matrix_j = is_square_j and is_orthogonal_j and is_determinant_one_j\n \n # Check if both rotation matrices are accurate\n if is_rotation_matrix_i and is_rotation_matrix_j:\n accurate_samples.append((i, j))\n \n# Calculate accuracy percentage\nif len(rotation_matrices) > 1:\n accuracy_percentage = len(accurate_samples) / (len(rotation_matrices) * (len(rotation_matrices) - 1) / 2) * 100\nelse:\n accuracy_percentage = 0.0\n\nprint(\"Accuracy Percentage: {:.2f}%\".format(accuracy_percentage))\n","metadata":{"_kg_hide-output":false,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-05-25T02:15:45.446898Z","iopub.execute_input":"2023-05-25T02:15:45.447930Z","iopub.status.idle":"2023-05-25T02:15:45.589760Z","shell.execute_reply.started":"2023-05-25T02:15:45.447892Z","shell.execute_reply":"2023-05-25T02:15:45.588644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Calculate the mean Average Accuracy (mAA):\n","metadata":{}},{"cell_type":"code","source":"mAA = np.mean(accurate_samples)","metadata":{"execution":{"iopub.status.busy":"2023-05-25T02:16:46.949009Z","iopub.execute_input":"2023-05-25T02:16:46.949438Z","iopub.status.idle":"2023-05-25T02:16:46.955604Z","shell.execute_reply.started":"2023-05-25T02:16:46.949405Z","shell.execute_reply":"2023-05-25T02:16:46.954421Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Submission**","metadata":{}},{"cell_type":"code","source":"submission_data.to_csv('submission.csv', index=False)\nsubmission_data","metadata":{"execution":{"iopub.status.busy":"2023-05-25T02:16:51.214182Z","iopub.execute_input":"2023-05-25T02:16:51.214587Z","iopub.status.idle":"2023-05-25T02:16:51.232903Z","shell.execute_reply.started":"2023-05-25T02:16:51.214557Z","shell.execute_reply":"2023-05-25T02:16:51.231583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"
\n\n
\nYour upvote is a great way to show your support and help others discover this valuable resource.","metadata":{}},{"cell_type":"markdown","source":"
📌 Note: If you forks my notebook, please don't forget to upvote it.