{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":49349,"databundleVersionId":5447706,"sourceType":"competition"}],"dockerImageVersionId":30458,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Problem:\n\nThe goal of this competition is to reconstruct accurate 3D maps from an unstructured collection of images taken from various viewpoints, lighting conditions, and other environmental factors. The challenge lies in generating 3D reconstructions of different types of scenes and accurately pose the images, which can have wide applications in photography, cultural heritage preservation, and many services across Google.\n\n## Data:\nThe dataset consists of images taken near the same location, along with 3D reconstructions and ground truth labels for the training data. The file structure includes:\n\n - sample_submission.csv: A valid, randomly-generated sample submission with fields such as image_path, dataset, scene, rotation_matrix, and translation_vector.\n\n - [train/test]///images: A batch of images taken near the same location. Some of the training datasets may contain a folder named images_full with additional images.\n\n - train///sfm: A 3D reconstruction for this batch of images, which can be opened with COLMAP, a 3D structure-from-motion library bundled with this competition.\n\n - train///LICENSE.txt: The license for the dataset.\n\n - train/train_labels.csv: A list of images in these datasets, with ground truth information such as dataset, scene, image_path, rotation_matrix, and translation_vector.\n\nUnderstanding the data involves recognizing the variations in the images, such as different viewpoints, lighting conditions, and other environmental factors. It is crucial to preprocess the images and extract useful features to reconstruct the 3D scene effectively.\n\nBy thoroughly understanding the problem and dataset, you can better approach the competition and develop a strategy to achieve the best possible results in reconstructing accurate 3D maps from the given images.","metadata":{}},{"cell_type":"markdown","source":"## Requirements","metadata":{}},{"cell_type":"code","source":"%pip install -U albumentations\n#%pip install opensfm","metadata":{"execution":{"iopub.status.busy":"2023-04-12T21:22:53.74875Z","iopub.execute_input":"2023-04-12T21:22:53.749169Z","iopub.status.idle":"2023-04-12T21:23:03.106468Z","shell.execute_reply.started":"2023-04-12T21:22:53.749131Z","shell.execute_reply":"2023-04-12T21:23:03.105102Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport cv2\nfrom albumentations import Compose, Resize, Normalize, HorizontalFlip, RandomBrightnessContrast, MotionBlur\n#from opensfm import dataset, features, matching, reconstruction, types\n\nfrom mpl_toolkits.mplot3d import Axes3D","metadata":{"execution":{"iopub.status.busy":"2023-04-12T21:23:03.109648Z","iopub.execute_input":"2023-04-12T21:23:03.11005Z","iopub.status.idle":"2023-04-12T21:23:03.119247Z","shell.execute_reply.started":"2023-04-12T21:23:03.110014Z","shell.execute_reply":"2023-04-12T21:23:03.116531Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_images_path = \"/kaggle/input/image-matching-challenge-2023/train/\"\ntrain_labels_path = \"/kaggle/input/image-matching-challenge-2023/train/train_labels.csv\"","metadata":{"execution":{"iopub.status.busy":"2023-04-12T19:02:00.125641Z","iopub.execute_input":"2023-04-12T19:02:00.127066Z","iopub.status.idle":"2023-04-12T19:02:00.131935Z","shell.execute_reply.started":"2023-04-12T19:02:00.126997Z","shell.execute_reply":"2023-04-12T19:02:00.130864Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def load_image(image_path):\n    img = cv2.imread(image_path)\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    return img\n\ndef preprocess_image(image, img_size=(512, 512), augment=False):\n    transforms = [\n        Resize(img_size[0], img_size[1]),\n        Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0)\n    ]\n    \n    if augment:\n        transforms.extend([\n            HorizontalFlip(p=0.5),\n            RandomBrightnessContrast(brightness_limit=0.2, contrast_limit=0.2, p=0.5),\n            MotionBlur(blur_limit=7, p=0.3)\n        ])\n    \n    augmentation_pipeline = Compose(transforms)\n    preprocessed_image = augmentation_pipeline(image=image)['image']\n    \n    return np.float32(preprocessed_image)","metadata":{"execution":{"iopub.status.busy":"2023-04-12T20:59:13.690059Z","iopub.execute_input":"2023-04-12T20:59:13.690456Z","iopub.status.idle":"2023-04-12T20:59:13.699194Z","shell.execute_reply.started":"2023-04-12T20:59:13.690404Z","shell.execute_reply":"2023-04-12T20:59:13.697783Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test_image_path =\"/kaggle/input/image-matching-challenge-2023/train/haiper/bike/images/image_004.jpeg\"\nimage = load_image(test_image_path)\npreprocessed_image = preprocess_image(image, augment=False)\nplt.figure(figsize=(12, 6))\nplt.subplot(1, 2, 1)\nplt.title(\"Original Image\")\nplt.imshow(image)\n\nplt.subplot(1, 2, 2)\nplt.title(\"Preprocessed Image\")\nplt.imshow((preprocessed_image * 255).astype(np.uint8))  # Convert back to the original scale and data type\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-12T20:21:54.235646Z","iopub.execute_input":"2023-04-12T20:21:54.235976Z","iopub.status.idle":"2023-04-12T20:21:54.980779Z","shell.execute_reply.started":"2023-04-12T20:21:54.23595Z","shell.execute_reply":"2023-04-12T20:21:54.979578Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df = pd.read_csv(train_labels_path)","metadata":{"execution":{"iopub.status.busy":"2023-04-12T19:02:00.23794Z","iopub.execute_input":"2023-04-12T19:02:00.238312Z","iopub.status.idle":"2023-04-12T19:02:00.250583Z","shell.execute_reply.started":"2023-04-12T19:02:00.238278Z","shell.execute_reply":"2023-04-12T19:02:00.248865Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-04-12T19:10:18.177714Z","iopub.execute_input":"2023-04-12T19:10:18.178069Z","iopub.status.idle":"2023-04-12T19:10:18.18627Z","shell.execute_reply.started":"2023-04-12T19:10:18.178041Z","shell.execute_reply":"2023-04-12T19:10:18.185114Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df.describe()","metadata":{"execution":{"iopub.status.busy":"2023-04-12T19:10:30.657903Z","iopub.execute_input":"2023-04-12T19:10:30.658268Z","iopub.status.idle":"2023-04-12T19:10:30.844382Z","shell.execute_reply.started":"2023-04-12T19:10:30.658238Z","shell.execute_reply":"2023-04-12T19:10:30.843448Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"for col in df.scene.unique().tolist():\n    display(df[df['scene']==col].head(2))","metadata":{"execution":{"iopub.status.busy":"2023-04-12T19:02:00.379581Z","iopub.execute_input":"2023-04-12T19:02:00.379934Z","iopub.status.idle":"2023-04-12T19:02:00.441723Z","shell.execute_reply.started":"2023-04-12T19:02:00.379902Z","shell.execute_reply":"2023-04-12T19:02:00.440483Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"for col in df.dataset.unique().tolist():\n    display(df[df['dataset']==col].head(2))","metadata":{"execution":{"iopub.status.busy":"2023-04-12T19:02:00.505279Z","iopub.execute_input":"2023-04-12T19:02:00.505866Z","iopub.status.idle":"2023-04-12T19:02:00.536696Z","shell.execute_reply.started":"2023-04-12T19:02:00.505832Z","shell.execute_reply":"2023-04-12T19:02:00.535686Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def string_to_vex(rot_mat_str):\n    return np.array([list(map(float, row.split(';'))) for row in rot_mat_str.split()])\n\ndf['rotation_matrix_vex'] = df['rotation_matrix'].apply(string_to_vex)\ndf['translation_vector_vex'] = df['translation_vector'].apply(string_to_vex)\n","metadata":{"execution":{"iopub.status.busy":"2023-04-12T19:02:00.639288Z","iopub.execute_input":"2023-04-12T19:02:00.639631Z","iopub.status.idle":"2023-04-12T19:02:00.651135Z","shell.execute_reply.started":"2023-04-12T19:02:00.639605Z","shell.execute_reply":"2023-04-12T19:02:00.649663Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def plot_3D(df, col, selection=None):\n    fig = plt.figure()\n    ax = fig.add_subplot(111, projection='3d')\n\n    x = df[col].apply(lambda x: x[0][0])\n    y = df[col].apply(lambda x: x[0][1])\n    z = df[col].apply(lambda x: x[0][2])\n\n    ax.scatter(x, y, z, c='red', marker='o')\n\n    # Add features to the plot\n    ax.set_xlabel('X-axis')\n    ax.set_ylabel('Y-axis')\n    ax.set_zlabel('Z-axis')\n\n    ax.set_title('3D Scatter Plot of '+col)\n    if selection:\n        ax.set_title('3D Scatter Plot of '+col+' for '+selection)\n\n    ax.grid(color='gray', linestyle='-', linewidth=0.5, alpha=0.5)\n\n    ax.set_box_aspect([1, 1, 1])\n\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-12T19:25:23.270877Z","iopub.execute_input":"2023-04-12T19:25:23.271243Z","iopub.status.idle":"2023-04-12T19:25:23.279916Z","shell.execute_reply.started":"2023-04-12T19:25:23.271212Z","shell.execute_reply":"2023-04-12T19:25:23.278825Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"for col in df.scene.unique().tolist():\n    plot_3D(df[df['scene']==col], 'rotation_matrix_vex', col)\n    plot_3D(df[df['scene']==col], 'translation_vector_vex', col)\n","metadata":{"execution":{"iopub.status.busy":"2023-04-12T19:25:23.906351Z","iopub.execute_input":"2023-04-12T19:25:23.906783Z","iopub.status.idle":"2023-04-12T19:25:26.212031Z","shell.execute_reply.started":"2023-04-12T19:25:23.906745Z","shell.execute_reply":"2023-04-12T19:25:26.210894Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"for col in df.dataset.unique().tolist():\n    plot_3D(df[df['dataset']==col], 'rotation_matrix_vex', col)\n    plot_3D(df[df['dataset']==col], 'translation_vector_vex', col)\n","metadata":{"execution":{"iopub.status.busy":"2023-04-12T19:27:31.029905Z","iopub.execute_input":"2023-04-12T19:27:31.030464Z","iopub.status.idle":"2023-04-12T19:27:32.363168Z","shell.execute_reply.started":"2023-04-12T19:27:31.030392Z","shell.execute_reply":"2023-04-12T19:27:32.362183Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Features matching with ORB","metadata":{}},{"cell_type":"markdown","source":"ORB feature detector and descriptor extractor with the Brute-Force Matcher:","metadata":{}},{"cell_type":"code","source":"def match_all_images(images, reference_image, method='ORB', match_ratio=0.75):\n    if method == 'ORB':\n        feature_detector = cv2.ORB_create()\n    elif method == 'SIFT':\n        feature_detector = cv2.SIFT_create()\n    else:\n        raise ValueError(f\"Invalid method: {method}\")\n\n    reference_keypoints, reference_descriptors = feature_detector.detectAndCompute(reference_image, None)\n\n    if reference_descriptors is None:\n        return None, None\n\n    if reference_descriptors.dtype != np.float32:\n        reference_descriptors = reference_descriptors.astype(np.float32)\n\n    normType = cv2.NORM_HAMMING if reference_descriptors.dtype == np.uint8 else cv2.NORM_L2\n\n    matcher = cv2.BFMatcher(normType)\n\n    keypoints_list = []\n    good_matches_list = []\n\n    for img in tqdm(images):\n        keypoints, descriptors = feature_detector.detectAndCompute(img, None)\n\n        if descriptors is None:\n            keypoints_list.append(None)\n            good_matches_list.append(None)\n            continue\n\n        if descriptors.dtype != np.float32:\n            descriptors = descriptors.astype(np.float32)\n\n        matches = matcher.knnMatch(reference_descriptors, descriptors, k=2)\n\n        good_matches = []\n        for m, n in matches:\n            if m.distance < match_ratio * n.distance:\n                good_matches.append(m)\n\n        keypoints_list.append(keypoints)\n        good_matches_list.append(good_matches)\n\n    return keypoints_list, good_matches_list\n\n\ndef draw_all_matches(images, reference_image, keypoints_list, good_matches_list):\n    reference_keypoints, _ = keypoints_list[0], good_matches_list[0]\n\n    fig, axs = plt.subplots(len(images), 1, figsize=(10, 20))\n\n    for i, (img, keypoints, good_matches) in tqdm(enumerate(zip(images, keypoints_list, good_matches_list))):\n        if keypoints is None or good_matches is None:\n            continue\n\n        result = cv2.drawMatches(reference_image, reference_keypoints, img, keypoints, good_matches, None, flags=cv2.DrawMatchesFlags_NOT_DRAW_SINGLE_POINTS)\n\n        axs[i].imshow(result)\n        axs[i].set_title(f\"Matches with image {i+1}\")\n\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-12T21:10:11.148023Z","iopub.execute_input":"2023-04-12T21:10:11.148446Z","iopub.status.idle":"2023-04-12T21:10:11.163168Z","shell.execute_reply.started":"2023-04-12T21:10:11.148395Z","shell.execute_reply":"2023-04-12T21:10:11.161599Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from tqdm import tqdm","metadata":{"execution":{"iopub.status.busy":"2023-04-12T20:35:24.782947Z","iopub.execute_input":"2023-04-12T20:35:24.783287Z","iopub.status.idle":"2023-04-12T20:35:24.788617Z","shell.execute_reply.started":"2023-04-12T20:35:24.783257Z","shell.execute_reply":"2023-04-12T20:35:24.787445Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"path_cd = \"/kaggle/input/image-matching-challenge-2023/train/haiper/bike/images_full/\"\nimage_filenames = os.listdir(path_cd)\nimage_filenames.sort()\nimages = np.zeros((len(image_filenames), 1920,1440,3), dtype=np.uint8)\nfor i, filename in tqdm(enumerate(image_filenames)):\n    image_path = os.path.join(path_cd, filename)\n    image = load_image(image_path)\n    images[i] = np.array(image)","metadata":{"execution":{"iopub.status.busy":"2023-04-12T20:59:50.46251Z","iopub.execute_input":"2023-04-12T20:59:50.462892Z","iopub.status.idle":"2023-04-12T20:59:56.019334Z","shell.execute_reply.started":"2023-04-12T20:59:50.462858Z","shell.execute_reply":"2023-04-12T20:59:56.017644Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"reference_image = images[0]\n\nkeypoints_list, good_matches_list = match_all_images(images, reference_image, method='ORB', match_ratio=.75)\n\ndraw_all_matches(images, reference_image, keypoints_list, good_matches_list)","metadata":{"execution":{"iopub.status.busy":"2023-04-12T21:10:17.983112Z","iopub.execute_input":"2023-04-12T21:10:17.985121Z","iopub.status.idle":"2023-04-12T21:11:30.46776Z","shell.execute_reply.started":"2023-04-12T21:10:17.985031Z","shell.execute_reply":"2023-04-12T21:11:30.466109Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"unique_scenes = df.scene.unique().tolist()\n\nn_rows = len(unique_scenes)\nn_cols = 10\n\nfig, axes = plt.subplots(n_rows, n_cols, figsize=(n_cols * 4, n_rows * 4))\n\nfor row, scene in enumerate(unique_scenes):\n    global_paths = df[df['scene'] == scene]['image_path'].head(n_cols).tolist()\n    for col, path in enumerate(global_paths):\n        img = load_image(train_images_path + path)\n        #preprocessed_image = preprocess_image(img, augment=False)\n\n        axes[row, col].imshow(img)\n        if col == 0:\n            axes[row, col].set_ylabel(f\"{scene} scene\", fontsize=14)\n        axes[row, col].axis('off')\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-12T20:26:14.897665Z","iopub.execute_input":"2023-04-12T20:26:14.898041Z","iopub.status.idle":"2023-04-12T20:27:30.991781Z","shell.execute_reply.started":"2023-04-12T20:26:14.898006Z","shell.execute_reply":"2023-04-12T20:27:30.990741Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"unique_scenes = df.dataset.unique().tolist()\n\nn_rows = len(unique_scenes)\nn_cols = 10\n\nfig, axes = plt.subplots(n_rows, n_cols, figsize=(n_cols * 4, n_rows * 4))\n\nfor row, scene in enumerate(unique_scenes):\n    print(scene)\n    global_paths = df[df['dataset'] == scene]['image_path'].head(n_cols).tolist()\n    for col, path in enumerate(global_paths):\n        img = mpimg.imread(train_images_path + path)\n\n        axes[row, col].imshow(img)\n        if col == 0:\n            axes[row, col].set_ylabel(f\"{scene} scene\", fontsize=14)\n        axes[row, col].axis('off')\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-12T19:48:44.760836Z","iopub.execute_input":"2023-04-12T19:48:44.761274Z","iopub.status.idle":"2023-04-12T19:48:54.313764Z","shell.execute_reply.started":"2023-04-12T19:48:44.761233Z","shell.execute_reply":"2023-04-12T19:48:54.312582Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"execution":{"iopub.status.busy":"2023-04-12T19:49:27.465016Z","iopub.execute_input":"2023-04-12T19:49:27.465434Z","iopub.status.idle":"2023-04-12T19:49:27.472658Z","shell.execute_reply.started":"2023-04-12T19:49:27.465379Z","shell.execute_reply":"2023-04-12T19:49:27.471703Z"},"trusted":true},"outputs":[],"execution_count":null}]}