\nMy code is partly a combination of the baseline provided by the host and the notebook by @chankhavu at IMC 2022. I extend my appreciation to both for providing the codes.
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9249230%2F1036693590b6655d8d48d588bfd69e9c%2FIMC-solution.png?generation=1686658721858306&alt=media\")
(June 21st: Added additional explanation.)
\nFirstly, I'd like to express my gratitude to the hosts and Kaggle staff for conducting the IMC 2023 competition. The task was both exciting and challenging, which made it a pleasure to engage with over the two months.
\nSuperPoint/SuperGlue proved to be exceptionally accurate and quick.
\nMy code is partly a combination of the baseline provided by the host and the notebook by @chankhavu at IMC 2022. I extend my appreciation to both for providing the codes.
\n
Considering the large number of image combinations, an effective screening method was necessary. I noticed that the number of matches made by SG is significantly low (<10) when an image pair is unsuitable for stereo matching. Consequently, I decided to bypass the process if the number of matches achieved by SG (longside = 1200) fell below a certain threshold (in this case, 30 matches). This strategy significantly reduced processing time, allowing for more pair trials within the given timeframe, leading to a noticeable improvement (LB: +0.08).
\nProcuring meaningful matches from pairs with unsorted image orientations, such as those found in Cyprus, proved to be challenging. Therefore, I incorporated a rotation process into the screening procedure, resulting in further improvement (LB: +0.04).
\nEach image was divided into four sections, each generating its own set of keypoints, followed by the execution of matchings across all pair combinations (4x4 = 16 pairs) with a batched process for SP/SG (longside = 1400).
\nWith image splitting, the number increased to almost 3 times larger than the single one as shown below,
\nactually, which does not depend on the original size of the image.
\nJust increasing the input size of an image cannot achieve this.
\nI obtained similar benefits at the other scenes.
\nThis method proved to be more effective and time-efficient than traditional TTA in my case (LB: +0.01~0.02).
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9249230%2F59ee7b75b96f413775ba3f8f7e6fc6c0%2Fsplit_2.png?generation=1687299587316264&alt=media\")
After comparing various models, DKM v3 emerged as a relatively lightweight and effective choice when used in conjunction with SG (LB: +0.01~0.04).
\nSuperGlue could not create correct matches at the stair in the image pair below (see the yellow region), while it provided good matches at the other objects such as the arch and pillar.
\nOn the other hand, DKM can detect the correct corresponding points of the stair, which suggests that these matchers are complementary to each other.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9249230%2F4a040a43881adc0532ccc0e5b4878c17%2FDKM_2.png?generation=1687299538792519&alt=media\")
Both matching and mapping/reconstruction are time-intensive tasks. However, the former utilizes both the GPU and a single CPU, while the latter only requires CPU resources. Therefore, implementing parallel processing using the queue library improved time efficiency by approximately 20~30%. This concept was inspired by a gold-prize solution at IMC 2022.
\nRemember to set mapper_options.num_threads = 1, which can also help avoid OOM during reconstruction.
urban / kyiv-puppet-theater (26 images, 325 pairs) -> mAA=0.921538, mAA_q=0.987385, mAA_t=0.921538
\nurban -> mAA=0.921538
heritage / dioscuri (174 images, 15051 pairs) -> mAA=0.594950, mAA_q=0.689662, mAA_t=0.602279
\nheritage / cyprus (30 images, 435 pairs) -> mAA=0.706437, mAA_q=0.727126, mAA_t=0.724828
\nheritage / wall (43 images, 903 pairs) -> mAA=0.805980, mAA_q=0.935105, mAA_t=0.824917
\nheritage -> mAA=0.702456
haiper / bike (15 images, 105 pairs) -> mAA=0.933333, mAA_q=0.999048, mAA_t=0.933333
\nhaiper / chairs (16 images, 120 pairs) -> mAA=0.981667, mAA_q=0.999167, mAA_t=0.981667
\nhaiper / fountain (23 images, 253 pairs) -> mAA=0.999605, mAA_q=1.000000, mAA_t=0.999605
\nhaiper -> mAA=0.971535
Final metric -> mAA=0.865176
\nPublic LB: 0.471
\nPrivate LB: 0.534
It should be noted that the submission showed best the highest local and public score, and also resulted in the best private score among my submissions. While I struggled with the randomness of Colmap, I now recognize that the dataset was useful and served as a valuable reference in aiming for the correct goal.
\n(June 21st: Added additional explanation.)
\nFirstly, I'd like to express my gratitude to the hosts and Kaggle staff for conducting the IMC 2023 competition. The task was both exciting and challenging, which made it a pleasure to engage with over the two months.
\nSuperPoint/SuperGlue proved to be exceptionally accurate and quick.
\nMy code is partly a combination of the baseline provided by the host and the notebook by @chankhavu at IMC 2022. I extend my appreciation to both for providing the codes.
\n
Considering the large number of image combinations, an effective screening method was necessary. I noticed that the number of matches made by SG is significantly low (<10) when an image pair is unsuitable for stereo matching. Consequently, I decided to bypass the process if the number of matches achieved by SG (longside = 1200) fell below a certain threshold (in this case, 30 matches). This strategy significantly reduced processing time, allowing for more pair trials within the given timeframe, leading to a noticeable improvement (LB: +0.08).
\nProcuring meaningful matches from pairs with unsorted image orientations, such as those found in Cyprus, proved to be challenging. Therefore, I incorporated a rotation process into the screening procedure, resulting in further improvement (LB: +0.04).
\nEach image was divided into four sections, each generating its own set of keypoints, followed by the execution of matchings across all pair combinations (4x4 = 16 pairs) with a batched process for SP/SG (longside = 1400).
\nWith image splitting, the number increased to almost 3 times larger than the single one as shown below,
\nactually, which does not depend on the original size of the image.
\nJust increasing the input size of an image cannot achieve this.
\nI obtained similar benefits at the other scenes.
\nThis method proved to be more effective and time-efficient than traditional TTA in my case (LB: +0.01~0.02).
\n
After comparing various models, DKM v3 emerged as a relatively lightweight and effective choice when used in conjunction with SG (LB: +0.01~0.04).
\nSuperGlue could not create correct matches at the stair in the image pair below (see the yellow region), while it provided good matches at the other objects such as the arch and pillar.
\nOn the other hand, DKM can detect the correct corresponding points of the stair, which suggests that these matchers are complementary to each other.
\n
Both matching and mapping/reconstruction are time-intensive tasks. However, the former utilizes both the GPU and a single CPU, while the latter only requires CPU resources. Therefore, implementing parallel processing using the queue library improved time efficiency by approximately 20~30%. This concept was inspired by a gold-prize solution at IMC 2022.
\nRemember to set mapper_options.num_threads = 1, which can also help avoid OOM during reconstruction.
urban / kyiv-puppet-theater (26 images, 325 pairs) -> mAA=0.921538, mAA_q=0.987385, mAA_t=0.921538
\nurban -> mAA=0.921538
heritage / dioscuri (174 images, 15051 pairs) -> mAA=0.594950, mAA_q=0.689662, mAA_t=0.602279
\nheritage / cyprus (30 images, 435 pairs) -> mAA=0.706437, mAA_q=0.727126, mAA_t=0.724828
\nheritage / wall (43 images, 903 pairs) -> mAA=0.805980, mAA_q=0.935105, mAA_t=0.824917
\nheritage -> mAA=0.702456
haiper / bike (15 images, 105 pairs) -> mAA=0.933333, mAA_q=0.999048, mAA_t=0.933333
\nhaiper / chairs (16 images, 120 pairs) -> mAA=0.981667, mAA_q=0.999167, mAA_t=0.981667
\nhaiper / fountain (23 images, 253 pairs) -> mAA=0.999605, mAA_q=1.000000, mAA_t=0.999605
\nhaiper -> mAA=0.971535
Final metric -> mAA=0.865176
\nPublic LB: 0.471
\nPrivate LB: 0.534
It should be noted that the submission showed best the highest local and public score, and also resulted in the best private score among my submissions. While I struggled with the randomness of Colmap, I now recognize that the dataset was useful and served as a valuable reference in aiming for the correct goal.
\nGreat writeup, and congrats! Even more impressive because that is solo!
\nIt is interesting that DKM worked for you, despite other participants failed to make it good.
\nTraditional question - do you have some table with numbers for the \"Other models such as LoFTR, SE2-LoFTR, disk, QuadTreeAttention, OpenGlue, and SILK \"?
The reason I am inviting on this question is that making a reality check for the new (and older) methods is one of the competition goals.
", "rawMarkdown": "Great writeup, and congrats! Even more impressive because that is solo!\n\nIt is interesting that DKM worked for you, despite other participants failed to make it good. \nTraditional question - do you have some table with numbers for the \"Other models such as LoFTR, SE2-LoFTR, disk, QuadTreeAttention, OpenGlue, and SILK \"?\n\nThe reason I am inviting on this question is that making a reality check for the new (and older) methods is one of the competition goals.", "votes": null }, { "id": "2300871", "postDate": "06/13/2023 13:51:30", "content": "@roniheka Cool that you had success with DKM! Have you tried RoMa (our new matcher)? https://github.com/Parskatt/RoMa
\nInstallation should be quite similar as DKM (xformers library is optional). It might be slightly slower than DKM due to new backbone, but should not be majorly different.
\nI'm wondering what types of gains you would have, as we found it to give large improvements on IMC2022.
", "rawMarkdown": "roniheka Cool that you had success with DKM! Have you tried RoMa (our new matcher)? https://github.com/Parskatt/RoMa\n\nInstallation should be quite similar as DKM (xformers library is optional). It might be slightly slower than DKM due to new backbone, but should not be majorly different.\n\nI'm wondering what types of gains you would have, as we found it to give large improvements on IMC2022.", "votes": null }, { "id": "2301127", "postDate": "06/13/2023 16:25:40", "content": "@johanedstedt Thanks for your great contributions with new SOTA libraries. Sorry, I'm not the author of the proposed solution, but I have a few related words.
\nOur team that took 2nd place (write-up https://www.kaggle.com/competitions/image-matching-challenge-2023/discussion/416873 ) experimented a lot with different sparse/dense matchers, and DKM, in particular. Although we do not use it in our final solution, we evaluated DKM V3 in IMC22 challenge as a late submission, and got 0.836/0.819 public/private score without extra work, impressive result. It is even better than QuadTreeAttention and AspanFormer (0.826/0.816) in 2D matching. One of the reasons why not to proceed with a lot of experiments here, is a pretty big running time of DKM, unfortunately. And, a little randomness caused by random sampling here. Anyway, DKM V3 shows superior result in IMC22, thanks!
One thing you can do with DKM is to evaluate the warp at the position of the keypoints (e.g. by grid_sample), then you do not have to deal with sampling issues :)
", "rawMarkdown": "One thing you can do with DKM is to evaluate the warp at the position of the keypoints (e.g. by grid_sample), then you do not have to deal with sampling issues :)", "votes": null }, { "id": "2301403", "postDate": "06/13/2023 22:11:08", "content": "Thank you for your kind words of congratulations. I'm truly happy!
\nHere are the tables to compare the effects of model choice onto local score.
\nSince I haven't evaluated all models at the same time, the benchmark values differ. This is why the table is not consolidated into one…
| Exp. 1 | \nLocal mAA | \n
|---|---|
| baseline (disk+SuperGlue+DKM) | \n0.613 | \n
| +LoFTR | \n0.604 | \n
| +QuadTreeAttention | \n0.609 | \n
| OpenGlue instead of SuperGlue | \n0.504 | \n
| Exp. 2 | \nLocal mAA | \n
|---|---|
| baseline (only SuperGlue) | \n0.740 | \n
| +DKM | \n0.783 | \n
| +SILK | \n0.749 | \n
| +DKM+SILK | \n0.768 | \n
| Exp. 3 | \nLocal mAA | \n
|---|---|
| baseline (only SuperGlue) | \n0.792 | \n
| +DKM | \n0.848 | \n
| +disk | \n0.802 | \n
| +SE2-LoFTR | \n0.750 | \n
In my situation, DKM is indeed the best for ensembling, and Disk and SILK are the second best.
", "rawMarkdown": "Thank you for your kind words of congratulations. I'm truly happy! \n\nHere are the tables to compare the effects of model choice onto local score.\nSince I haven't evaluated all models at the same time, the benchmark values differ. This is why the table is not consolidated into one...\n\n\n\n| Exp. 1\t| Local mAA| \n| --- | --- |\n| baseline (disk+SuperGlue+DKM)\t| 0.613| \n| +LoFTR\t| 0.604| \n| +QuadTreeAttention\t| 0.609| \n| OpenGlue instead of SuperGlue\t| 0.504| \n\n|Exp. 2\t|Local mAA|\n| --- | --- |\n|baseline (only SuperGlue)\t|0.740|\n|+DKM\t|0.783|\n|+SILK\t|0.749|\n|+DKM+SILK\t|0.768|\n\n|Exp. 3\t|Local mAA|\n| --- | --- |\n|baseline (only SuperGlue)\t|0.792|\n|+DKM\t|0.848|\n|+disk\t|0.802|\n|+SE2-LoFTR\t|0.750|\n\nIn my situation, DKM is indeed the best for ensembling, and Disk and SILK are the second best.", "votes": null }, { "id": "2301409", "postDate": "06/13/2023 22:25:18", "content": "Thank you very much for your kind comments, and introduction to RoMa!
\nAlthough I haven't tested it, I expect it to work as well as DKM.
\nI've provided the tables of local mAA at the reply to old-ufo. Please refer to it.
Congrats on gold medal, Great work!
\nRegarding your last point about using fundamental matrix to estimate R and T. I believe that you had to calculate the essential matrix from it by multiplying intrinsics of the two cameras from both sides, then from essential matrix you can get R and T.
\nThere are two problems with this approach:
\n1) The intrinsics for the unregistered cameras are unknown
\n2) even if known, the translation part will be correct up to some scale, which might different from the one in your model, so normalizing all errors will lead to errors in translation when calculating the mAA for translation.
\nA working solution would include global PnP with BA after that for each image find the correspondences between its 2D points and 3D points in your model and apply PnP with estimating intrinsics as well then followed with BA.
\nUnfortunately, this approach also couldnot improve our LB although it boosted Cyprus and Dioscuri by registering correctly 2 and 6 additional images respectively, provide +0.02 validation improvement.
Congratulations to you, too!
\nAnd thank you for the supplement and pointing out the issues.
As you mentioned, the lack of internal parameters of the camera was indeed the main reason that made calculations difficult.
\nActually, I tried something like this; In the output of Colmap's incremental mapping, there are (estimated) camera matrices for registered images. For images that were not registered, I assumed the camera matrices were the same as the registered images and fed them into the ComputeEssentialMatrix function (found in this notebook). However, it was rarely the case that it fell within the threshold of the evaluation function…
\nReading your explanation, I've understood an additional reason why it didn't work out.
Great writeup, and congrats! Even more impressive because that is solo!
\nIt is interesting that DKM worked for you, despite other participants failed to make it good.
\nTraditional question - do you have some table with numbers for the \"Other models such as LoFTR, SE2-LoFTR, disk, QuadTreeAttention, OpenGlue, and SILK \"?
The reason I am inviting on this question is that making a reality check for the new (and older) methods is one of the competition goals.
", "votes": null, "replies": [ { "id": 2301403, "author_name": "roniheka", "author_url": "", "post_date": "06/13/2023 22:11:08", "content": "Thank you for your kind words of congratulations. I'm truly happy!
\nHere are the tables to compare the effects of model choice onto local score.
\nSince I haven't evaluated all models at the same time, the benchmark values differ. This is why the table is not consolidated into one…
| Exp. 1 | \nLocal mAA | \n
|---|---|
| baseline (disk+SuperGlue+DKM) | \n0.613 | \n
| +LoFTR | \n0.604 | \n
| +QuadTreeAttention | \n0.609 | \n
| OpenGlue instead of SuperGlue | \n0.504 | \n
| Exp. 2 | \nLocal mAA | \n
|---|---|
| baseline (only SuperGlue) | \n0.740 | \n
| +DKM | \n0.783 | \n
| +SILK | \n0.749 | \n
| +DKM+SILK | \n0.768 | \n
| Exp. 3 | \nLocal mAA | \n
|---|---|
| baseline (only SuperGlue) | \n0.792 | \n
| +DKM | \n0.848 | \n
| +disk | \n0.802 | \n
| +SE2-LoFTR | \n0.750 | \n
In my situation, DKM is indeed the best for ensembling, and Disk and SILK are the second best.
", "votes": null, "replies": [] } ] }, { "id": 2300871, "author_name": "johanedstedt", "author_url": "", "post_date": "06/13/2023 13:51:30", "content": "@roniheka Cool that you had success with DKM! Have you tried RoMa (our new matcher)? https://github.com/Parskatt/RoMa
\nInstallation should be quite similar as DKM (xformers library is optional). It might be slightly slower than DKM due to new backbone, but should not be majorly different.
\nI'm wondering what types of gains you would have, as we found it to give large improvements on IMC2022.
", "votes": null, "replies": [ { "id": 2301127, "author_name": "igorlashkov", "author_url": "", "post_date": "06/13/2023 16:25:40", "content": "@johanedstedt Thanks for your great contributions with new SOTA libraries. Sorry, I'm not the author of the proposed solution, but I have a few related words.
\nOur team that took 2nd place (write-up https://www.kaggle.com/competitions/image-matching-challenge-2023/discussion/416873 ) experimented a lot with different sparse/dense matchers, and DKM, in particular. Although we do not use it in our final solution, we evaluated DKM V3 in IMC22 challenge as a late submission, and got 0.836/0.819 public/private score without extra work, impressive result. It is even better than QuadTreeAttention and AspanFormer (0.826/0.816) in 2D matching. One of the reasons why not to proceed with a lot of experiments here, is a pretty big running time of DKM, unfortunately. And, a little randomness caused by random sampling here. Anyway, DKM V3 shows superior result in IMC22, thanks!
One thing you can do with DKM is to evaluate the warp at the position of the keypoints (e.g. by grid_sample), then you do not have to deal with sampling issues :)
", "votes": null, "replies": [] } ] }, { "id": 2301409, "author_name": "roniheka", "author_url": "", "post_date": "06/13/2023 22:25:18", "content": "Thank you very much for your kind comments, and introduction to RoMa!
\nAlthough I haven't tested it, I expect it to work as well as DKM.
\nI've provided the tables of local mAA at the reply to old-ufo. Please refer to it.
Congrats on gold medal, Great work!
\nRegarding your last point about using fundamental matrix to estimate R and T. I believe that you had to calculate the essential matrix from it by multiplying intrinsics of the two cameras from both sides, then from essential matrix you can get R and T.
\nThere are two problems with this approach:
\n1) The intrinsics for the unregistered cameras are unknown
\n2) even if known, the translation part will be correct up to some scale, which might different from the one in your model, so normalizing all errors will lead to errors in translation when calculating the mAA for translation.
\nA working solution would include global PnP with BA after that for each image find the correspondences between its 2D points and 3D points in your model and apply PnP with estimating intrinsics as well then followed with BA.
\nUnfortunately, this approach also couldnot improve our LB although it boosted Cyprus and Dioscuri by registering correctly 2 and 6 additional images respectively, provide +0.02 validation improvement.
Congratulations to you, too!
\nAnd thank you for the supplement and pointing out the issues.
As you mentioned, the lack of internal parameters of the camera was indeed the main reason that made calculations difficult.
\nActually, I tried something like this; In the output of Colmap's incremental mapping, there are (estimated) camera matrices for registered images. For images that were not registered, I assumed the camera matrices were the same as the registered images and fed them into the ComputeEssentialMatrix function (found in this notebook). However, it was rarely the case that it fell within the threshold of the evaluation function…
\nReading your explanation, I've understood an additional reason why it didn't work out.