![\"architecture\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5065877%2F68911db7c4cc430dec05670cd196a960%2Fslide_architecture.png?generation=1687202186098466&alt=media\")
We presented our solution at CVPR 2023. Watch on YouTube here
\nOur team would like to deeply appreciate the Kaggle staff, Google Research, and Haiper for hosting the continuation of this exciting image matching challenge, as well as everyone here who compete and shared great discussions. My congratulations to all participants!
\nThe work we describe here is truly a joint effort of @yamsam, @remekkinas, and @vostankovich. I’m grateful of being a part of this hardworking, cohesive, and skilled team. Thanks a lot, guys! I learned a lot from you.
\nThe best submission that was used for final scoring in private LB finished on the last day of the competition. We weren’t completely sure about this submission because it was not clear how much randomness was in it. We had a practice to re-run the same notebook code multiple times to see what scores we can get. We discussed the solution which was implemented on the last day, trusted it and it worked out. The other interesting fact is that our 2nd selected submission for final evaluation scored 0.497/0.542 that also allows us to take 2nd place. This selected second submission is the same as the 1st one except the “Run reconstruction multiple times” trick, that is described below. Anyway, the difference between the best submission (0.562) and the 2nd one is noticable.
\nIn general, throughout our code submissions every time we fight with the randomness coming from COLMAP responsible for scene reconstruction. Our final solution is based on the use of COLMAP and pretrained SuperPoint/SuperGlue models running on different resolutions for every image in the scene. We apply a bunch of different tricks aimed at different parts of COLMAP-based pipeline in order to stabilize our solution and reach the final score.
\n
• TTA multi crop, no success. The idea was to split image into multiple crops and extract matches in order to find similar images in the scene and determine the best image pairs.
\n• Square-sized images.
\n• Bigger image size (e.g., 1600) for SP/SG.
\n• Manual RANSAC instead of using COLMAP internal implementation. We run experiments by disabling geometric verification, but the score was not good.
\n• NMS filtering to reduce number of points by using ANMS.
\n• Filter least significant image pairs by number of outliers instead of relying on raw matches. It was quite important to look for certain number of matches for an image pair. Run experiments using SP/SG and Loftr. We got higher mAA with Loftr, but, probably, more effort needed here to make it work properly, not enough time.
\n• Downscale scene images before passing them to COLMAP.
\n• Pixel-Perfect Structure-from-Motion. It was a promising method to evaluate as we got a good boost locally with PixSfm, using a single image size of 1280, and it boosted our score from 0.71727 to 0.76253. Then, we managed to install this framework successfully and run it in Kaggle environment, but could not beat our best score at that moment. It is a heavyweight framework taking too much RAM, and we could run it only for scenes having at most ~30 images. A bit upset because we spent tons of hours compiling all this stuff.
\n• Adaptive image sizes. Say, if the longest image side >= 1536 for most images in the scene, we use higher image resolution for matchers ensemble e.g. [1280, 1408, 1536]. Otherwise, a default one is applied [1280,1088,1376]. Did not have enough time to test this idea. It worked locally for cyprus and wall that have big resolution. One of our last submissions implementing this idea crashed with internal error.
\n• Different detectors, matchers. We tested Loftr, QuadreeAttention, AspanFormer, DKM v3, GlueStick (keypoints + lines), Patch2Pix, KeyNetAffNetHardNet, DISK, PatchNetVLAD. We also experimented with the confidence matching thresholds and number of matches, but no boost here. Eventually, SP/SG was the best choice for us. Probably, the reason why many dense-based methods did not work out for us is because of the low performance of the “repeatability” metric and high noise in the matches.
\n• Different CNNs to find the most similar images in the scene and generate corresponding image pairs (NetVLAD, different pretrained timm-based backbones, CosPlace etc). We even specifically trained a model to find similar images in the scene, but no success here. Later, we gave up using this strategy at all.
\n• Different keypoint/matching refinement methods (e.g., recently published AdaMatcher, Patch2Pix ), but did not have enough time. AdaMatcher seems a promising idea to try, a quote from their paper “as a refinement network for SP/SG we observe a noticeable improvement in AUC”
| Method | \nPrivate LB | \nPublic LB | \nΔ Private LB | \n
|---|---|---|---|
| Baseline | \n0.382 | \n0.317 | \n– | \n
| Manual RANSAC | \n0.336 | \n0.286 | \n–0.046 | \n
| Image size 1024 → 1280 | \n0.407 | \n0.345 | \n+0.025 | \n
| Image size 1376, similarity=None | \n0.489 | \n0.426 | \n+0.026 | \n
| Exhaustive matching, 8/10 | \n0.486 | \n0.441 | \n-0.003 | \n
| TTA, Image sizes [840, 1024, 1280] | \n0.491 | \n0.447 | \n+0.002 | \n
| TTA, Image sizes [1088, 1280, 1376] | \n0.523 | \n0.475 | \n+0.032 | \n
| Manual image initialization | \n0.529 | \n0.492 | \n+0.006 | \n
| Rotate detection | \n0.542 | \n0.497 | \n+0.013 | \n
| Multi-run reconstruction, matches thr [100, 125, 75, 100] | \n0.562 | \n0.506 | \n+0.02 | \n
As a reference, this is one of our latest metric reports using train dataset:
\nurban / kyiv-puppet-theater (26 images, 325 pairs) -> mAA=0.921538, mAA_q=0.991077, mAA_t=0.921846
\nurban -> mAA=0.921538
heritage / cyprus (30 images, 435 pairs) -> mAA=0.514713, mAA_q=0.525287, mAA_t=0.543678
\nheritage / wall (43 images, 903 pairs) -> mAA=0.495792, mAA_q=0.875637, mAA_t=0.509302
\nheritage -> mAA=0.505252
haiper / bike (15 images, 105 pairs) -> mAA=0.940952, mAA_q=0.999048, mAA_t=0.940952
\nhaiper / chairs (16 images, 120 pairs) -> mAA=0.834167, mAA_q=0.863333, mAA_t=0.839167
\nhaiper / fountain (23 images, 253 pairs) -> mAA=0.999605, mAA_q=1.000000, mAA_t=0.999605
\nhaiper -> mAA=0.924908
Final metric -> mAA=0.783900
\nFinally, we had two submissions running on the last day. One of them succeeded and allowed us to get 2nd place, but the other one did not fit the time limit, unexpectedly. Sometimes it is stressful to make a submission on the last day of the competition.
\nSpecial thanks to the authors of the following projects:
\n• COLMAP
\n• SuperPoint
\n• SuperGlue
\n• check_orientation
We presented our solution at CVPR 2023. Watch on YouTube here
\nOur team would like to deeply appreciate the Kaggle staff, Google Research, and Haiper for hosting the continuation of this exciting image matching challenge, as well as everyone here who compete and shared great discussions. My congratulations to all participants!
\nThe work we describe here is truly a joint effort of @yamsam, @remekkinas, and @vostankovich. I’m grateful of being a part of this hardworking, cohesive, and skilled team. Thanks a lot, guys! I learned a lot from you.
\nThe best submission that was used for final scoring in private LB finished on the last day of the competition. We weren’t completely sure about this submission because it was not clear how much randomness was in it. We had a practice to re-run the same notebook code multiple times to see what scores we can get. We discussed the solution which was implemented on the last day, trusted it and it worked out. The other interesting fact is that our 2nd selected submission for final evaluation scored 0.497/0.542 that also allows us to take 2nd place. This selected second submission is the same as the 1st one except the “Run reconstruction multiple times” trick, that is described below. Anyway, the difference between the best submission (0.562) and the 2nd one is noticable.
\nIn general, throughout our code submissions every time we fight with the randomness coming from COLMAP responsible for scene reconstruction. Our final solution is based on the use of COLMAP and pretrained SuperPoint/SuperGlue models running on different resolutions for every image in the scene. We apply a bunch of different tricks aimed at different parts of COLMAP-based pipeline in order to stabilize our solution and reach the final score.
\n
• TTA multi crop, no success. The idea was to split image into multiple crops and extract matches in order to find similar images in the scene and determine the best image pairs.
\n• Square-sized images.
\n• Bigger image size (e.g., 1600) for SP/SG.
\n• Manual RANSAC instead of using COLMAP internal implementation. We run experiments by disabling geometric verification, but the score was not good.
\n• NMS filtering to reduce number of points by using ANMS.
\n• Filter least significant image pairs by number of outliers instead of relying on raw matches. It was quite important to look for certain number of matches for an image pair. Run experiments using SP/SG and Loftr. We got higher mAA with Loftr, but, probably, more effort needed here to make it work properly, not enough time.
\n• Downscale scene images before passing them to COLMAP.
\n• Pixel-Perfect Structure-from-Motion. It was a promising method to evaluate as we got a good boost locally with PixSfm, using a single image size of 1280, and it boosted our score from 0.71727 to 0.76253. Then, we managed to install this framework successfully and run it in Kaggle environment, but could not beat our best score at that moment. It is a heavyweight framework taking too much RAM, and we could run it only for scenes having at most ~30 images. A bit upset because we spent tons of hours compiling all this stuff.
\n• Adaptive image sizes. Say, if the longest image side >= 1536 for most images in the scene, we use higher image resolution for matchers ensemble e.g. [1280, 1408, 1536]. Otherwise, a default one is applied [1280,1088,1376]. Did not have enough time to test this idea. It worked locally for cyprus and wall that have big resolution. One of our last submissions implementing this idea crashed with internal error.
\n• Different detectors, matchers. We tested Loftr, QuadreeAttention, AspanFormer, DKM v3, GlueStick (keypoints + lines), Patch2Pix, KeyNetAffNetHardNet, DISK, PatchNetVLAD. We also experimented with the confidence matching thresholds and number of matches, but no boost here. Eventually, SP/SG was the best choice for us. Probably, the reason why many dense-based methods did not work out for us is because of the low performance of the “repeatability” metric and high noise in the matches.
\n• Different CNNs to find the most similar images in the scene and generate corresponding image pairs (NetVLAD, different pretrained timm-based backbones, CosPlace etc). We even specifically trained a model to find similar images in the scene, but no success here. Later, we gave up using this strategy at all.
\n• Different keypoint/matching refinement methods (e.g., recently published AdaMatcher, Patch2Pix ), but did not have enough time. AdaMatcher seems a promising idea to try, a quote from their paper “as a refinement network for SP/SG we observe a noticeable improvement in AUC”
| Method | \nPrivate LB | \nPublic LB | \nΔ Private LB | \n
|---|---|---|---|
| Baseline | \n0.382 | \n0.317 | \n– | \n
| Manual RANSAC | \n0.336 | \n0.286 | \n–0.046 | \n
| Image size 1024 → 1280 | \n0.407 | \n0.345 | \n+0.025 | \n
| Image size 1376, similarity=None | \n0.489 | \n0.426 | \n+0.026 | \n
| Exhaustive matching, 8/10 | \n0.486 | \n0.441 | \n-0.003 | \n
| TTA, Image sizes [840, 1024, 1280] | \n0.491 | \n0.447 | \n+0.002 | \n
| TTA, Image sizes [1088, 1280, 1376] | \n0.523 | \n0.475 | \n+0.032 | \n
| Manual image initialization | \n0.529 | \n0.492 | \n+0.006 | \n
| Rotate detection | \n0.542 | \n0.497 | \n+0.013 | \n
| Multi-run reconstruction, matches thr [100, 125, 75, 100] | \n0.562 | \n0.506 | \n+0.02 | \n
As a reference, this is one of our latest metric reports using train dataset:
\nurban / kyiv-puppet-theater (26 images, 325 pairs) -> mAA=0.921538, mAA_q=0.991077, mAA_t=0.921846
\nurban -> mAA=0.921538
heritage / cyprus (30 images, 435 pairs) -> mAA=0.514713, mAA_q=0.525287, mAA_t=0.543678
\nheritage / wall (43 images, 903 pairs) -> mAA=0.495792, mAA_q=0.875637, mAA_t=0.509302
\nheritage -> mAA=0.505252
haiper / bike (15 images, 105 pairs) -> mAA=0.940952, mAA_q=0.999048, mAA_t=0.940952
\nhaiper / chairs (16 images, 120 pairs) -> mAA=0.834167, mAA_q=0.863333, mAA_t=0.839167
\nhaiper / fountain (23 images, 253 pairs) -> mAA=0.999605, mAA_q=1.000000, mAA_t=0.999605
\nhaiper -> mAA=0.924908
Final metric -> mAA=0.783900
\nFinally, we had two submissions running on the last day. One of them succeeded and allowed us to get 2nd place, but the other one did not fit the time limit, unexpectedly. Sometimes it is stressful to make a submission on the last day of the competition.
\nSpecial thanks to the authors of the following projects:
\n• COLMAP
\n• SuperPoint
\n• SuperGlue
\n• check_orientation
Congrats with great result!
\n\n\nLoftr, QuadreeAttention, AspanFormer, DKM v3, GlueStick (keypoints + lines), Patch2Pix, KeyNetAffNetHardNet, DISK, PatchNetVLAD
\n
Could you please share the numbers?
\nThe same for the global descriptor thing, please?
", "rawMarkdown": "Congrats with great result!\n\n>Loftr, QuadreeAttention, AspanFormer, DKM v3, GlueStick (keypoints + lines), Patch2Pix, KeyNetAffNetHardNet, DISK, PatchNetVLAD\n\nCould you please share the numbers?\n\nThe same for the global descriptor thing, please?", "votes": null }, { "id": "2300600", "postDate": "06/13/2023 09:34:45", "content": "Hey team, I just want to say a big thank you. Your work during this contest was amazing. We came up with so many ideas and things kept changing all the time.
\nSpecial thanks to @vostankovich. You never gave up. I did - I quit when trying to set up pixfm, and that's not like me. I've learned from this. It's important to listen to other people's ideas and not to think that I always know best.
\nFrom the start, @igorlashkov said we should work on the rotation. I tried a few times but it didn't help me. But guess what? In the end, working on the rotation was a key part of our solution. So, big thanks to Igla for not giving up!
\nAnd thank you, @oldufo for talking about 3D reconstruction. I first started looking into this last year, just out of interest. The last contest made me more interested. It's a great way to learn more and get better at what we do.
\nI'm really looking forward to next year and IMC 2024. Let's do it!
\nOne more from my side - the way I was trying to understand SfM pipeline was Meshroom. SfM was new for me so looking only in example notebooks was not enough for me. During the competition I was experimenting a lot playing with different settings to understand \"what if\".
\n![\"\"](\"https://i.ibb.co/dgTymVc/Meshroom.jpg\")
The matchers were tested during start/mid of competition when we had very simple pipeline. We just tested one by one and discarded any matcher that was worse than SP/SG. All of the listed models are worse by a large margin, at least 15-20% lower than SP/SG. Some of the models also were just too slow or memory consuming along with the bad score. The closest one was LoFTR as I remember, but even ensembling it with SP/SG didn't give us any boost. We can rerun some matchers in our final pipeline for sure if needed to get the numbers.
\nThe same thing with global descriptors, we don't have numbers for them, we just discovered one time that SG can be a retrieval itself and the approach is robust enough and outperforms other models. SP/SG are really fast in half mode and with caching, so we actually completely agreed not to use any additional retrieval models for pairs search.
", "rawMarkdown": "The matchers were tested during start/mid of competition when we had very simple pipeline. We just tested one by one and discarded any matcher that was worse than SP/SG. All of the listed models are worse by a large margin, at least 15-20% lower than SP/SG. Some of the models also were just too slow or memory consuming along with the bad score. The closest one was LoFTR as I remember, but even ensembling it with SP/SG didn't give us any boost. We can rerun some matchers in our final pipeline for sure if needed to get the numbers.\n\nThe same thing with global descriptors, we don't have numbers for them, we just discovered one time that SG can be a retrieval itself and the approach is robust enough and outperforms other models. SP/SG are really fast in half mode and with caching, so we actually completely agreed not to use any additional retrieval models for pairs search.", "votes": null }, { "id": "2300617", "postDate": "06/13/2023 09:52:24", "content": "Congrats on 2nd place! good work!
\nPixSFM was really tricky! It gave boost to our validation score up to +0.1 on mAA, but unfortunately it didn't improve our LB.
\nYou can try it using this notebook!
\nhttps://www.kaggle.com/jaafarmahmoud1/pixel-perfect-sfm-on-kaggle
Hi @igorlashkov , congratulations for the prize and gold. I have a question
\nHow to run reconstruction multiple times? I think the reconstruction is in pycolmap.incremental_mapping, which consumes much time. How can you control your submission within 9 hours?
Thanks for the detailed write up! pretty cool that sp and sg worked with small image sizes as well. In our case we used 1600 for small images and 3200 for large ones
\nwe have also tried iterative reconstruction but with changing the camera type it gave +0.001 boost so we ignored it especially after getting a reproducable pycolmap results, it seems that it worth further exploring
\nCongrats on the second place you really did a great job !!
Use of cache allowed us significantly increase speed of our solution. We mostly cached 3 heavy things – images resized and packed in CUDA tensor beforehand, SP keypoints, and SG matches saved in memory on a 1st run. Also, if we see that the N of matches for a certain image pair is low, we do not proceed with other image sizes, exit earlier. Special thanks to @vostankovich for making our solution blazing fast!
", "rawMarkdown": "Use of cache allowed us significantly increase speed of our solution. We mostly cached 3 heavy things – images resized and packed in CUDA tensor beforehand, SP keypoints, and SG matches saved in memory on a 1st run. Also, if we see that the N of matches for a certain image pair is low, we do not proceed with other image sizes, exit earlier. Special thanks to @vostankovich for making our solution blazing fast!", "votes": null }, { "id": "2301152", "postDate": "06/13/2023 16:49:41", "content": "Congratulations on the amazing result. A lot of new & great stuff in this write up for me to read up on!
", "rawMarkdown": "Congratulations on the amazing result. A lot of new & great stuff in this write up for me to read up on!", "votes": null }, { "id": "2301391", "postDate": "06/13/2023 21:24:25", "content": "Congratulations for the excellent result and gold. It provided lots of idea to me, very helpful !!!
", "rawMarkdown": "Congratulations for the excellent result and gold. It provided lots of idea to me, very helpful !!!", "votes": null }, { "id": "2301678", "postDate": "06/14/2023 05:05:56", "content": "I'm glad you found the information detailed and interesting! It's impressive to see how techniques like super-resolution (SP) and style transfer (SG) can work effectively even with small image sizes. In your case, using 1600 for small images and 3200 for large ones seems like a reasonable approach. It's always fascinating to explore different strategies and see how they can contribute to the overall performance in image-related challenges.
", "rawMarkdown": "I'm glad you found the information detailed and interesting! It's impressive to see how techniques like super-resolution (SP) and style transfer (SG) can work effectively even with small image sizes. In your case, using 1600 for small images and 3200 for large ones seems like a reasonable approach. It's always fascinating to explore different strategies and see how they can contribute to the overall performance in image-related challenges.", "votes": null }, { "id": "2319801", "postDate": "06/27/2023 10:17:10", "content": "Hi @igorlashkov . Can you please share the notebook with me? Thanks
", "rawMarkdown": "Hi @igorlashkov . Can you please share the notebook with me? Thanks", "votes": null }, { "id": "2613301", "postDate": "01/22/2024 02:04:09", "content": "@remekkinas,
\nThank you very much to you and your team for this amazing contribution. I am trying to reimplement your solution for learning purposes. Wanted to know if the submission notebook/code is already public.
Congrats with great result!
\n\n\nLoftr, QuadreeAttention, AspanFormer, DKM v3, GlueStick (keypoints + lines), Patch2Pix, KeyNetAffNetHardNet, DISK, PatchNetVLAD
\n
Could you please share the numbers?
\nThe same for the global descriptor thing, please?
", "votes": null, "replies": [ { "id": 2300614, "author_name": "vostankovich", "author_url": "", "post_date": "06/13/2023 09:49:13", "content": "The matchers were tested during start/mid of competition when we had very simple pipeline. We just tested one by one and discarded any matcher that was worse than SP/SG. All of the listed models are worse by a large margin, at least 15-20% lower than SP/SG. Some of the models also were just too slow or memory consuming along with the bad score. The closest one was LoFTR as I remember, but even ensembling it with SP/SG didn't give us any boost. We can rerun some matchers in our final pipeline for sure if needed to get the numbers.
\nThe same thing with global descriptors, we don't have numbers for them, we just discovered one time that SG can be a retrieval itself and the approach is robust enough and outperforms other models. SP/SG are really fast in half mode and with caching, so we actually completely agreed not to use any additional retrieval models for pairs search.
", "votes": null, "replies": [] } ] }, { "id": 2300600, "author_name": "remekkinas", "author_url": "", "post_date": "06/13/2023 09:34:45", "content": "Hey team, I just want to say a big thank you. Your work during this contest was amazing. We came up with so many ideas and things kept changing all the time.
\nSpecial thanks to @vostankovich. You never gave up. I did - I quit when trying to set up pixfm, and that's not like me. I've learned from this. It's important to listen to other people's ideas and not to think that I always know best.
\nFrom the start, @igorlashkov said we should work on the rotation. I tried a few times but it didn't help me. But guess what? In the end, working on the rotation was a key part of our solution. So, big thanks to Igla for not giving up!
\nAnd thank you, @oldufo for talking about 3D reconstruction. I first started looking into this last year, just out of interest. The last contest made me more interested. It's a great way to learn more and get better at what we do.
\nI'm really looking forward to next year and IMC 2024. Let's do it!
\nOne more from my side - the way I was trying to understand SfM pipeline was Meshroom. SfM was new for me so looking only in example notebooks was not enough for me. During the competition I was experimenting a lot playing with different settings to understand \"what if\".
\n
Congrats on 2nd place! good work!
\nPixSFM was really tricky! It gave boost to our validation score up to +0.1 on mAA, but unfortunately it didn't improve our LB.
\nYou can try it using this notebook!
\nhttps://www.kaggle.com/jaafarmahmoud1/pixel-perfect-sfm-on-kaggle
Hi @igorlashkov , congratulations for the prize and gold. I have a question
\nHow to run reconstruction multiple times? I think the reconstruction is in pycolmap.incremental_mapping, which consumes much time. How can you control your submission within 9 hours?
Use of cache allowed us significantly increase speed of our solution. We mostly cached 3 heavy things – images resized and packed in CUDA tensor beforehand, SP keypoints, and SG matches saved in memory on a 1st run. Also, if we see that the N of matches for a certain image pair is low, we do not proceed with other image sizes, exit earlier. Special thanks to @vostankovich for making our solution blazing fast!
", "votes": null, "replies": [ { "id": 2319801, "author_name": "narshima199712", "author_url": "", "post_date": "06/27/2023 10:17:10", "content": "Hi @igorlashkov . Can you please share the notebook with me? Thanks
", "votes": null, "replies": [] } ] } ] }, { "id": 2301071, "author_name": "ammarali32", "author_url": "", "post_date": "06/13/2023 15:45:47", "content": "Thanks for the detailed write up! pretty cool that sp and sg worked with small image sizes as well. In our case we used 1600 for small images and 3200 for large ones
\nwe have also tried iterative reconstruction but with changing the camera type it gave +0.001 boost so we ignored it especially after getting a reproducable pycolmap results, it seems that it worth further exploring
\nCongrats on the second place you really did a great job !!
Congratulations on the amazing result. A lot of new & great stuff in this write up for me to read up on!
", "votes": null, "replies": [] }, { "id": 2301391, "author_name": "kaisenzhang", "author_url": "", "post_date": "06/13/2023 21:24:25", "content": "Congratulations for the excellent result and gold. It provided lots of idea to me, very helpful !!!
", "votes": null, "replies": [] }, { "id": 2301678, "author_name": "poojach7611", "author_url": "", "post_date": "06/14/2023 05:05:56", "content": "I'm glad you found the information detailed and interesting! It's impressive to see how techniques like super-resolution (SP) and style transfer (SG) can work effectively even with small image sizes. In your case, using 1600 for small images and 3200 for large ones seems like a reasonable approach. It's always fascinating to explore different strategies and see how they can contribute to the overall performance in image-related challenges.
", "votes": null, "replies": [] }, { "id": 2613301, "author_name": "mukit0", "author_url": "", "post_date": "01/22/2024 02:04:09", "content": "@remekkinas,
\nThank you very much to you and your team for this amazing contribution. I am trying to reimplement your solution for learning purposes. Wanted to know if the submission notebook/code is already public.