I tried running this metric on the full training set (without randomly dropping images), but it seems to take forever for pond, which has more than 700 images.
The questions are:
\nI tried running this metric on the full training set (without randomly dropping images), but it seems to take forever for pond, which has more than 700 images.
The questions are:
\nThe number of scenes per image for the test set was chosen to allow metric evaluation within an hour. Actually, I don't think that anyone could be able to finish on a Kaggle notebook a pose estimation for scene of 700 images within the competition time constraints.
", "rawMarkdown": "The number of scenes per image for the test set was chosen to allow metric evaluation within an hour. Actually, I don't think that anyone could be able to finish on a Kaggle notebook a pose estimation for scene of 700 images within the competition time constraints.", "votes": null }, { "id": "2758960", "postDate": "04/18/2024 12:11:22", "content": "@fabiobellavia Thanks for the quick response!
\nJust to clarify, does this mean that the number of images in the test set is small enough to finish within an hour, or does the metric use only a subset of images in the test set?
I tried to evaluate the baseline with the full data set, but it was impossible.
\nI modified the script a bit to show an ETA, and it looks like it will take about four full days to evaluate the full data set of ponds.
\nFor now, for \"pond\" and \"lizard\", I am checking scores on a randomly selected sub-dataset.
Same for me. I discard images more than 100.
\nIt seems the metric runs something like RANSAC in n^3 loop…
I haven't never fully inspected the metric, but the time complexity is certainly above O(N^4) with heavy computations. That being said, it's quite possible to run it on scenes with less than 100 images, and that's likely the number of images in every scene in the test set.
", "rawMarkdown": "I haven't never fully inspected the metric, but the time complexity is certainly above O(N^4) with heavy computations. That being said, it's quite possible to run it on scenes with less than 100 images, and that's likely the number of images in every scene in the test set.", "votes": null }, { "id": "2758990", "postDate": "04/18/2024 12:32:25", "content": "The first you said.
", "rawMarkdown": "The first you said.", "votes": null }, { "id": "2759011", "postDate": "04/18/2024 12:44:59", "content": "You can create your own “test” set from training, by the analogy with the “church” scene : by selecting a subset <100 images with covisibility and random ordering
", "rawMarkdown": "You can create your own “test” set from training, by the analogy with the “church” scene : by selecting a subset <100 images with covisibility and random ordering", "votes": null }, { "id": "2759111", "postDate": "04/18/2024 13:48:45", "content": "Orgs said test scenes include up to 100 images here. I guess we should not expect more than 100 images in any scene.
", "rawMarkdown": "Orgs said test scenes include up to 100 images [here](https://www.kaggle.com/competitions/image-matching-challenge-2024/discussion/486584). I guess we should not expect more than 100 images in any scene.", "votes": null }, { "id": "2759914", "postDate": "04/19/2024 02:26:06", "content": "Just for reference, it took around 8 min for my env to run validation for sampled dataset up to 110 in each scene.
", "rawMarkdown": "Just for reference, it took around 8 min for my env to run validation for sampled dataset up to 110 in each scene.", "votes": null }, { "id": "2760416", "postDate": "04/19/2024 09:43:56", "content": "Thanks, I missed that!
", "rawMarkdown": "Thanks, I missed that!", "votes": null }, { "id": "2760418", "postDate": "04/19/2024 09:46:02", "content": "Interesting. But how can we replicate the covisibility of the hidden test set?
", "rawMarkdown": "Interesting. But how can we replicate the covisibility of the hidden test set?", "votes": null }, { "id": "2763671", "postDate": "04/20/2024 17:06:47", "content": "Good question. Last year participants didn't have even a single scene in the \"open test\" set ;)
", "rawMarkdown": "Good question. Last year participants didn't have even a single scene in the \"open test\" set ;)", "votes": null }, { "id": "2767887", "postDate": "04/22/2024 14:54:33", "content": "Hi, did anyone try calculating the validation with somehow sampled train data?
\nI sampled 110 images per scene and the validation result is quite different from 0.15 in this notebook.
@clearwaterkzk
\ndo you mean it's much better than 0.15?
\nI think it's natural for the score to increase if you increase the number of images, as it makes it easier to triangulate.
\n\ndo you mean it's much better than 0.15?
\n
Yes, I got much higher result, around 0.28X just by increasing images.
\n\n\nI think it's natural for the score to increase if you increase the number of images, as it makes it easier to triangulate.
\n
I see ! Thanks for sharing.
", "rawMarkdown": ">do you mean it's much better than 0.15?\n\nYes, I got much higher result, around 0.28X just by increasing images.\n\n\n>I think it's natural for the score to increase if you increase the number of images, as it makes it easier to triangulate.\n\nI see ! Thanks for sharing.", "votes": null }, { "id": "2774024", "postDate": "04/25/2024 03:16:49", "content": "Increasing the number of images, do I just need to increase the n_samples?
", "rawMarkdown": "Increasing the number of images, do I just need to increase the n_samples?", "votes": null }, { "id": "2774042", "postDate": "04/25/2024 03:26:37", "content": "\n\nIncreasing the number of images, do I just need to increase the n_samples?
\n
Yes.
\nJust increasing n_samples will result in higher validation score while it takes more time to evaluate.
The number of scenes per image for the test set was chosen to allow metric evaluation within an hour. Actually, I don't think that anyone could be able to finish on a Kaggle notebook a pose estimation for scene of 700 images within the competition time constraints.
", "votes": null, "replies": [ { "id": 2758960, "author_name": "bamps53", "author_url": "", "post_date": "04/18/2024 12:11:22", "content": "@fabiobellavia Thanks for the quick response!
\nJust to clarify, does this mean that the number of images in the test set is small enough to finish within an hour, or does the metric use only a subset of images in the test set?
I haven't never fully inspected the metric, but the time complexity is certainly above O(N^4) with heavy computations. That being said, it's quite possible to run it on scenes with less than 100 images, and that's likely the number of images in every scene in the test set.
", "votes": null, "replies": [] }, { "id": 2758990, "author_name": "fabiobellavia", "author_url": "", "post_date": "04/18/2024 12:32:25", "content": "The first you said.
", "votes": null, "replies": [] } ] } ] }, { "id": 2758961, "author_name": "kashiwaba", "author_url": "", "post_date": "04/18/2024 12:11:48", "content": "I tried to evaluate the baseline with the full data set, but it was impossible.
\nI modified the script a bit to show an ETA, and it looks like it will take about four full days to evaluate the full data set of ponds.
\nFor now, for \"pond\" and \"lizard\", I am checking scores on a randomly selected sub-dataset.
Same for me. I discard images more than 100.
\nIt seems the metric runs something like RANSAC in n^3 loop…
You can create your own “test” set from training, by the analogy with the “church” scene : by selecting a subset <100 images with covisibility and random ordering
", "votes": null, "replies": [ { "id": 2760418, "author_name": "bamps53", "author_url": "", "post_date": "04/19/2024 09:46:02", "content": "Interesting. But how can we replicate the covisibility of the hidden test set?
", "votes": null, "replies": [ { "id": 2763671, "author_name": "oldufo", "author_url": "", "post_date": "04/20/2024 17:06:47", "content": "Good question. Last year participants didn't have even a single scene in the \"open test\" set ;)
", "votes": null, "replies": [] } ] } ] }, { "id": 2759914, "author_name": "clearwaterkzk", "author_url": "", "post_date": "04/19/2024 02:26:06", "content": "Just for reference, it took around 8 min for my env to run validation for sampled dataset up to 110 in each scene.
", "votes": null, "replies": [] } ] }, { "id": 2759111, "author_name": "vostankovich", "author_url": "", "post_date": "04/18/2024 13:48:45", "content": "Orgs said test scenes include up to 100 images here. I guess we should not expect more than 100 images in any scene.
", "votes": null, "replies": [ { "id": 2760416, "author_name": "bamps53", "author_url": "", "post_date": "04/19/2024 09:43:56", "content": "Thanks, I missed that!
", "votes": null, "replies": [] } ] }, { "id": 2767887, "author_name": "clearwaterkzk", "author_url": "", "post_date": "04/22/2024 14:54:33", "content": "Hi, did anyone try calculating the validation with somehow sampled train data?
\nI sampled 110 images per scene and the validation result is quite different from 0.15 in this notebook.
@clearwaterkzk
\ndo you mean it's much better than 0.15?
\nI think it's natural for the score to increase if you increase the number of images, as it makes it easier to triangulate.
\n\ndo you mean it's much better than 0.15?
\n
Yes, I got much higher result, around 0.28X just by increasing images.
\n\n\nI think it's natural for the score to increase if you increase the number of images, as it makes it easier to triangulate.
\n
I see ! Thanks for sharing.
", "votes": null, "replies": [ { "id": 2774024, "author_name": "huangxiao309", "author_url": "", "post_date": "04/25/2024 03:16:49", "content": "Increasing the number of images, do I just need to increase the n_samples?
", "votes": null, "replies": [ { "id": 2774042, "author_name": "clearwaterkzk", "author_url": "", "post_date": "04/25/2024 03:26:37", "content": "\n\nIncreasing the number of images, do I just need to increase the n_samples?
\n
Yes.
\nJust increasing n_samples will result in higher validation score while it takes more time to evaluate.