Hi everyone,
\nWe investigated this report and have confirmed that the competition launched with a bug in the metric code. We'll post a patched copy of the metric momentarily and will begin rescoring existing submissions later today. This metric takes quite a bit longer to evaluate than typical competition metrics so I may need to rescore submissions in batches over multiple days. Submissions will remain open but the leaderboard will not be meaningful until the process is complete. I will make a follow up post once there's more to report.
\nIn terms of what to expect: many submissions were not affected at all by the bug and will retain their existing score. Submissions that did trigger the bug will have lower scores, sometimes by a large factor (2x).
\nAs for the bug itself:
\nThe metric has two stages. First, it greedily matches the clusters provided by the user, to the ground truth clusters. Second, it computes the accuracy of the estimated image poses for each of the clusters. The issue is in the greedy assignment step. Under some conditions can lead to an improper association between a ground truth scene and the outliers cluster. This cascades into the computation of mAA and clustering score. The problem lies in the initialization of the tables used in the greedy assignment step, which are initially set to zero. When filling the table, entries corresponding to the outlier cluster are also set to 0. In some situations, the best cluster and the outliers cluster are given the same score in the greedy assignment, and the outlier cluster is wrongly selected instead. Specifically, this happens when (a) the best non-outlier cluster for a scene has a score of zero, and (b) the submission file contains \"outlier\" rows before non-outlier rows. The fix is simple: initialize the tables to -1 instead.
\nWe would like to send a great deal of thanks to the users who noticed this and reached out, especially to @gowillgo and @petermoorhouse.
\nApologies again for the disruption and thank you in advance for your patience.
\nEdit 1: The rescores are under way.
\nEdit 2: The rescores are now 99.9% complete. The final dozen or so submissions should be done very soon.
\nEdit 3: You should now also see faster metric scoring times.
", "messages": [ { "id": "3208075", "postDate": "05/23/2025 16:50:33", "content": "Hi everyone,
\nWe investigated this report and have confirmed that the competition launched with a bug in the metric code. We'll post a patched copy of the metric momentarily and will begin rescoring existing submissions later today. This metric takes quite a bit longer to evaluate than typical competition metrics so I may need to rescore submissions in batches over multiple days. Submissions will remain open but the leaderboard will not be meaningful until the process is complete. I will make a follow up post once there's more to report.
\nIn terms of what to expect: many submissions were not affected at all by the bug and will retain their existing score. Submissions that did trigger the bug will have lower scores, sometimes by a large factor (2x).
\nAs for the bug itself:
\nThe metric has two stages. First, it greedily matches the clusters provided by the user, to the ground truth clusters. Second, it computes the accuracy of the estimated image poses for each of the clusters. The issue is in the greedy assignment step. Under some conditions can lead to an improper association between a ground truth scene and the outliers cluster. This cascades into the computation of mAA and clustering score. The problem lies in the initialization of the tables used in the greedy assignment step, which are initially set to zero. When filling the table, entries corresponding to the outlier cluster are also set to 0. In some situations, the best cluster and the outliers cluster are given the same score in the greedy assignment, and the outlier cluster is wrongly selected instead. Specifically, this happens when (a) the best non-outlier cluster for a scene has a score of zero, and (b) the submission file contains \"outlier\" rows before non-outlier rows. The fix is simple: initialize the tables to -1 instead.
\nWe would like to send a great deal of thanks to the users who noticed this and reached out, especially to @gowillgo and @petermoorhouse.
\nApologies again for the disruption and thank you in advance for your patience.
\nEdit 1: The rescores are under way.
\nEdit 2: The rescores are now 99.9% complete. The final dozen or so submissions should be done very soon.
\nEdit 3: You should now also see faster metric scoring times.
", "rawMarkdown": "Hi everyone,\n\nWe investigated [this report](https://www.kaggle.com/competitions/image-matching-challenge-2025/discussion/579625) and have confirmed that the competition launched with a bug in the metric code. We'll post a patched copy of the metric momentarily and will begin rescoring existing submissions later today. This metric takes quite a bit longer to evaluate than typical competition metrics so I may need to rescore submissions in batches over multiple days. **Submissions will remain open but the leaderboard will not be meaningful until the process is complete**. I will make a follow up post once there's more to report. \n\nIn terms of what to expect: many submissions were not affected at all by the bug and will retain their existing score. Submissions that did trigger the bug will have lower scores, sometimes by a large factor (2x). \n\nAs for the bug itself:\n\nThe metric has two stages. First, it greedily matches the clusters provided by the user, to the ground truth clusters. Second, it computes the accuracy of the estimated image poses for each of the clusters. The issue is in the greedy assignment step. Under some conditions can lead to an improper association between a ground truth scene and the outliers cluster. This cascades into the computation of mAA and clustering score. The problem lies in the initialization of the tables used in the greedy assignment step, which are initially set to zero. When filling the table, entries corresponding to the outlier cluster are also set to 0. In some situations, the best cluster and the outliers cluster are given the same score in the greedy assignment, and the outlier cluster is wrongly selected instead. Specifically, this happens when (a) the best non-outlier cluster for a scene has a score of zero, and (b) the submission file contains \"outlier\" rows before non-outlier rows. The fix is simple: initialize the tables to -1 instead.\n\nWe would like to send a great deal of thanks to the users who noticed this and reached out, especially to @gowillgo and @petermoorhouse.\n\nApologies again for the disruption and thank you in advance for your patience.\n\nEdit 1: The rescores are under way. \n\nEdit 2: The rescores are now 99.9% complete. The final dozen or so submissions should be done very soon.\n\nEdit 3: You should now also see faster metric scoring times.", "votes": null }, { "id": "3208267", "postDate": "05/23/2025 20:38:28", "content": "The fixed version of the metric is available at
\nhttps://www.kaggle.com/datasets/eduardtrulls/imc25-utils/data?select=metric.py
Hi, @sohier , @eduardtrulls , @oldufo ,
\nThank you for conducting the rescoring.
\nThis might be specific to my case, but after rescoring, I’ve encountered Submission Scoring Error on several of my previous submissions that had successfully scored before.
\nAs I mentioned earlier in [https://www.kaggle.com/competitions/image-matching-challenge-2025/discussion/579064#3203775], my submissions tend to take a relatively long time during the scoring phase after the CSV has submitted, and I suspect they may be exceeding a time limit for scoring phase.
\n(There was a similar mention about time limits during the scoring phase in a previous competition as well: [https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/459969])
Additionally, I’ve observed that when I submitted the same Notebook multiple times, sometimes the scoring completed successfully, and other times it resulted in a Submission Scoring Error.
\nGiven that the Notebook always finishes running and the row count check for sample_submission.csv passes on rerun, I believe it’s unlikely that there’s an issue with the Notebook itself.
I assume there may be a time limit of around 60 minutes currently,
\nbut would it be possible to consider extending it slightly (e.g. 90 minutes)?
\n(or if there are any other possible reasons why the same submission.csv might succeed or fail, I’d appreciate it if you could let me know.)
It’s frustrating when the same submission.csv passes or fails depending on the run…🥲
\nI’d greatly appreciate your consideration.
There is a 60 minute metric runtime limit, yes. I can take a closer look at your case in order to confirm no unexpected issues cropped up but won't be able to dig in until after the long weekend. We did roll out a metric performance improvement so hopefully you won't run into the same problem as much going forward.
", "rawMarkdown": "There is a 60 minute metric runtime limit, yes. I can take a closer look at your case in order to confirm no unexpected issues cropped up but won't be able to dig in until after the long weekend. We did roll out a metric performance improvement so hopefully you won't run into the same problem as much going forward.", "votes": null }, { "id": "3208368", "postDate": "05/24/2025 02:11:30", "content": "@sohier Thank you for your reply and taking measure for metric performance improvement.
\nI’ll try resubmitting the notebook that encountered a Submission Scoring Error.
\nIf the problem continues, I’d appreciate the chance to follow up next week…
@ryosukesaito This this been resolved, correct? The metric should be faster now.
", "rawMarkdown": "ryosukesaito This this been resolved, correct? The metric should be faster now.", "votes": null }, { "id": "3208515", "postDate": "05/24/2025 09:15:17", "content": "@eduardtrulls @sohier,
\nI just confirmed the scoring time has been significantly reduced, and the submission now completes in time.
\nThank you for your prompt support!
Thank you for conducting the rescoring.
", "rawMarkdown": "Thank you for conducting the rescoring.", "votes": null } ], "comments": [ { "id": 3208267, "author_name": "oldufo", "author_url": "", "post_date": "05/23/2025 20:38:28", "content": "The fixed version of the metric is available at
\nhttps://www.kaggle.com/datasets/eduardtrulls/imc25-utils/data?select=metric.py
Hi, @sohier , @eduardtrulls , @oldufo ,
\nThank you for conducting the rescoring.
\nThis might be specific to my case, but after rescoring, I’ve encountered Submission Scoring Error on several of my previous submissions that had successfully scored before.
\nAs I mentioned earlier in [https://www.kaggle.com/competitions/image-matching-challenge-2025/discussion/579064#3203775], my submissions tend to take a relatively long time during the scoring phase after the CSV has submitted, and I suspect they may be exceeding a time limit for scoring phase.
\n(There was a similar mention about time limits during the scoring phase in a previous competition as well: [https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/459969])
Additionally, I’ve observed that when I submitted the same Notebook multiple times, sometimes the scoring completed successfully, and other times it resulted in a Submission Scoring Error.
\nGiven that the Notebook always finishes running and the row count check for sample_submission.csv passes on rerun, I believe it’s unlikely that there’s an issue with the Notebook itself.
I assume there may be a time limit of around 60 minutes currently,
\nbut would it be possible to consider extending it slightly (e.g. 90 minutes)?
\n(or if there are any other possible reasons why the same submission.csv might succeed or fail, I’d appreciate it if you could let me know.)
It’s frustrating when the same submission.csv passes or fails depending on the run…🥲
\nI’d greatly appreciate your consideration.
There is a 60 minute metric runtime limit, yes. I can take a closer look at your case in order to confirm no unexpected issues cropped up but won't be able to dig in until after the long weekend. We did roll out a metric performance improvement so hopefully you won't run into the same problem as much going forward.
", "votes": null, "replies": [ { "id": 3208368, "author_name": "ryosukesaito", "author_url": "", "post_date": "05/24/2025 02:11:30", "content": "@sohier Thank you for your reply and taking measure for metric performance improvement.
\nI’ll try resubmitting the notebook that encountered a Submission Scoring Error.
\nIf the problem continues, I’d appreciate the chance to follow up next week…
@ryosukesaito This this been resolved, correct? The metric should be faster now.
", "votes": null, "replies": [ { "id": 3208515, "author_name": "ryosukesaito", "author_url": "", "post_date": "05/24/2025 09:15:17", "content": "@eduardtrulls @sohier,
\nI just confirmed the scoring time has been significantly reduced, and the submission now completes in time.
\nThank you for your prompt support!
Thank you for conducting the rescoring.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3208075": "Hi everyone,\n\nWe investigated [this report](https://www.kaggle.com/competitions/image-matching-challenge-2025/discussion/579625) and have confirmed that the competition launched with a bug in the metric code. We'll post a patched copy of the metric momentarily and will begin rescoring existing submissions later today. This metric takes quite a bit longer to evaluate than typical competition metrics so I may need to rescore submissions in batches over multiple days. **Submissions will remain open but the leaderboard will not be meaningful until the process is complete**. I will make a follow up post once there's more to report. \n\nIn terms of what to expect: many submissions were not affected at all by the bug and will retain their existing score. Submissions that did trigger the bug will have lower scores, sometimes by a large factor (2x). \n\nAs for the bug itself:\n\nThe metric has two stages. First, it greedily matches the clusters provided by the user, to the ground truth clusters. Second, it computes the accuracy of the estimated image poses for each of the clusters. The issue is in the greedy assignment step. Under some conditions can lead to an improper association between a ground truth scene and the outliers cluster. This cascades into the computation of mAA and clustering score. The problem lies in the initialization of the tables used in the greedy assignment step, which are initially set to zero. When filling the table, entries corresponding to the outlier cluster are also set to 0. In some situations, the best cluster and the outliers cluster are given the same score in the greedy assignment, and the outlier cluster is wrongly selected instead. Specifically, this happens when (a) the best non-outlier cluster for a scene has a score of zero, and (b) the submission file contains \"outlier\" rows before non-outlier rows. The fix is simple: initialize the tables to -1 instead.\n\nWe would like to send a great deal of thanks to the users who noticed this and reached out, especially to @gowillgo and @petermoorhouse.\n\nApologies again for the disruption and thank you in advance for your patience.\n\nEdit 1: The rescores are under way. \n\nEdit 2: The rescores are now 99.9% complete. The final dozen or so submissions should be done very soon.\n\nEdit 3: You should now also see faster metric scoring times.", "3208267": "The fixed version of the metric is available at \nhttps://www.kaggle.com/datasets/eduardtrulls/imc25-utils/data?select=metric.py", "3208351": "Hi, @sohier , @eduardtrulls , @oldufo ,\n\nThank you for conducting the rescoring.\n\nThis might be specific to my case, but after rescoring, I’ve encountered **Submission Scoring Error** on several of my previous submissions that had successfully scored before.\n\nAs I mentioned earlier in [https://www.kaggle.com/competitions/image-matching-challenge-2025/discussion/579064#3203775], my submissions tend to take a relatively long time during the scoring phase after the CSV has submitted, and I suspect they may be exceeding a time limit for scoring phase.\n(There was a similar mention about time limits during the scoring phase in a previous competition as well: [https://www.kaggle.com/competitions/blood-vessel-segmentation/discussion/459969])\n\nAdditionally, I’ve observed that when I submitted the same Notebook multiple times, sometimes the scoring completed successfully, and other times it resulted in a Submission Scoring Error.\nGiven that the Notebook always finishes running and the row count check for sample_submission.csv passes on rerun, I believe it’s unlikely that there’s an issue with the Notebook itself.\n\nI assume there may be a time limit of around 60 minutes currently,\nbut would it be possible to consider extending it slightly (e.g. 90 minutes)?\n(or if there are any other possible reasons why the same submission.csv might succeed or fail, I’d appreciate it if you could let me know.)\n\nIt’s frustrating when the same submission.csv passes or fails depending on the run...🥲\nI’d greatly appreciate your consideration.", "3208366": "There is a 60 minute metric runtime limit, yes. I can take a closer look at your case in order to confirm no unexpected issues cropped up but won't be able to dig in until after the long weekend. We did roll out a metric performance improvement so hopefully you won't run into the same problem as much going forward.", "3208368": "sohier Thank you for your reply and taking measure for metric performance improvement.\n\nI’ll try resubmitting the notebook that encountered a Submission Scoring Error.\nIf the problem continues, I’d appreciate the chance to follow up next week...", "3208481": "ryosukesaito This this been resolved, correct? The metric should be faster now.", "3208515": "eduardtrulls @sohier,\nI just confirmed the scoring time has been significantly reduced, and the submission now completes in time.\nThank you for your prompt support!", "3209386": "Thank you for conducting the rescoring." }, "source": "meta" }