{
  "id": 401273,
  "title": "Actual mAA thresholds",
  "url": "/competitions/image-matching-challenge-2023/discussion/401273",
  "author_name": "",
  "post_date": "2023-04-12T14:46:28.733954600Z",
  "votes": 3,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Would it be possible to share a couple other examples of mAA thresholds for some of the test scenes? It would be nice to gauge the relative importance of relative orientation vs relative translation? If not, can we consider the thresholds on the \"evaluation\" page as average? Those were the thresholds used last year - and was there some feeling that they were too loose? On average, are they tighter this year? thanks!</p>",
  "messages": [
    {
      "id": "2219404",
      "postDate": "04/12/2023 14:46:28",
      "content": "<p>Would it be possible to share a couple other examples of mAA thresholds for some of the test scenes? It would be nice to gauge the relative importance of relative orientation vs relative translation? If not, can we consider the thresholds on the \"evaluation\" page as average? Those were the thresholds used last year - and was there some feeling that they were too loose? On average, are they tighter this year? thanks!</p>",
      "rawMarkdown": "Would it be possible to share a couple other examples of mAA thresholds for some of the test scenes? It would be nice to gauge the relative importance of relative orientation vs relative translation? If not, can we consider the thresholds on the \"evaluation\" page as average? Those were the thresholds used last year - and was there some feeling that they were too loose? On average, are they tighter this year? thanks!",
      "votes": null
    },
    {
      "id": "2219424",
      "postDate": "04/12/2023 14:52:56",
      "content": "<p>I don't think we are going to share threshold for the test scenes, that would be a huge leak. However, I can guarantee that thresholds are not too loose. <br>\nI also can recommend you to run some baselines on the training set and see what you can get with different thresholds. </p>",
      "rawMarkdown": "I don't think we are going to share threshold for the test scenes, that would be a huge leak. However, I can guarantee that thresholds are not too loose. \nI also can recommend you to run some baselines on the training set and see what you can get with different thresholds.",
      "votes": null
    },
    {
      "id": "2219669",
      "postDate": "04/12/2023 19:02:46",
      "content": "<p>Thanks, Dmytro!</p>\n<p>I noticed Eduard did exactly as you suggested here:<br>\n<a href=\"https://www.kaggle.com/code/eduardtrulls/imc2023-evaluation/notebook\" target=\"_blank\">https://www.kaggle.com/code/eduardtrulls/imc2023-evaluation/notebook</a></p>\n<p>So last year the thresholds were:<br>\nthresholds_r = np.linspace(1, 10, 10)  # In degrees.<br>\nthresholds_t = np.geomspace(0.2, 5, 10)  # In meters.</p>\n<p>And this year, this is maybe a median of the several dataset-dependent thresholds that Eduard picked:<br>\nnp.linspace(0.2, 10, 10)<br>\nnp.geomspace(0.05, 1, 10)</p>\n<p>Probably more tightening on the translation error thresholds this year ;)</p>",
      "rawMarkdown": "Thanks, Dmytro!\n\nI noticed Eduard did exactly as you suggested here:\nhttps://www.kaggle.com/code/eduardtrulls/imc2023-evaluation/notebook\n\nSo last year the thresholds were:\nthresholds_r = np.linspace(1, 10, 10)  # In degrees.\nthresholds_t = np.geomspace(0.2, 5, 10)  # In meters.\n\nAnd this year, this is maybe a median of the several dataset-dependent thresholds that Eduard picked:\nnp.linspace(0.2, 10, 10)\nnp.geomspace(0.05, 1, 10)\n\nProbably more tightening on the translation error thresholds this year ;)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2219424,
      "author_name": "oldufo",
      "author_url": "",
      "post_date": "04/12/2023 14:52:56",
      "content": "<p>I don't think we are going to share threshold for the test scenes, that would be a huge leak. However, I can guarantee that thresholds are not too loose. <br>\nI also can recommend you to run some baselines on the training set and see what you can get with different thresholds. </p>",
      "votes": null,
      "replies": [
        {
          "id": 2219669,
          "author_name": "dingmuti",
          "author_url": "",
          "post_date": "04/12/2023 19:02:46",
          "content": "<p>Thanks, Dmytro!</p>\n<p>I noticed Eduard did exactly as you suggested here:<br>\n<a href=\"https://www.kaggle.com/code/eduardtrulls/imc2023-evaluation/notebook\" target=\"_blank\">https://www.kaggle.com/code/eduardtrulls/imc2023-evaluation/notebook</a></p>\n<p>So last year the thresholds were:<br>\nthresholds_r = np.linspace(1, 10, 10)  # In degrees.<br>\nthresholds_t = np.geomspace(0.2, 5, 10)  # In meters.</p>\n<p>And this year, this is maybe a median of the several dataset-dependent thresholds that Eduard picked:<br>\nnp.linspace(0.2, 10, 10)<br>\nnp.geomspace(0.05, 1, 10)</p>\n<p>Probably more tightening on the translation error thresholds this year ;)</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2219404": "Would it be possible to share a couple other examples of mAA thresholds for some of the test scenes? It would be nice to gauge the relative importance of relative orientation vs relative translation? If not, can we consider the thresholds on the \"evaluation\" page as average? Those were the thresholds used last year - and was there some feeling that they were too loose? On average, are they tighter this year? thanks!",
    "2219424": "I don't think we are going to share threshold for the test scenes, that would be a huge leak. However, I can guarantee that thresholds are not too loose. \nI also can recommend you to run some baselines on the training set and see what you can get with different thresholds.",
    "2219669": "Thanks, Dmytro!\n\nI noticed Eduard did exactly as you suggested here:\nhttps://www.kaggle.com/code/eduardtrulls/imc2023-evaluation/notebook\n\nSo last year the thresholds were:\nthresholds_r = np.linspace(1, 10, 10)  # In degrees.\nthresholds_t = np.geomspace(0.2, 5, 10)  # In meters.\n\nAnd this year, this is maybe a median of the several dataset-dependent thresholds that Eduard picked:\nnp.linspace(0.2, 10, 10)\nnp.geomspace(0.05, 1, 10)\n\nProbably more tightening on the translation error thresholds this year ;)"
  },
  "source": "meta"
}