{
  "id": 412958,
  "title": "Validation vs LB scores",
  "url": "/competitions/image-matching-challenge-2023/discussion/412958",
  "author_name": "Gunes Evitan",
  "post_date": "2023-05-26T05:15:56.664000",
  "votes": 27,
  "comment_count": 57,
  "views": 0,
  "content": "<p>Here are my validation scores for every scene and the public leaderboard score listed. It looks like improvements on easier scenes like haiper doesn't translate to leaderboard. There are also scenes like dioscuri on which I can't get any improvement at all. I think public and private leaderboard is similar to heritage dataset. Do you have similar results? </p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>mAA</th>\n<th>mAA Rotation</th>\n<th>mAA Translation</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>bike</td>\n<td>0.921</td>\n<td>0.999</td>\n<td>0.921</td>\n</tr>\n<tr>\n<td>chairs</td>\n<td>0.976</td>\n<td>0.999</td>\n<td>0.976</td>\n</tr>\n<tr>\n<td>fountain</td>\n<td>0.998</td>\n<td>1</td>\n<td>0.998</td>\n</tr>\n<tr>\n<td>dioscuri</td>\n<td>0.001</td>\n<td>0.005</td>\n<td>0.02</td>\n</tr>\n<tr>\n<td>cyprus</td>\n<td>0.2535</td>\n<td>0.2701</td>\n<td>0.2839</td>\n</tr>\n<tr>\n<td>wall</td>\n<td>0.8753</td>\n<td>0.9503</td>\n<td>0.8880</td>\n</tr>\n<tr>\n<td>kyiv-puppet-theater</td>\n<td>0.2501</td>\n<td>0.4390</td>\n<td>0.3418</td>\n</tr>\n<tr>\n<td>haiper</td>\n<td>0.9652</td>\n<td>0.999</td>\n<td>0.9652</td>\n</tr>\n<tr>\n<td>heritage</td>\n<td>0.3769</td>\n<td>0.4086</td>\n<td>0.3977</td>\n</tr>\n<tr>\n<td>urban</td>\n<td>0.2501</td>\n<td>0.4390</td>\n<td>0.3418</td>\n</tr>\n<tr>\n<td><strong>Validation</strong></td>\n<td><strong>0.5307</strong></td>\n<td><strong>0.6157</strong></td>\n<td><strong>0.5681</strong></td>\n</tr>\n<tr>\n<td><strong>Leaderboard</strong></td>\n<td><strong>0.214</strong></td>\n<td></td>\n<td></td>\n</tr>\n</tbody>\n</table>",
  "messages": [
    {
      "id": 2274552,
      "postDate": "2023-05-26T05:15:56.663Z",
      "content": "<p>Here are my validation scores for every scene and the public leaderboard score listed. It looks like improvements on easier scenes like haiper doesn't translate to leaderboard. There are also scenes like dioscuri on which I can't get any improvement at all. I think public and private leaderboard is similar to heritage dataset. Do you have similar results? </p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>mAA</th>\n<th>mAA Rotation</th>\n<th>mAA Translation</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>bike</td>\n<td>0.921</td>\n<td>0.999</td>\n<td>0.921</td>\n</tr>\n<tr>\n<td>chairs</td>\n<td>0.976</td>\n<td>0.999</td>\n<td>0.976</td>\n</tr>\n<tr>\n<td>fountain</td>\n<td>0.998</td>\n<td>1</td>\n<td>0.998</td>\n</tr>\n<tr>\n<td>dioscuri</td>\n<td>0.001</td>\n<td>0.005</td>\n<td>0.02</td>\n</tr>\n<tr>\n<td>cyprus</td>\n<td>0.2535</td>\n<td>0.2701</td>\n<td>0.2839</td>\n</tr>\n<tr>\n<td>wall</td>\n<td>0.8753</td>\n<td>0.9503</td>\n<td>0.8880</td>\n</tr>\n<tr>\n<td>kyiv-puppet-theater</td>\n<td>0.2501</td>\n<td>0.4390</td>\n<td>0.3418</td>\n</tr>\n<tr>\n<td>haiper</td>\n<td>0.9652</td>\n<td>0.999</td>\n<td>0.9652</td>\n</tr>\n<tr>\n<td>heritage</td>\n<td>0.3769</td>\n<td>0.4086</td>\n<td>0.3977</td>\n</tr>\n<tr>\n<td>urban</td>\n<td>0.2501</td>\n<td>0.4390</td>\n<td>0.3418</td>\n</tr>\n<tr>\n<td><strong>Validation</strong></td>\n<td><strong>0.5307</strong></td>\n<td><strong>0.6157</strong></td>\n<td><strong>0.5681</strong></td>\n</tr>\n<tr>\n<td><strong>Leaderboard</strong></td>\n<td><strong>0.214</strong></td>\n<td></td>\n<td></td>\n</tr>\n</tbody>\n</table>",
      "rawMarkdown": "Here are my validation scores for every scene and the public leaderboard score listed. It looks like improvements on easier scenes like haiper doesn't translate to leaderboard. There are also scenes like dioscuri on which I can't get any improvement at all. I think public and private leaderboard is similar to heritage dataset. Do you have similar results? \n\n\n|                     | mAA    | mAA Rotation | mAA Translation |\n|---------------------|--------|--------------|-----------------|\n| bike                | 0.921  | 0.999        | 0.921           |\n| chairs              | 0.976  | 0.999        | 0.976           |\n| fountain            | 0.998  | 1            | 0.998           |\n| dioscuri            | 0.001  | 0.005        | 0.02            |\n| cyprus              | 0.2535 | 0.2701       | 0.2839          |\n| wall                | 0.8753 | 0.9503       | 0.8880          |\n| kyiv-puppet-theater | 0.2501 | 0.4390       | 0.3418          |\n| haiper              | 0.9652 | 0.999        | 0.9652          |\n| heritage            | 0.3769 | 0.4086       | 0.3977          |\n| urban               | 0.2501 | 0.4390       | 0.3418          |\n| **Validation**               | **0.5307** | **0.6157**       | **0.5681**          |\n| **Leaderboard**               | **0.214** |||\n\n",
      "votes": 26
    },
    {
      "id": 2280942,
      "postDate": "2023-05-30T13:23:13.890Z",
      "content": "<p>LB：0.467<br>\n*** METRICS ***<br>\nurban / kyiv-puppet-theater (26 images, 325 pairs) -&gt; mAA=0.632308, mAA_q=0.823077, mAA_t=0.674462<br>\nurban -&gt; mAA=0.632308</p>\n<p>heritage / dioscuri (174 images, 15051 pairs) -&gt; mAA=0.680167, mAA_q=0.799356, mAA_t=0.686001<br>\nheritage / cyprus (30 images, 435 pairs) -&gt; mAA=0.056092, mAA_q=0.057471, mAA_t=0.075402<br>\nheritage / wall (43 images, 903 pairs) -&gt; mAA=0.726910, mAA_q=0.922038, mAA_t=0.739867<br>\nheritage -&gt; mAA=0.487723</p>\n<p>haiper / bike (15 images, 105 pairs) -&gt; mAA=0.925714, mAA_q=0.999048, mAA_t=0.925714<br>\nhaiper / chairs (16 images, 120 pairs) -&gt; mAA=0.903333, mAA_q=0.986667, mAA_t=0.906667<br>\nhaiper / fountain (23 images, 253 pairs) -&gt; mAA=1.000000, mAA_q=1.000000, mAA_t=1.000000<br>\nhaiper -&gt; mAA=0.943016</p>\n<p>Final metric -&gt; mAA=0.687682 (t: 2.7530465126037598 sec.)</p>",
      "rawMarkdown": "LB：0.467\n*** METRICS ***\nurban / kyiv-puppet-theater (26 images, 325 pairs) -> mAA=0.632308, mAA_q=0.823077, mAA_t=0.674462\nurban -> mAA=0.632308\n\nheritage / dioscuri (174 images, 15051 pairs) -> mAA=0.680167, mAA_q=0.799356, mAA_t=0.686001\nheritage / cyprus (30 images, 435 pairs) -> mAA=0.056092, mAA_q=0.057471, mAA_t=0.075402\nheritage / wall (43 images, 903 pairs) -> mAA=0.726910, mAA_q=0.922038, mAA_t=0.739867\nheritage -> mAA=0.487723\n\nhaiper / bike (15 images, 105 pairs) -> mAA=0.925714, mAA_q=0.999048, mAA_t=0.925714\nhaiper / chairs (16 images, 120 pairs) -> mAA=0.903333, mAA_q=0.986667, mAA_t=0.906667\nhaiper / fountain (23 images, 253 pairs) -> mAA=1.000000, mAA_q=1.000000, mAA_t=1.000000\nhaiper -> mAA=0.943016\n\nFinal metric -> mAA=0.687682 (t: 2.7530465126037598 sec.)\n",
      "votes": 14,
      "replies": [
        {
          "id": 2287489,
          "postDate": "2023-06-04T14:25:24.177Z",
          "content": "<p>really helpful,thanks</p>",
          "rawMarkdown": "really helpful,thanks"
        },
        {
          "id": 2288192,
          "postDate": "2023-06-05T08:13:29.903Z",
          "content": "<p>Look helpful for me, cheers</p>",
          "rawMarkdown": "Look helpful for me, cheers"
        },
        {
          "id": 2295318,
          "postDate": "2023-06-10T19:27:36.127Z",
          "content": "<p>Thank you. How long did your submission run? </p>",
          "rawMarkdown": "Thank you. How long did your submission run? "
        }
      ]
    },
    {
      "id": 2274955,
      "postDate": "2023-05-26T11:59:22.700Z",
      "content": "<p>accroding to my last a few try, whether to skip scenes larger than 40 images can lead to remarkable difference(like 0.05 points). Given that my code did poor in these large scene, i guess large scenes really counts in final score. </p>",
      "rawMarkdown": "accroding to my last a few try, whether to skip scenes larger than 40 images can lead to remarkable difference(like 0.05 points). Given that my code did poor in these large scene, i guess large scenes really counts in final score. ",
      "votes": 7,
      "replies": [
        {
          "id": 2281538,
          "postDate": "2023-05-31T01:25:41.743Z",
          "content": "<p>Hi, have you found a way to reduce the randomness of pycolmap? I noticed that you have been able to lock the error of skipping scenes to 0.05, while I have submitted the same code multiple times and the score sometimes varies by more than 0.05.</p>",
          "rawMarkdown": "Hi, have you found a way to reduce the randomness of pycolmap? I noticed that you have been able to lock the error of skipping scenes to 0.05, while I have submitted the same code multiple times and the score sometimes varies by more than 0.05.",
          "votes": 4,
          "replies": [
            {
              "id": 2291903,
              "postDate": "2023-06-07T23:38:56.187Z",
              "content": "<p>I have not.Randomness exists, what I can do is only try multiple times to eliminate its effect</p>",
              "rawMarkdown": "I have not.Randomness exists, what I can do is only try multiple times to eliminate its effect",
              "votes": 1
            },
            {
              "id": 2291969,
              "postDate": "2023-06-08T02:14:39.570Z",
              "content": "<p>Thank you for your response.</p>",
              "rawMarkdown": "Thank you for your response."
            }
          ]
        },
        {
          "id": 2290941,
          "postDate": "2023-06-07T08:08:23.807Z",
          "content": "<blockquote>\n  <p>accroding to my last a few try, whether to skip scenes larger than 40 images can lead to remarkable difference(like 0.05 points). Given that my code did poor in these large scene, i guess large scenes really counts in final score.</p>\n</blockquote>\n<p>What do you mean with \"skip scenes\"? Do you not submit anything at all for scenes with more than 40 images?</p>",
          "rawMarkdown": "> accroding to my last a few try, whether to skip scenes larger than 40 images can lead to remarkable difference(like 0.05 points). Given that my code did poor in these large scene, i guess large scenes really counts in final score.\n\nWhat do you mean with \"skip scenes\"? Do you not submit anything at all for scenes with more than 40 images?",
          "replies": [
            {
              "id": 2291905,
              "postDate": "2023-06-07T23:42:40.267Z",
              "content": "<p>As you may found, these scenes are time consumming.\"skip scenes\" simply means creating a random but valid result without computation</p>",
              "rawMarkdown": "As you may found, these scenes are time consumming.\"skip scenes\" simply means creating a random but valid result without computation",
              "votes": 2
            }
          ]
        }
      ]
    },
    {
      "id": 2276879,
      "postDate": "2023-05-27T09:30:16.067Z",
      "content": "<p>my lb : 0.338</p>\n<pre><code>haiper / bike (15 images, 105 pairs) -&gt; mAA=0.921905, mAA_q=1.000000, mAA_t=0.921905\nhaiper / chairs (16 images, 120 pairs) -&gt; mAA=0.766667, mAA_q=0.846667, mAA_t=0.777500\nhaiper / fountain (23 images, 253 pairs) -&gt; mAA=0.999210, mAA_q=1.000000, mAA_t=0.999210\nhaiper -&gt; mAA=0.895927\n\nheritage / cyprus (30 images, 435 pairs) -&gt; mAA=0.021839, mAA_q=0.022989, mAA_t=0.027816\nheritage / dioscuri (174 images, 15051 pairs) -&gt; mAA=0.435572, mAA_q=0.535353, mAA_t=0.442469\nheritage / wall (43 images, 903 pairs) -&gt; mAA=0.175748, mAA_q=0.442636, mAA_t=0.188815\nheritage -&gt; mAA=0.211053\n\nurban / kyiv-puppet-theater (26 images, 325 pairs) -&gt; mAA=0.755077, mAA_q=0.915077, mAA_t=0.780923\nurban -&gt; mAA=0.755077\n\nFinal metric -&gt; mAA=0.620686 (t: 3.0341649055480957 sec.)\n</code></pre>",
      "rawMarkdown": "my lb : 0.338\n\n```\nhaiper / bike (15 images, 105 pairs) -> mAA=0.921905, mAA_q=1.000000, mAA_t=0.921905\nhaiper / chairs (16 images, 120 pairs) -> mAA=0.766667, mAA_q=0.846667, mAA_t=0.777500\nhaiper / fountain (23 images, 253 pairs) -> mAA=0.999210, mAA_q=1.000000, mAA_t=0.999210\nhaiper -> mAA=0.895927\n\nheritage / cyprus (30 images, 435 pairs) -> mAA=0.021839, mAA_q=0.022989, mAA_t=0.027816\nheritage / dioscuri (174 images, 15051 pairs) -> mAA=0.435572, mAA_q=0.535353, mAA_t=0.442469\nheritage / wall (43 images, 903 pairs) -> mAA=0.175748, mAA_q=0.442636, mAA_t=0.188815\nheritage -> mAA=0.211053\n\nurban / kyiv-puppet-theater (26 images, 325 pairs) -> mAA=0.755077, mAA_q=0.915077, mAA_t=0.780923\nurban -> mAA=0.755077\n\nFinal metric -> mAA=0.620686 (t: 3.0341649055480957 sec.)\n```",
      "votes": 8,
      "replies": [
        {
          "id": 2276911,
          "postDate": "2023-05-27T10:01:35.200Z",
          "content": "<p>Nice. Higher urban score is correlated with lb score as I thought.</p>",
          "rawMarkdown": "Nice. Higher urban score is correlated with lb score as I thought.",
          "votes": 5
        },
        {
          "id": 2276941,
          "postDate": "2023-05-27T10:23:51.040Z",
          "content": "<p>thanks for sharing</p>",
          "rawMarkdown": "thanks for sharing"
        },
        {
          "id": 2277042,
          "postDate": "2023-05-27T11:43:08.187Z",
          "content": "<p>thanks for your sharing</p>",
          "rawMarkdown": "thanks for your sharing"
        }
      ]
    },
    {
      "id": 2295648,
      "postDate": "2023-06-11T07:15:37.630Z",
      "content": "<p>For me, a higher CV brings worse LB.</p>\n<p>urban / kyiv-puppet-theater (26 images, 325 pairs) -&gt; mAA=0.915077, mAA_q=0.979692, mAA_t=0.924923<br>\nurban -&gt; mAA=0.915077</p>\n<p>heritage / dioscuri (174 images, 15051 pairs) -&gt; mAA=0.508478, mAA_q=0.637612, mAA_t=0.514238<br>\nheritage / cyprus (30 images, 435 pairs) -&gt; mAA=0.022299, mAA_q=0.022989, mAA_t=0.028046<br>\nheritage / wall (43 images, 903 pairs) -&gt; mAA=0.523810, mAA_q=0.882946, mAA_t=0.533666<br>\nheritage -&gt; mAA=0.351529</p>\n<p>haiper / bike (15 images, 105 pairs) -&gt; mAA=0.951429, mAA_q=1.000000, mAA_t=0.951429<br>\nhaiper / chairs (16 images, 120 pairs) -&gt; mAA=0.929167, mAA_q=0.995000, mAA_t=0.931667<br>\nhaiper / fountain (23 images, 253 pairs) -&gt; mAA=0.999605, mAA_q=1.000000, mAA_t=0.999605<br>\nhaiper -&gt; mAA=0.960067</p>\n<p>Final metric -&gt; mAA=0.742224 (t: 2.0051422119140625 sec.)</p>\n<p>LB: 0.37😂</p>",
      "rawMarkdown": "For me, a higher CV brings worse LB.\n\nurban / kyiv-puppet-theater (26 images, 325 pairs) -> mAA=0.915077, mAA_q=0.979692, mAA_t=0.924923\nurban -> mAA=0.915077\n\nheritage / dioscuri (174 images, 15051 pairs) -> mAA=0.508478, mAA_q=0.637612, mAA_t=0.514238\nheritage / cyprus (30 images, 435 pairs) -> mAA=0.022299, mAA_q=0.022989, mAA_t=0.028046\nheritage / wall (43 images, 903 pairs) -> mAA=0.523810, mAA_q=0.882946, mAA_t=0.533666\nheritage -> mAA=0.351529\n\nhaiper / bike (15 images, 105 pairs) -> mAA=0.951429, mAA_q=1.000000, mAA_t=0.951429\nhaiper / chairs (16 images, 120 pairs) -> mAA=0.929167, mAA_q=0.995000, mAA_t=0.931667\nhaiper / fountain (23 images, 253 pairs) -> mAA=0.999605, mAA_q=1.000000, mAA_t=0.999605\nhaiper -> mAA=0.960067\n\nFinal metric -> mAA=0.742224 (t: 2.0051422119140625 sec.)\n\nLB: 0.37😂",
      "votes": 6,
      "replies": [
        {
          "id": 2295655,
          "postDate": "2023-06-11T07:21:22.107Z",
          "content": "<blockquote>\n  <p>For me, a higher CV brings worse LB.</p>\n  <p>urban / kyiv-puppet-theater (26 images, 325 pairs) -&gt; mAA=0.915077, mAA_q=0.979692, mAA_t=0.924923<br>\n  urban -&gt; mAA=0.915077</p>\n  <p>heritage / dioscuri (174 images, 15051 pairs) -&gt; mAA=0.508478, mAA_q=0.637612, mAA_t=0.514238<br>\n  heritage / cyprus (30 images, 435 pairs) -&gt; mAA=0.022299, mAA_q=0.022989, mAA_t=0.028046<br>\n  heritage / wall (43 images, 903 pairs) -&gt; mAA=0.523810, mAA_q=0.882946, mAA_t=0.533666<br>\n  heritage -&gt; mAA=0.351529</p>\n  <p>haiper / bike (15 images, 105 pairs) -&gt; mAA=0.951429, mAA_q=1.000000, mAA_t=0.951429<br>\n  haiper / chairs (16 images, 120 pairs) -&gt; mAA=0.929167, mAA_q=0.995000, mAA_t=0.931667<br>\n  haiper / fountain (23 images, 253 pairs) -&gt; mAA=0.999605, mAA_q=1.000000, mAA_t=0.999605<br>\n  haiper -&gt; mAA=0.960067</p>\n  <p>Final metric -&gt; mAA=0.742224 (t: 2.0051422119140625 sec.)</p>\n  <p>LB: 0.37😂</p>\n</blockquote>\n<p>Same for us…</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F443651%2Ffb62a4598acc83d811a595f9f77d9f2c%2Fcv_scores.png?generation=1686468079902863&amp;alt=media\" alt=\"\"></p>",
          "rawMarkdown": "> For me, a higher CV brings worse LB.\n> \n> urban / kyiv-puppet-theater (26 images, 325 pairs) -> mAA=0.915077, mAA_q=0.979692, mAA_t=0.924923\n> urban -> mAA=0.915077\n> \n> heritage / dioscuri (174 images, 15051 pairs) -> mAA=0.508478, mAA_q=0.637612, mAA_t=0.514238\n> heritage / cyprus (30 images, 435 pairs) -> mAA=0.022299, mAA_q=0.022989, mAA_t=0.028046\n> heritage / wall (43 images, 903 pairs) -> mAA=0.523810, mAA_q=0.882946, mAA_t=0.533666\n> heritage -> mAA=0.351529\n> \n> haiper / bike (15 images, 105 pairs) -> mAA=0.951429, mAA_q=1.000000, mAA_t=0.951429\n> haiper / chairs (16 images, 120 pairs) -> mAA=0.929167, mAA_q=0.995000, mAA_t=0.931667\n> haiper / fountain (23 images, 253 pairs) -> mAA=0.999605, mAA_q=1.000000, mAA_t=0.999605\n> haiper -> mAA=0.960067\n> \n> Final metric -> mAA=0.742224 (t: 2.0051422119140625 sec.)\n> \n> LB: 0.37😂\n\nSame for us...\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F443651%2Ffb62a4598acc83d811a595f9f77d9f2c%2Fcv_scores.png?generation=1686468079902863&alt=media)\n",
          "votes": 3,
          "replies": [
            {
              "id": 2295673,
              "postDate": "2023-06-11T07:32:21.250Z",
              "content": "<p>How much time does it take for you guys to run on the leaderboard? Have you experienced significant slowdown during the past 2 days? All my notebooks timed out though they used to finish running. </p>",
              "rawMarkdown": "How much time does it take for you guys to run on the leaderboard? Have you experienced significant slowdown during the past 2 days? All my notebooks timed out though they used to finish running. ",
              "votes": 1
            },
            {
              "id": 2295692,
              "postDate": "2023-06-11T07:39:08.043Z",
              "content": "<p>Yes, my submission has timed out. I have never had it before.</p>",
              "rawMarkdown": "Yes, my submission has timed out. I have never had it before.",
              "votes": 1
            },
            {
              "id": 2295694,
              "postDate": "2023-06-11T07:43:06.713Z",
              "content": "<p>Hi <a href=\"https://www.kaggle.com/group16\" target=\"_blank\">@group16</a> , Your CV is too great, I can't even believe my eyes 😂.  Are you sure you get the CV score in a right way?</p>",
              "rawMarkdown": "Hi @group16 , Your CV is too great, I can't even believe my eyes 😂.  Are you sure you get the CV score in a right way?",
              "votes": 1
            },
            {
              "id": 2295703,
              "postDate": "2023-06-11T07:51:20.050Z",
              "content": "<blockquote>\n  <p>Hi <a href=\"https://www.kaggle.com/group16\" target=\"_blank\">@group16</a> , Your CV is too great, I can't even believe my eyes 😂.  Are you sure you get the CV score in a right way?</p>\n</blockquote>\n<p>Hi <a href=\"https://www.kaggle.com/forcewithme\" target=\"_blank\">@forcewithme</a>,</p>\n<p>We do think that is correct indeed :). To say that we are a little bit frustrated that there is such a discrepancy between the training datasets &amp; scenes and the test ones, would be the understatement of the year.</p>\n<p>I think we at least might win the participation trophy of having the best CV scores in this competition.</p>",
              "rawMarkdown": "> Hi @group16 , Your CV is too great, I can't even believe my eyes 😂.  Are you sure you get the CV score in a right way?\n\nHi @forcewithme,\n\nWe do think that is correct indeed :). To say that we are a little bit frustrated that there is such a discrepancy between the training datasets & scenes and the test ones, would be the understatement of the year.\n\nI think we at least might win the participation trophy of having the best CV scores in this competition.",
              "votes": 1
            },
            {
              "id": 2295704,
              "postDate": "2023-06-11T07:51:30.420Z",
              "content": "<p>We always tried to boost our validation score and I was getting 0.79 mAA at some point but it didn't work out on the public leaderboard. I think a little shake might happen at the end.</p>",
              "rawMarkdown": "We always tried to boost our validation score and I was getting 0.79 mAA at some point but it didn't work out on the public leaderboard. I think a little shake might happen at the end.",
              "votes": 2
            },
            {
              "id": 2295708,
              "postDate": "2023-06-11T07:54:41.107Z",
              "content": "<p>Wow, however I find there's some randomness in my submissions (up to 0.025 on lb), for my cv it's even bigger, for the same code sometimes the cv is good sometimes it's not that good. </p>",
              "rawMarkdown": "Wow, however I find there's some randomness in my submissions (up to 0.025 on lb), for my cv it's even bigger, for the same code sometimes the cv is good sometimes it's not that good. ",
              "votes": 1
            },
            {
              "id": 2295711,
              "postDate": "2023-06-11T07:56:50.987Z",
              "content": "<p>I guess everyone is working on this last minute lol which may have slowed down the Kaggle server, not very fair for those who started late but understandable…</p>",
              "rawMarkdown": "I guess everyone is working on this last minute lol which may have slowed down the Kaggle server, not very fair for those who started late but understandable...",
              "votes": 1
            },
            {
              "id": 2295714,
              "postDate": "2023-06-11T07:59:38.773Z",
              "content": "<blockquote>\n  <p>To say that we are a little bit frustrated that there is such a discrepancy between the training datasets &amp; scenes and the test ones</p>\n</blockquote>\n<p>Yeah, a competition with almost no correlation between CV and LB is always frustrated. I have the same feeling  for the last month. </p>",
              "rawMarkdown": ">To say that we are a little bit frustrated that there is such a discrepancy between the training datasets & scenes and the test ones\n\nYeah, a competition with almost no correlation between CV and LB is always frustrated. I have the same feeling  for the last month. "
            },
            {
              "id": 2295718,
              "postDate": "2023-06-11T08:01:46.330Z",
              "content": "<p>I also encountered this problem. I repeatedly submitted the same code without exactly the same score.</p>",
              "rawMarkdown": "I also encountered this problem. I repeatedly submitted the same code without exactly the same score."
            },
            {
              "id": 2295720,
              "postDate": "2023-06-11T08:03:40.197Z",
              "content": "<p>Yeah cuz Colmap reconstruction has randomness, I think one way to alleviate it would be to generate more reconstruction models then choose the best one, but it takes more time. </p>",
              "rawMarkdown": "Yeah cuz Colmap reconstruction has randomness, I think one way to alleviate it would be to generate more reconstruction models then choose the best one, but it takes more time. ",
              "votes": 1
            },
            {
              "id": 2295728,
              "postDate": "2023-06-11T08:07:33.747Z",
              "content": "<p>I think the final ranking will change a lot, because the final submitted code will be re-run.</p>",
              "rawMarkdown": "I think the final ranking will change a lot, because the final submitted code will be re-run.",
              "votes": -1
            },
            {
              "id": 2295733,
              "postDate": "2023-06-11T08:08:56.960Z",
              "content": "<p>Hi <a href=\"https://www.kaggle.com/gunesevitan\" target=\"_blank\">@gunesevitan</a> , I think the shake depends on how they split the dataset, which they didn't give an answer in <a href=\"https://www.kaggle.com/competitions/image-matching-challenge-2023/discussion/411778\" target=\"_blank\">my post</a>.</p>\n<p>IMO, if they split the leaderboard by image pair, which means the public and private Leaderboard <strong>have overlap on scenes</strong>, there will be little shake. Although I can't figure out how to get the score for both leaderboards if there is an overlap on scenes. Also, I think if all the scenes are included in both leaderboards,  it's just an overfitting game. </p>\n<p>But on the other hand, if they split the dataset by <code>scene</code> or <code>dataset</code>，which means that <strong>there won't be overlap</strong> between scenes on the Leaderboards, there may be a super huge shake, or not, depends on the discrepancy between the scenes in both Leaderboards.</p>\n<p>So it's really a hard choice for them. Let's wait and see what will happen.👀</p>",
              "rawMarkdown": "Hi @gunesevitan , I think the shake depends on how they split the dataset, which they didn't give an answer in [my post](https://www.kaggle.com/competitions/image-matching-challenge-2023/discussion/411778).\n\nIMO, if they split the leaderboard by image pair, which means the public and private Leaderboard **have overlap on scenes**, there will be little shake. Although I can't figure out how to get the score for both leaderboards if there is an overlap on scenes. Also, I think if all the scenes are included in both leaderboards,  it's just an overfitting game. \n\nBut on the other hand, if they split the dataset by `scene` or `dataset`，which means that **there won't be overlap** between scenes on the Leaderboards, there may be a super huge shake, or not, depends on the discrepancy between the scenes in both Leaderboards.\n\nSo it's really a hard choice for them. Let's wait and see what will happen.👀",
              "votes": 3
            },
            {
              "id": 2295743,
              "postDate": "2023-06-11T08:14:10.877Z",
              "content": "<blockquote>\n  <p>I think the final ranking will change a lot, because the final submitted code will be re-run.</p>\n</blockquote>\n<p>It will not be re-run. Your private score is already calculated every time you submit &amp; only revealed after the competition. You will see that after this competition, every submission will have an associated private score.</p>",
              "rawMarkdown": "> I think the final ranking will change a lot, because the final submitted code will be re-run.\n\nIt will not be re-run. Your private score is already calculated every time you submit & only revealed after the competition. You will see that after this competition, every submission will have an associated private score.",
              "votes": 2
            },
            {
              "id": 2295799,
              "postDate": "2023-06-11T08:45:35.270Z",
              "content": "<p>This is the official explanation:“ In this competition, we will privately re-run your selected Notebook Version with a hidden test set substituted into the competition dataset. We then extract your chosen Output File from the re-run and use that to determine your score.”<br>\nDoesn't that mean it will re-run our code? Forgive me for participating in the kaggle competition for the first time.</p>",
              "rawMarkdown": "This is the official explanation:“ In this competition, we will privately re-run your selected Notebook Version with a hidden test set substituted into the competition dataset. We then extract your chosen Output File from the re-run and use that to determine your score.”\nDoesn't that mean it will re-run our code? Forgive me for participating in the kaggle competition for the first time."
            },
            {
              "id": 2295801,
              "postDate": "2023-06-11T08:48:31.627Z",
              "content": "<blockquote>\n  <p>This is the official explanation:“ In this competition, we will privately re-run your selected Notebook Version with a hidden test set substituted into the competition dataset. We then extract your chosen Output File from the re-run and use that to determine your score.”<br>\n  Doesn't that mean it will re-run our code? Forgive me for participating in the kaggle competition for the first time.</p>\n</blockquote>\n<p>No need to apologise! Normally, both public and private test score are calculated as soon as you submit!</p>",
              "rawMarkdown": "> This is the official explanation:“ In this competition, we will privately re-run your selected Notebook Version with a hidden test set substituted into the competition dataset. We then extract your chosen Output File from the re-run and use that to determine your score.”\n> Doesn't that mean it will re-run our code? Forgive me for participating in the kaggle competition for the first time.\n\nNo need to apologise! Normally, both public and private test score are calculated as soon as you submit!"
            },
            {
              "id": 2295813,
              "postDate": "2023-06-11T09:04:43.750Z",
              "content": "<p>Yes, it's usually like this, but I think this game will be re-run, because it requires us to submit the code.</p>",
              "rawMarkdown": "Yes, it's usually like this, but I think this game will be re-run, because it requires us to submit the code."
            },
            {
              "id": 2295816,
              "postDate": "2023-06-11T09:13:20.530Z",
              "content": "<blockquote>\n  <p>Yes, it's usually like this, but I think this game will be re-run, because it requires us to submit the code.</p>\n</blockquote>\n<p>This would be the first competition where this would be done then (except for some exceptional re-scores that needed to happen in some competitions). In all code competitions in the past, it has been like I explained above :)</p>",
              "rawMarkdown": "> Yes, it's usually like this, but I think this game will be re-run, because it requires us to submit the code.\n\nThis would be the first competition where this would be done then (except for some exceptional re-scores that needed to happen in some competitions). In all code competitions in the past, it has been like I explained above :)"
            },
            {
              "id": 2295828,
              "postDate": "2023-06-11T09:40:47.833Z",
              "rawMarkdown": "",
              "isDeleted": true
            },
            {
              "id": 2295831,
              "postDate": "2023-06-11T09:45:06.167Z",
              "content": "<p><a href=\"https://www.kaggle.com/bent1e\" target=\"_blank\">@bent1e</a> Where did you see that they will re-run the code? I cannot find it anywhere</p>",
              "rawMarkdown": "@bent1e Where did you see that they will re-run the code? I cannot find it anywhere",
              "votes": 1
            },
            {
              "id": 2295833,
              "postDate": "2023-06-11T09:49:03.150Z",
              "content": "<p>When you click Submit Predictions, you will see it</p>",
              "rawMarkdown": "When you click Submit Predictions, you will see it"
            },
            {
              "id": 2295862,
              "postDate": "2023-06-11T10:20:53.997Z",
              "content": "<p><a href=\"https://www.kaggle.com/bent1e\" target=\"_blank\">@bent1e</a> . When you <code>commit and save</code> the notebook, it's the first run. When you submit the notebook, it's the second run(or the re-run). So there won't be a re-run after the end of the competition. </p>",
              "rawMarkdown": "@bent1e . When you `commit and save` the notebook, it's the first run. When you submit the notebook, it's the second run(or the re-run). So there won't be a re-run after the end of the competition. ",
              "votes": 3
            },
            {
              "id": 2295974,
              "postDate": "2023-06-11T12:48:56.950Z",
              "content": "<p>I think that besides the colmap randomness, it could be because of the robustness of the matches too, not really sure.</p>",
              "rawMarkdown": "I think that besides the colmap randomness, it could be because of the robustness of the matches too, not really sure."
            }
          ]
        },
        {
          "id": 2295678,
          "postDate": "2023-06-11T07:35:45.430Z",
          "content": "<p>That seems weird 🤒</p>",
          "rawMarkdown": "That seems weird 🤒"
        }
      ]
    },
    {
      "id": 2276683,
      "postDate": "2023-05-27T06:03:07.220Z",
      "content": "<p>*** METRICS ***<br>\nurban / kyiv-puppet-theater (26 images, 325 pairs) -&gt; mAA=0.803692, mAA_q=0.888923, mAA_t=0.829231<br>\nurban -&gt; mAA=0.803692</p>\n<p>heritage / dioscuri (174 images, 15051 pairs) -&gt; mAA=0.001787, mAA_q=0.003535, mAA_t=0.014006<br>\nheritage / cyprus (30 images, 435 pairs) -&gt; mAA=0.013793, mAA_q=0.013793, mAA_t=0.019080<br>\nheritage / wall (43 images, 903 pairs) -&gt; mAA=0.091251, mAA_q=0.547397, mAA_t=0.094574<br>\nheritage -&gt; mAA=0.035611</p>\n<p>haiper / bike (15 images, 105 pairs) -&gt; mAA=0.323810, mAA_q=0.382857, mAA_t=0.323810<br>\nhaiper / chairs (16 images, 120 pairs) -&gt; mAA=0.084167, mAA_q=0.132500, mAA_t=0.088333<br>\nhaiper / fountain (23 images, 253 pairs) -&gt; mAA=1.000000, mAA_q=1.000000, mAA_t=1.000000<br>\nhaiper -&gt; mAA=0.469325</p>\n<p>Final metric -&gt; mAA=0.436209 <br>\nI was shocked that you weer able to achieve such good results on the haiper dataset</p>",
      "rawMarkdown": "*** METRICS ***\nurban / kyiv-puppet-theater (26 images, 325 pairs) -> mAA=0.803692, mAA_q=0.888923, mAA_t=0.829231\nurban -> mAA=0.803692\n\nheritage / dioscuri (174 images, 15051 pairs) -> mAA=0.001787, mAA_q=0.003535, mAA_t=0.014006\nheritage / cyprus (30 images, 435 pairs) -> mAA=0.013793, mAA_q=0.013793, mAA_t=0.019080\nheritage / wall (43 images, 903 pairs) -> mAA=0.091251, mAA_q=0.547397, mAA_t=0.094574\nheritage -> mAA=0.035611\n\nhaiper / bike (15 images, 105 pairs) -> mAA=0.323810, mAA_q=0.382857, mAA_t=0.323810\nhaiper / chairs (16 images, 120 pairs) -> mAA=0.084167, mAA_q=0.132500, mAA_t=0.088333\nhaiper / fountain (23 images, 253 pairs) -> mAA=1.000000, mAA_q=1.000000, mAA_t=1.000000\nhaiper -> mAA=0.469325\n\nFinal metric -> mAA=0.436209 \nI was shocked that you weer able to achieve such good results on the haiper dataset",
      "votes": 3,
      "replies": [
        {
          "id": 2283022,
          "postDate": "2023-06-01T02:43:14.403Z",
          "content": "<p>I have the same problem, why the ratings of bike and chairs are so low</p>",
          "rawMarkdown": "I have the same problem, why the ratings of bike and chairs are so low",
          "votes": 2
        },
        {
          "id": 2283250,
          "postDate": "2023-06-01T06:36:47.147Z",
          "content": "<p>so did I. I wonder there are any bugs that make the poor performance on haiper dataset</p>",
          "rawMarkdown": "so did I. I wonder there are any bugs that make the poor performance on haiper dataset",
          "votes": 1
        }
      ]
    },
    {
      "id": 2297749,
      "postDate": "2023-06-12T19:11:50.047Z",
      "content": "<pre><code>*** METRICS ***\nurban / kyiv-puppet-theater (26 images, 325 pairs) -&gt; =0.892308, =0.973846, =0.906462\nurban -&gt; =0.892308\n\nheritage / dioscuri (174 images, 15051 pairs) -&gt; =0.716630, =0.825752, =0.725560\nheritage / cyprus (30 images, 435 pairs) -&gt; =0.637701, =0.646897, =0.673563\nheritage / wall (43 images, 903 pairs) -&gt; =0.912292, =0.950609, =0.934330\nheritage -&gt; =0.755541\n\nhaiper / bike (15 images, 105 pairs) -&gt; =0.921905, =0.999048, =0.921905\nhaiper / chairs (16 images, 120 pairs) -&gt; =0.965000, =0.995000, =0.966667\nhaiper / fountain (23 images, 253 pairs) -&gt; =0.999605, =1.000000, =0.999605\nhaiper -&gt; =0.962170\n\nFinal metric -&gt; =0.870006 (t: 1.9735429286956787 sec.)\n\n0.87000626\n</code></pre>\n<p>LB :485</p>",
      "rawMarkdown": "```\n*** METRICS ***\nurban / kyiv-puppet-theater (26 images, 325 pairs) -> mAA=0.892308, mAA_q=0.973846, mAA_t=0.906462\nurban -> mAA=0.892308\n\nheritage / dioscuri (174 images, 15051 pairs) -> mAA=0.716630, mAA_q=0.825752, mAA_t=0.725560\nheritage / cyprus (30 images, 435 pairs) -> mAA=0.637701, mAA_q=0.646897, mAA_t=0.673563\nheritage / wall (43 images, 903 pairs) -> mAA=0.912292, mAA_q=0.950609, mAA_t=0.934330\nheritage -> mAA=0.755541\n\nhaiper / bike (15 images, 105 pairs) -> mAA=0.921905, mAA_q=0.999048, mAA_t=0.921905\nhaiper / chairs (16 images, 120 pairs) -> mAA=0.965000, mAA_q=0.995000, mAA_t=0.966667\nhaiper / fountain (23 images, 253 pairs) -> mAA=0.999605, mAA_q=1.000000, mAA_t=0.999605\nhaiper -> mAA=0.962170\n\nFinal metric -> mAA=0.870006 (t: 1.9735429286956787 sec.)\n\n0.87000626\n```\nLB :485\n",
      "votes": 4
    },
    {
      "id": 2274979,
      "postDate": "2023-05-26T12:09:45.260Z",
      "content": "<p>LB：0.26<br>\n*** METRICS ***<br>\nurban / kyiv-puppet-theater (26 images, 325 pairs) -&gt; mAA=0.610154, mAA_q=0.808000, mAA_t=0.653846<br>\nurban -&gt; mAA=0.610154</p>\n<p>heritage / dioscuri (174 images, 15051 pairs) -&gt; mAA=0.001787, mAA_q=0.003535, mAA_t=0.014006<br>\nheritage / cyprus (30 images, 435 pairs) -&gt; mAA=0.541839, mAA_q=0.561379, mAA_t=0.548966<br>\nheritage / wall (43 images, 903 pairs) -&gt; mAA=0.302326, mAA_q=0.682060, mAA_t=0.304983<br>\nheritage -&gt; mAA=0.281984</p>\n<p>haiper / bike (15 images, 105 pairs) -&gt; mAA=0.908571, mAA_q=0.993333, mAA_t=0.908571<br>\nhaiper / chairs (16 images, 120 pairs) -&gt; mAA=0.647500, mAA_q=0.802500, mAA_t=0.656667<br>\nhaiper / fountain (23 images, 253 pairs) -&gt; mAA=0.999605, mAA_q=1.000000, mAA_t=0.999605<br>\nhaiper -&gt; mAA=0.851892</p>",
      "rawMarkdown": "LB：0.26\n*** METRICS ***\nurban / kyiv-puppet-theater (26 images, 325 pairs) -> mAA=0.610154, mAA_q=0.808000, mAA_t=0.653846\nurban -> mAA=0.610154\n\nheritage / dioscuri (174 images, 15051 pairs) -> mAA=0.001787, mAA_q=0.003535, mAA_t=0.014006\nheritage / cyprus (30 images, 435 pairs) -> mAA=0.541839, mAA_q=0.561379, mAA_t=0.548966\nheritage / wall (43 images, 903 pairs) -> mAA=0.302326, mAA_q=0.682060, mAA_t=0.304983\nheritage -> mAA=0.281984\n\nhaiper / bike (15 images, 105 pairs) -> mAA=0.908571, mAA_q=0.993333, mAA_t=0.908571\nhaiper / chairs (16 images, 120 pairs) -> mAA=0.647500, mAA_q=0.802500, mAA_t=0.656667\nhaiper / fountain (23 images, 253 pairs) -> mAA=0.999605, mAA_q=1.000000, mAA_t=0.999605\nhaiper -> mAA=0.851892",
      "votes": 3,
      "replies": [
        {
          "id": 2276727,
          "postDate": "2023-05-27T06:41:50.467Z",
          "content": "<p>May I ask if you are using PyColmap for incremental reconstruction? Or are there other methods for global reconstruction that you are using?</p>",
          "rawMarkdown": "May I ask if you are using PyColmap for incremental reconstruction? Or are there other methods for global reconstruction that you are using?",
          "replies": [
            {
              "id": 2276867,
              "postDate": "2023-05-27T09:16:41.003Z",
              "content": "<p>yeah, it seems to me the only way to do reconstruction</p>",
              "rawMarkdown": "yeah, it seems to me the only way to do reconstruction",
              "votes": 2
            },
            {
              "id": 2276984,
              "postDate": "2023-05-27T11:04:41.803Z",
              "content": "<p>Thank you for your sharing!😃</p>",
              "rawMarkdown": "Thank you for your sharing!😃"
            }
          ]
        }
      ]
    },
    {
      "id": 2274671,
      "postDate": "2023-05-26T07:17:52.483Z",
      "content": "<p>I find this problem as well,there is a big gap between LB_Public and CV.<br>\nTraining data about 328<br>\nLB_Public data about 484<br>\nLB_provate data about 616<br>\nbased on the num of data,the gap is something we can't ignore. I would like to know how the top teams  deal with this problem.</p>",
      "rawMarkdown": "I find this problem as well,there is a big gap between LB_Public and CV.\nTraining data about 328\nLB_Public data about 484\nLB_provate data about 616\nbased on the num of data,the gap is something we can't ignore. I would like to know how the top teams  deal with this problem.",
      "votes": 3
    },
    {
      "id": 2274825,
      "postDate": "2023-05-26T09:18:24.193Z",
      "content": "<p>Yes, I agree with what you're saying. Certain datasets may indeed hold a larger proportion in the LB, possibly similar to the heritage dataset or others.</p>\n<p>urban / kyiv-puppet-theater (26 images, 325 pairs) -&gt; mAA=0.574154, mAA_q=0.754769, mAA_t=0.604615<br>\n<strong>urban -&gt; mAA=0.574154</strong><br>\nheritage / dioscuri (174 images, 15051 pairs) -&gt; mAA=0.132091, mAA_q=0.152933, mAA_t=0.142309<br>\nheritage / cyprus (30 images, 435 pairs) -&gt; mAA=0.012874, mAA_q=0.019310, mAA_t=0.025747<br>\nheritage / wall (43 images, 903 pairs) -&gt; mAA=0.079291, mAA_q=0.528682, mAA_t=0.080066<br>\n<strong>heritage -&gt; mAA=0.074752</strong><br>\nhaiper / bike (15 images, 105 pairs) -&gt; mAA=0.192381, mAA_q=0.200000, mAA_t=0.192381<br>\nhaiper / chairs (16 images, 120 pairs) -&gt; mAA=0.526667, mAA_q=0.624167, mAA_t=0.530000<br>\nhaiper / fountain (23 images, 253 pairs) -&gt; mAA=1.000000, mAA_q=1.000000, mAA_t=1.000000<br>\n<strong>haiper -&gt; mAA=0.573016</strong><br>\n<strong>Final metric -&gt; mAA=0.407307</strong> </p>\n<p><strong>LeaderBoard 0.141</strong></p>",
      "rawMarkdown": "Yes, I agree with what you're saying. Certain datasets may indeed hold a larger proportion in the LB, possibly similar to the heritage dataset or others.\n\nurban / kyiv-puppet-theater (26 images, 325 pairs) -> mAA=0.574154, mAA_q=0.754769, mAA_t=0.604615\n**urban -> mAA=0.574154**\nheritage / dioscuri (174 images, 15051 pairs) -> mAA=0.132091, mAA_q=0.152933, mAA_t=0.142309\nheritage / cyprus (30 images, 435 pairs) -> mAA=0.012874, mAA_q=0.019310, mAA_t=0.025747\nheritage / wall (43 images, 903 pairs) -> mAA=0.079291, mAA_q=0.528682, mAA_t=0.080066\n**heritage -> mAA=0.074752**\nhaiper / bike (15 images, 105 pairs) -> mAA=0.192381, mAA_q=0.200000, mAA_t=0.192381\nhaiper / chairs (16 images, 120 pairs) -> mAA=0.526667, mAA_q=0.624167, mAA_t=0.530000\nhaiper / fountain (23 images, 253 pairs) -> mAA=1.000000, mAA_q=1.000000, mAA_t=1.000000\n**haiper -> mAA=0.573016**\n**Final metric -> mAA=0.407307** \n\n**LeaderBoard 0.141**",
      "votes": 4
    },
    {
      "id": 2276675,
      "postDate": "2023-05-27T05:54:29.527Z",
      "content": "<p>You get a good performance on haiper dataset,maybe there is much  dataset like haiper on LB_private,trust your CV.</p>",
      "rawMarkdown": "You get a good performance on haiper dataset,maybe there is much  dataset like haiper on LB_private,trust your CV.",
      "votes": 1,
      "replies": [
        {
          "id": 2276691,
          "postDate": "2023-05-27T06:15:57.667Z",
          "content": "<p>Your urban score is really good too. I don't think there will be scenes like haiper in the private leaderboard. If you check haiper scenes visually, you can see that they are kinda too easy like very few images, small image size and images are literally taken around an object. Based on my 4 year Kaggle experience, I can say that I've never seen any private leaderboard with such easy examples in any competition.</p>",
          "rawMarkdown": "Your urban score is really good too. I don't think there will be scenes like haiper in the private leaderboard. If you check haiper scenes visually, you can see that they are kinda too easy like very few images, small image size and images are literally taken around an object. Based on my 4 year Kaggle experience, I can say that I've never seen any private leaderboard with such easy examples in any competition.",
          "votes": 5,
          "replies": [
            {
              "id": 2276741,
              "postDate": "2023-05-27T06:58:59.987Z",
              "content": "<p>On other discussion you said you guess the intention was weighting each dataset equally . It seemed little bit weird to me as well.So maybe we should trust our CV and focus on hard dataset such as urban with more photos  in a huge scene.</p>",
              "rawMarkdown": "On other discussion you said you guess the intention was weighting each dataset equally . It seemed little bit weird to me as well.So maybe we should trust our CV and focus on hard dataset such as urban with more photos  in a huge scene.",
              "votes": 1
            }
          ]
        }
      ]
    },
    {
      "id": 2281647,
      "postDate": "2023-05-31T03:51:09.820Z",
      "content": "<p>my lb: 0.348</p>\n<p>*** METRICS ***<br>\nurban / kyiv-puppet-theater (26 images, 325 pairs) -&gt; mAA=0.659385, mAA_q=0.836923, mAA_t=0.688308<br>\nurban -&gt; mAA=0.659385</p>\n<p>heritage / dioscuri (174 images, 15051 pairs) -&gt; mAA=0.404631, mAA_q=0.507096, mAA_t=0.410491<br>\nheritage / cyprus (30 images, 435 pairs) -&gt; mAA=0.022989, mAA_q=0.022989, mAA_t=0.029195<br>\nheritage / wall (43 images, 903 pairs) -&gt; mAA=0.864673, mAA_q=0.937431, mAA_t=0.878516<br>\nheritage -&gt; mAA=0.430764</p>\n<p>haiper / bike (15 images, 105 pairs) -&gt; mAA=0.921905, mAA_q=0.999048, mAA_t=0.921905<br>\nhaiper / chairs (16 images, 120 pairs) -&gt; mAA=0.776667, mAA_q=0.860000, mAA_t=0.778333<br>\nhaiper / fountain (23 images, 253 pairs) -&gt; mAA=1.000000, mAA_q=1.000000, mAA_t=1.000000<br>\nhaiper -&gt; mAA=0.899524</p>\n<p>Final metric -&gt; mAA=0.663224 (t: 0.6245906352996826 sec.)</p>\n<p>0.6632242</p>",
      "rawMarkdown": "my lb: 0.348\n\n*** METRICS ***\nurban / kyiv-puppet-theater (26 images, 325 pairs) -> mAA=0.659385, mAA_q=0.836923, mAA_t=0.688308\nurban -> mAA=0.659385\n\nheritage / dioscuri (174 images, 15051 pairs) -> mAA=0.404631, mAA_q=0.507096, mAA_t=0.410491\nheritage / cyprus (30 images, 435 pairs) -> mAA=0.022989, mAA_q=0.022989, mAA_t=0.029195\nheritage / wall (43 images, 903 pairs) -> mAA=0.864673, mAA_q=0.937431, mAA_t=0.878516\nheritage -> mAA=0.430764\n\nhaiper / bike (15 images, 105 pairs) -> mAA=0.921905, mAA_q=0.999048, mAA_t=0.921905\nhaiper / chairs (16 images, 120 pairs) -> mAA=0.776667, mAA_q=0.860000, mAA_t=0.778333\nhaiper / fountain (23 images, 253 pairs) -> mAA=1.000000, mAA_q=1.000000, mAA_t=1.000000\nhaiper -> mAA=0.899524\n\nFinal metric -> mAA=0.663224 (t: 0.6245906352996826 sec.)\n\n0.6632242",
      "votes": 2,
      "replies": [
        {
          "id": 2281675,
          "postDate": "2023-05-31T04:29:43.390Z",
          "content": "<p>Thanks for sharing</p>",
          "rawMarkdown": "Thanks for sharing"
        }
      ]
    },
    {
      "id": 2280853,
      "postDate": "2023-05-30T12:21:11.840Z",
      "content": "<p>LB：0.18<br>\n*** METRICS ***<br>\nurban / kyiv-puppet-theater (26 images, 325 pairs) -&gt; mAA=0.844615, mAA_q=0.915385, mAA_t=0.854461<br>\nurban -&gt; mAA=0.844615</p>\n<p>heritage / dioscuri (174 images, 15051 pairs) -&gt; mAA=0.149532, mAA_q=0.242469, mAA_t=0.161929<br>\nheritage / cyprus (30 images, 435 pairs) -&gt; mAA=0.012874, mAA_q=0.013793, mAA_t=0.017471<br>\nheritage / wall (43 images, 903 pairs) -&gt; mAA=0.049059, mAA_q=0.517276, mAA_t=0.049280<br>\nheritage -&gt; mAA=0.070488</p>\n<p>haiper / bike (15 images, 105 pairs) -&gt; mAA=0.944762, mAA_q=1.000000, mAA_t=0.944762<br>\nhaiper / chairs (16 images, 120 pairs) -&gt; mAA=0.838333, mAA_q=0.867500, mAA_t=0.841667<br>\nhaiper / fountain (23 images, 253 pairs) -&gt; mAA=0.999605, mAA_q=1.000000, mAA_t=0.999605<br>\nhaiper -&gt; mAA=0.927567</p>\n<p>Final metric -&gt; mAA=0.614223 (t: 1.932884931564331 sec.)</p>",
      "rawMarkdown": "LB：0.18\n*** METRICS ***\nurban / kyiv-puppet-theater (26 images, 325 pairs) -> mAA=0.844615, mAA_q=0.915385, mAA_t=0.854461\nurban -> mAA=0.844615\n\nheritage / dioscuri (174 images, 15051 pairs) -> mAA=0.149532, mAA_q=0.242469, mAA_t=0.161929\nheritage / cyprus (30 images, 435 pairs) -> mAA=0.012874, mAA_q=0.013793, mAA_t=0.017471\nheritage / wall (43 images, 903 pairs) -> mAA=0.049059, mAA_q=0.517276, mAA_t=0.049280\nheritage -> mAA=0.070488\n\nhaiper / bike (15 images, 105 pairs) -> mAA=0.944762, mAA_q=1.000000, mAA_t=0.944762\nhaiper / chairs (16 images, 120 pairs) -> mAA=0.838333, mAA_q=0.867500, mAA_t=0.841667\nhaiper / fountain (23 images, 253 pairs) -> mAA=0.999605, mAA_q=1.000000, mAA_t=0.999605\nhaiper -> mAA=0.927567\n\nFinal metric -> mAA=0.614223 (t: 1.932884931564331 sec.)",
      "votes": 2
    },
    {
      "id": 2276889,
      "postDate": "2023-05-27T09:36:57.193Z",
      "content": "<p>I know that the haiper data set is a relatively simple scenario, but it is still difficult for me to do a good result, even not as good as urban. Why is that?</p>",
      "rawMarkdown": "I know that the haiper data set is a relatively simple scenario, but it is still difficult for me to do a good result, even not as good as urban. Why is that?",
      "replies": [
        {
          "id": 2283352,
          "postDate": "2023-06-01T07:58:46.387Z",
          "content": "<p>It seems you  have resolve this problem.</p>",
          "rawMarkdown": "It seems you  have resolve this problem."
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 2280942,
      "author_name": "junfang1",
      "author_url": "",
      "post_date": "2023-05-30T13:23:13.890000",
      "content": "<p>LB：0.467<br>\n*** METRICS ***<br>\nurban / kyiv-puppet-theater (26 images, 325 pairs) -&gt; mAA=0.632308, mAA_q=0.823077, mAA_t=0.674462<br>\nurban -&gt; mAA=0.632308</p>\n<p>heritage / dioscuri (174 images, 15051 pairs) -&gt; mAA=0.680167, mAA_q=0.799356, mAA_t=0.686001<br>\nheritage / cyprus (30 images, 435 pairs) -&gt; mAA=0.056092, mAA_q=0.057471, mAA_t=0.075402<br>\nheritage / wall (43 images, 903 pairs) -&gt; mAA=0.726910, mAA_q=0.922038, mAA_t=0.739867<br>\nheritage -&gt; mAA=0.487723</p>\n<p>haiper / bike (15 images, 105 pairs) -&gt; mAA=0.925714, mAA_q=0.999048, mAA_t=0.925714<br>\nhaiper / chairs (16 images, 120 pairs) -&gt; mAA=0.903333, mAA_q=0.986667, mAA_t=0.906667<br>\nhaiper / fountain (23 images, 253 pairs) -&gt; mAA=1.000000, mAA_q=1.000000, mAA_t=1.000000<br>\nhaiper -&gt; mAA=0.943016</p>\n<p>Final metric -&gt; mAA=0.687682 (t: 2.7530465126037598 sec.)</p>",
      "votes": 14,
      "replies": [
        {
          "id": 2287489,
          "author_name": "Seg_paper",
          "author_url": "",
          "post_date": "2023-06-04T14:25:24.177000",
          "content": "<p>really helpful,thanks</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 2288192,
          "author_name": "kk",
          "author_url": "",
          "post_date": "2023-06-05T08:13:29.903000",
          "content": "<p>Look helpful for me, cheers</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 2295318,
          "author_name": "Yuxiang Huang",
          "author_url": "",
          "post_date": "2023-06-10T19:27:36.127000",
          "content": "<p>Thank you. How long did your submission run? </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2274955,
      "author_name": "niubiren",
      "author_url": "",
      "post_date": "2023-05-26T11:59:22.700000",
      "content": "<p>accroding to my last a few try, whether to skip scenes larger than 40 images can lead to remarkable difference(like 0.05 points). Given that my code did poor in these large scene, i guess large scenes really counts in final score. </p>",
      "votes": 7,
      "replies": [
        {
          "id": 2281538,
          "author_name": "leon5566",
          "author_url": "",
          "post_date": "2023-05-31T01:25:41.743000",
          "content": "<p>Hi, have you found a way to reduce the randomness of pycolmap? I noticed that you have been able to lock the error of skipping scenes to 0.05, while I have submitted the same code multiple times and the score sometimes varies by more than 0.05.</p>",
          "votes": 4,
          "replies": [
            {
              "id": 2291903,
              "author_name": "niubiren",
              "author_url": "",
              "post_date": "2023-06-07T23:38:56.187000",
              "content": "<p>I have not.Randomness exists, what I can do is only try multiple times to eliminate its effect</p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 2291969,
              "author_name": "leon5566",
              "author_url": "",
              "post_date": "2023-06-08T02:14:39.570000",
              "content": "<p>Thank you for your response.</p>",
              "votes": 0,
              "replies": []
            }
          ]
        },
        {
          "id": 2290941,
          "author_name": "Gilles Vandewiele",
          "author_url": "",
          "post_date": "2023-06-07T08:08:23.807000",
          "content": "<blockquote>\n  <p>accroding to my last a few try, whether to skip scenes larger than 40 images can lead to remarkable difference(like 0.05 points). Given that my code did poor in these large scene, i guess large scenes really counts in final score.</p>\n</blockquote>\n<p>What do you mean with \"skip scenes\"? Do you not submit anything at all for scenes with more than 40 images?</p>",
          "votes": 0,
          "replies": [
            {
              "id": 2291905,
              "author_name": "niubiren",
              "author_url": "",
              "post_date": "2023-06-07T23:42:40.267000",
              "content": "<p>As you may found, these scenes are time consumming.\"skip scenes\" simply means creating a random but valid result without computation</p>",
              "votes": 2,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2276879,
      "author_name": "motono0223",
      "author_url": "",
      "post_date": "2023-05-27T09:30:16.067000",
      "content": "<p>my lb : 0.338</p>\n<pre><code>haiper / bike (15 images, 105 pairs) -&gt; mAA=0.921905, mAA_q=1.000000, mAA_t=0.921905\nhaiper / chairs (16 images, 120 pairs) -&gt; mAA=0.766667, mAA_q=0.846667, mAA_t=0.777500\nhaiper / fountain (23 images, 253 pairs) -&gt; mAA=0.999210, mAA_q=1.000000, mAA_t=0.999210\nhaiper -&gt; mAA=0.895927\n\nheritage / cyprus (30 images, 435 pairs) -&gt; mAA=0.021839, mAA_q=0.022989, mAA_t=0.027816\nheritage / dioscuri (174 images, 15051 pairs) -&gt; mAA=0.435572, mAA_q=0.535353, mAA_t=0.442469\nheritage / wall (43 images, 903 pairs) -&gt; mAA=0.175748, mAA_q=0.442636, mAA_t=0.188815\nheritage -&gt; mAA=0.211053\n\nurban / kyiv-puppet-theater (26 images, 325 pairs) -&gt; mAA=0.755077, mAA_q=0.915077, mAA_t=0.780923\nurban -&gt; mAA=0.755077\n\nFinal metric -&gt; mAA=0.620686 (t: 3.0341649055480957 sec.)\n</code></pre>",
      "votes": 8,
      "replies": [
        {
          "id": 2276911,
          "author_name": "Gunes Evitan",
          "author_url": "",
          "post_date": "2023-05-27T10:01:35.200000",
          "content": "<p>Nice. Higher urban score is correlated with lb score as I thought.</p>",
          "votes": 5,
          "replies": []
        },
        {
          "id": 2276941,
          "author_name": "niubiren",
          "author_url": "",
          "post_date": "2023-05-27T10:23:51.040000",
          "content": "<p>thanks for sharing</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 2277042,
          "author_name": "吕舟",
          "author_url": "",
          "post_date": "2023-05-27T11:43:08.187000",
          "content": "<p>thanks for your sharing</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2295648,
      "author_name": "ForcewithMe",
      "author_url": "",
      "post_date": "2023-06-11T07:15:37.630000",
      "content": "<p>For me, a higher CV brings worse LB.</p>\n<p>urban / kyiv-puppet-theater (26 images, 325 pairs) -&gt; mAA=0.915077, mAA_q=0.979692, mAA_t=0.924923<br>\nurban -&gt; mAA=0.915077</p>\n<p>heritage / dioscuri (174 images, 15051 pairs) -&gt; mAA=0.508478, mAA_q=0.637612, mAA_t=0.514238<br>\nheritage / cyprus (30 images, 435 pairs) -&gt; mAA=0.022299, mAA_q=0.022989, mAA_t=0.028046<br>\nheritage / wall (43 images, 903 pairs) -&gt; mAA=0.523810, mAA_q=0.882946, mAA_t=0.533666<br>\nheritage -&gt; mAA=0.351529</p>\n<p>haiper / bike (15 images, 105 pairs) -&gt; mAA=0.951429, mAA_q=1.000000, mAA_t=0.951429<br>\nhaiper / chairs (16 images, 120 pairs) -&gt; mAA=0.929167, mAA_q=0.995000, mAA_t=0.931667<br>\nhaiper / fountain (23 images, 253 pairs) -&gt; mAA=0.999605, mAA_q=1.000000, mAA_t=0.999605<br>\nhaiper -&gt; mAA=0.960067</p>\n<p>Final metric -&gt; mAA=0.742224 (t: 2.0051422119140625 sec.)</p>\n<p>LB: 0.37😂</p>",
      "votes": 6,
      "replies": [
        {
          "id": 2295655,
          "author_name": "Gilles Vandewiele",
          "author_url": "",
          "post_date": "2023-06-11T07:21:22.107000",
          "content": "<blockquote>\n  <p>For me, a higher CV brings worse LB.</p>\n  <p>urban / kyiv-puppet-theater (26 images, 325 pairs) -&gt; mAA=0.915077, mAA_q=0.979692, mAA_t=0.924923<br>\n  urban -&gt; mAA=0.915077</p>\n  <p>heritage / dioscuri (174 images, 15051 pairs) -&gt; mAA=0.508478, mAA_q=0.637612, mAA_t=0.514238<br>\n  heritage / cyprus (30 images, 435 pairs) -&gt; mAA=0.022299, mAA_q=0.022989, mAA_t=0.028046<br>\n  heritage / wall (43 images, 903 pairs) -&gt; mAA=0.523810, mAA_q=0.882946, mAA_t=0.533666<br>\n  heritage -&gt; mAA=0.351529</p>\n  <p>haiper / bike (15 images, 105 pairs) -&gt; mAA=0.951429, mAA_q=1.000000, mAA_t=0.951429<br>\n  haiper / chairs (16 images, 120 pairs) -&gt; mAA=0.929167, mAA_q=0.995000, mAA_t=0.931667<br>\n  haiper / fountain (23 images, 253 pairs) -&gt; mAA=0.999605, mAA_q=1.000000, mAA_t=0.999605<br>\n  haiper -&gt; mAA=0.960067</p>\n  <p>Final metric -&gt; mAA=0.742224 (t: 2.0051422119140625 sec.)</p>\n  <p>LB: 0.37😂</p>\n</blockquote>\n<p>Same for us…</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F443651%2Ffb62a4598acc83d811a595f9f77d9f2c%2Fcv_scores.png?generation=1686468079902863&amp;alt=media\" alt=\"\"></p>",
          "votes": 3,
          "replies": [
            {
              "id": 2295673,
              "author_name": "Yuxiang Huang",
              "author_url": "",
              "post_date": "2023-06-11T07:32:21.250000",
              "content": "<p>How much time does it take for you guys to run on the leaderboard? Have you experienced significant slowdown during the past 2 days? All my notebooks timed out though they used to finish running. </p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 2295692,
              "author_name": "bent1e",
              "author_url": "",
              "post_date": "2023-06-11T07:39:08.043000",
              "content": "<p>Yes, my submission has timed out. I have never had it before.</p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 2295694,
              "author_name": "ForcewithMe",
              "author_url": "",
              "post_date": "2023-06-11T07:43:06.713000",
              "content": "<p>Hi <a href=\"https://www.kaggle.com/group16\" target=\"_blank\">@group16</a> , Your CV is too great, I can't even believe my eyes 😂.  Are you sure you get the CV score in a right way?</p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 2295703,
              "author_name": "Gilles Vandewiele",
              "author_url": "",
              "post_date": "2023-06-11T07:51:20.050000",
              "content": "<blockquote>\n  <p>Hi <a href=\"https://www.kaggle.com/group16\" target=\"_blank\">@group16</a> , Your CV is too great, I can't even believe my eyes 😂.  Are you sure you get the CV score in a right way?</p>\n</blockquote>\n<p>Hi <a href=\"https://www.kaggle.com/forcewithme\" target=\"_blank\">@forcewithme</a>,</p>\n<p>We do think that is correct indeed :). To say that we are a little bit frustrated that there is such a discrepancy between the training datasets &amp; scenes and the test ones, would be the understatement of the year.</p>\n<p>I think we at least might win the participation trophy of having the best CV scores in this competition.</p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 2295704,
              "author_name": "Gunes Evitan",
              "author_url": "",
              "post_date": "2023-06-11T07:51:30.420000",
              "content": "<p>We always tried to boost our validation score and I was getting 0.79 mAA at some point but it didn't work out on the public leaderboard. I think a little shake might happen at the end.</p>",
              "votes": 2,
              "replies": []
            },
            {
              "id": 2295708,
              "author_name": "Yuxiang Huang",
              "author_url": "",
              "post_date": "2023-06-11T07:54:41.107000",
              "content": "<p>Wow, however I find there's some randomness in my submissions (up to 0.025 on lb), for my cv it's even bigger, for the same code sometimes the cv is good sometimes it's not that good. </p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 2295711,
              "author_name": "Yuxiang Huang",
              "author_url": "",
              "post_date": "2023-06-11T07:56:50.987000",
              "content": "<p>I guess everyone is working on this last minute lol which may have slowed down the Kaggle server, not very fair for those who started late but understandable…</p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 2295714,
              "author_name": "ForcewithMe",
              "author_url": "",
              "post_date": "2023-06-11T07:59:38.773000",
              "content": "<blockquote>\n  <p>To say that we are a little bit frustrated that there is such a discrepancy between the training datasets &amp; scenes and the test ones</p>\n</blockquote>\n<p>Yeah, a competition with almost no correlation between CV and LB is always frustrated. I have the same feeling  for the last month. </p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2295718,
              "author_name": "bent1e",
              "author_url": "",
              "post_date": "2023-06-11T08:01:46.330000",
              "content": "<p>I also encountered this problem. I repeatedly submitted the same code without exactly the same score.</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2295720,
              "author_name": "Yuxiang Huang",
              "author_url": "",
              "post_date": "2023-06-11T08:03:40.197000",
              "content": "<p>Yeah cuz Colmap reconstruction has randomness, I think one way to alleviate it would be to generate more reconstruction models then choose the best one, but it takes more time. </p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 2295728,
              "author_name": "bent1e",
              "author_url": "",
              "post_date": "2023-06-11T08:07:33.747000",
              "content": "<p>I think the final ranking will change a lot, because the final submitted code will be re-run.</p>",
              "votes": -1,
              "replies": []
            },
            {
              "id": 2295733,
              "author_name": "ForcewithMe",
              "author_url": "",
              "post_date": "2023-06-11T08:08:56.960000",
              "content": "<p>Hi <a href=\"https://www.kaggle.com/gunesevitan\" target=\"_blank\">@gunesevitan</a> , I think the shake depends on how they split the dataset, which they didn't give an answer in <a href=\"https://www.kaggle.com/competitions/image-matching-challenge-2023/discussion/411778\" target=\"_blank\">my post</a>.</p>\n<p>IMO, if they split the leaderboard by image pair, which means the public and private Leaderboard <strong>have overlap on scenes</strong>, there will be little shake. Although I can't figure out how to get the score for both leaderboards if there is an overlap on scenes. Also, I think if all the scenes are included in both leaderboards,  it's just an overfitting game. </p>\n<p>But on the other hand, if they split the dataset by <code>scene</code> or <code>dataset</code>，which means that <strong>there won't be overlap</strong> between scenes on the Leaderboards, there may be a super huge shake, or not, depends on the discrepancy between the scenes in both Leaderboards.</p>\n<p>So it's really a hard choice for them. Let's wait and see what will happen.👀</p>",
              "votes": 3,
              "replies": []
            },
            {
              "id": 2295743,
              "author_name": "Gilles Vandewiele",
              "author_url": "",
              "post_date": "2023-06-11T08:14:10.877000",
              "content": "<blockquote>\n  <p>I think the final ranking will change a lot, because the final submitted code will be re-run.</p>\n</blockquote>\n<p>It will not be re-run. Your private score is already calculated every time you submit &amp; only revealed after the competition. You will see that after this competition, every submission will have an associated private score.</p>",
              "votes": 2,
              "replies": []
            },
            {
              "id": 2295799,
              "author_name": "bent1e",
              "author_url": "",
              "post_date": "2023-06-11T08:45:35.270000",
              "content": "<p>This is the official explanation:“ In this competition, we will privately re-run your selected Notebook Version with a hidden test set substituted into the competition dataset. We then extract your chosen Output File from the re-run and use that to determine your score.”<br>\nDoesn't that mean it will re-run our code? Forgive me for participating in the kaggle competition for the first time.</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2295801,
              "author_name": "Gilles Vandewiele",
              "author_url": "",
              "post_date": "2023-06-11T08:48:31.627000",
              "content": "<blockquote>\n  <p>This is the official explanation:“ In this competition, we will privately re-run your selected Notebook Version with a hidden test set substituted into the competition dataset. We then extract your chosen Output File from the re-run and use that to determine your score.”<br>\n  Doesn't that mean it will re-run our code? Forgive me for participating in the kaggle competition for the first time.</p>\n</blockquote>\n<p>No need to apologise! Normally, both public and private test score are calculated as soon as you submit!</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2295813,
              "author_name": "bent1e",
              "author_url": "",
              "post_date": "2023-06-11T09:04:43.750000",
              "content": "<p>Yes, it's usually like this, but I think this game will be re-run, because it requires us to submit the code.</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2295816,
              "author_name": "Gilles Vandewiele",
              "author_url": "",
              "post_date": "2023-06-11T09:13:20.530000",
              "content": "<blockquote>\n  <p>Yes, it's usually like this, but I think this game will be re-run, because it requires us to submit the code.</p>\n</blockquote>\n<p>This would be the first competition where this would be done then (except for some exceptional re-scores that needed to happen in some competitions). In all code competitions in the past, it has been like I explained above :)</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2295828,
              "author_name": "",
              "author_url": "",
              "post_date": "2023-06-11T09:40:47.833000",
              "content": "",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2295831,
              "author_name": "Yuxiang Huang",
              "author_url": "",
              "post_date": "2023-06-11T09:45:06.167000",
              "content": "<p><a href=\"https://www.kaggle.com/bent1e\" target=\"_blank\">@bent1e</a> Where did you see that they will re-run the code? I cannot find it anywhere</p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 2295833,
              "author_name": "bent1e",
              "author_url": "",
              "post_date": "2023-06-11T09:49:03.150000",
              "content": "<p>When you click Submit Predictions, you will see it</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2295862,
              "author_name": "ForcewithMe",
              "author_url": "",
              "post_date": "2023-06-11T10:20:53.997000",
              "content": "<p><a href=\"https://www.kaggle.com/bent1e\" target=\"_blank\">@bent1e</a> . When you <code>commit and save</code> the notebook, it's the first run. When you submit the notebook, it's the second run(or the re-run). So there won't be a re-run after the end of the competition. </p>",
              "votes": 3,
              "replies": []
            },
            {
              "id": 2295974,
              "author_name": "_jovoooooo",
              "author_url": "",
              "post_date": "2023-06-11T12:48:56.950000",
              "content": "<p>I think that besides the colmap randomness, it could be because of the robustness of the matches too, not really sure.</p>",
              "votes": 0,
              "replies": []
            }
          ]
        },
        {
          "id": 2295678,
          "author_name": "Pami_loveDL",
          "author_url": "",
          "post_date": "2023-06-11T07:35:45.430000",
          "content": "<p>That seems weird 🤒</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2276683,
      "author_name": "吕舟",
      "author_url": "",
      "post_date": "2023-05-27T06:03:07.220000",
      "content": "<p>*** METRICS ***<br>\nurban / kyiv-puppet-theater (26 images, 325 pairs) -&gt; mAA=0.803692, mAA_q=0.888923, mAA_t=0.829231<br>\nurban -&gt; mAA=0.803692</p>\n<p>heritage / dioscuri (174 images, 15051 pairs) -&gt; mAA=0.001787, mAA_q=0.003535, mAA_t=0.014006<br>\nheritage / cyprus (30 images, 435 pairs) -&gt; mAA=0.013793, mAA_q=0.013793, mAA_t=0.019080<br>\nheritage / wall (43 images, 903 pairs) -&gt; mAA=0.091251, mAA_q=0.547397, mAA_t=0.094574<br>\nheritage -&gt; mAA=0.035611</p>\n<p>haiper / bike (15 images, 105 pairs) -&gt; mAA=0.323810, mAA_q=0.382857, mAA_t=0.323810<br>\nhaiper / chairs (16 images, 120 pairs) -&gt; mAA=0.084167, mAA_q=0.132500, mAA_t=0.088333<br>\nhaiper / fountain (23 images, 253 pairs) -&gt; mAA=1.000000, mAA_q=1.000000, mAA_t=1.000000<br>\nhaiper -&gt; mAA=0.469325</p>\n<p>Final metric -&gt; mAA=0.436209 <br>\nI was shocked that you weer able to achieve such good results on the haiper dataset</p>",
      "votes": 3,
      "replies": [
        {
          "id": 2283022,
          "author_name": "Kawhi_Hong",
          "author_url": "",
          "post_date": "2023-06-01T02:43:14.403000",
          "content": "<p>I have the same problem, why the ratings of bike and chairs are so low</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 2283250,
          "author_name": "Dive Deeper",
          "author_url": "",
          "post_date": "2023-06-01T06:36:47.147000",
          "content": "<p>so did I. I wonder there are any bugs that make the poor performance on haiper dataset</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2297749,
      "author_name": "Jaafar Mahmoud",
      "author_url": "",
      "post_date": "2023-06-12T19:11:50.047000",
      "content": "<pre><code>*** METRICS ***\nurban / kyiv-puppet-theater (26 images, 325 pairs) -&gt; =0.892308, =0.973846, =0.906462\nurban -&gt; =0.892308\n\nheritage / dioscuri (174 images, 15051 pairs) -&gt; =0.716630, =0.825752, =0.725560\nheritage / cyprus (30 images, 435 pairs) -&gt; =0.637701, =0.646897, =0.673563\nheritage / wall (43 images, 903 pairs) -&gt; =0.912292, =0.950609, =0.934330\nheritage -&gt; =0.755541\n\nhaiper / bike (15 images, 105 pairs) -&gt; =0.921905, =0.999048, =0.921905\nhaiper / chairs (16 images, 120 pairs) -&gt; =0.965000, =0.995000, =0.966667\nhaiper / fountain (23 images, 253 pairs) -&gt; =0.999605, =1.000000, =0.999605\nhaiper -&gt; =0.962170\n\nFinal metric -&gt; =0.870006 (t: 1.9735429286956787 sec.)\n\n0.87000626\n</code></pre>\n<p>LB :485</p>",
      "votes": 4,
      "replies": []
    },
    {
      "id": 2274979,
      "author_name": "niubiren",
      "author_url": "",
      "post_date": "2023-05-26T12:09:45.260000",
      "content": "<p>LB：0.26<br>\n*** METRICS ***<br>\nurban / kyiv-puppet-theater (26 images, 325 pairs) -&gt; mAA=0.610154, mAA_q=0.808000, mAA_t=0.653846<br>\nurban -&gt; mAA=0.610154</p>\n<p>heritage / dioscuri (174 images, 15051 pairs) -&gt; mAA=0.001787, mAA_q=0.003535, mAA_t=0.014006<br>\nheritage / cyprus (30 images, 435 pairs) -&gt; mAA=0.541839, mAA_q=0.561379, mAA_t=0.548966<br>\nheritage / wall (43 images, 903 pairs) -&gt; mAA=0.302326, mAA_q=0.682060, mAA_t=0.304983<br>\nheritage -&gt; mAA=0.281984</p>\n<p>haiper / bike (15 images, 105 pairs) -&gt; mAA=0.908571, mAA_q=0.993333, mAA_t=0.908571<br>\nhaiper / chairs (16 images, 120 pairs) -&gt; mAA=0.647500, mAA_q=0.802500, mAA_t=0.656667<br>\nhaiper / fountain (23 images, 253 pairs) -&gt; mAA=0.999605, mAA_q=1.000000, mAA_t=0.999605<br>\nhaiper -&gt; mAA=0.851892</p>",
      "votes": 3,
      "replies": [
        {
          "id": 2276727,
          "author_name": "Pami_loveDL",
          "author_url": "",
          "post_date": "2023-05-27T06:41:50.467000",
          "content": "<p>May I ask if you are using PyColmap for incremental reconstruction? Or are there other methods for global reconstruction that you are using?</p>",
          "votes": 0,
          "replies": [
            {
              "id": 2276867,
              "author_name": "niubiren",
              "author_url": "",
              "post_date": "2023-05-27T09:16:41.003000",
              "content": "<p>yeah, it seems to me the only way to do reconstruction</p>",
              "votes": 2,
              "replies": []
            },
            {
              "id": 2276984,
              "author_name": "Pami_loveDL",
              "author_url": "",
              "post_date": "2023-05-27T11:04:41.803000",
              "content": "<p>Thank you for your sharing!😃</p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2274671,
      "author_name": "吕舟",
      "author_url": "",
      "post_date": "2023-05-26T07:17:52.483000",
      "content": "<p>I find this problem as well,there is a big gap between LB_Public and CV.<br>\nTraining data about 328<br>\nLB_Public data about 484<br>\nLB_provate data about 616<br>\nbased on the num of data,the gap is something we can't ignore. I would like to know how the top teams  deal with this problem.</p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 2274825,
      "author_name": "Pami_loveDL",
      "author_url": "",
      "post_date": "2023-05-26T09:18:24.193000",
      "content": "<p>Yes, I agree with what you're saying. Certain datasets may indeed hold a larger proportion in the LB, possibly similar to the heritage dataset or others.</p>\n<p>urban / kyiv-puppet-theater (26 images, 325 pairs) -&gt; mAA=0.574154, mAA_q=0.754769, mAA_t=0.604615<br>\n<strong>urban -&gt; mAA=0.574154</strong><br>\nheritage / dioscuri (174 images, 15051 pairs) -&gt; mAA=0.132091, mAA_q=0.152933, mAA_t=0.142309<br>\nheritage / cyprus (30 images, 435 pairs) -&gt; mAA=0.012874, mAA_q=0.019310, mAA_t=0.025747<br>\nheritage / wall (43 images, 903 pairs) -&gt; mAA=0.079291, mAA_q=0.528682, mAA_t=0.080066<br>\n<strong>heritage -&gt; mAA=0.074752</strong><br>\nhaiper / bike (15 images, 105 pairs) -&gt; mAA=0.192381, mAA_q=0.200000, mAA_t=0.192381<br>\nhaiper / chairs (16 images, 120 pairs) -&gt; mAA=0.526667, mAA_q=0.624167, mAA_t=0.530000<br>\nhaiper / fountain (23 images, 253 pairs) -&gt; mAA=1.000000, mAA_q=1.000000, mAA_t=1.000000<br>\n<strong>haiper -&gt; mAA=0.573016</strong><br>\n<strong>Final metric -&gt; mAA=0.407307</strong> </p>\n<p><strong>LeaderBoard 0.141</strong></p>",
      "votes": 4,
      "replies": []
    },
    {
      "id": 2276675,
      "author_name": "吕舟",
      "author_url": "",
      "post_date": "2023-05-27T05:54:29.527000",
      "content": "<p>You get a good performance on haiper dataset,maybe there is much  dataset like haiper on LB_private,trust your CV.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2276691,
          "author_name": "Gunes Evitan",
          "author_url": "",
          "post_date": "2023-05-27T06:15:57.667000",
          "content": "<p>Your urban score is really good too. I don't think there will be scenes like haiper in the private leaderboard. If you check haiper scenes visually, you can see that they are kinda too easy like very few images, small image size and images are literally taken around an object. Based on my 4 year Kaggle experience, I can say that I've never seen any private leaderboard with such easy examples in any competition.</p>",
          "votes": 5,
          "replies": [
            {
              "id": 2276741,
              "author_name": "吕舟",
              "author_url": "",
              "post_date": "2023-05-27T06:58:59.987000",
              "content": "<p>On other discussion you said you guess the intention was weighting each dataset equally . It seemed little bit weird to me as well.So maybe we should trust our CV and focus on hard dataset such as urban with more photos  in a huge scene.</p>",
              "votes": 1,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2281647,
      "author_name": "Bach Ngo",
      "author_url": "",
      "post_date": "2023-05-31T03:51:09.820000",
      "content": "<p>my lb: 0.348</p>\n<p>*** METRICS ***<br>\nurban / kyiv-puppet-theater (26 images, 325 pairs) -&gt; mAA=0.659385, mAA_q=0.836923, mAA_t=0.688308<br>\nurban -&gt; mAA=0.659385</p>\n<p>heritage / dioscuri (174 images, 15051 pairs) -&gt; mAA=0.404631, mAA_q=0.507096, mAA_t=0.410491<br>\nheritage / cyprus (30 images, 435 pairs) -&gt; mAA=0.022989, mAA_q=0.022989, mAA_t=0.029195<br>\nheritage / wall (43 images, 903 pairs) -&gt; mAA=0.864673, mAA_q=0.937431, mAA_t=0.878516<br>\nheritage -&gt; mAA=0.430764</p>\n<p>haiper / bike (15 images, 105 pairs) -&gt; mAA=0.921905, mAA_q=0.999048, mAA_t=0.921905<br>\nhaiper / chairs (16 images, 120 pairs) -&gt; mAA=0.776667, mAA_q=0.860000, mAA_t=0.778333<br>\nhaiper / fountain (23 images, 253 pairs) -&gt; mAA=1.000000, mAA_q=1.000000, mAA_t=1.000000<br>\nhaiper -&gt; mAA=0.899524</p>\n<p>Final metric -&gt; mAA=0.663224 (t: 0.6245906352996826 sec.)</p>\n<p>0.6632242</p>",
      "votes": 2,
      "replies": [
        {
          "id": 2281675,
          "author_name": "DL",
          "author_url": "",
          "post_date": "2023-05-31T04:29:43.390000",
          "content": "<p>Thanks for sharing</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2280853,
      "author_name": "bent1e",
      "author_url": "",
      "post_date": "2023-05-30T12:21:11.840000",
      "content": "<p>LB：0.18<br>\n*** METRICS ***<br>\nurban / kyiv-puppet-theater (26 images, 325 pairs) -&gt; mAA=0.844615, mAA_q=0.915385, mAA_t=0.854461<br>\nurban -&gt; mAA=0.844615</p>\n<p>heritage / dioscuri (174 images, 15051 pairs) -&gt; mAA=0.149532, mAA_q=0.242469, mAA_t=0.161929<br>\nheritage / cyprus (30 images, 435 pairs) -&gt; mAA=0.012874, mAA_q=0.013793, mAA_t=0.017471<br>\nheritage / wall (43 images, 903 pairs) -&gt; mAA=0.049059, mAA_q=0.517276, mAA_t=0.049280<br>\nheritage -&gt; mAA=0.070488</p>\n<p>haiper / bike (15 images, 105 pairs) -&gt; mAA=0.944762, mAA_q=1.000000, mAA_t=0.944762<br>\nhaiper / chairs (16 images, 120 pairs) -&gt; mAA=0.838333, mAA_q=0.867500, mAA_t=0.841667<br>\nhaiper / fountain (23 images, 253 pairs) -&gt; mAA=0.999605, mAA_q=1.000000, mAA_t=0.999605<br>\nhaiper -&gt; mAA=0.927567</p>\n<p>Final metric -&gt; mAA=0.614223 (t: 1.932884931564331 sec.)</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 2276889,
      "author_name": "bent1e",
      "author_url": "",
      "post_date": "2023-05-27T09:36:57.193000",
      "content": "<p>I know that the haiper data set is a relatively simple scenario, but it is still difficult for me to do a good result, even not as good as urban. Why is that?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 2283352,
          "author_name": "吕舟",
          "author_url": "",
          "post_date": "2023-06-01T07:58:46.387000",
          "content": "<p>It seems you  have resolve this problem.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2274552": "Here are my validation scores for every scene and the public leaderboard score listed. It looks like improvements on easier scenes like haiper doesn't translate to leaderboard. There are also scenes like dioscuri on which I can't get any improvement at all. I think public and private leaderboard is similar to heritage dataset. Do you have similar results? \n\n\n|                     | mAA    | mAA Rotation | mAA Translation |\n|---------------------|--------|--------------|-----------------|\n| bike                | 0.921  | 0.999        | 0.921           |\n| chairs              | 0.976  | 0.999        | 0.976           |\n| fountain            | 0.998  | 1            | 0.998           |\n| dioscuri            | 0.001  | 0.005        | 0.02            |\n| cyprus              | 0.2535 | 0.2701       | 0.2839          |\n| wall                | 0.8753 | 0.9503       | 0.8880          |\n| kyiv-puppet-theater | 0.2501 | 0.4390       | 0.3418          |\n| haiper              | 0.9652 | 0.999        | 0.9652          |\n| heritage            | 0.3769 | 0.4086       | 0.3977          |\n| urban               | 0.2501 | 0.4390       | 0.3418          |\n| **Validation**               | **0.5307** | **0.6157**       | **0.5681**          |\n| **Leaderboard**               | **0.214** |||\n\n",
    "2280942": "LB：0.467\n*** METRICS ***\nurban / kyiv-puppet-theater (26 images, 325 pairs) -> mAA=0.632308, mAA_q=0.823077, mAA_t=0.674462\nurban -> mAA=0.632308\n\nheritage / dioscuri (174 images, 15051 pairs) -> mAA=0.680167, mAA_q=0.799356, mAA_t=0.686001\nheritage / cyprus (30 images, 435 pairs) -> mAA=0.056092, mAA_q=0.057471, mAA_t=0.075402\nheritage / wall (43 images, 903 pairs) -> mAA=0.726910, mAA_q=0.922038, mAA_t=0.739867\nheritage -> mAA=0.487723\n\nhaiper / bike (15 images, 105 pairs) -> mAA=0.925714, mAA_q=0.999048, mAA_t=0.925714\nhaiper / chairs (16 images, 120 pairs) -> mAA=0.903333, mAA_q=0.986667, mAA_t=0.906667\nhaiper / fountain (23 images, 253 pairs) -> mAA=1.000000, mAA_q=1.000000, mAA_t=1.000000\nhaiper -> mAA=0.943016\n\nFinal metric -> mAA=0.687682 (t: 2.7530465126037598 sec.)\n",
    "2274955": "accroding to my last a few try, whether to skip scenes larger than 40 images can lead to remarkable difference(like 0.05 points). Given that my code did poor in these large scene, i guess large scenes really counts in final score. ",
    "2276879": "my lb : 0.338\n\n```\nhaiper / bike (15 images, 105 pairs) -> mAA=0.921905, mAA_q=1.000000, mAA_t=0.921905\nhaiper / chairs (16 images, 120 pairs) -> mAA=0.766667, mAA_q=0.846667, mAA_t=0.777500\nhaiper / fountain (23 images, 253 pairs) -> mAA=0.999210, mAA_q=1.000000, mAA_t=0.999210\nhaiper -> mAA=0.895927\n\nheritage / cyprus (30 images, 435 pairs) -> mAA=0.021839, mAA_q=0.022989, mAA_t=0.027816\nheritage / dioscuri (174 images, 15051 pairs) -> mAA=0.435572, mAA_q=0.535353, mAA_t=0.442469\nheritage / wall (43 images, 903 pairs) -> mAA=0.175748, mAA_q=0.442636, mAA_t=0.188815\nheritage -> mAA=0.211053\n\nurban / kyiv-puppet-theater (26 images, 325 pairs) -> mAA=0.755077, mAA_q=0.915077, mAA_t=0.780923\nurban -> mAA=0.755077\n\nFinal metric -> mAA=0.620686 (t: 3.0341649055480957 sec.)\n```",
    "2295648": "For me, a higher CV brings worse LB.\n\nurban / kyiv-puppet-theater (26 images, 325 pairs) -> mAA=0.915077, mAA_q=0.979692, mAA_t=0.924923\nurban -> mAA=0.915077\n\nheritage / dioscuri (174 images, 15051 pairs) -> mAA=0.508478, mAA_q=0.637612, mAA_t=0.514238\nheritage / cyprus (30 images, 435 pairs) -> mAA=0.022299, mAA_q=0.022989, mAA_t=0.028046\nheritage / wall (43 images, 903 pairs) -> mAA=0.523810, mAA_q=0.882946, mAA_t=0.533666\nheritage -> mAA=0.351529\n\nhaiper / bike (15 images, 105 pairs) -> mAA=0.951429, mAA_q=1.000000, mAA_t=0.951429\nhaiper / chairs (16 images, 120 pairs) -> mAA=0.929167, mAA_q=0.995000, mAA_t=0.931667\nhaiper / fountain (23 images, 253 pairs) -> mAA=0.999605, mAA_q=1.000000, mAA_t=0.999605\nhaiper -> mAA=0.960067\n\nFinal metric -> mAA=0.742224 (t: 2.0051422119140625 sec.)\n\nLB: 0.37😂",
    "2276683": "*** METRICS ***\nurban / kyiv-puppet-theater (26 images, 325 pairs) -> mAA=0.803692, mAA_q=0.888923, mAA_t=0.829231\nurban -> mAA=0.803692\n\nheritage / dioscuri (174 images, 15051 pairs) -> mAA=0.001787, mAA_q=0.003535, mAA_t=0.014006\nheritage / cyprus (30 images, 435 pairs) -> mAA=0.013793, mAA_q=0.013793, mAA_t=0.019080\nheritage / wall (43 images, 903 pairs) -> mAA=0.091251, mAA_q=0.547397, mAA_t=0.094574\nheritage -> mAA=0.035611\n\nhaiper / bike (15 images, 105 pairs) -> mAA=0.323810, mAA_q=0.382857, mAA_t=0.323810\nhaiper / chairs (16 images, 120 pairs) -> mAA=0.084167, mAA_q=0.132500, mAA_t=0.088333\nhaiper / fountain (23 images, 253 pairs) -> mAA=1.000000, mAA_q=1.000000, mAA_t=1.000000\nhaiper -> mAA=0.469325\n\nFinal metric -> mAA=0.436209 \nI was shocked that you weer able to achieve such good results on the haiper dataset",
    "2297749": "```\n*** METRICS ***\nurban / kyiv-puppet-theater (26 images, 325 pairs) -> mAA=0.892308, mAA_q=0.973846, mAA_t=0.906462\nurban -> mAA=0.892308\n\nheritage / dioscuri (174 images, 15051 pairs) -> mAA=0.716630, mAA_q=0.825752, mAA_t=0.725560\nheritage / cyprus (30 images, 435 pairs) -> mAA=0.637701, mAA_q=0.646897, mAA_t=0.673563\nheritage / wall (43 images, 903 pairs) -> mAA=0.912292, mAA_q=0.950609, mAA_t=0.934330\nheritage -> mAA=0.755541\n\nhaiper / bike (15 images, 105 pairs) -> mAA=0.921905, mAA_q=0.999048, mAA_t=0.921905\nhaiper / chairs (16 images, 120 pairs) -> mAA=0.965000, mAA_q=0.995000, mAA_t=0.966667\nhaiper / fountain (23 images, 253 pairs) -> mAA=0.999605, mAA_q=1.000000, mAA_t=0.999605\nhaiper -> mAA=0.962170\n\nFinal metric -> mAA=0.870006 (t: 1.9735429286956787 sec.)\n\n0.87000626\n```\nLB :485\n",
    "2274979": "LB：0.26\n*** METRICS ***\nurban / kyiv-puppet-theater (26 images, 325 pairs) -> mAA=0.610154, mAA_q=0.808000, mAA_t=0.653846\nurban -> mAA=0.610154\n\nheritage / dioscuri (174 images, 15051 pairs) -> mAA=0.001787, mAA_q=0.003535, mAA_t=0.014006\nheritage / cyprus (30 images, 435 pairs) -> mAA=0.541839, mAA_q=0.561379, mAA_t=0.548966\nheritage / wall (43 images, 903 pairs) -> mAA=0.302326, mAA_q=0.682060, mAA_t=0.304983\nheritage -> mAA=0.281984\n\nhaiper / bike (15 images, 105 pairs) -> mAA=0.908571, mAA_q=0.993333, mAA_t=0.908571\nhaiper / chairs (16 images, 120 pairs) -> mAA=0.647500, mAA_q=0.802500, mAA_t=0.656667\nhaiper / fountain (23 images, 253 pairs) -> mAA=0.999605, mAA_q=1.000000, mAA_t=0.999605\nhaiper -> mAA=0.851892",
    "2274671": "I find this problem as well,there is a big gap between LB_Public and CV.\nTraining data about 328\nLB_Public data about 484\nLB_provate data about 616\nbased on the num of data,the gap is something we can't ignore. I would like to know how the top teams  deal with this problem.",
    "2274825": "Yes, I agree with what you're saying. Certain datasets may indeed hold a larger proportion in the LB, possibly similar to the heritage dataset or others.\n\nurban / kyiv-puppet-theater (26 images, 325 pairs) -> mAA=0.574154, mAA_q=0.754769, mAA_t=0.604615\n**urban -> mAA=0.574154**\nheritage / dioscuri (174 images, 15051 pairs) -> mAA=0.132091, mAA_q=0.152933, mAA_t=0.142309\nheritage / cyprus (30 images, 435 pairs) -> mAA=0.012874, mAA_q=0.019310, mAA_t=0.025747\nheritage / wall (43 images, 903 pairs) -> mAA=0.079291, mAA_q=0.528682, mAA_t=0.080066\n**heritage -> mAA=0.074752**\nhaiper / bike (15 images, 105 pairs) -> mAA=0.192381, mAA_q=0.200000, mAA_t=0.192381\nhaiper / chairs (16 images, 120 pairs) -> mAA=0.526667, mAA_q=0.624167, mAA_t=0.530000\nhaiper / fountain (23 images, 253 pairs) -> mAA=1.000000, mAA_q=1.000000, mAA_t=1.000000\n**haiper -> mAA=0.573016**\n**Final metric -> mAA=0.407307** \n\n**LeaderBoard 0.141**",
    "2276675": "You get a good performance on haiper dataset,maybe there is much  dataset like haiper on LB_private,trust your CV.",
    "2281647": "my lb: 0.348\n\n*** METRICS ***\nurban / kyiv-puppet-theater (26 images, 325 pairs) -> mAA=0.659385, mAA_q=0.836923, mAA_t=0.688308\nurban -> mAA=0.659385\n\nheritage / dioscuri (174 images, 15051 pairs) -> mAA=0.404631, mAA_q=0.507096, mAA_t=0.410491\nheritage / cyprus (30 images, 435 pairs) -> mAA=0.022989, mAA_q=0.022989, mAA_t=0.029195\nheritage / wall (43 images, 903 pairs) -> mAA=0.864673, mAA_q=0.937431, mAA_t=0.878516\nheritage -> mAA=0.430764\n\nhaiper / bike (15 images, 105 pairs) -> mAA=0.921905, mAA_q=0.999048, mAA_t=0.921905\nhaiper / chairs (16 images, 120 pairs) -> mAA=0.776667, mAA_q=0.860000, mAA_t=0.778333\nhaiper / fountain (23 images, 253 pairs) -> mAA=1.000000, mAA_q=1.000000, mAA_t=1.000000\nhaiper -> mAA=0.899524\n\nFinal metric -> mAA=0.663224 (t: 0.6245906352996826 sec.)\n\n0.6632242",
    "2280853": "LB：0.18\n*** METRICS ***\nurban / kyiv-puppet-theater (26 images, 325 pairs) -> mAA=0.844615, mAA_q=0.915385, mAA_t=0.854461\nurban -> mAA=0.844615\n\nheritage / dioscuri (174 images, 15051 pairs) -> mAA=0.149532, mAA_q=0.242469, mAA_t=0.161929\nheritage / cyprus (30 images, 435 pairs) -> mAA=0.012874, mAA_q=0.013793, mAA_t=0.017471\nheritage / wall (43 images, 903 pairs) -> mAA=0.049059, mAA_q=0.517276, mAA_t=0.049280\nheritage -> mAA=0.070488\n\nhaiper / bike (15 images, 105 pairs) -> mAA=0.944762, mAA_q=1.000000, mAA_t=0.944762\nhaiper / chairs (16 images, 120 pairs) -> mAA=0.838333, mAA_q=0.867500, mAA_t=0.841667\nhaiper / fountain (23 images, 253 pairs) -> mAA=0.999605, mAA_q=1.000000, mAA_t=0.999605\nhaiper -> mAA=0.927567\n\nFinal metric -> mAA=0.614223 (t: 1.932884931564331 sec.)",
    "2276889": "I know that the haiper data set is a relatively simple scenario, but it is still difficult for me to do a good result, even not as good as urban. Why is that?"
  }
}