{
  "id": 575160,
  "title": "Public LB and Local Score",
  "url": "/competitions/image-matching-challenge-2025/discussion/575160",
  "author_name": "",
  "post_date": "2025-04-26T12:46:26.633226400Z",
  "votes": 52,
  "comment_count": 5,
  "views": 0,
  "content": "<p>I share my lb score and local score.</p>\n<p></p><ul><br>\n<li><p>public LB = 43.82</p><p></p>\n<table>\n<thead>\n<tr>\n<th>dataset</th>\n<th># of images</th>\n<th>Score</th>\n<th>mAA</th>\n<th>clusterness</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>imc2023_haiper</td>\n<td>54</td>\n<td>65.84</td>\n<td>68.33</td>\n<td>63.53</td>\n</tr>\n<tr>\n<td>imc2025-amy_gardens</td>\n<td>200</td>\n<td>25.66</td>\n<td>14.72</td>\n<td>100.00</td>\n</tr>\n<tr>\n<td>imc2025-fbk_vineyard</td>\n<td>200</td>\n<td>23.80</td>\n<td>18.51</td>\n<td>33.33</td>\n</tr>\n<tr>\n<td>imc2025-ETs</td>\n<td>22</td>\n<td>100.00</td>\n<td>100.00</td>\n<td>100.00</td>\n</tr>\n<tr>\n<td>imc2025-stairs</td>\n<td>51</td>\n<td>5.26</td>\n<td>2.78</td>\n<td>50.00</td>\n</tr>\n</tbody>\n</table>\n<p></p></li><br>\n<li><p>public LB = 33.86</p><p></p>\n<table>\n<thead>\n<tr>\n<th>dataset</th>\n<th># of images</th>\n<th>Score</th>\n<th>mAA</th>\n<th>clusterness</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>imc2023_haiper</td>\n<td>54</td>\n<td>65.58</td>\n<td>67.78</td>\n<td>63.53</td>\n</tr>\n<tr>\n<td>pt_brandenburg_british_buckingham</td>\n<td>225</td>\n<td>67.05</td>\n<td>76.16</td>\n<td>59.89</td>\n</tr>\n<tr>\n<td>pt_stpeters_stpauls</td>\n<td>200</td>\n<td>59.54</td>\n<td>73.58</td>\n<td>50.00</td>\n</tr>\n<tr>\n<td>imc2025-amy_gardens</td>\n<td>200</td>\n<td>24.11</td>\n<td>13.71</td>\n<td>100.00</td>\n</tr>\n<tr>\n<td>imc2025-fbk_vineyard</td>\n<td>200</td>\n<td>29.86</td>\n<td>22.40</td>\n<td>34.54</td>\n</tr>\n<tr>\n<td>imc2025-ETs</td>\n<td>22</td>\n<td>68.35</td>\n<td>51.92</td>\n<td>100.00</td>\n</tr>\n<tr>\n<td>imc2025-stairs</td>\n<td>51</td>\n<td>0.00</td>\n<td>0.00</td>\n<td>61.54</td>\n</tr>\n</tbody>\n</table>\n<p></p></li><br>\n<li><p>public LB = 28.49<br></p>\n<table>\n<thead>\n<tr>\n<th>dataset</th>\n<th># of images</th>\n<th>Score</th>\n<th>mAA</th>\n<th>clusterness</th>\n<th><br></th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>imc2023_haiper</td>\n<td>54</td>\n<td>65.58</td>\n<td>67.78</td>\n<td>63.53</td>\n<td><br></td>\n</tr>\n<tr>\n<td>imc2023_heritage</td>\n<td>209</td>\n<td>87.45</td>\n<td>77.70</td>\n<td>100.00</td>\n<td><br></td>\n</tr>\n<tr>\n<td>imc2023_theather_imc2024_church</td>\n<td>76</td>\n<td>65.71</td>\n<td>50.43</td>\n<td>100.00</td>\n<td><br></td>\n</tr>\n<tr>\n<td>imc2024_dioscuri_baalshamin</td>\n<td>138</td>\n<td>90.77</td>\n<td>83.10</td>\n<td>100.00</td>\n<td><br></td>\n</tr>\n<tr>\n<td>imc2024_lizard_pond</td>\n<td>214</td>\n<td>67.05</td>\n<td>40.43</td>\n<td>100.00</td>\n<td><br></td>\n</tr>\n<tr>\n<td>pt_brandenburg_british_buckingham</td>\n<td>225</td>\n<td>41.17</td>\n<td>53.82</td>\n<td>33.33</td>\n<td><br></td>\n</tr>\n<tr>\n<td>pt_piazzasanmarco_grandplace</td>\n<td>168</td>\n<td>85.61</td>\n<td>74.85</td>\n<td>100.00</td>\n<td><br></td>\n</tr>\n<tr>\n<td>pt_sacrecoeur_trevi_tajmahal</td>\n<td>225</td>\n<td>70.53</td>\n<td>85.53</td>\n<td>60.00</td>\n<td><br></td>\n</tr>\n<tr>\n<td>pt_stpeters_stpauls</td>\n<td>200</td>\n<td>60.08</td>\n<td>73.26</td>\n<td>50.00</td>\n<td><br></td>\n</tr>\n<tr>\n<td>imc2025-amy_gardens</td>\n<td>200</td>\n<td>25.47</td>\n<td>14.59</td>\n<td>100.00</td>\n<td><br></td>\n</tr>\n<tr>\n<td>imc2025-fbk_vineyard</td>\n<td>200</td>\n<td>42.59</td>\n<td>38.96</td>\n<td>46.97</td>\n<td><br></td>\n</tr>\n<tr>\n<td>imc2025-ETs</td>\n<td>22</td>\n<td>64.94</td>\n<td>48.08</td>\n<td>100.00</td>\n<td><br></td>\n</tr>\n<tr>\n<td>imc2025-stairs</td>\n<td>51</td>\n<td>2.17</td>\n<td>1.11</td>\n<td>50.00</td>\n<td><p></p></td></tr></tbody></table></li>\n\n\n</ul>\n\n\n\n\n\n\n\n.",
  "messages": [
    {
      "id": "3187730",
      "postDate": "04/26/2025 12:46:26",
      "content": "<p>I share my lb score and local score.</p>\n<p></p><ul><br>\n<li><p>public LB = 43.82</p><p></p>\n<table>\n<thead>\n<tr>\n<th>dataset</th>\n<th># of images</th>\n<th>Score</th>\n<th>mAA</th>\n<th>clusterness</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>imc2023_haiper</td>\n<td>54</td>\n<td>65.84</td>\n<td>68.33</td>\n<td>63.53</td>\n</tr>\n<tr>\n<td>imc2025-amy_gardens</td>\n<td>200</td>\n<td>25.66</td>\n<td>14.72</td>\n<td>100.00</td>\n</tr>\n<tr>\n<td>imc2025-fbk_vineyard</td>\n<td>200</td>\n<td>23.80</td>\n<td>18.51</td>\n<td>33.33</td>\n</tr>\n<tr>\n<td>imc2025-ETs</td>\n<td>22</td>\n<td>100.00</td>\n<td>100.00</td>\n<td>100.00</td>\n</tr>\n<tr>\n<td>imc2025-stairs</td>\n<td>51</td>\n<td>5.26</td>\n<td>2.78</td>\n<td>50.00</td>\n</tr>\n</tbody>\n</table>\n<p></p></li><br>\n<li><p>public LB = 33.86</p><p></p>\n<table>\n<thead>\n<tr>\n<th>dataset</th>\n<th># of images</th>\n<th>Score</th>\n<th>mAA</th>\n<th>clusterness</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>imc2023_haiper</td>\n<td>54</td>\n<td>65.58</td>\n<td>67.78</td>\n<td>63.53</td>\n</tr>\n<tr>\n<td>pt_brandenburg_british_buckingham</td>\n<td>225</td>\n<td>67.05</td>\n<td>76.16</td>\n<td>59.89</td>\n</tr>\n<tr>\n<td>pt_stpeters_stpauls</td>\n<td>200</td>\n<td>59.54</td>\n<td>73.58</td>\n<td>50.00</td>\n</tr>\n<tr>\n<td>imc2025-amy_gardens</td>\n<td>200</td>\n<td>24.11</td>\n<td>13.71</td>\n<td>100.00</td>\n</tr>\n<tr>\n<td>imc2025-fbk_vineyard</td>\n<td>200</td>\n<td>29.86</td>\n<td>22.40</td>\n<td>34.54</td>\n</tr>\n<tr>\n<td>imc2025-ETs</td>\n<td>22</td>\n<td>68.35</td>\n<td>51.92</td>\n<td>100.00</td>\n</tr>\n<tr>\n<td>imc2025-stairs</td>\n<td>51</td>\n<td>0.00</td>\n<td>0.00</td>\n<td>61.54</td>\n</tr>\n</tbody>\n</table>\n<p></p></li><br>\n<li><p>public LB = 28.49<br></p>\n<table>\n<thead>\n<tr>\n<th>dataset</th>\n<th># of images</th>\n<th>Score</th>\n<th>mAA</th>\n<th>clusterness</th>\n<th><br></th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>imc2023_haiper</td>\n<td>54</td>\n<td>65.58</td>\n<td>67.78</td>\n<td>63.53</td>\n<td><br></td>\n</tr>\n<tr>\n<td>imc2023_heritage</td>\n<td>209</td>\n<td>87.45</td>\n<td>77.70</td>\n<td>100.00</td>\n<td><br></td>\n</tr>\n<tr>\n<td>imc2023_theather_imc2024_church</td>\n<td>76</td>\n<td>65.71</td>\n<td>50.43</td>\n<td>100.00</td>\n<td><br></td>\n</tr>\n<tr>\n<td>imc2024_dioscuri_baalshamin</td>\n<td>138</td>\n<td>90.77</td>\n<td>83.10</td>\n<td>100.00</td>\n<td><br></td>\n</tr>\n<tr>\n<td>imc2024_lizard_pond</td>\n<td>214</td>\n<td>67.05</td>\n<td>40.43</td>\n<td>100.00</td>\n<td><br></td>\n</tr>\n<tr>\n<td>pt_brandenburg_british_buckingham</td>\n<td>225</td>\n<td>41.17</td>\n<td>53.82</td>\n<td>33.33</td>\n<td><br></td>\n</tr>\n<tr>\n<td>pt_piazzasanmarco_grandplace</td>\n<td>168</td>\n<td>85.61</td>\n<td>74.85</td>\n<td>100.00</td>\n<td><br></td>\n</tr>\n<tr>\n<td>pt_sacrecoeur_trevi_tajmahal</td>\n<td>225</td>\n<td>70.53</td>\n<td>85.53</td>\n<td>60.00</td>\n<td><br></td>\n</tr>\n<tr>\n<td>pt_stpeters_stpauls</td>\n<td>200</td>\n<td>60.08</td>\n<td>73.26</td>\n<td>50.00</td>\n<td><br></td>\n</tr>\n<tr>\n<td>imc2025-amy_gardens</td>\n<td>200</td>\n<td>25.47</td>\n<td>14.59</td>\n<td>100.00</td>\n<td><br></td>\n</tr>\n<tr>\n<td>imc2025-fbk_vineyard</td>\n<td>200</td>\n<td>42.59</td>\n<td>38.96</td>\n<td>46.97</td>\n<td><br></td>\n</tr>\n<tr>\n<td>imc2025-ETs</td>\n<td>22</td>\n<td>64.94</td>\n<td>48.08</td>\n<td>100.00</td>\n<td><br></td>\n</tr>\n<tr>\n<td>imc2025-stairs</td>\n<td>51</td>\n<td>2.17</td>\n<td>1.11</td>\n<td>50.00</td>\n<td><p></p></td></tr></tbody></table></li>\n\n\n</ul>\n\n\n\n\n\n\n\n.",
      "rawMarkdown": "I share my lb score and local score.\n\n- public LB = 43.82\n| dataset  | # of images | Score | mAA  | clusterness |\n|---------------------|----:|------:|------:|-------:|\n| imc2023_haiper      |  54 | 65.84 | 68.33 | 63.53  |\n| imc2025-amy_gardens | 200 | 25.66 | 14.72 | 100.00 |\n| imc2025-fbk_vineyard| 200 | 23.80 | 18.51 |  33.33 |\n| imc2025-ETs         |  22 |100.00 |100.00 | 100.00 |\n| imc2025-stairs |  51 |  5.26 |  2.78 |  50.00 |\n\n- public LB = 33.86\n| dataset  | # of images | Score | mAA  | clusterness |\n|---------------------|----:|------:|------:|-------:|\n| imc2023_haiper      |  54 | 65.58 | 67.78 | 63.53  |\n| pt_brandenburg_british_buckingham | 225 | 67.05 | 76.16 | 59.89 |\n| pt_stpeters_stpauls | 200 | 59.54 | 73.58 | 50.00  |\n| imc2025-amy_gardens | 200 | 24.11 | 13.71 | 100.00 |\n| imc2025-fbk_vineyard| 200 | 29.86 | 22.40 |  34.54 |\n| imc2025-ETs         |  22 | 68.35 | 51.92 | 100.00 |\n| imc2025-stairs |  51 |  0.00 |  0.00 |  61.54 |\n\n- public LB = 28.49\n| dataset  | # of images | Score | mAA  | clusterness |\n|---------------------|----:|------:|------:|-------:|\n| imc2023_haiper      |  54 | 65.58 | 67.78 | 63.53  |\n| imc2023_heritage    | 209 | 87.45 | 77.70 | 100.00 |\n| imc2023_theather_imc2024_church    | 76 | 65.71 | 50.43 | 100.00 |\n| imc2024_dioscuri_baalshamin    | 138 | 90.77 | 83.10 | 100.00 |\n| imc2024_lizard_pond | 214 | 67.05 | 40.43 | 100.00 |\n| pt_brandenburg_british_buckingham | 225 | 41.17 | 53.82 | 33.33 |\n| pt_piazzasanmarco_grandplace | 168 | 85.61 | 74.85 | 100.00 |\n| pt_sacrecoeur_trevi_tajmahal | 225 | 70.53 | 85.53 | 60.00 |\n| pt_stpeters_stpauls | 200 | 60.08 | 73.26 |  50.00 |\n| imc2025-amy_gardens | 200 | 25.47 | 14.59 | 100.00 |\n| imc2025-fbk_vineyard| 200 | 42.59 | 38.96 |  46.97 |\n| imc2025-ETs         |  22 | 64.94 | 48.08 | 100.00 |\n| imc2025-stairs |  51 |  2.17 |  1.11 |  50.00 |\n\n.",
      "votes": null
    },
    {
      "id": "3187918",
      "postDate": "04/26/2025 17:43:23",
      "content": "<p>nice work man!</p>",
      "rawMarkdown": "nice work man!",
      "votes": null
    },
    {
      "id": "3203893",
      "postDate": "05/17/2025 12:43:57",
      "content": "<p>Thanks for sharing, are your submission scores stable between different submissions?</p>",
      "rawMarkdown": "Thanks for sharing, are your submission scores stable between different submissions?",
      "votes": null
    },
    {
      "id": "3207601",
      "postDate": "05/23/2025 03:01:11",
      "content": "<p>Really appreciate you sharing these. It's interesting that as your score for <strong>\"fbk_vineyard\"</strong> goes down, your LB score goes up. The <strong>\"stairs\"</strong> dataset also appears to be much more difficult than the test datasets. Your improvements on <strong>\"ETs\"</strong> seem to show the most correlation with the leaderboard score, outside of that dataset it's hard to see where the +15 in LB score is coming from.</p>\n<p>If we just look at just the five datasets that are given for each submission: haiper, amy_gardens, ETs, fbk_vineyard &amp; stairs, we can make this overview:</p>\n<table>\n<thead>\n<tr>\n<th>submission</th>\n<th>Public LB</th>\n<th>avg. score</th>\n<th>avg. mAA</th>\n<th>avg. clusterness</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>v3</td>\n<td>43.82</td>\n<td>44.112</td>\n<td>40.868</td>\n<td>69.372</td>\n</tr>\n<tr>\n<td>v2</td>\n<td>33.86</td>\n<td>37.58</td>\n<td>31.162</td>\n<td>71.922</td>\n</tr>\n<tr>\n<td>v1</td>\n<td>28.49</td>\n<td>40.15</td>\n<td>34.104</td>\n<td>72.1</td>\n</tr>\n</tbody>\n</table>\n<p><br></p>\n<p>My main takeaway from this is that we should focus on improving mAA over clusterness, but it's surprising to see the v2 submission outperform the v1 given its weaker performance locally. Currently, there are two gold medalists with 10 or fewer submissions - I wonder if they've found a better way to evaluate their notebooks locally.</p>",
      "rawMarkdown": "Really appreciate you sharing these. It's interesting that as your score for **\"fbk_vineyard\"** goes down, your LB score goes up. The **\"stairs\"** dataset also appears to be much more difficult than the test datasets. Your improvements on **\"ETs\"** seem to show the most correlation with the leaderboard score, outside of that dataset it's hard to see where the +15 in LB score is coming from.\n\nIf we just look at just the five datasets that are given for each submission: haiper, amy_gardens, ETs, fbk_vineyard & stairs, we can make this overview:\n\n|submission |Public LB\t|avg. score\t|avg. mAA\t|avg. clusterness|\n| --- | --- |--- |--- |--- |\n|v3\t|43.82\t|44.112\t|40.868\t|69.372|\n|v2\t|33.86\t|37.58\t|31.162\t|71.922|\n|v1\t|28.49\t|40.15\t|34.104\t|72.1|\n\n<br>\n\nMy main takeaway from this is that we should focus on improving mAA over clusterness, but it's surprising to see the v2 submission outperform the v1 given its weaker performance locally. Currently, there are two gold medalists with 10 or fewer submissions - I wonder if they've found a better way to evaluate their notebooks locally.",
      "votes": null
    },
    {
      "id": "3207753",
      "postDate": "05/23/2025 07:34:09",
      "content": "<p>It works on me, nice! Thanks a lot!</p>",
      "rawMarkdown": "It works on me, nice! Thanks a lot!",
      "votes": null
    },
    {
      "id": "3213886",
      "postDate": "05/30/2025 16:13:15",
      "content": "<p>From the results of the experiment you shared, it seems that there is some correlation between LB and Public Score.</p>",
      "rawMarkdown": "From the results of the experiment you shared, it seems that there is some correlation between LB and Public Score.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3187918,
      "author_name": "mohanapavanbezawada",
      "author_url": "",
      "post_date": "04/26/2025 17:43:23",
      "content": "<p>nice work man!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3203893,
      "author_name": "yuanzhezhou",
      "author_url": "",
      "post_date": "05/17/2025 12:43:57",
      "content": "<p>Thanks for sharing, are your submission scores stable between different submissions?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3207601,
      "author_name": "petermoorhouse",
      "author_url": "",
      "post_date": "05/23/2025 03:01:11",
      "content": "<p>Really appreciate you sharing these. It's interesting that as your score for <strong>\"fbk_vineyard\"</strong> goes down, your LB score goes up. The <strong>\"stairs\"</strong> dataset also appears to be much more difficult than the test datasets. Your improvements on <strong>\"ETs\"</strong> seem to show the most correlation with the leaderboard score, outside of that dataset it's hard to see where the +15 in LB score is coming from.</p>\n<p>If we just look at just the five datasets that are given for each submission: haiper, amy_gardens, ETs, fbk_vineyard &amp; stairs, we can make this overview:</p>\n<table>\n<thead>\n<tr>\n<th>submission</th>\n<th>Public LB</th>\n<th>avg. score</th>\n<th>avg. mAA</th>\n<th>avg. clusterness</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>v3</td>\n<td>43.82</td>\n<td>44.112</td>\n<td>40.868</td>\n<td>69.372</td>\n</tr>\n<tr>\n<td>v2</td>\n<td>33.86</td>\n<td>37.58</td>\n<td>31.162</td>\n<td>71.922</td>\n</tr>\n<tr>\n<td>v1</td>\n<td>28.49</td>\n<td>40.15</td>\n<td>34.104</td>\n<td>72.1</td>\n</tr>\n</tbody>\n</table>\n<p><br></p>\n<p>My main takeaway from this is that we should focus on improving mAA over clusterness, but it's surprising to see the v2 submission outperform the v1 given its weaker performance locally. Currently, there are two gold medalists with 10 or fewer submissions - I wonder if they've found a better way to evaluate their notebooks locally.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3207753,
      "author_name": "",
      "author_url": "",
      "post_date": "05/23/2025 07:34:09",
      "content": "<p>It works on me, nice! Thanks a lot!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3213886,
      "author_name": "michaelchen2025",
      "author_url": "",
      "post_date": "05/30/2025 16:13:15",
      "content": "<p>From the results of the experiment you shared, it seems that there is some correlation between LB and Public Score.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3187730": "I share my lb score and local score.\n\n- public LB = 43.82\n| dataset  | # of images | Score | mAA  | clusterness |\n|---------------------|----:|------:|------:|-------:|\n| imc2023_haiper      |  54 | 65.84 | 68.33 | 63.53  |\n| imc2025-amy_gardens | 200 | 25.66 | 14.72 | 100.00 |\n| imc2025-fbk_vineyard| 200 | 23.80 | 18.51 |  33.33 |\n| imc2025-ETs         |  22 |100.00 |100.00 | 100.00 |\n| imc2025-stairs |  51 |  5.26 |  2.78 |  50.00 |\n\n- public LB = 33.86\n| dataset  | # of images | Score | mAA  | clusterness |\n|---------------------|----:|------:|------:|-------:|\n| imc2023_haiper      |  54 | 65.58 | 67.78 | 63.53  |\n| pt_brandenburg_british_buckingham | 225 | 67.05 | 76.16 | 59.89 |\n| pt_stpeters_stpauls | 200 | 59.54 | 73.58 | 50.00  |\n| imc2025-amy_gardens | 200 | 24.11 | 13.71 | 100.00 |\n| imc2025-fbk_vineyard| 200 | 29.86 | 22.40 |  34.54 |\n| imc2025-ETs         |  22 | 68.35 | 51.92 | 100.00 |\n| imc2025-stairs |  51 |  0.00 |  0.00 |  61.54 |\n\n- public LB = 28.49\n| dataset  | # of images | Score | mAA  | clusterness |\n|---------------------|----:|------:|------:|-------:|\n| imc2023_haiper      |  54 | 65.58 | 67.78 | 63.53  |\n| imc2023_heritage    | 209 | 87.45 | 77.70 | 100.00 |\n| imc2023_theather_imc2024_church    | 76 | 65.71 | 50.43 | 100.00 |\n| imc2024_dioscuri_baalshamin    | 138 | 90.77 | 83.10 | 100.00 |\n| imc2024_lizard_pond | 214 | 67.05 | 40.43 | 100.00 |\n| pt_brandenburg_british_buckingham | 225 | 41.17 | 53.82 | 33.33 |\n| pt_piazzasanmarco_grandplace | 168 | 85.61 | 74.85 | 100.00 |\n| pt_sacrecoeur_trevi_tajmahal | 225 | 70.53 | 85.53 | 60.00 |\n| pt_stpeters_stpauls | 200 | 60.08 | 73.26 |  50.00 |\n| imc2025-amy_gardens | 200 | 25.47 | 14.59 | 100.00 |\n| imc2025-fbk_vineyard| 200 | 42.59 | 38.96 |  46.97 |\n| imc2025-ETs         |  22 | 64.94 | 48.08 | 100.00 |\n| imc2025-stairs |  51 |  2.17 |  1.11 |  50.00 |\n\n.",
    "3187918": "nice work man!",
    "3203893": "Thanks for sharing, are your submission scores stable between different submissions?",
    "3207601": "Really appreciate you sharing these. It's interesting that as your score for **\"fbk_vineyard\"** goes down, your LB score goes up. The **\"stairs\"** dataset also appears to be much more difficult than the test datasets. Your improvements on **\"ETs\"** seem to show the most correlation with the leaderboard score, outside of that dataset it's hard to see where the +15 in LB score is coming from.\n\nIf we just look at just the five datasets that are given for each submission: haiper, amy_gardens, ETs, fbk_vineyard & stairs, we can make this overview:\n\n|submission |Public LB\t|avg. score\t|avg. mAA\t|avg. clusterness|\n| --- | --- |--- |--- |--- |\n|v3\t|43.82\t|44.112\t|40.868\t|69.372|\n|v2\t|33.86\t|37.58\t|31.162\t|71.922|\n|v1\t|28.49\t|40.15\t|34.104\t|72.1|\n\n<br>\n\nMy main takeaway from this is that we should focus on improving mAA over clusterness, but it's surprising to see the v2 submission outperform the v1 given its weaker performance locally. Currently, there are two gold medalists with 10 or fewer submissions - I wonder if they've found a better way to evaluate their notebooks locally.",
    "3207753": "It works on me, nice! Thanks a lot!",
    "3213886": "From the results of the experiment you shared, it seems that there is some correlation between LB and Public Score."
  },
  "source": "meta"
}