{
  "id": 583131,
  "title": "Viewing and understanding the churn across the leaderboards",
  "url": "/competitions/byu-locating-bacterial-flagellar-motors-2025/discussion/583131",
  "author_name": "",
  "post_date": "2025-06-05T00:49:20.301933700Z",
  "votes": 14,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hello all,</p>\n<p>We have a moderate level of churn here as shown in the figure below-</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8273630%2F27e62c453a0f03211e3b5d476b720ddb%2FLB.png?generation=1749083494556393&amp;alt=media\" alt=\"\"></p>\n<p><br>Let's delve into the figure here and understand the results in a bit of detail-</p>\n<table>\n<thead>\n<tr>\n<th>Region Label</th>\n<th>Comments</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>R1</td>\n<td>Top 10% in public + private leaderboard <br> Congratulations to these teams for their superior achievements across both leaderboards!</td>\n</tr>\n<tr>\n<td>R2</td>\n<td>Teams that did well on the public leaderboard but fell off the perch on the private leaderboard <br> This pool is dense, indicating some level of churn in the competition</td>\n</tr>\n<tr>\n<td>R3</td>\n<td>This is the reverse of R2 - congrats to these teams for moving up in the churn from a modest public LB position!</td>\n</tr>\n<tr>\n<td>R4</td>\n<td>Similar to R2 but outside the top10% rankers!</td>\n</tr>\n</tbody>\n</table>\n<p><br>Let's delve into the top10% ranks and fathom their submission and score patterns!</p>\n<table>\n<thead>\n<tr>\n<th>Medal zone</th>\n<th>Last rank</th>\n<th>Submissions - min-mean-max</th>\n<th>Stability details</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Gold</td>\n<td>12</td>\n<td>16-198-455</td>\n<td>Stable teams = 6 <br> Teams entering from below = 6</td>\n</tr>\n<tr>\n<td>Silver</td>\n<td>58</td>\n<td>46 - 202 - 460</td>\n<td>Stable teams = 17 <br> Teams entering from below = 25 <br> Teams entering from above = 4</td>\n</tr>\n<tr>\n<td>Bronze</td>\n<td>117</td>\n<td>6 - 94 - 364</td>\n<td>Stable teams = 13 <br> Teams entering from below =  38 <br> Teams entering from above = 8</td>\n</tr>\n</tbody>\n</table>\n<p><br> The number of average submissions consistently decreases as we move from gold - bronze regions!</p>\n<h2>Other events and analysis</h2>\n<table>\n<thead>\n<tr>\n<th>Event</th>\n<th>Public LB rank</th>\n<th>Private LB rank</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Overall - highest upward movement in private LB</td>\n<td>459</td>\n<td>33</td>\n</tr>\n<tr>\n<td>R1 - highest upward movement in private LB</td>\n<td>98</td>\n<td>18</td>\n</tr>\n<tr>\n<td>Overall - biggest fall in private LB</td>\n<td>88</td>\n<td>618</td>\n</tr>\n<tr>\n<td>R1 - biggest fall in private LB</td>\n<td>29</td>\n<td>115</td>\n</tr>\n</tbody>\n</table>\n<h2>Concluding remarks</h2>\n<p>Congrats to the winners and best regards to one and all! <br>\nWishing you the best for your future goals on Kaggle and otherwise! </p>\n<p>Regards,<br>\nRavi R</p>",
  "messages": [
    {
      "id": "3217374",
      "postDate": "06/05/2025 00:49:20",
      "content": "<p>Hello all,</p>\n<p>We have a moderate level of churn here as shown in the figure below-</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8273630%2F27e62c453a0f03211e3b5d476b720ddb%2FLB.png?generation=1749083494556393&amp;alt=media\" alt=\"\"></p>\n<p><br>Let's delve into the figure here and understand the results in a bit of detail-</p>\n<table>\n<thead>\n<tr>\n<th>Region Label</th>\n<th>Comments</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>R1</td>\n<td>Top 10% in public + private leaderboard <br> Congratulations to these teams for their superior achievements across both leaderboards!</td>\n</tr>\n<tr>\n<td>R2</td>\n<td>Teams that did well on the public leaderboard but fell off the perch on the private leaderboard <br> This pool is dense, indicating some level of churn in the competition</td>\n</tr>\n<tr>\n<td>R3</td>\n<td>This is the reverse of R2 - congrats to these teams for moving up in the churn from a modest public LB position!</td>\n</tr>\n<tr>\n<td>R4</td>\n<td>Similar to R2 but outside the top10% rankers!</td>\n</tr>\n</tbody>\n</table>\n<p><br>Let's delve into the top10% ranks and fathom their submission and score patterns!</p>\n<table>\n<thead>\n<tr>\n<th>Medal zone</th>\n<th>Last rank</th>\n<th>Submissions - min-mean-max</th>\n<th>Stability details</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Gold</td>\n<td>12</td>\n<td>16-198-455</td>\n<td>Stable teams = 6 <br> Teams entering from below = 6</td>\n</tr>\n<tr>\n<td>Silver</td>\n<td>58</td>\n<td>46 - 202 - 460</td>\n<td>Stable teams = 17 <br> Teams entering from below = 25 <br> Teams entering from above = 4</td>\n</tr>\n<tr>\n<td>Bronze</td>\n<td>117</td>\n<td>6 - 94 - 364</td>\n<td>Stable teams = 13 <br> Teams entering from below =  38 <br> Teams entering from above = 8</td>\n</tr>\n</tbody>\n</table>\n<p><br> The number of average submissions consistently decreases as we move from gold - bronze regions!</p>\n<h2>Other events and analysis</h2>\n<table>\n<thead>\n<tr>\n<th>Event</th>\n<th>Public LB rank</th>\n<th>Private LB rank</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Overall - highest upward movement in private LB</td>\n<td>459</td>\n<td>33</td>\n</tr>\n<tr>\n<td>R1 - highest upward movement in private LB</td>\n<td>98</td>\n<td>18</td>\n</tr>\n<tr>\n<td>Overall - biggest fall in private LB</td>\n<td>88</td>\n<td>618</td>\n</tr>\n<tr>\n<td>R1 - biggest fall in private LB</td>\n<td>29</td>\n<td>115</td>\n</tr>\n</tbody>\n</table>\n<h2>Concluding remarks</h2>\n<p>Congrats to the winners and best regards to one and all! <br>\nWishing you the best for your future goals on Kaggle and otherwise! </p>\n<p>Regards,<br>\nRavi R</p>",
      "rawMarkdown": "Hello all,\n\nWe have a moderate level of churn here as shown in the figure below-\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8273630%2F27e62c453a0f03211e3b5d476b720ddb%2FLB.png?generation=1749083494556393&alt=media)\n\n<br>Let's delve into the figure here and understand the results in a bit of detail-\n\n|Region Label|Comments|\n| --------- | ----------- |\n|R1 | Top 10% in public + private leaderboard <br> Congratulations to these teams for their superior achievements across both leaderboards! | \n|R2 | Teams that did well on the public leaderboard but fell off the perch on the private leaderboard <br> This pool is dense, indicating some level of churn in the competition | \n|R3 | This is the reverse of R2 - congrats to these teams for moving up in the churn from a modest public LB position! | \n|R4 | Similar to R2 but outside the top10% rankers! | \n\n<br>Let's delve into the top10% ranks and fathom their submission and score patterns!\n\n|Medal zone   |Last rank |Submissions - min-mean-max|Stability details|\n|----|---------|---------| ----------- |\n|Gold  |12| 16-198-455       | Stable teams = 6 <br> Teams entering from below = 6 |\n|Silver |58| 46 - 202 - 460 | Stable teams = 17 <br> Teams entering from below = 25 <br> Teams entering from above = 4 |\n|Bronze |117| 6 - 94 - 364  | Stable teams = 13 <br> Teams entering from below =  38 <br> Teams entering from above = 8 |\n\n<br> The number of average submissions consistently decreases as we move from gold - bronze regions!\n\n## Other events and analysis\n\n|Event|Public LB rank|Private LB rank|\n| -------- | ---------- | ---------- |\n| Overall - highest upward movement in private LB | 459 | 33 |\n| R1 - highest upward movement in private LB | 98 | 18 |\n| Overall - biggest fall in private LB | 88| 618|\n| R1 - biggest fall in private LB | 29| 115 |\n\n## Concluding remarks\n\nCongrats to the winners and best regards to one and all! \nWishing you the best for your future goals on Kaggle and otherwise! \n\nRegards,\nRavi R",
      "votes": null
    },
    {
      "id": "3217387",
      "postDate": "06/05/2025 01:15:32",
      "content": "<p>I generated this plot among others earlier today. I’ll share all my plots later. I am curious what the straight lines right beside the line of best fit are. Public notebooks? If so why are there random offshoots in some places?</p>",
      "rawMarkdown": "I generated this plot among others earlier today. I’ll share all my plots later. I am curious what the straight lines right beside the line of best fit are. Public notebooks? If so why are there random offshoots in some places?",
      "votes": null
    },
    {
      "id": "3217418",
      "postDate": "06/05/2025 02:07:33",
      "content": "<p>It's theoretically possible for teams to have submitted and gotten exactly the same score as public notebooks on the public set but different scores on the private set</p>",
      "rawMarkdown": "It's theoretically possible for teams to have submitted and gotten exactly the same score as public notebooks on the public set but different scores on the private set",
      "votes": null
    },
    {
      "id": "3217509",
      "postDate": "06/05/2025 04:53:50",
      "content": "<p>I'm not surprised the results. Annotation based on human preference with a lot of mistakes is not stable for validation. </p>",
      "rawMarkdown": "I'm not surprised the results. Annotation based on human preference with a lot of mistakes is not stable for validation.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3217387,
      "author_name": "andrewjdarley",
      "author_url": "",
      "post_date": "06/05/2025 01:15:32",
      "content": "<p>I generated this plot among others earlier today. I’ll share all my plots later. I am curious what the straight lines right beside the line of best fit are. Public notebooks? If so why are there random offshoots in some places?</p>",
      "votes": null,
      "replies": [
        {
          "id": 3217418,
          "author_name": "ryches",
          "author_url": "",
          "post_date": "06/05/2025 02:07:33",
          "content": "<p>It's theoretically possible for teams to have submitted and gotten exactly the same score as public notebooks on the public set but different scores on the private set</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3217509,
      "author_name": "tom99763",
      "author_url": "",
      "post_date": "06/05/2025 04:53:50",
      "content": "<p>I'm not surprised the results. Annotation based on human preference with a lot of mistakes is not stable for validation. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3217374": "Hello all,\n\nWe have a moderate level of churn here as shown in the figure below-\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8273630%2F27e62c453a0f03211e3b5d476b720ddb%2FLB.png?generation=1749083494556393&alt=media)\n\n<br>Let's delve into the figure here and understand the results in a bit of detail-\n\n|Region Label|Comments|\n| --------- | ----------- |\n|R1 | Top 10% in public + private leaderboard <br> Congratulations to these teams for their superior achievements across both leaderboards! | \n|R2 | Teams that did well on the public leaderboard but fell off the perch on the private leaderboard <br> This pool is dense, indicating some level of churn in the competition | \n|R3 | This is the reverse of R2 - congrats to these teams for moving up in the churn from a modest public LB position! | \n|R4 | Similar to R2 but outside the top10% rankers! | \n\n<br>Let's delve into the top10% ranks and fathom their submission and score patterns!\n\n|Medal zone   |Last rank |Submissions - min-mean-max|Stability details|\n|----|---------|---------| ----------- |\n|Gold  |12| 16-198-455       | Stable teams = 6 <br> Teams entering from below = 6 |\n|Silver |58| 46 - 202 - 460 | Stable teams = 17 <br> Teams entering from below = 25 <br> Teams entering from above = 4 |\n|Bronze |117| 6 - 94 - 364  | Stable teams = 13 <br> Teams entering from below =  38 <br> Teams entering from above = 8 |\n\n<br> The number of average submissions consistently decreases as we move from gold - bronze regions!\n\n## Other events and analysis\n\n|Event|Public LB rank|Private LB rank|\n| -------- | ---------- | ---------- |\n| Overall - highest upward movement in private LB | 459 | 33 |\n| R1 - highest upward movement in private LB | 98 | 18 |\n| Overall - biggest fall in private LB | 88| 618|\n| R1 - biggest fall in private LB | 29| 115 |\n\n## Concluding remarks\n\nCongrats to the winners and best regards to one and all! \nWishing you the best for your future goals on Kaggle and otherwise! \n\nRegards,\nRavi R",
    "3217387": "I generated this plot among others earlier today. I’ll share all my plots later. I am curious what the straight lines right beside the line of best fit are. Public notebooks? If so why are there random offshoots in some places?",
    "3217418": "It's theoretically possible for teams to have submitted and gotten exactly the same score as public notebooks on the public set but different scores on the private set",
    "3217509": "I'm not surprised the results. Annotation based on human preference with a lot of mistakes is not stable for validation."
  },
  "source": "meta"
}