{
  "id": 567316,
  "title": "Mar9 update analysis - almost complete churn across bronze zone!",
  "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/567316",
  "author_name": "",
  "post_date": "2025-03-09T16:43:16.267418500Z",
  "votes": 17,
  "comment_count": 7,
  "views": 0,
  "content": "<p>Hello all,</p>\n<p>We have a leaderboard update as on date and the figure below delves into the regions of interest, between this update and the leaderboard update in mid-February.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8273630%2F5a8800a9fe59ffe2da0dc1042bacda21%2FMar9.png?generation=1741537823868685&amp;alt=media\" alt=\"\"></p>\n<p><br>We see a lot of churn here compared to a relatively stable leaderboard over the <a href=\"https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/562435\" target=\"_blank\">last time</a></p>\n<table>\n<thead>\n<tr>\n<th>Region label</th>\n<th>Description</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>R1</td>\n<td>* This region illustrates the top 10% participants in the competition that remained stable across the 2 updates. This region is quick thick, indicating good stability! Congratulations to these participants for the stable performance</td>\n</tr>\n<tr>\n<td>R2</td>\n<td>* This region illustrates the participants falling off the perch in this update from a top 10% public LB position in the previous update <br> * This is one of the key areas to consider in forecasting competitions! <br> * This region is quite dense, illustrating that a lot of the top 10% scorers in the previous LB update have fallen off the perch here!</td>\n</tr>\n<tr>\n<td>R3</td>\n<td>* This region illustrates the participants rising in the private leaderboard by beating the churn! Congratulations for the performance and hope for the best! <br> * This region is also quite dense, with a lot of participants rising through, akin to the fallers in region R2</td>\n</tr>\n<tr>\n<td>R4</td>\n<td>* This region is similar to R2, but occurs outside of the medal zone and illustrates a fall off the perch in the private leaderboard</td>\n</tr>\n</tbody>\n</table>\n<h2>Analysis of the top 10% teams for stability</h2>\n<table>\n<thead>\n<tr>\n<th>Medal zone label</th>\n<th>Last rank</th>\n<th>Comments</th>\n<th>Stability Ratio</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Gold medal</td>\n<td>17</td>\n<td>* Stable teams = 13  <br> * Entering from below =  4</td>\n<td>13/17 = 76/47%</td>\n</tr>\n<tr>\n<td>Silver medal</td>\n<td>187</td>\n<td>* Stable teams = 119  <br> * Entering from below = 47 <br> * Entering from above =  4</td>\n<td>119/  170 = 70%</td>\n</tr>\n<tr>\n<td>Bronze medal</td>\n<td>375</td>\n<td>* Stable teams =  1 <br> * Entering from below = 187  <br> * Entering from above = 0</td>\n<td>1 / 188 = 0.53%</td>\n</tr>\n</tbody>\n</table>\n<p><br>Wow! The bronze medal zone got completely churned with almost the entire region entering from an erstwhile lower rank! Amazing! <br>\nStability ratio across the gold region is quite strong, illustrating that the top scoring solution are exhibiting immunity to the leaderboard updates, at least as on date!</p>\n<h2>Other adjutant observations</h2>\n<table>\n<thead>\n<tr>\n<th>Observation</th>\n<th>Previous update</th>\n<th>Current update</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>* Highest upward movement across the leaderboards - overall</td>\n<td>2013</td>\n<td>40</td>\n</tr>\n<tr>\n<td>* Highest downward movement across the leaderboards - overall</td>\n<td>819</td>\n<td>3350</td>\n</tr>\n<tr>\n<td>* Highest upward movement across the leaderboards in R1 region</td>\n<td>2013</td>\n<td>40</td>\n</tr>\n<tr>\n<td>* Highest downward movement across the leaderboards and finishing in R1</td>\n<td>204</td>\n<td>350</td>\n</tr>\n</tbody>\n</table>\n<p><br>Finally, wishing you the best for the future updates, happy learning and best regards! For those in R2, please note that we have a number of updates remaining and you could still make it across the 5 months remaining!</p>\n<p>All the best!</p>\n<p>Regards,<br>\nRavi Ramakrishnan</p>",
  "messages": [
    {
      "id": "3145285",
      "postDate": "03/09/2025 16:43:16",
      "content": "<p>Hello all,</p>\n<p>We have a leaderboard update as on date and the figure below delves into the regions of interest, between this update and the leaderboard update in mid-February.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8273630%2F5a8800a9fe59ffe2da0dc1042bacda21%2FMar9.png?generation=1741537823868685&amp;alt=media\" alt=\"\"></p>\n<p><br>We see a lot of churn here compared to a relatively stable leaderboard over the <a href=\"https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/562435\" target=\"_blank\">last time</a></p>\n<table>\n<thead>\n<tr>\n<th>Region label</th>\n<th>Description</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>R1</td>\n<td>* This region illustrates the top 10% participants in the competition that remained stable across the 2 updates. This region is quick thick, indicating good stability! Congratulations to these participants for the stable performance</td>\n</tr>\n<tr>\n<td>R2</td>\n<td>* This region illustrates the participants falling off the perch in this update from a top 10% public LB position in the previous update <br> * This is one of the key areas to consider in forecasting competitions! <br> * This region is quite dense, illustrating that a lot of the top 10% scorers in the previous LB update have fallen off the perch here!</td>\n</tr>\n<tr>\n<td>R3</td>\n<td>* This region illustrates the participants rising in the private leaderboard by beating the churn! Congratulations for the performance and hope for the best! <br> * This region is also quite dense, with a lot of participants rising through, akin to the fallers in region R2</td>\n</tr>\n<tr>\n<td>R4</td>\n<td>* This region is similar to R2, but occurs outside of the medal zone and illustrates a fall off the perch in the private leaderboard</td>\n</tr>\n</tbody>\n</table>\n<h2>Analysis of the top 10% teams for stability</h2>\n<table>\n<thead>\n<tr>\n<th>Medal zone label</th>\n<th>Last rank</th>\n<th>Comments</th>\n<th>Stability Ratio</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Gold medal</td>\n<td>17</td>\n<td>* Stable teams = 13  <br> * Entering from below =  4</td>\n<td>13/17 = 76/47%</td>\n</tr>\n<tr>\n<td>Silver medal</td>\n<td>187</td>\n<td>* Stable teams = 119  <br> * Entering from below = 47 <br> * Entering from above =  4</td>\n<td>119/  170 = 70%</td>\n</tr>\n<tr>\n<td>Bronze medal</td>\n<td>375</td>\n<td>* Stable teams =  1 <br> * Entering from below = 187  <br> * Entering from above = 0</td>\n<td>1 / 188 = 0.53%</td>\n</tr>\n</tbody>\n</table>\n<p><br>Wow! The bronze medal zone got completely churned with almost the entire region entering from an erstwhile lower rank! Amazing! <br>\nStability ratio across the gold region is quite strong, illustrating that the top scoring solution are exhibiting immunity to the leaderboard updates, at least as on date!</p>\n<h2>Other adjutant observations</h2>\n<table>\n<thead>\n<tr>\n<th>Observation</th>\n<th>Previous update</th>\n<th>Current update</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>* Highest upward movement across the leaderboards - overall</td>\n<td>2013</td>\n<td>40</td>\n</tr>\n<tr>\n<td>* Highest downward movement across the leaderboards - overall</td>\n<td>819</td>\n<td>3350</td>\n</tr>\n<tr>\n<td>* Highest upward movement across the leaderboards in R1 region</td>\n<td>2013</td>\n<td>40</td>\n</tr>\n<tr>\n<td>* Highest downward movement across the leaderboards and finishing in R1</td>\n<td>204</td>\n<td>350</td>\n</tr>\n</tbody>\n</table>\n<p><br>Finally, wishing you the best for the future updates, happy learning and best regards! For those in R2, please note that we have a number of updates remaining and you could still make it across the 5 months remaining!</p>\n<p>All the best!</p>\n<p>Regards,<br>\nRavi Ramakrishnan</p>",
      "rawMarkdown": "Hello all,\n\nWe have a leaderboard update as on date and the figure below delves into the regions of interest, between this update and the leaderboard update in mid-February.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8273630%2F5a8800a9fe59ffe2da0dc1042bacda21%2FMar9.png?generation=1741537823868685&alt=media)\n\n<br>We see a lot of churn here compared to a relatively stable leaderboard over the [last time](https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/562435)\n\n| Region label | Description |\n| --- | --- | \n| R1 |  * This region illustrates the top 10% participants in the competition that remained stable across the 2 updates. This region is quick thick, indicating good stability! Congratulations to these participants for the stable performance |\n| R2 |  * This region illustrates the participants falling off the perch in this update from a top 10% public LB position in the previous update <br> * This is one of the key areas to consider in forecasting competitions! <br> * This region is quite dense, illustrating that a lot of the top 10% scorers in the previous LB update have fallen off the perch here! |\n| R3 |  * This region illustrates the participants rising in the private leaderboard by beating the churn! Congratulations for the performance and hope for the best! <br> * This region is also quite dense, with a lot of participants rising through, akin to the fallers in region R2|\n| R4 |  * This region is similar to R2, but occurs outside of the medal zone and illustrates a fall off the perch in the private leaderboard |\n\n## Analysis of the top 10% teams for stability\n\n| Medal zone label | Last rank  | Comments | Stability Ratio | \n| --- | --- | ----- | ----------------- | \n| Gold medal   | 17 | * Stable teams = 13  <br> * Entering from below =  4| 13/17 = 76/47% | \n| Silver medal  | 187| * Stable teams = 119  <br> * Entering from below = 47 <br> * Entering from above =  4| 119/  170 = 70% | \n| Bronze medal  | 375| * Stable teams =  1 <br> * Entering from below = 187  <br> * Entering from above = 0  | 1 / 188 = 0.53% | \n\n<br>Wow! The bronze medal zone got completely churned with almost the entire region entering from an erstwhile lower rank! Amazing! \nStability ratio across the gold region is quite strong, illustrating that the top scoring solution are exhibiting immunity to the leaderboard updates, at least as on date!\n\n## Other adjutant observations\n\n| Observation | Previous update  | Current update | \n| --- | --- | ---- | \n| * Highest upward movement across the leaderboards - overall | 2013 | 40 | \n| * Highest downward movement across the leaderboards - overall |  819|  3350| \n| * Highest upward movement across the leaderboards in R1 region |2013 | 40 | \n| * Highest downward movement across the leaderboards and finishing in R1 | 204 | 350| \n\n<br>Finally, wishing you the best for the future updates, happy learning and best regards! For those in R2, please note that we have a number of updates remaining and you could still make it across the 5 months remaining!\n\nAll the best!\n\nRegards,\nRavi Ramakrishnan",
      "votes": null
    },
    {
      "id": "3145912",
      "postDate": "03/10/2025 10:59:22",
      "content": "<p>The change in bronze medal zone is really shocking!!</p>",
      "rawMarkdown": "The change in bronze medal zone is really shocking!!",
      "votes": null
    },
    {
      "id": "3145957",
      "postDate": "03/10/2025 11:57:39",
      "content": "<p>Maybe bronze medal all forked public notebooks or something? or something happened in this data like adding a new symbol ?</p>",
      "rawMarkdown": "Maybe bronze medal all forked public notebooks or something? or something happened in this data like adding a new symbol ?",
      "votes": null
    },
    {
      "id": "3146250",
      "postDate": "03/10/2025 17:30:54",
      "content": "<p>After this update 178 to 561 all have the same score of 0.8004, possible it's a group people that used the same public notebook and got shuffled</p>",
      "rawMarkdown": "After this update 178 to 561 all have the same score of 0.8004, possible it's a group people that used the same public notebook and got shuffled",
      "votes": null
    },
    {
      "id": "3146315",
      "postDate": "03/10/2025 18:21:31",
      "content": "<p>Perhaps not retraining the model is also a reason - a trend shift perhaps caused a drop <a href=\"https://www.kaggle.com/redfoongus\" target=\"_blank\">@redfoongus</a> <a href=\"https://www.kaggle.com/julianmukaj\" target=\"_blank\">@julianmukaj</a> </p>",
      "rawMarkdown": "Perhaps not retraining the model is also a reason - a trend shift perhaps caused a drop @redfoongus @julianmukaj",
      "votes": null
    },
    {
      "id": "3146789",
      "postDate": "03/11/2025 08:49:29",
      "content": "<blockquote>\n  <p>Highest upward movement across the leaderboards - overall    2013    40</p>\n</blockquote>\n<p>Haha, it's me. I have no idea why I got the 2013th rank in the first update or why I am at 40th now. My public was like 199th.</p>\n<p>Looking at my 2 selected submissions. One with a 30-day lookback online learning, and another with a 60-day lookback. The 30-day one performs a lot better… So I am guessing it is related to the recent market situation changes… In other words, if the market goes back to normal soon, my subbmision might drop a lot again.</p>",
      "rawMarkdown": "> Highest upward movement across the leaderboards - overall\t2013\t40\n\nHaha, it's me. I have no idea why I got the 2013th rank in the first update or why I am at 40th now. My public was like 199th.\n\nLooking at my 2 selected submissions. One with a 30-day lookback online learning, and another with a 60-day lookback. The 30-day one performs a lot better... So I am guessing it is related to the recent market situation changes... In other words, if the market goes back to normal soon, my subbmision might drop a lot again.",
      "votes": null
    },
    {
      "id": "3146929",
      "postDate": "03/11/2025 12:21:49",
      "content": "<p>Trend reversals and slight changes in existing symbol information can create this level of shift in time series forecast competitions! Your case is common in such assignments <a href=\"https://www.kaggle.com/kingychiu\" target=\"_blank\">@kingychiu</a> </p>",
      "rawMarkdown": "Trend reversals and slight changes in existing symbol information can create this level of shift in time series forecast competitions! Your case is common in such assignments @kingychiu",
      "votes": null
    },
    {
      "id": "3148023",
      "postDate": "03/12/2025 16:49:47",
      "content": "<p>What cause the total number of Silver + Bronze medal change from 562 to 358? <br>\nIf it is due to many disqualification, then are you able to trace how many was disqualified?</p>",
      "rawMarkdown": "What cause the total number of Silver + Bronze medal change from 562 to 358? \nIf it is due to many disqualification, then are you able to trace how many was disqualified?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3145912,
      "author_name": "yich723",
      "author_url": "",
      "post_date": "03/10/2025 10:59:22",
      "content": "<p>The change in bronze medal zone is really shocking!!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3145957,
      "author_name": "julianmukaj",
      "author_url": "",
      "post_date": "03/10/2025 11:57:39",
      "content": "<p>Maybe bronze medal all forked public notebooks or something? or something happened in this data like adding a new symbol ?</p>",
      "votes": null,
      "replies": [
        {
          "id": 3146250,
          "author_name": "redfoongus",
          "author_url": "",
          "post_date": "03/10/2025 17:30:54",
          "content": "<p>After this update 178 to 561 all have the same score of 0.8004, possible it's a group people that used the same public notebook and got shuffled</p>",
          "votes": null,
          "replies": [
            {
              "id": 3146315,
              "author_name": "ravi20076",
              "author_url": "",
              "post_date": "03/10/2025 18:21:31",
              "content": "<p>Perhaps not retraining the model is also a reason - a trend shift perhaps caused a drop <a href=\"https://www.kaggle.com/redfoongus\" target=\"_blank\">@redfoongus</a> <a href=\"https://www.kaggle.com/julianmukaj\" target=\"_blank\">@julianmukaj</a> </p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3146789,
      "author_name": "kingychiu",
      "author_url": "",
      "post_date": "03/11/2025 08:49:29",
      "content": "<blockquote>\n  <p>Highest upward movement across the leaderboards - overall    2013    40</p>\n</blockquote>\n<p>Haha, it's me. I have no idea why I got the 2013th rank in the first update or why I am at 40th now. My public was like 199th.</p>\n<p>Looking at my 2 selected submissions. One with a 30-day lookback online learning, and another with a 60-day lookback. The 30-day one performs a lot better… So I am guessing it is related to the recent market situation changes… In other words, if the market goes back to normal soon, my subbmision might drop a lot again.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3146929,
          "author_name": "ravi20076",
          "author_url": "",
          "post_date": "03/11/2025 12:21:49",
          "content": "<p>Trend reversals and slight changes in existing symbol information can create this level of shift in time series forecast competitions! Your case is common in such assignments <a href=\"https://www.kaggle.com/kingychiu\" target=\"_blank\">@kingychiu</a> </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3148023,
      "author_name": "chewkokwahibrainai",
      "author_url": "",
      "post_date": "03/12/2025 16:49:47",
      "content": "<p>What cause the total number of Silver + Bronze medal change from 562 to 358? <br>\nIf it is due to many disqualification, then are you able to trace how many was disqualified?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3145285": "Hello all,\n\nWe have a leaderboard update as on date and the figure below delves into the regions of interest, between this update and the leaderboard update in mid-February.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8273630%2F5a8800a9fe59ffe2da0dc1042bacda21%2FMar9.png?generation=1741537823868685&alt=media)\n\n<br>We see a lot of churn here compared to a relatively stable leaderboard over the [last time](https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/562435)\n\n| Region label | Description |\n| --- | --- | \n| R1 |  * This region illustrates the top 10% participants in the competition that remained stable across the 2 updates. This region is quick thick, indicating good stability! Congratulations to these participants for the stable performance |\n| R2 |  * This region illustrates the participants falling off the perch in this update from a top 10% public LB position in the previous update <br> * This is one of the key areas to consider in forecasting competitions! <br> * This region is quite dense, illustrating that a lot of the top 10% scorers in the previous LB update have fallen off the perch here! |\n| R3 |  * This region illustrates the participants rising in the private leaderboard by beating the churn! Congratulations for the performance and hope for the best! <br> * This region is also quite dense, with a lot of participants rising through, akin to the fallers in region R2|\n| R4 |  * This region is similar to R2, but occurs outside of the medal zone and illustrates a fall off the perch in the private leaderboard |\n\n## Analysis of the top 10% teams for stability\n\n| Medal zone label | Last rank  | Comments | Stability Ratio | \n| --- | --- | ----- | ----------------- | \n| Gold medal   | 17 | * Stable teams = 13  <br> * Entering from below =  4| 13/17 = 76/47% | \n| Silver medal  | 187| * Stable teams = 119  <br> * Entering from below = 47 <br> * Entering from above =  4| 119/  170 = 70% | \n| Bronze medal  | 375| * Stable teams =  1 <br> * Entering from below = 187  <br> * Entering from above = 0  | 1 / 188 = 0.53% | \n\n<br>Wow! The bronze medal zone got completely churned with almost the entire region entering from an erstwhile lower rank! Amazing! \nStability ratio across the gold region is quite strong, illustrating that the top scoring solution are exhibiting immunity to the leaderboard updates, at least as on date!\n\n## Other adjutant observations\n\n| Observation | Previous update  | Current update | \n| --- | --- | ---- | \n| * Highest upward movement across the leaderboards - overall | 2013 | 40 | \n| * Highest downward movement across the leaderboards - overall |  819|  3350| \n| * Highest upward movement across the leaderboards in R1 region |2013 | 40 | \n| * Highest downward movement across the leaderboards and finishing in R1 | 204 | 350| \n\n<br>Finally, wishing you the best for the future updates, happy learning and best regards! For those in R2, please note that we have a number of updates remaining and you could still make it across the 5 months remaining!\n\nAll the best!\n\nRegards,\nRavi Ramakrishnan",
    "3145912": "The change in bronze medal zone is really shocking!!",
    "3145957": "Maybe bronze medal all forked public notebooks or something? or something happened in this data like adding a new symbol ?",
    "3146250": "After this update 178 to 561 all have the same score of 0.8004, possible it's a group people that used the same public notebook and got shuffled",
    "3146315": "Perhaps not retraining the model is also a reason - a trend shift perhaps caused a drop @redfoongus @julianmukaj",
    "3146789": "> Highest upward movement across the leaderboards - overall\t2013\t40\n\nHaha, it's me. I have no idea why I got the 2013th rank in the first update or why I am at 40th now. My public was like 199th.\n\nLooking at my 2 selected submissions. One with a 30-day lookback online learning, and another with a 60-day lookback. The 30-day one performs a lot better... So I am guessing it is related to the recent market situation changes... In other words, if the market goes back to normal soon, my subbmision might drop a lot again.",
    "3146929": "Trend reversals and slight changes in existing symbol information can create this level of shift in time series forecast competitions! Your case is common in such assignments @kingychiu",
    "3148023": "What cause the total number of Silver + Bronze medal change from 562 to 358? \nIf it is due to many disqualification, then are you able to trace how many was disqualified?"
  },
  "source": "meta"
}