{
  "id": 562410,
  "title": "First private leaderboard looks suprisingly stable",
  "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/562410",
  "author_name": "",
  "post_date": "2025-02-11T14:12:57.118380900Z",
  "votes": 2,
  "comment_count": 8,
  "views": 0,
  "content": "<p>Thanks to the new setup (new api and problem setup), I guess it's safe to say after this competition, the era of huge shakeup in financial data science competitions is over. Kudos to kaggle and Jane Street teams (of course also previous Optiver one)!<br>\nGood luck for the following updates, everyone. 😀</p>",
  "messages": [
    {
      "id": "3121353",
      "postDate": "02/11/2025 14:12:57",
      "content": "<p>Thanks to the new setup (new api and problem setup), I guess it's safe to say after this competition, the era of huge shakeup in financial data science competitions is over. Kudos to kaggle and Jane Street teams (of course also previous Optiver one)!<br>\nGood luck for the following updates, everyone. 😀</p>",
      "rawMarkdown": "Thanks to the new setup (new api and problem setup), I guess it's safe to say after this competition, the era of huge shakeup in financial data science competitions is over. Kudos to kaggle and Jane Street teams (of course also previous Optiver one)!\nGood luck for the following updates, everyone. 😀",
      "votes": null
    },
    {
      "id": "3121357",
      "postDate": "02/11/2025 14:18:01",
      "content": "<p>I think its still early to say that, only 60 days given and 150 more days to go.. my two selected submissions have converged a bit in score and I see some teams have dropped 5+ places.. Will get shakier over next month or two I think 😀</p>",
      "rawMarkdown": "I think its still early to say that, only 60 days given and 150 more days to go.. my two selected submissions have converged a bit in score and I see some teams have dropped 5+ places.. Will get shakier over next month or two I think 😀",
      "votes": null
    },
    {
      "id": "3121363",
      "postDate": "02/11/2025 14:21:55",
      "content": "<p>Technically, there are only 20 effective dates used in this first private update and 100 more days to go, but you get the point. I mean shakeuping around 5 places should be surprising, right? 😀</p>",
      "rawMarkdown": "Technically, there are only 20 effective dates used in this first private update and 100 more days to go, but you get the point. I mean shakeuping around 5 places should be surprising, right? 😀",
      "votes": null
    },
    {
      "id": "3121378",
      "postDate": "02/11/2025 14:41:11",
      "content": "<p>The top 10 definitely looks relatively stable. Outside of the top 10 I think it's a bit shaky though, i'm seeing some teams that have climbed 40+ places and others that have dropped much lower than where they were before.</p>",
      "rawMarkdown": "The top 10 definitely looks relatively stable. Outside of the top 10 I think it's a bit shaky though, i'm seeing some teams that have climbed 40+ places and others that have dropped much lower than where they were before.",
      "votes": null
    },
    {
      "id": "3121386",
      "postDate": "02/11/2025 14:47:06",
      "content": "<p>This is also expected. Since from the solutions sharing of teams placed arond 20~50, some of them ensembed their solutions from models with online learning and models without online learning. In this case, a relative \"large\" drop from public leaderboard to prive is not really surprising. <br>\nThe shakeup I refer to is something you could not predict and pure luck, which is really scary. 🫠</p>",
      "rawMarkdown": "This is also expected. Since from the solutions sharing of teams placed arond 20~50, some of them ensembed their solutions from models with online learning and models without online learning. In this case, a relative \"large\" drop from public leaderboard to prive is not really surprising. \nThe shakeup I refer to is something you could not predict and pure luck, which is really scary. 🫠",
      "votes": null
    },
    {
      "id": "3121437",
      "postDate": "02/11/2025 15:48:59",
      "content": "<p>Yeah.. I don't know how people trusted models that didn't use online learning, they're almost certainly going to degrade in performance over time. </p>\n<p>I too am glad that the LB is behaving relatively well, although you never truly know when it comes to financial data. I remember finding a couple \"black swan\" events in the training data where my model would out of nowhere score extremely poor on specific date_ids (like R^2=-0.5).</p>",
      "rawMarkdown": "Yeah.. I don't know how people trusted models that didn't use online learning, they're almost certainly going to degrade in performance over time. \n\nI too am glad that the LB is behaving relatively well, although you never truly know when it comes to financial data. I remember finding a couple \"black swan\" events in the training data where my model would out of nowhere score extremely poor on specific date_ids (like R^2=-0.5).",
      "votes": null
    },
    {
      "id": "3121444",
      "postDate": "02/11/2025 15:54:23",
      "content": "<blockquote>\n  <p>Yeah.. I don't know how people trusted models that didn't use online learning, they're almost certainly going to degrade in performance over time.</p>\n</blockquote>\n<p>My offline model performed better than my online model in this update. 20 days may be small to draw conclusions. Also non-optimal online learning may perform worse than offline models.</p>",
      "rawMarkdown": "> Yeah.. I don't know how people trusted models that didn't use online learning, they're almost certainly going to degrade in performance over time.\n\nMy offline model performed better than my online model in this update. 20 days may be small to draw conclusions. Also non-optimal online learning may perform worse than offline models.",
      "votes": null
    },
    {
      "id": "3121525",
      "postDate": "02/11/2025 17:19:34",
      "content": "<p>In my experience with testing offline models, they started to substantially degrade in performance after around 100-200 new date_ids. It's possible that I was just training it poorly though, I didn't really do that much testing with offline models.</p>\n<blockquote>\n  <p>Also non-optimal online learning may perform worse than offline models.</p>\n</blockquote>\n<p>Yeah, that could be true. </p>",
      "rawMarkdown": "In my experience with testing offline models, they started to substantially degrade in performance after around 100-200 new date_ids. It's possible that I was just training it poorly though, I didn't really do that much testing with offline models.\n\n> Also non-optimal online learning may perform worse than offline models.\n\nYeah, that could be true.",
      "votes": null
    },
    {
      "id": "3128590",
      "postDate": "02/19/2025 17:22:32",
      "content": "<p>My final public score and first private score are exactly the same for my best submission, surprisingly stable indeed…</p>",
      "rawMarkdown": "My final public score and first private score are exactly the same for my best submission, surprisingly stable indeed...",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3121357,
      "author_name": "julianmukaj",
      "author_url": "",
      "post_date": "02/11/2025 14:18:01",
      "content": "<p>I think its still early to say that, only 60 days given and 150 more days to go.. my two selected submissions have converged a bit in score and I see some teams have dropped 5+ places.. Will get shakier over next month or two I think 😀</p>",
      "votes": null,
      "replies": [
        {
          "id": 3121363,
          "author_name": "lihaorocky",
          "author_url": "",
          "post_date": "02/11/2025 14:21:55",
          "content": "<p>Technically, there are only 20 effective dates used in this first private update and 100 more days to go, but you get the point. I mean shakeuping around 5 places should be surprising, right? 😀</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3121378,
      "author_name": "johnpayne0",
      "author_url": "",
      "post_date": "02/11/2025 14:41:11",
      "content": "<p>The top 10 definitely looks relatively stable. Outside of the top 10 I think it's a bit shaky though, i'm seeing some teams that have climbed 40+ places and others that have dropped much lower than where they were before.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3121386,
          "author_name": "lihaorocky",
          "author_url": "",
          "post_date": "02/11/2025 14:47:06",
          "content": "<p>This is also expected. Since from the solutions sharing of teams placed arond 20~50, some of them ensembed their solutions from models with online learning and models without online learning. In this case, a relative \"large\" drop from public leaderboard to prive is not really surprising. <br>\nThe shakeup I refer to is something you could not predict and pure luck, which is really scary. 🫠</p>",
          "votes": null,
          "replies": [
            {
              "id": 3121437,
              "author_name": "johnpayne0",
              "author_url": "",
              "post_date": "02/11/2025 15:48:59",
              "content": "<p>Yeah.. I don't know how people trusted models that didn't use online learning, they're almost certainly going to degrade in performance over time. </p>\n<p>I too am glad that the LB is behaving relatively well, although you never truly know when it comes to financial data. I remember finding a couple \"black swan\" events in the training data where my model would out of nowhere score extremely poor on specific date_ids (like R^2=-0.5).</p>",
              "votes": null,
              "replies": [
                {
                  "id": 3121444,
                  "author_name": "aerdem4",
                  "author_url": "",
                  "post_date": "02/11/2025 15:54:23",
                  "content": "<blockquote>\n  <p>Yeah.. I don't know how people trusted models that didn't use online learning, they're almost certainly going to degrade in performance over time.</p>\n</blockquote>\n<p>My offline model performed better than my online model in this update. 20 days may be small to draw conclusions. Also non-optimal online learning may perform worse than offline models.</p>",
                  "votes": null,
                  "replies": [
                    {
                      "id": 3121525,
                      "author_name": "johnpayne0",
                      "author_url": "",
                      "post_date": "02/11/2025 17:19:34",
                      "content": "<p>In my experience with testing offline models, they started to substantially degrade in performance after around 100-200 new date_ids. It's possible that I was just training it poorly though, I didn't really do that much testing with offline models.</p>\n<blockquote>\n  <p>Also non-optimal online learning may perform worse than offline models.</p>\n</blockquote>\n<p>Yeah, that could be true. </p>",
                      "votes": null,
                      "replies": []
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "id": 3128590,
      "author_name": "sroger",
      "author_url": "",
      "post_date": "02/19/2025 17:22:32",
      "content": "<p>My final public score and first private score are exactly the same for my best submission, surprisingly stable indeed…</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3121353": "Thanks to the new setup (new api and problem setup), I guess it's safe to say after this competition, the era of huge shakeup in financial data science competitions is over. Kudos to kaggle and Jane Street teams (of course also previous Optiver one)!\nGood luck for the following updates, everyone. 😀",
    "3121357": "I think its still early to say that, only 60 days given and 150 more days to go.. my two selected submissions have converged a bit in score and I see some teams have dropped 5+ places.. Will get shakier over next month or two I think 😀",
    "3121363": "Technically, there are only 20 effective dates used in this first private update and 100 more days to go, but you get the point. I mean shakeuping around 5 places should be surprising, right? 😀",
    "3121378": "The top 10 definitely looks relatively stable. Outside of the top 10 I think it's a bit shaky though, i'm seeing some teams that have climbed 40+ places and others that have dropped much lower than where they were before.",
    "3121386": "This is also expected. Since from the solutions sharing of teams placed arond 20~50, some of them ensembed their solutions from models with online learning and models without online learning. In this case, a relative \"large\" drop from public leaderboard to prive is not really surprising. \nThe shakeup I refer to is something you could not predict and pure luck, which is really scary. 🫠",
    "3121437": "Yeah.. I don't know how people trusted models that didn't use online learning, they're almost certainly going to degrade in performance over time. \n\nI too am glad that the LB is behaving relatively well, although you never truly know when it comes to financial data. I remember finding a couple \"black swan\" events in the training data where my model would out of nowhere score extremely poor on specific date_ids (like R^2=-0.5).",
    "3121444": "> Yeah.. I don't know how people trusted models that didn't use online learning, they're almost certainly going to degrade in performance over time.\n\nMy offline model performed better than my online model in this update. 20 days may be small to draw conclusions. Also non-optimal online learning may perform worse than offline models.",
    "3121525": "In my experience with testing offline models, they started to substantially degrade in performance after around 100-200 new date_ids. It's possible that I was just training it poorly though, I didn't really do that much testing with offline models.\n\n> Also non-optimal online learning may perform worse than offline models.\n\nYeah, that could be true.",
    "3128590": "My final public score and first private score are exactly the same for my best submission, surprisingly stable indeed..."
  },
  "source": "meta"
}