{
  "id": 554336,
  "title": "Much slower of online inference due to platform issue?",
  "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/554336",
  "author_name": "",
  "post_date": "2025-01-01T00:48:01.639301100Z",
  "votes": 8,
  "comment_count": 10,
  "views": 0,
  "content": "<p>My submission not change much which should run less then 80mins , but latest runs using much longger. Do you also face this problem?  <br>\nBTW, I'm using this notebook to monitor submission running time usage. <a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-submission-time\" target=\"_blank\">https://www.kaggle.com/code/yasufuminakama/fb3-submission-time</a>  </p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F42245%2F77abd45765e3a0d88aebc0b0c1c54212%2Fslow.png?generation=1735692434835116&amp;alt=media\" alt=\"\"></p>",
  "messages": [
    {
      "id": "3085271",
      "postDate": "01/01/2025 00:48:01",
      "content": "<p>My submission not change much which should run less then 80mins , but latest runs using much longger. Do you also face this problem?  <br>\nBTW, I'm using this notebook to monitor submission running time usage. <a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-submission-time\" target=\"_blank\">https://www.kaggle.com/code/yasufuminakama/fb3-submission-time</a>  </p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F42245%2F77abd45765e3a0d88aebc0b0c1c54212%2Fslow.png?generation=1735692434835116&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "My submission not change much which should run less then 80mins , but latest runs using much longger. Do you also face this problem?  \nBTW, I'm using this notebook to monitor submission running time usage. https://www.kaggle.com/code/yasufuminakama/fb3-submission-time  \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F42245%2F77abd45765e3a0d88aebc0b0c1c54212%2Fslow.png?generation=1735692434835116&alt=media)",
      "votes": null
    },
    {
      "id": "3085342",
      "postDate": "01/01/2025 04:10:44",
      "content": "<p>It happened to me a few times when they were doing a lot of rescores earlier. Maybe it’s because kaggle servers are loaded</p>",
      "rawMarkdown": "It happened to me a few times when they were doing a lot of rescores earlier. Maybe it’s because kaggle servers are loaded",
      "votes": null
    },
    {
      "id": "3085358",
      "postDate": "01/01/2025 04:51:52",
      "content": "<p>Yeah I was wonering it is due to the new 0.0079LB notebook…</p>",
      "rawMarkdown": "Yeah I was wonering it is due to the new 0.0079LB notebook...",
      "votes": null
    },
    {
      "id": "3086071",
      "postDate": "01/02/2025 00:39:58",
      "content": "<p>It does occur on occasion. I encountered the same issue a few days back. <br>\n<a href=\"https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/553291\" target=\"_blank\">https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/553291</a><br>\nTypically, it gets resolved within several days, yet it remains rather bothersome nonetheless. </p>",
      "rawMarkdown": "It does occur on occasion. I encountered the same issue a few days back. \nhttps://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/553291\nTypically, it gets resolved within several days, yet it remains rather bothersome nonetheless.",
      "votes": null
    },
    {
      "id": "3089585",
      "postDate": "01/06/2025 08:52:16",
      "content": "<p>I am experiencing something similar. I wonder if they fill the future dates with dummy data recently.</p>",
      "rawMarkdown": "I am experiencing something similar. I wonder if they fill the future dates with dummy data recently.",
      "votes": null
    },
    {
      "id": "3089588",
      "postDate": "01/06/2025 08:59:28",
      "content": "<p>Same here. Submission today took 2-3x longer than usual</p>",
      "rawMarkdown": "Same here. Submission today took 2-3x longer than usual",
      "votes": null
    },
    {
      "id": "3089592",
      "postDate": "01/06/2025 09:08:03",
      "content": "<p>I think you are right,  maybe we can use \"is_scored\" column to verify this.</p>",
      "rawMarkdown": "I think you are right,  maybe we can use \"is_scored\" column to verify this.",
      "votes": null
    },
    {
      "id": "3089630",
      "postDate": "01/06/2025 10:38:15",
      "content": "<p>I can confirm the future dates with dummy data are not filled yet. I also faced the similar problem. I tried submitted 3 notebooks the same time 10 hours ago, the one which was supposed to be finished within 7 hours was indeed finished in 7 hours, while the other 2 notebooks (supposed to be finished in 4 hours), one was finished after 9 hours and another still running. So I guess the problem is just the sever is fully loaded towards the end of the competition. </p>",
      "rawMarkdown": "I can confirm the future dates with dummy data are not filled yet. I also faced the similar problem. I tried submitted 3 notebooks the same time 10 hours ago, the one which was supposed to be finished within 7 hours was indeed finished in 7 hours, while the other 2 notebooks (supposed to be finished in 4 hours), one was finished after 9 hours and another still running. So I guess the problem is just the sever is fully loaded towards the end of the competition.",
      "votes": null
    },
    {
      "id": "3089649",
      "postDate": "01/06/2025 11:20:06",
      "content": "<p>No they didn't fill the dataset yet. The server is heavily loaded torwards the end of the competition, and submissions are queued to wait for scoring. My experience is that the queue time can be 2-4 hours or even longer in some rare cases.</p>",
      "rawMarkdown": "No they didn't fill the dataset yet. The server is heavily loaded torwards the end of the competition, and submissions are queued to wait for scoring. My experience is that the queue time can be 2-4 hours or even longer in some rare cases.",
      "votes": null
    },
    {
      "id": "3089813",
      "postDate": "01/06/2025 15:01:51",
      "content": "<p>Same issue here, I have some submissions running for &gt;9 hours, but somehow not timeout yet.</p>\n<p>Usual time should be ~5 hours.</p>",
      "rawMarkdown": "Same issue here, I have some submissions running for >9 hours, but somehow not timeout yet.\n\nUsual time should be ~5 hours.",
      "votes": null
    },
    {
      "id": "3090227",
      "postDate": "01/07/2025 03:30:12",
      "content": "<p>I submitted yesterday, it was running 0.03s per test dataframe with the synthetic dataset. But the submission has been running for 6hours +</p>",
      "rawMarkdown": "I submitted yesterday, it was running 0.03s per test dataframe with the synthetic dataset. But the submission has been running for 6hours +",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3085342,
      "author_name": "snehalverma10",
      "author_url": "",
      "post_date": "01/01/2025 04:10:44",
      "content": "<p>It happened to me a few times when they were doing a lot of rescores earlier. Maybe it’s because kaggle servers are loaded</p>",
      "votes": null,
      "replies": [
        {
          "id": 3085358,
          "author_name": "goldenlock",
          "author_url": "",
          "post_date": "01/01/2025 04:51:52",
          "content": "<p>Yeah I was wonering it is due to the new 0.0079LB notebook…</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3086071,
      "author_name": "carrotwait",
      "author_url": "",
      "post_date": "01/02/2025 00:39:58",
      "content": "<p>It does occur on occasion. I encountered the same issue a few days back. <br>\n<a href=\"https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/553291\" target=\"_blank\">https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/553291</a><br>\nTypically, it gets resolved within several days, yet it remains rather bothersome nonetheless. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3089585,
      "author_name": "aerdem4",
      "author_url": "",
      "post_date": "01/06/2025 08:52:16",
      "content": "<p>I am experiencing something similar. I wonder if they fill the future dates with dummy data recently.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3089588,
          "author_name": "snehalverma10",
          "author_url": "",
          "post_date": "01/06/2025 08:59:28",
          "content": "<p>Same here. Submission today took 2-3x longer than usual</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 3089592,
          "author_name": "carrotwait",
          "author_url": "",
          "post_date": "01/06/2025 09:08:03",
          "content": "<p>I think you are right,  maybe we can use \"is_scored\" column to verify this.</p>",
          "votes": null,
          "replies": [
            {
              "id": 3089630,
              "author_name": "lihaorocky",
              "author_url": "",
              "post_date": "01/06/2025 10:38:15",
              "content": "<p>I can confirm the future dates with dummy data are not filled yet. I also faced the similar problem. I tried submitted 3 notebooks the same time 10 hours ago, the one which was supposed to be finished within 7 hours was indeed finished in 7 hours, while the other 2 notebooks (supposed to be finished in 4 hours), one was finished after 9 hours and another still running. So I guess the problem is just the sever is fully loaded towards the end of the competition. </p>",
              "votes": null,
              "replies": []
            }
          ]
        },
        {
          "id": 3089649,
          "author_name": "shiyili",
          "author_url": "",
          "post_date": "01/06/2025 11:20:06",
          "content": "<p>No they didn't fill the dataset yet. The server is heavily loaded torwards the end of the competition, and submissions are queued to wait for scoring. My experience is that the queue time can be 2-4 hours or even longer in some rare cases.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 3089813,
      "author_name": "williamwu88",
      "author_url": "",
      "post_date": "01/06/2025 15:01:51",
      "content": "<p>Same issue here, I have some submissions running for &gt;9 hours, but somehow not timeout yet.</p>\n<p>Usual time should be ~5 hours.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3090227,
      "author_name": "kingychiu",
      "author_url": "",
      "post_date": "01/07/2025 03:30:12",
      "content": "<p>I submitted yesterday, it was running 0.03s per test dataframe with the synthetic dataset. But the submission has been running for 6hours +</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3085271": "My submission not change much which should run less then 80mins , but latest runs using much longger. Do you also face this problem?  \nBTW, I'm using this notebook to monitor submission running time usage. https://www.kaggle.com/code/yasufuminakama/fb3-submission-time  \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F42245%2F77abd45765e3a0d88aebc0b0c1c54212%2Fslow.png?generation=1735692434835116&alt=media)",
    "3085342": "It happened to me a few times when they were doing a lot of rescores earlier. Maybe it’s because kaggle servers are loaded",
    "3085358": "Yeah I was wonering it is due to the new 0.0079LB notebook...",
    "3086071": "It does occur on occasion. I encountered the same issue a few days back. \nhttps://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/553291\nTypically, it gets resolved within several days, yet it remains rather bothersome nonetheless.",
    "3089585": "I am experiencing something similar. I wonder if they fill the future dates with dummy data recently.",
    "3089588": "Same here. Submission today took 2-3x longer than usual",
    "3089592": "I think you are right,  maybe we can use \"is_scored\" column to verify this.",
    "3089630": "I can confirm the future dates with dummy data are not filled yet. I also faced the similar problem. I tried submitted 3 notebooks the same time 10 hours ago, the one which was supposed to be finished within 7 hours was indeed finished in 7 hours, while the other 2 notebooks (supposed to be finished in 4 hours), one was finished after 9 hours and another still running. So I guess the problem is just the sever is fully loaded towards the end of the competition.",
    "3089649": "No they didn't fill the dataset yet. The server is heavily loaded torwards the end of the competition, and submissions are queued to wait for scoring. My experience is that the queue time can be 2-4 hours or even longer in some rare cases.",
    "3089813": "Same issue here, I have some submissions running for >9 hours, but somehow not timeout yet.\n\nUsual time should be ~5 hours.",
    "3090227": "I submitted yesterday, it was running 0.03s per test dataframe with the synthetic dataset. But the submission has been running for 6hours +"
  },
  "source": "meta"
}