As several people have pointed out, submission runtimes have become unusually long in the last few days. What's happening behind the scenes is that submissions have been waiting in a queue for compute to become available. This wait time is frustrating but is separate from the 8 hour runtime limit, which only begins once a submission exits the queue.
\nWe increased the size of the relevant compute pool yesterday and again this morning; it's now more than twice the capacity of a few days ago. The backlog is shrinking rapidly and appears to be on track to be cleared later today. We'll continue to monitor the situation in case the submission volume increases further.
\nUpdate: the backlog is clear for now. It is possible there will be a further surge in submissions as we approach the submission deadline.
", "messages": [ { "id": "3090822", "postDate": "01/07/2025 17:29:34", "content": "As several people have pointed out, submission runtimes have become unusually long in the last few days. What's happening behind the scenes is that submissions have been waiting in a queue for compute to become available. This wait time is frustrating but is separate from the 8 hour runtime limit, which only begins once a submission exits the queue.
\nWe increased the size of the relevant compute pool yesterday and again this morning; it's now more than twice the capacity of a few days ago. The backlog is shrinking rapidly and appears to be on track to be cleared later today. We'll continue to monitor the situation in case the submission volume increases further.
\nUpdate: the backlog is clear for now. It is possible there will be a further surge in submissions as we approach the submission deadline.
", "rawMarkdown": "As several people have pointed out, submission runtimes have become unusually long in the last few days. What's happening behind the scenes is that submissions have been waiting in a queue for compute to become available. This wait time is frustrating but is separate from the 8 hour runtime limit, which only begins once a submission exits the queue.\n\nWe increased the size of the relevant compute pool yesterday and again this morning; it's now more than twice the capacity of a few days ago. The backlog is shrinking rapidly and appears to be on track to be cleared later today. We'll continue to monitor the situation in case the submission volume increases further. \n\nUpdate: the backlog is clear for now. It is possible there will be a further surge in submissions as we approach the submission deadline.", "votes": null }, { "id": "3090841", "postDate": "01/07/2025 17:40:22", "content": "\n\nsubmissions have been waiting in a queue for compute to become available
\n
@sohier If my solution is purely CPU-based, using only the two stock CPU cores that a Kaggle notebook is typically allotted for, would it still be subject to this \"waiting in a queue for compute to become available\" or would it be run immediately after submitting?
", "rawMarkdown": ">submissions have been waiting in a queue for compute to become available\n\n@sohier If my solution is purely CPU-based, using only the two stock CPU cores that a Kaggle notebook is typically allotted for, would it still be subject to this \"waiting in a queue for compute to become available\" or would it be run immediately after submitting?", "votes": null }, { "id": "3090858", "postDate": "01/07/2025 17:55:19", "content": "I'm not positive it was the only pool affected, but most of the issues were with submissions using P100s.
", "rawMarkdown": "I'm not positive it was the only pool affected, but most of the issues were with submissions using P100s.", "votes": null }, { "id": "3090862", "postDate": "01/07/2025 18:07:19", "content": "I can confirm that my CPU only submission is taking longer now.
", "rawMarkdown": "I can confirm that my CPU only submission is taking longer now.", "votes": null }, { "id": "3090872", "postDate": "01/07/2025 18:16:48", "content": "Thanks for the report, we'll take a look.
", "rawMarkdown": "Thanks for the report, we'll take a look.", "votes": null }, { "id": "3090880", "postDate": "01/07/2025 18:23:36", "content": "@gunesevitan there's no queue for CPU submissions. Can you DM me a link to the notebook where you're seeing runtime differences?
", "rawMarkdown": "gunesevitan there's no queue for CPU submissions. Can you DM me a link to the notebook where you're seeing runtime differences?", "votes": null }, { "id": "3092262", "postDate": "01/09/2025 11:13:57", "content": "Will notebooks with cpu runtimes still run on a p100 during submissions?
", "rawMarkdown": "Will notebooks with cpu runtimes still run on a p100 during submissions?", "votes": null }, { "id": "3092289", "postDate": "01/09/2025 12:19:05", "content": "\"Each of my predictions (a dataframe) takes around 1.37 seconds. Can I predict successfully? Sorry, I’m really confused; the runtime of each of my notebooks is timing out.\"
", "rawMarkdown": "\"Each of my predictions (a dataframe) takes around 1.37 seconds. Can I predict successfully? Sorry, I’m really confused; the runtime of each of my notebooks is timing out.\"", "votes": null }, { "id": "3092454", "postDate": "01/09/2025 15:05:22", "content": "I'm kinda sure that it happened again @sohier. I used two submissions today. LightGBM model using first 600 trees took 1.5-2 hours and after couple hours same LightGBM model using first 500 trees took 3 hours for some reason.
", "rawMarkdown": "I'm kinda sure that it happened again @sohier. I used two submissions today. LightGBM model using first 600 trees took 1.5-2 hours and after couple hours same LightGBM model using first 500 trees took 3 hours for some reason.", "votes": null }, { "id": "3092733", "postDate": "01/09/2025 23:54:59", "content": "@gunesevitan thanks, we're taking a look.
", "rawMarkdown": "gunesevitan thanks, we're taking a look.", "votes": null } ], "comments": [ { "id": 3090841, "author_name": "revealer", "author_url": "", "post_date": "01/07/2025 17:40:22", "content": "\n\nsubmissions have been waiting in a queue for compute to become available
\n
@sohier If my solution is purely CPU-based, using only the two stock CPU cores that a Kaggle notebook is typically allotted for, would it still be subject to this \"waiting in a queue for compute to become available\" or would it be run immediately after submitting?
", "votes": null, "replies": [ { "id": 3090858, "author_name": "sohier", "author_url": "", "post_date": "01/07/2025 17:55:19", "content": "I'm not positive it was the only pool affected, but most of the issues were with submissions using P100s.
", "votes": null, "replies": [] }, { "id": 3090862, "author_name": "gunesevitan", "author_url": "", "post_date": "01/07/2025 18:07:19", "content": "I can confirm that my CPU only submission is taking longer now.
", "votes": null, "replies": [ { "id": 3090872, "author_name": "sohier", "author_url": "", "post_date": "01/07/2025 18:16:48", "content": "Thanks for the report, we'll take a look.
", "votes": null, "replies": [ { "id": 3090880, "author_name": "sohier", "author_url": "", "post_date": "01/07/2025 18:23:36", "content": "@gunesevitan there's no queue for CPU submissions. Can you DM me a link to the notebook where you're seeing runtime differences?
", "votes": null, "replies": [ { "id": 3092262, "author_name": "leonshams", "author_url": "", "post_date": "01/09/2025 11:13:57", "content": "Will notebooks with cpu runtimes still run on a p100 during submissions?
", "votes": null, "replies": [] }, { "id": 3092454, "author_name": "gunesevitan", "author_url": "", "post_date": "01/09/2025 15:05:22", "content": "I'm kinda sure that it happened again @sohier. I used two submissions today. LightGBM model using first 600 trees took 1.5-2 hours and after couple hours same LightGBM model using first 500 trees took 3 hours for some reason.
", "votes": null, "replies": [ { "id": 3092733, "author_name": "sohier", "author_url": "", "post_date": "01/09/2025 23:54:59", "content": "@gunesevitan thanks, we're taking a look.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] }, { "id": 3092289, "author_name": "tykmm1", "author_url": "", "post_date": "01/09/2025 12:19:05", "content": "\"Each of my predictions (a dataframe) takes around 1.37 seconds. Can I predict successfully? Sorry, I’m really confused; the runtime of each of my notebooks is timing out.\"
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3090822": "As several people have pointed out, submission runtimes have become unusually long in the last few days. What's happening behind the scenes is that submissions have been waiting in a queue for compute to become available. This wait time is frustrating but is separate from the 8 hour runtime limit, which only begins once a submission exits the queue.\n\nWe increased the size of the relevant compute pool yesterday and again this morning; it's now more than twice the capacity of a few days ago. The backlog is shrinking rapidly and appears to be on track to be cleared later today. We'll continue to monitor the situation in case the submission volume increases further. \n\nUpdate: the backlog is clear for now. It is possible there will be a further surge in submissions as we approach the submission deadline.", "3090841": ">submissions have been waiting in a queue for compute to become available\n\n@sohier If my solution is purely CPU-based, using only the two stock CPU cores that a Kaggle notebook is typically allotted for, would it still be subject to this \"waiting in a queue for compute to become available\" or would it be run immediately after submitting?", "3090858": "I'm not positive it was the only pool affected, but most of the issues were with submissions using P100s.", "3090862": "I can confirm that my CPU only submission is taking longer now.", "3090872": "Thanks for the report, we'll take a look.", "3090880": "gunesevitan there's no queue for CPU submissions. Can you DM me a link to the notebook where you're seeing runtime differences?", "3092262": "Will notebooks with cpu runtimes still run on a p100 during submissions?", "3092289": "\"Each of my predictions (a dataframe) takes around 1.37 seconds. Can I predict successfully? Sorry, I’m really confused; the runtime of each of my notebooks is timing out.\"", "3092454": "I'm kinda sure that it happened again @sohier. I used two submissions today. LightGBM model using first 600 trees took 1.5-2 hours and after couple hours same LightGBM model using first 500 trees took 3 hours for some reason.", "3092733": "gunesevitan thanks, we're taking a look." }, "source": "meta" }