{ "id": 107323, "title": "Kernel timeouts status", "url": "/competitions/aptos2019-blindness-detection/discussion/107323", "author_name": "Sohier Dane", "post_date": "2019-09-03T16:35:08.938000", "votes": 31, "comment_count": 73, "views": 0, "content": "

Update 3 (13:45 PM PST September 5th):\n- Our engineers located and alleviated an I/O bottleneck that was affecting this competition. New submissions should run at the previously normal speeds, but you may see unexpected timeout errors on currently running submissions as we can't move them off of the congested resource without canceling the run entirely.

\n\n

Update 2 (12:20 PM PST September 4th):\n- The problem seems to be reoccurring; we're seeing a partial slowdown in kernel runtimes again and our engineers are reviewing the case. Most submissions are still completing successfully, but those that were close to the time limit may not get through at the moment.

\n\n

Update (September 3rd):\n- The good news: we have implemented a temporary measure now to mitigate the slowdown while we continue to investigate the root cause. Please let us know if you see any unexpected timeout errors with kernels submitted AFTER this post was made at roughly 17:30 PST.\n- The bad news: the mitigation involved migrating the submissions work to different machines, and we accidentally killed many running submissions in the process. I would recommend restarting any kernels that unexpectedly failed. Sorry for the inconvenience, but hopefully this leaves us on stable ground going into the last days of the competition. We are extending the deadline by two days so impacted users have enough time to resubmit.

\n\n

Original post: As several of you have noticed, some kernels are running more slowly than usual. Our engineers are looking into the issue now. I'll post updates once we have more information.

", "messages": [ { "id": 616992, "postDate": "2019-09-03T16:35:08.940Z", "content": "

Update 3 (13:45 PM PST September 5th):\n- Our engineers located and alleviated an I/O bottleneck that was affecting this competition. New submissions should run at the previously normal speeds, but you may see unexpected timeout errors on currently running submissions as we can't move them off of the congested resource without canceling the run entirely.

\n\n

Update 2 (12:20 PM PST September 4th):\n- The problem seems to be reoccurring; we're seeing a partial slowdown in kernel runtimes again and our engineers are reviewing the case. Most submissions are still completing successfully, but those that were close to the time limit may not get through at the moment.

\n\n

Update (September 3rd):\n- The good news: we have implemented a temporary measure now to mitigate the slowdown while we continue to investigate the root cause. Please let us know if you see any unexpected timeout errors with kernels submitted AFTER this post was made at roughly 17:30 PST.\n- The bad news: the mitigation involved migrating the submissions work to different machines, and we accidentally killed many running submissions in the process. I would recommend restarting any kernels that unexpectedly failed. Sorry for the inconvenience, but hopefully this leaves us on stable ground going into the last days of the competition. We are extending the deadline by two days so impacted users have enough time to resubmit.

\n\n

Original post: As several of you have noticed, some kernels are running more slowly than usual. Our engineers are looking into the issue now. I'll post updates once we have more information.

", "rawMarkdown": "Update 3 (13:45 PM PST September 5th):\n- Our engineers located and alleviated an I/O bottleneck that was affecting this competition. New submissions should run at the previously normal speeds, but you may see unexpected timeout errors on currently running submissions as we can't move them off of the congested resource without canceling the run entirely.\n\nUpdate 2 (12:20 PM PST September 4th):\n- The problem seems to be reoccurring; we're seeing a partial slowdown in kernel runtimes again and our engineers are reviewing the case. Most submissions are still completing successfully, but those that were close to the time limit may not get through at the moment.\n\nUpdate (September 3rd):\n- The good news: we have implemented a temporary measure now to mitigate the slowdown while we continue to investigate the root cause. Please let us know if you see any unexpected timeout errors with kernels submitted AFTER this post was made at roughly 17:30 PST.\n- The bad news: the mitigation involved migrating the submissions work to different machines, and we accidentally killed many running submissions in the process. I would recommend restarting any kernels that unexpectedly failed. Sorry for the inconvenience, but hopefully this leaves us on stable ground going into the last days of the competition. [We are extending the deadline by two days](https://www.kaggle.com/c/aptos2019-blindness-detection/discussion/107369#latest-617285) so impacted users have enough time to resubmit.\n\nOriginal post: As several of you have noticed, some kernels are running more slowly than usual. Our engineers are looking into the issue now. I'll post updates once we have more information.", "votes": 30 }, { "id": 619790, "postDate": "2019-09-06T15:16:55.810Z", "content": "

Kernel Timeout error again after 3 hours runing. Anyone else having the same problem?

", "rawMarkdown": "Kernel Timeout error again after 3 hours runing. Anyone else having the same problem?", "votes": 5, "replies": [ { "id": 619829, "postDate": "2019-09-06T16:22:22.973Z", "content": "

Same here\nKernel Timeout again・・・

", "rawMarkdown": "Same here\nKernel Timeout again・・・", "votes": 3 }, { "id": 620348, "postDate": "2019-09-07T11:10:24.417Z", "content": "

Me, too.\nThis is stressful :'(

", "rawMarkdown": "Me, too.\nThis is stressful :'(", "votes": 1 } ] }, { "id": 619109, "postDate": "2019-09-05T20:47:11.867Z", "content": "

New submissions should run at normal speed again, please see the update to the main post for details.

", "rawMarkdown": "New submissions should run at normal speed again, please see the [update to the main post for details](https://www.kaggle.com/c/aptos2019-blindness-detection/discussion/107323).", "votes": 5, "replies": [ { "id": 619630, "postDate": "2019-09-06T11:28:19.907Z", "content": "

We still encountered submission time variance when we submitted 12 hours ago. Almost same code, but one finished 8 hours and the other kernel timed out. There still be an issue.

", "rawMarkdown": "We still encountered submission time variance when we submitted 12 hours ago. Almost same code, but one finished 8 hours and the other kernel timed out. There still be an issue.", "votes": 6 }, { "id": 619747, "postDate": "2019-09-06T14:39:37.460Z", "content": "

Yes, the issue is still there

", "rawMarkdown": "Yes, the issue is still there" }, { "id": 619752, "postDate": "2019-09-06T14:41:43.757Z", "content": "

30 minutes kernel, but when submitting it's running for about 4 hours

", "rawMarkdown": "30 minutes kernel, but when submitting it's running for about 4 hours", "votes": 3 }, { "id": 619753, "postDate": "2019-09-06T14:42:00.843Z", "content": "

@sohier , your fix did not solve the problem completely. Can you please respond.

", "rawMarkdown": "@sohier , your fix did not solve the problem completely. Can you please respond.", "votes": 3 }, { "id": 619967, "postDate": "2019-09-06T20:19:51.687Z", "content": "

Timeouts aren't down to zero, but I can confirm that overall rates have dropped substantially since we implemented the fix. I'd encourage anyone who has been holding onto a longer running submission to make the attempt.

", "rawMarkdown": "Timeouts aren't down to zero, but I can confirm that overall rates have dropped substantially since we implemented the fix. I'd encourage anyone who has been holding onto a longer running submission to make the attempt." }, { "id": 619975, "postDate": "2019-09-06T20:36:59.910Z", "content": "

Kernel commits complete, but when I open new version to check the notebook it won't load. Now when I refresh any open notebook (only view not running) nothing load anymore.

", "rawMarkdown": "Kernel commits complete, but when I open new version to check the notebook it won't load. Now when I refresh any open notebook (only view not running) nothing load anymore. ", "votes": 1 }, { "id": 619979, "postDate": "2019-09-06T20:46:49.413Z", "content": "

nvm I can see it on my laptop, but not on my desktop anymore... weird :p guess I'll restart browser or pc

", "rawMarkdown": "nvm I can see it on my laptop, but not on my desktop anymore... weird :p guess I'll restart browser or pc" }, { "id": 620717, "postDate": "2019-09-07T22:11:37.657Z", "content": "

I seem to be getting bit by the timeout error. I submitted an inference kernel that took sub 15 minutes to run on the 15% test subset. But instead of completing in a timely fashion its taking over double the amount of time I thought it would take to run on the full dataset.

\n\n

Edit: And it timed out after nine hours. Extrapolating from the sub 15 minute commit before submission it should have finished between 3-6 hours. This extrapolation worked earlier in the week.

", "rawMarkdown": "I seem to be getting bit by the timeout error. I submitted an inference kernel that took sub 15 minutes to run on the 15% test subset. But instead of completing in a timely fashion its taking over double the amount of time I thought it would take to run on the full dataset.\n\nEdit: And it timed out after nine hours. Extrapolating from the sub 15 minute commit before submission it should have finished between 3-6 hours. This extrapolation worked earlier in the week." } ] }, { "id": 617276, "postDate": "2019-09-04T00:54:47.140Z", "content": "

This situation is a mess, but to be honest, I don't want to have this competition extended because I'm getting tired and want to sleep.......

", "rawMarkdown": "This situation is a mess, but to be honest, I don't want to have this competition extended because I'm getting tired and want to sleep.......", "votes": 6, "replies": [ { "id": 617319, "postDate": "2019-09-04T02:20:39.717Z", "content": "

I knew that would happen again.....🙄 Good luck everyone, work hard 2 more days!

", "rawMarkdown": "I knew that would happen again.....🙄 Good luck everyone, work hard 2 more days!" }, { "id": 617346, "postDate": "2019-09-04T03:24:36.903Z", "content": "

I have the same feeling. Learned much less than expected on kaggle. The key to success is tuning hyperparameters, lb probing and ensembling more, larger models!!!😨 😨

", "rawMarkdown": "I have the same feeling. Learned much less than expected on kaggle. The key to success is tuning hyperparameters, lb probing and ensembling more, larger models!!!😨 😨 " } ] }, { "id": 617004, "postDate": "2019-09-03T16:46:47.430Z", "content": "

Great to know that you are working on it. Please solve it asap.\nHave you thought about the possibility of giving us some extra time (some few more days) to compensate all the time we have wasted due to the different problems with commits and submittions (and also considering the 1 GPU limit)?

", "rawMarkdown": "Great to know that you are working on it. Please solve it asap.\nHave you thought about the possibility of giving us some extra time (some few more days) to compensate all the time we have wasted due to the different problems with commits and submittions (and also considering the 1 GPU limit)? ", "votes": 4, "replies": [ { "id": 617185, "postDate": "2019-09-03T20:46:00.103Z", "content": "

Yeah, I am unable to make possibly my best submissions.

", "rawMarkdown": "Yeah, I am unable to make possibly my best submissions.", "votes": 2 } ] }, { "id": 617245, "postDate": "2019-09-03T23:53:12.397Z", "content": "

My kernel recently submitted was timed out without waiting 9 hours. I wonder why.\n\"\"

", "rawMarkdown": "My kernel recently submitted was timed out without waiting 9 hours. I wonder why.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2208353%2F3727c4aca6848c33da8d3b3ed0506159%2Ftimeout_%202019-09-04%20085008.jpg?generation=1567554923612468&alt=media)", "votes": 3, "replies": [ { "id": 617250, "postDate": "2019-09-04T00:00:10.433Z", "content": "

About 10 minutes ago, all my submissions timed out at the same time.

", "rawMarkdown": "About 10 minutes ago, all my submissions timed out at the same time." }, { "id": 617251, "postDate": "2019-09-04T00:01:49.007Z", "content": "

Same timing. I also confirmed this phenomenon about 10-15 minutes ago.

", "rawMarkdown": "Same timing. I also confirmed this phenomenon about 10-15 minutes ago." }, { "id": 617253, "postDate": "2019-09-04T00:02:15.370Z", "content": "

They must have stopped all of the submissions for some reason.

", "rawMarkdown": "They must have stopped all of the submissions for some reason." } ] }, { "id": 618805, "postDate": "2019-09-05T14:08:31.207Z", "content": "

It seems to work now but it is so slow: notebook running less than 1 hour is being submitted all 9 hours!

", "rawMarkdown": "It seems to work now but it is so slow: notebook running less than 1 hour is being submitted all 9 hours! ", "votes": 4, "replies": [ { "id": 618821, "postDate": "2019-09-05T14:35:08.010Z", "content": "

@demonplus How many minutes took your notebook before submission? Mine is 35 min and it took 8h.

", "rawMarkdown": "@demonplus How many minutes took your notebook before submission? Mine is 35 min and it took 8h.", "votes": 1 }, { "id": 618988, "postDate": "2019-09-05T17:53:51.473Z", "content": "

@raimonds1993 Mine was about 50 min and submission took 9 hours, I already thought it would be error but still it was ok. Awful it takes so much time to submit.

", "rawMarkdown": "@raimonds1993 Mine was about 50 min and submission took 9 hours, I already thought it would be error but still it was ok. Awful it takes so much time to submit.", "votes": 2 } ] }, { "id": 617003, "postDate": "2019-09-03T16:46:20.170Z", "content": "

Is it possible to rerun that submissions without losing new attempts?

", "rawMarkdown": "Is it possible to rerun that submissions without losing new attempts? ", "votes": 4 }, { "id": 617199, "postDate": "2019-09-03T21:08:06.733Z", "content": "

With this like the 5th time the kernels are not working in this kernels-only competition, I think Kaggle should consider the extending the deadline by a week as I discuss here

", "rawMarkdown": "With this like the 5th time the kernels are not working in this kernels-only competition, I think Kaggle should consider the extending the deadline by a week as I discuss [here](https://www.kaggle.com/c/aptos2019-blindness-detection/discussion/105916#latest-609765)", "votes": 1 }, { "id": 620724, "postDate": "2019-09-07T22:50:29.050Z", "content": "

Same here\nKernel Timeout...

", "rawMarkdown": "Same here\nKernel Timeout...", "votes": 1 }, { "id": 619076, "postDate": "2019-09-05T19:57:36.833Z", "content": "

my kernel eventually timed out.. i lost submission for the day and 1 more tomorrow..\nnot sure if it was thought initially that live run of kernel on huge private data could make things slow and would overload existing amount of resources..\nSuch kind of outages dont go well with expectation of smooth experience . \nParticipants have spent huge time in waiting for kernel to complete ,on the top of it on outages to get resolved.

", "rawMarkdown": "my kernel eventually timed out.. i lost submission for the day and 1 more tomorrow..\nnot sure if it was thought initially that live run of kernel on huge private data could make things slow and would overload existing amount of resources..\nSuch kind of outages dont go well with expectation of smooth experience . \nParticipants have spent huge time in waiting for kernel to complete ,on the top of it on outages to get resolved.\n\n", "votes": 1 }, { "id": 618819, "postDate": "2019-09-05T14:34:51.367Z", "content": "

All subs running through... will report if something fails for me. Apparently, it is a problem for edge cases running long.

", "rawMarkdown": "All subs running through... will report if something fails for me. Apparently, it is a problem for edge cases running long.", "votes": 1 }, { "id": 618578, "postDate": "2019-09-05T10:18:31.577Z", "content": "

submission kernel still running ...

", "rawMarkdown": "submission kernel still running ...", "votes": 1 }, { "id": 618812, "postDate": "2019-09-05T14:20:38.523Z", "content": "

The issue still exists.. My kernels which usually takes around 2 hours to complete is still running since past 4 hours.

", "rawMarkdown": "The issue still exists.. My kernels which usually takes around 2 hours to complete is still running since past 4 hours.", "votes": 2 }, { "id": 618625, "postDate": "2019-09-05T11:22:57.587Z", "content": "

Almost the same as the previous.\nI submitted two kernels with almost the same content roughly simultaneously. (15:19GMT+9 and 15:32GMT+9)\nOne has completed processing (successful) in 2 hours, but the other has been processing for more than 4 hours now.

\n\n

I would appreciate if you could tell us when the system is heavy.\nThank you.

", "rawMarkdown": "Almost the same as the previous.\nI submitted two kernels with almost the same content roughly simultaneously. (15:19GMT+9 and 15:32GMT+9)\nOne has completed processing (successful) in 2 hours, but the other has been processing for more than 4 hours now.\n\nI would appreciate if you could tell us when the system is heavy.\nThank you.", "votes": 2, "replies": [ { "id": 618897, "postDate": "2019-09-05T15:55:55.690Z", "content": "

its been 5 hrs running with same kernel.. \nIts running 4 times each time takes ideally 30-40 min so not more than 3 hrs it should taken for 4 times run

", "rawMarkdown": "its been 5 hrs running with same kernel.. \nIts running 4 times each time takes ideally 30-40 min so not more than 3 hrs it should taken for 4 times run " }, { "id": 618937, "postDate": "2019-09-05T17:00:35.177Z", "content": "

Looks like this kernel will find its futile fate...

", "rawMarkdown": "Looks like this kernel will find its futile fate... " } ] }, { "id": 617795, "postDate": "2019-09-04T13:43:41.593Z", "content": "

@sohier the problem hasn't gone. My kernel with commiting time ~20 min was being submitting during ~4 hours. Kernels with commiting time ~70 min just timed out (meanwhile almost the same ones were submitted earlier)

", "rawMarkdown": "@sohier the problem hasn't gone. My kernel with commiting time ~20 min was being submitting during ~4 hours. Kernels with commiting time ~70 min just timed out (meanwhile almost the same ones were submitted earlier)", "votes": 2, "replies": [ { "id": 618018, "postDate": "2019-09-04T18:25:11.830Z", "content": "

@sohier same problem, My kernel with committing time ~30 min has been running for over 4 hours and still no result. My second Kernels with committing time ~70 min timed out.

", "rawMarkdown": "@sohier same problem, My kernel with committing time ~30 min has been running for over 4 hours and still no result. My second Kernels with committing time ~70 min timed out.", "votes": 1 } ] }, { "id": 617780, "postDate": "2019-09-04T13:20:08.980Z", "content": "

I am still running kernels that take much more to complete than before - is the issue really fixed? I think they will time out...

", "rawMarkdown": "I am still running kernels that take much more to complete than before - is the issue really fixed? I think they will time out...", "votes": 2 }, { "id": 617234, "postDate": "2019-09-03T23:07:33.933Z", "content": "

I lost a lot of submission in a few days.\nI will not be able to make the best submission.\nI would like to extend the due date a few days or increase the maximum number of submissions per day.

\n\n

\"\".png?generation=1567551903664904&alt=media)

", "rawMarkdown": "I lost a lot of submission in a few days.\nI will not be able to make the best submission.\nI would like to extend the due date a few days or increase the maximum number of submissions per day.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2930293%2Fae45ce748ad4a914f89e4eb1acb78e00%2F().png?generation=1567551903664904&alt=media)", "votes": 2 }, { "id": 618603, "postDate": "2019-09-05T10:56:37.417Z", "content": "

My kernel takes around 3 hours to run when i commit but when submitting it is still not completing with in 9 hours.

", "rawMarkdown": "My kernel takes around 3 hours to run when i commit but when submitting it is still not completing with in 9 hours.", "replies": [ { "id": 618639, "postDate": "2019-09-05T11:35:18.197Z", "content": "

and it will stay like that :)\nduring \"commit\" the code runs against the \"public test set\" (around 2000 images)\nduring \"submit\" the \"private test set\" is used, this is around 13000 images, and from my experience it takes around 7-8 times as long as commit

\n\n

if you have commit time of 3 hours, it would need around 24 hours -> you need to stay below 1 hour on commit

", "rawMarkdown": "and it will stay like that :)\nduring \"commit\" the code runs against the \"public test set\" (around 2000 images)\nduring \"submit\" the \"private test set\" is used, this is around 13000 images, and from my experience it takes around 7-8 times as long as commit\n\nif you have commit time of 3 hours, it would need around 24 hours -> you need to stay below 1 hour on commit", "votes": 2 }, { "id": 618665, "postDate": "2019-09-05T12:00:33.967Z", "content": "

Is this an assumption or kaggle officials have confirmed this in a thread?\nSince i can see some public kernels with more than 3 hours runtime and still being scored ?

", "rawMarkdown": "Is this an assumption or kaggle officials have confirmed this in a thread?\nSince i can see some public kernels with more than 3 hours runtime and still being scored ?" }, { "id": 618782, "postDate": "2019-09-05T13:40:42.027Z", "content": "

This is how it works, yes. You need to calculate at least runtime of commit * 7 < 9 to be confident that it will run through.

", "rawMarkdown": "This is how it works, yes. You need to calculate at least runtime of commit * 7 < 9 to be confident that it will run through." }, { "id": 618784, "postDate": "2019-09-05T13:43:00.393Z", "content": "

But then how so many public kernerls with runtime of more than 3hours went through ?

", "rawMarkdown": "But then how so many public kernerls with runtime of more than 3hours went through ?" }, { "id": 618823, "postDate": "2019-09-05T14:36:39.073Z", "content": "

Can you link an example?

", "rawMarkdown": "Can you link an example?" }, { "id": 618869, "postDate": "2019-09-05T15:17:13.200Z", "content": "

Sure, have a look at this, runtime is 4 hours and there is also public score attached to it, which means it ran successfully

\n\n

https://www.kaggle.com/chopinforest/efficientnetb4-fastai-blindness-detection

", "rawMarkdown": "Sure, have a look at this, runtime is 4 hours and there is also public score attached to it, which means it ran successfully\n\nhttps://www.kaggle.com/chopinforest/efficientnetb4-fastai-blindness-detection" }, { "id": 618877, "postDate": "2019-09-05T15:25:44.063Z", "content": "

The runtime comes from fitting the model, not from inference.

", "rawMarkdown": "The runtime comes from fitting the model, not from inference." }, { "id": 618932, "postDate": "2019-09-05T16:55:09.630Z", "content": "

Then what about this, this is scoring 5 times i think\nhttps://www.kaggle.com/ahoukang/aptos-vote

", "rawMarkdown": "Then what about this, this is scoring 5 times i think\nhttps://www.kaggle.com/ahoukang/aptos-vote" }, { "id": 619027, "postDate": "2019-09-05T18:59:32Z", "content": "

But the kernel fits, and fitting is not different in submission.

", "rawMarkdown": "But the kernel fits, and fitting is not different in submission.", "votes": 1 } ] }, { "id": 617266, "postDate": "2019-09-04T00:37:39.717Z", "content": "

Update:\n- The good news: we have implemented a temporary measure now to mitigate the slowdown while we continue to investigate the root cause. Please let us know if you see any unexpected timeout errors with kernels submitted AFTER this post was made at roughly 17:30 PST.\n- The bad news: the mitigation involved migrating the submissions work to different machines, and we accidentally killed many running submissions in the process. I would recommend restarting any kernels that unexpectedly failed. Sorry for the inconvenience, but hopefully this leaves us on stable ground going into the last 24 hours of the competition.

", "rawMarkdown": "Update:\n- The good news: we have implemented a temporary measure now to mitigate the slowdown while we continue to investigate the root cause. Please let us know if you see any unexpected timeout errors with kernels submitted AFTER this post was made at roughly 17:30 PST.\n- The bad news: the mitigation involved migrating the submissions work to different machines, and we accidentally killed many running submissions in the process. I would recommend restarting any kernels that unexpectedly failed. Sorry for the inconvenience, but hopefully this leaves us on stable ground going into the last 24 hours of the competition.\n", "votes": -2, "replies": [ { "id": 617267, "postDate": "2019-09-04T00:40:20.293Z", "content": "

The Kaggle Kernels and their many errors have made this a very bad experience. A week extension I think would be best for this competition.

", "rawMarkdown": "The Kaggle Kernels and their many errors have made this a very bad experience. A week extension I think would be best for this competition." }, { "id": 617271, "postDate": "2019-09-04T00:44:01.150Z", "content": "

Thank you for your response. There is one thing I really want to confirm.

\n\n

The deadline will be never postponed, right?

\n\n

It is very important for the operation of the remaining few submissions, so I would be happy if you could answer clearly.

", "rawMarkdown": "Thank you for your response. There is one thing I really want to confirm.\n\nThe deadline will be never postponed, right?\n\nIt is very important for the operation of the remaining few submissions, so I would be happy if you could answer clearly." }, { "id": 617282, "postDate": "2019-09-04T01:14:03.200Z", "content": "

@octpath0302 good question. We generally avoid deadline extensions, but we had to review this case given problems in the last 24 hours of the competition. The deadline has been extended by 2 days per this post.

", "rawMarkdown": "@octpath0302 good question. We generally avoid deadline extensions, but we had to review this case given problems in the last 24 hours of the competition. The deadline has been extended by 2 days per [this post](https://www.kaggle.com/c/aptos2019-blindness-detection/discussion/107369).", "votes": 1 }, { "id": 617444, "postDate": "2019-09-04T06:11:19.823Z", "content": "

There were still 48 hours left.

", "rawMarkdown": "There were still 48 hours left.", "votes": 1 } ] }, { "id": 617204, "postDate": "2019-09-03T21:44:17.473Z", "content": "

All my subs finished fine today btw.

", "rawMarkdown": "All my subs finished fine today btw.", "votes": -1, "replies": [ { "id": 617223, "postDate": "2019-09-03T22:27:21.507Z", "content": "

\"\"

", "rawMarkdown": "![](https://i.imgur.com/RGa5oDL.jpg)", "votes": 12 }, { "id": 617447, "postDate": "2019-09-04T06:18:42.503Z", "content": "

The ones over night failed because they were manually stopped apparently as posted in the topic, but the ones yesterday worked fine.

", "rawMarkdown": "The ones over night failed because they were manually stopped apparently as posted in the topic, but the ones yesterday worked fine. " } ] }, { "id": 620630, "postDate": "2019-09-07T18:54:23.643Z", "content": "

I can't even open a kernel, \"this kernel is powered off\" banner keep showing ...

", "rawMarkdown": "I can't even open a kernel, \"this kernel is powered off\" banner keep showing ..." }, { "id": 619101, "postDate": "2019-09-05T20:23:50.380Z", "content": "

@sohier, please let us know once the issue is fixed. Waiting for the resolution to press submit.

", "rawMarkdown": "@sohier, please let us know once the issue is fixed. Waiting for the resolution to press submit." }, { "id": 618463, "postDate": "2019-09-05T08:27:35.460Z", "content": "

I am unable to even open the kernel, let alone submit any thing. It's been like this since 24 hours at least.

", "rawMarkdown": "I am unable to even open the kernel, let alone submit any thing. It's been like this since 24 hours at least." }, { "id": 618454, "postDate": "2019-09-05T08:16:07.500Z", "content": "

I'm facing this problem too, you should post this message in other competitions discussion sections to let people know that you're working on solving this issue.

", "rawMarkdown": "I'm facing this problem too, you should post this message in other competitions discussion sections to let people know that you're working on solving this issue." }, { "id": 618062, "postDate": "2019-09-04T19:18:30.357Z", "content": "

I keep getting \"ResourceExhaustedError\" just one hour after submitting.

", "rawMarkdown": "I keep getting \"ResourceExhaustedError\" just one hour after submitting. " }, { "id": 617953, "postDate": "2019-09-04T16:50:35.023Z", "content": "

i m unable to open the notebooks in view mode.. all thait is open is open but i cant open it the eixsting versions

", "rawMarkdown": "i m unable to open the notebooks in view mode.. all thait is open is open but i cant open it the eixsting versions" }, { "id": 617766, "postDate": "2019-09-04T12:59:47.593Z", "content": "

@sohier I report a suspicious behavior of kernels again.\nI submitted two kernels with almost the same content roughly simultaneously. (17:20GMT+9 and 17:40GMT+9)\nOne has completed processing (successful) in less than 2 hours, and the other has been processing for more than 4 hours now.\nAre there any abnormalities such as system load still high?\nThank you.

\n\n

EDITED: The kernel that took a long time has finally completed(success). It took at least twice as long.

", "rawMarkdown": "@sohier I report a suspicious behavior of kernels again.\nI submitted two kernels with almost the same content roughly simultaneously. (17:20GMT+9 and 17:40GMT+9)\nOne has completed processing (successful) in less than 2 hours, and the other has been processing for more than 4 hours now.\nAre there any abnormalities such as system load still high?\nThank you.\n\nEDITED: The kernel that took a long time has finally completed(success). It took at least twice as long." }, { "id": 617750, "postDate": "2019-09-04T12:43:12.130Z", "content": "

Thanks @sohier.

\n\n

My kernel completed running in only 37 minutes yet my submission timed out with an error after 9hrs yesterday. I have re-submitted the same kernel after your post and will report back whether it succeeds or times out again.

", "rawMarkdown": "Thanks @sohier. \n\nMy kernel completed running in only 37 minutes yet my submission timed out with an error after 9hrs yesterday. I have re-submitted the same kernel after your post and will report back whether it succeeds or times out again.", "replies": [ { "id": 617986, "postDate": "2019-09-04T17:34:12.063Z", "content": "

@sohier , I am reporting back. My kernel submission ended after 9hrs with the following error:-

\n\n

> Submission CSV Not Found

\n\n

By the way my submission file is correctly named i.e. \"submission.csv\" as always.

", "rawMarkdown": "@sohier , I am reporting back. My kernel submission ended after 9hrs with the following error:-\n\n> Submission CSV Not Found\n\nBy the way my submission file is correctly named i.e. \"submission.csv\" as always." }, { "id": 618129, "postDate": "2019-09-04T22:16:22.970Z", "content": "

@sohier , my other inference kernel that took 34 minutes to run also failed submission with a Kernel Timeout error. This problem is very much around.

", "rawMarkdown": "@sohier , my other inference kernel that took 34 minutes to run also failed submission with a Kernel Timeout error. This problem is very much around." } ] }, { "id": 617679, "postDate": "2019-09-04T11:28:38.487Z", "content": "

kernel that worked yesterday fails with exit code 134 on commit, and fails without explanation on submission

", "rawMarkdown": "kernel that worked yesterday fails with exit code 134 on commit, and fails without explanation on submission" }, { "id": 617402, "postDate": "2019-09-04T05:04:42.170Z", "content": "

I am still getting submission error

", "rawMarkdown": "I am still getting submission error " }, { "id": 617354, "postDate": "2019-09-04T03:35:54.567Z", "content": "

Thanks for informing give opportunity to rerun...

\n\n

Best

", "rawMarkdown": "Thanks for informing give opportunity to rerun...\n\nBest" }, { "id": 617352, "postDate": "2019-09-04T03:32:40.127Z", "content": "

Yes my last six submissions I observed unexpected timeouts... no possibility to be... even they run in half of the limit time... my older top versions were running almost two times slower.

\n\n

Best

", "rawMarkdown": "Yes my last six submissions I observed unexpected timeouts... no possibility to be... even they run in half of the limit time... my older top versions were running almost two times slower.\n\nBest " }, { "id": 616994, "postDate": "2019-09-03T16:37:47.417Z", "content": "

Thanks, please pay attention at submission issue, this is critical for the competition I believe!

", "rawMarkdown": "Thanks, please pay attention at submission issue, this is critical for the competition I believe!" }, { "id": 620038, "postDate": "2019-09-06T22:43:54.120Z", "rawMarkdown": "", "isDeleted": true }, { "id": 618031, "postDate": "2019-09-04T18:42:35.837Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 619790, "author_name": "Carlos Prades K.", "author_url": "", "post_date": "2019-09-06T15:16:55.810000", "content": "

Kernel Timeout error again after 3 hours runing. Anyone else having the same problem?

", "votes": 5, "replies": [ { "id": 619829, "author_name": "KK", "author_url": "", "post_date": "2019-09-06T16:22:22.973000", "content": "

Same here\nKernel Timeout again・・・

", "votes": 3, "replies": [] }, { "id": 620348, "author_name": "GOSS", "author_url": "", "post_date": "2019-09-07T11:10:24.417000", "content": "

Me, too.\nThis is stressful :'(

", "votes": 1, "replies": [] } ] }, { "id": 619109, "author_name": "Sohier Dane", "author_url": "", "post_date": "2019-09-05T20:47:11.867000", "content": "

New submissions should run at normal speed again, please see the update to the main post for details.

", "votes": 5, "replies": [ { "id": 619630, "author_name": "Camaro", "author_url": "", "post_date": "2019-09-06T11:28:19.907000", "content": "

We still encountered submission time variance when we submitted 12 hours ago. Almost same code, but one finished 8 hours and the other kernel timed out. There still be an issue.

", "votes": 6, "replies": [] }, { "id": 619747, "author_name": "Ravi Kumar Vadapalli", "author_url": "", "post_date": "2019-09-06T14:39:37.460000", "content": "

Yes, the issue is still there

", "votes": 0, "replies": [] }, { "id": 619752, "author_name": "Ivan Razumov", "author_url": "", "post_date": "2019-09-06T14:41:43.757000", "content": "

30 minutes kernel, but when submitting it's running for about 4 hours

", "votes": 3, "replies": [] }, { "id": 619753, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-09-06T14:42:00.843000", "content": "

@sohier , your fix did not solve the problem completely. Can you please respond.

", "votes": 3, "replies": [] }, { "id": 619967, "author_name": "Sohier Dane", "author_url": "", "post_date": "2019-09-06T20:19:51.687000", "content": "

Timeouts aren't down to zero, but I can confirm that overall rates have dropped substantially since we implemented the fix. I'd encourage anyone who has been holding onto a longer running submission to make the attempt.

", "votes": 0, "replies": [] }, { "id": 619975, "author_name": "Sirong Huang", "author_url": "", "post_date": "2019-09-06T20:36:59.910000", "content": "

Kernel commits complete, but when I open new version to check the notebook it won't load. Now when I refresh any open notebook (only view not running) nothing load anymore.

", "votes": 1, "replies": [] }, { "id": 619979, "author_name": "Sirong Huang", "author_url": "", "post_date": "2019-09-06T20:46:49.413000", "content": "

nvm I can see it on my laptop, but not on my desktop anymore... weird :p guess I'll restart browser or pc

", "votes": 0, "replies": [] }, { "id": 620717, "author_name": "Benjamin Warner", "author_url": "", "post_date": "2019-09-07T22:11:37.657000", "content": "

I seem to be getting bit by the timeout error. I submitted an inference kernel that took sub 15 minutes to run on the 15% test subset. But instead of completing in a timely fashion its taking over double the amount of time I thought it would take to run on the full dataset.

\n\n

Edit: And it timed out after nine hours. Extrapolating from the sub 15 minute commit before submission it should have finished between 3-6 hours. This extrapolation worked earlier in the week.

", "votes": 0, "replies": [] } ] }, { "id": 617276, "author_name": "Camaro", "author_url": "", "post_date": "2019-09-04T00:54:47.140000", "content": "

This situation is a mess, but to be honest, I don't want to have this competition extended because I'm getting tired and want to sleep.......

", "votes": 6, "replies": [ { "id": 617319, "author_name": "Camaro", "author_url": "", "post_date": "2019-09-04T02:20:39.717000", "content": "

I knew that would happen again.....🙄 Good luck everyone, work hard 2 more days!

", "votes": 0, "replies": [] }, { "id": 617346, "author_name": "Homoalways", "author_url": "", "post_date": "2019-09-04T03:24:36.903000", "content": "

I have the same feeling. Learned much less than expected on kaggle. The key to success is tuning hyperparameters, lb probing and ensembling more, larger models!!!😨 😨

", "votes": 0, "replies": [] } ] }, { "id": 617004, "author_name": "Carlos Prades K.", "author_url": "", "post_date": "2019-09-03T16:46:47.430000", "content": "

Great to know that you are working on it. Please solve it asap.\nHave you thought about the possibility of giving us some extra time (some few more days) to compensate all the time we have wasted due to the different problems with commits and submittions (and also considering the 1 GPU limit)?

", "votes": 4, "replies": [ { "id": 617185, "author_name": "Saurabh7", "author_url": "", "post_date": "2019-09-03T20:46:00.103000", "content": "

Yeah, I am unable to make possibly my best submissions.

", "votes": 2, "replies": [] } ] }, { "id": 617245, "author_name": "oct_path", "author_url": "", "post_date": "2019-09-03T23:53:12.397000", "content": "

My kernel recently submitted was timed out without waiting 9 hours. I wonder why.\n\"\"

", "votes": 3, "replies": [ { "id": 617250, "author_name": "ilovescience", "author_url": "", "post_date": "2019-09-04T00:00:10.433000", "content": "

About 10 minutes ago, all my submissions timed out at the same time.

", "votes": 0, "replies": [] }, { "id": 617251, "author_name": "oct_path", "author_url": "", "post_date": "2019-09-04T00:01:49.007000", "content": "

Same timing. I also confirmed this phenomenon about 10-15 minutes ago.

", "votes": 0, "replies": [] }, { "id": 617253, "author_name": "ilovescience", "author_url": "", "post_date": "2019-09-04T00:02:15.370000", "content": "

They must have stopped all of the submissions for some reason.

", "votes": 0, "replies": [] } ] }, { "id": 618805, "author_name": "Anna Novikova", "author_url": "", "post_date": "2019-09-05T14:08:31.207000", "content": "

It seems to work now but it is so slow: notebook running less than 1 hour is being submitted all 9 hours!

", "votes": 4, "replies": [ { "id": 618821, "author_name": "Federico Raimondi Cominesi", "author_url": "", "post_date": "2019-09-05T14:35:08.010000", "content": "

@demonplus How many minutes took your notebook before submission? Mine is 35 min and it took 8h.

", "votes": 1, "replies": [] }, { "id": 618988, "author_name": "Anna Novikova", "author_url": "", "post_date": "2019-09-05T17:53:51.473000", "content": "

@raimonds1993 Mine was about 50 min and submission took 9 hours, I already thought it would be error but still it was ok. Awful it takes so much time to submit.

", "votes": 2, "replies": [] } ] }, { "id": 617003, "author_name": "empty", "author_url": "", "post_date": "2019-09-03T16:46:20.170000", "content": "

Is it possible to rerun that submissions without losing new attempts?

", "votes": 4, "replies": [] }, { "id": 617199, "author_name": "ilovescience", "author_url": "", "post_date": "2019-09-03T21:08:06.733000", "content": "

With this like the 5th time the kernels are not working in this kernels-only competition, I think Kaggle should consider the extending the deadline by a week as I discuss here

", "votes": 1, "replies": [] }, { "id": 620724, "author_name": "liuzhangzhen", "author_url": "", "post_date": "2019-09-07T22:50:29.050000", "content": "

Same here\nKernel Timeout...

", "votes": 1, "replies": [] }, { "id": 619076, "author_name": "Jaideep", "author_url": "", "post_date": "2019-09-05T19:57:36.833000", "content": "

my kernel eventually timed out.. i lost submission for the day and 1 more tomorrow..\nnot sure if it was thought initially that live run of kernel on huge private data could make things slow and would overload existing amount of resources..\nSuch kind of outages dont go well with expectation of smooth experience . \nParticipants have spent huge time in waiting for kernel to complete ,on the top of it on outages to get resolved.

", "votes": 1, "replies": [] }, { "id": 618819, "author_name": "Psi", "author_url": "", "post_date": "2019-09-05T14:34:51.367000", "content": "

All subs running through... will report if something fails for me. Apparently, it is a problem for edge cases running long.

", "votes": 1, "replies": [] }, { "id": 618578, "author_name": "ls", "author_url": "", "post_date": "2019-09-05T10:18:31.577000", "content": "

submission kernel still running ...

", "votes": 1, "replies": [] }, { "id": 618812, "author_name": "Jayasooryan K V", "author_url": "", "post_date": "2019-09-05T14:20:38.523000", "content": "

The issue still exists.. My kernels which usually takes around 2 hours to complete is still running since past 4 hours.

", "votes": 2, "replies": [] }, { "id": 618625, "author_name": "oct_path", "author_url": "", "post_date": "2019-09-05T11:22:57.587000", "content": "

Almost the same as the previous.\nI submitted two kernels with almost the same content roughly simultaneously. (15:19GMT+9 and 15:32GMT+9)\nOne has completed processing (successful) in 2 hours, but the other has been processing for more than 4 hours now.

\n\n

I would appreciate if you could tell us when the system is heavy.\nThank you.

", "votes": 2, "replies": [ { "id": 618897, "author_name": "Jaideep", "author_url": "", "post_date": "2019-09-05T15:55:55.690000", "content": "

its been 5 hrs running with same kernel.. \nIts running 4 times each time takes ideally 30-40 min so not more than 3 hrs it should taken for 4 times run

", "votes": 0, "replies": [] }, { "id": 618937, "author_name": "Jaideep", "author_url": "", "post_date": "2019-09-05T17:00:35.177000", "content": "

Looks like this kernel will find its futile fate...

", "votes": 0, "replies": [] } ] }, { "id": 617795, "author_name": "empty", "author_url": "", "post_date": "2019-09-04T13:43:41.593000", "content": "

@sohier the problem hasn't gone. My kernel with commiting time ~20 min was being submitting during ~4 hours. Kernels with commiting time ~70 min just timed out (meanwhile almost the same ones were submitted earlier)

", "votes": 2, "replies": [ { "id": 618018, "author_name": "Amil Gentili", "author_url": "", "post_date": "2019-09-04T18:25:11.830000", "content": "

@sohier same problem, My kernel with committing time ~30 min has been running for over 4 hours and still no result. My second Kernels with committing time ~70 min timed out.

", "votes": 1, "replies": [] } ] }, { "id": 617780, "author_name": "Μαριος Μιχαηλιδης KazAnova", "author_url": "", "post_date": "2019-09-04T13:20:08.980000", "content": "

I am still running kernels that take much more to complete than before - is the issue really fixed? I think they will time out...

", "votes": 2, "replies": [] }, { "id": 617234, "author_name": "Shaorooon", "author_url": "", "post_date": "2019-09-03T23:07:33.933000", "content": "

I lost a lot of submission in a few days.\nI will not be able to make the best submission.\nI would like to extend the due date a few days or increase the maximum number of submissions per day.

\n\n

\"\".png?generation=1567551903664904&alt=media)

", "votes": 2, "replies": [] }, { "id": 618603, "author_name": "Shubhinder", "author_url": "", "post_date": "2019-09-05T10:56:37.417000", "content": "

My kernel takes around 3 hours to run when i commit but when submitting it is still not completing with in 9 hours.

", "votes": 0, "replies": [ { "id": 618639, "author_name": "steelrose", "author_url": "", "post_date": "2019-09-05T11:35:18.197000", "content": "

and it will stay like that :)\nduring \"commit\" the code runs against the \"public test set\" (around 2000 images)\nduring \"submit\" the \"private test set\" is used, this is around 13000 images, and from my experience it takes around 7-8 times as long as commit

\n\n

if you have commit time of 3 hours, it would need around 24 hours -> you need to stay below 1 hour on commit

", "votes": 2, "replies": [] }, { "id": 618665, "author_name": "Shubhinder", "author_url": "", "post_date": "2019-09-05T12:00:33.967000", "content": "

Is this an assumption or kaggle officials have confirmed this in a thread?\nSince i can see some public kernels with more than 3 hours runtime and still being scored ?

", "votes": 0, "replies": [] }, { "id": 618782, "author_name": "Psi", "author_url": "", "post_date": "2019-09-05T13:40:42.027000", "content": "

This is how it works, yes. You need to calculate at least runtime of commit * 7 < 9 to be confident that it will run through.

", "votes": 0, "replies": [] }, { "id": 618784, "author_name": "Shubhinder", "author_url": "", "post_date": "2019-09-05T13:43:00.393000", "content": "

But then how so many public kernerls with runtime of more than 3hours went through ?

", "votes": 0, "replies": [] }, { "id": 618823, "author_name": "Psi", "author_url": "", "post_date": "2019-09-05T14:36:39.073000", "content": "

Can you link an example?

", "votes": 0, "replies": [] }, { "id": 618869, "author_name": "Shubhinder", "author_url": "", "post_date": "2019-09-05T15:17:13.200000", "content": "

Sure, have a look at this, runtime is 4 hours and there is also public score attached to it, which means it ran successfully

\n\n

https://www.kaggle.com/chopinforest/efficientnetb4-fastai-blindness-detection

", "votes": 0, "replies": [] }, { "id": 618877, "author_name": "Psi", "author_url": "", "post_date": "2019-09-05T15:25:44.063000", "content": "

The runtime comes from fitting the model, not from inference.

", "votes": 0, "replies": [] }, { "id": 618932, "author_name": "Shubhinder", "author_url": "", "post_date": "2019-09-05T16:55:09.630000", "content": "

Then what about this, this is scoring 5 times i think\nhttps://www.kaggle.com/ahoukang/aptos-vote

", "votes": 0, "replies": [] }, { "id": 619027, "author_name": "Psi", "author_url": "", "post_date": "2019-09-05T18:59:32", "content": "

But the kernel fits, and fitting is not different in submission.

", "votes": 1, "replies": [] } ] }, { "id": 617266, "author_name": "Sohier Dane", "author_url": "", "post_date": "2019-09-04T00:37:39.717000", "content": "

Update:\n- The good news: we have implemented a temporary measure now to mitigate the slowdown while we continue to investigate the root cause. Please let us know if you see any unexpected timeout errors with kernels submitted AFTER this post was made at roughly 17:30 PST.\n- The bad news: the mitigation involved migrating the submissions work to different machines, and we accidentally killed many running submissions in the process. I would recommend restarting any kernels that unexpectedly failed. Sorry for the inconvenience, but hopefully this leaves us on stable ground going into the last 24 hours of the competition.

", "votes": -2, "replies": [ { "id": 617267, "author_name": "ilovescience", "author_url": "", "post_date": "2019-09-04T00:40:20.293000", "content": "

The Kaggle Kernels and their many errors have made this a very bad experience. A week extension I think would be best for this competition.

", "votes": 0, "replies": [] }, { "id": 617271, "author_name": "oct_path", "author_url": "", "post_date": "2019-09-04T00:44:01.150000", "content": "

Thank you for your response. There is one thing I really want to confirm.

\n\n

The deadline will be never postponed, right?

\n\n

It is very important for the operation of the remaining few submissions, so I would be happy if you could answer clearly.

", "votes": 0, "replies": [] }, { "id": 617282, "author_name": "Sohier Dane", "author_url": "", "post_date": "2019-09-04T01:14:03.200000", "content": "

@octpath0302 good question. We generally avoid deadline extensions, but we had to review this case given problems in the last 24 hours of the competition. The deadline has been extended by 2 days per this post.

", "votes": 1, "replies": [] }, { "id": 617444, "author_name": "Psi", "author_url": "", "post_date": "2019-09-04T06:11:19.823000", "content": "

There were still 48 hours left.

", "votes": 1, "replies": [] } ] }, { "id": 617204, "author_name": "Psi", "author_url": "", "post_date": "2019-09-03T21:44:17.473000", "content": "

All my subs finished fine today btw.

", "votes": -1, "replies": [ { "id": 617223, "author_name": "empty", "author_url": "", "post_date": "2019-09-03T22:27:21.507000", "content": "

\"\"

", "votes": 12, "replies": [] }, { "id": 617447, "author_name": "Psi", "author_url": "", "post_date": "2019-09-04T06:18:42.503000", "content": "

The ones over night failed because they were manually stopped apparently as posted in the topic, but the ones yesterday worked fine.

", "votes": 0, "replies": [] } ] }, { "id": 620630, "author_name": "Nicolas H", "author_url": "", "post_date": "2019-09-07T18:54:23.643000", "content": "

I can't even open a kernel, \"this kernel is powered off\" banner keep showing ...

", "votes": 0, "replies": [] }, { "id": 619101, "author_name": "Ravi Kumar Vadapalli", "author_url": "", "post_date": "2019-09-05T20:23:50.380000", "content": "

@sohier, please let us know once the issue is fixed. Waiting for the resolution to press submit.

", "votes": 0, "replies": [] }, { "id": 618463, "author_name": "Basit Riaz Sheikh", "author_url": "", "post_date": "2019-09-05T08:27:35.460000", "content": "

I am unable to even open the kernel, let alone submit any thing. It's been like this since 24 hours at least.

", "votes": 0, "replies": [] }, { "id": 618454, "author_name": "Nicolas H", "author_url": "", "post_date": "2019-09-05T08:16:07.500000", "content": "

I'm facing this problem too, you should post this message in other competitions discussion sections to let people know that you're working on solving this issue.

", "votes": 0, "replies": [] }, { "id": 618062, "author_name": "Ram Muthiah", "author_url": "", "post_date": "2019-09-04T19:18:30.357000", "content": "

I keep getting \"ResourceExhaustedError\" just one hour after submitting.

", "votes": 0, "replies": [] }, { "id": 617953, "author_name": "Jaideep", "author_url": "", "post_date": "2019-09-04T16:50:35.023000", "content": "

i m unable to open the notebooks in view mode.. all thait is open is open but i cant open it the eixsting versions

", "votes": 0, "replies": [] }, { "id": 617766, "author_name": "oct_path", "author_url": "", "post_date": "2019-09-04T12:59:47.593000", "content": "

@sohier I report a suspicious behavior of kernels again.\nI submitted two kernels with almost the same content roughly simultaneously. (17:20GMT+9 and 17:40GMT+9)\nOne has completed processing (successful) in less than 2 hours, and the other has been processing for more than 4 hours now.\nAre there any abnormalities such as system load still high?\nThank you.

\n\n

EDITED: The kernel that took a long time has finally completed(success). It took at least twice as long.

", "votes": 0, "replies": [] }, { "id": 617750, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-09-04T12:43:12.130000", "content": "

Thanks @sohier.

\n\n

My kernel completed running in only 37 minutes yet my submission timed out with an error after 9hrs yesterday. I have re-submitted the same kernel after your post and will report back whether it succeeds or times out again.

", "votes": 0, "replies": [ { "id": 617986, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-09-04T17:34:12.063000", "content": "

@sohier , I am reporting back. My kernel submission ended after 9hrs with the following error:-

\n\n

> Submission CSV Not Found

\n\n

By the way my submission file is correctly named i.e. \"submission.csv\" as always.

", "votes": 0, "replies": [] }, { "id": 618129, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-09-04T22:16:22.970000", "content": "

@sohier , my other inference kernel that took 34 minutes to run also failed submission with a Kernel Timeout error. This problem is very much around.

", "votes": 0, "replies": [] } ] }, { "id": 617679, "author_name": "JBR", "author_url": "", "post_date": "2019-09-04T11:28:38.487000", "content": "

kernel that worked yesterday fails with exit code 134 on commit, and fails without explanation on submission

", "votes": 0, "replies": [] }, { "id": 617402, "author_name": "", "author_url": "", "post_date": "2019-09-04T05:04:42.170000", "content": "", "votes": 0, "replies": [] }, { "id": 617354, "author_name": "", "author_url": "", "post_date": "2019-09-04T03:35:54.567000", "content": "", "votes": 0, "replies": [] }, { "id": 617352, "author_name": "", "author_url": "", "post_date": "2019-09-04T03:32:40.127000", "content": "", "votes": 0, "replies": [] }, { "id": 616994, "author_name": "", "author_url": "", "post_date": "2019-09-03T16:37:47.417000", "content": "", "votes": 0, "replies": [] }, { "id": 620038, "author_name": "", "author_url": "", "post_date": "2019-09-06T22:43:54.120000", "content": "", "votes": 0, "replies": [] }, { "id": 618031, "author_name": "", "author_url": "", "post_date": "2019-09-04T18:42:35.837000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "616992": "Update 3 (13:45 PM PST September 5th):\n- Our engineers located and alleviated an I/O bottleneck that was affecting this competition. New submissions should run at the previously normal speeds, but you may see unexpected timeout errors on currently running submissions as we can't move them off of the congested resource without canceling the run entirely.\n\nUpdate 2 (12:20 PM PST September 4th):\n- The problem seems to be reoccurring; we're seeing a partial slowdown in kernel runtimes again and our engineers are reviewing the case. Most submissions are still completing successfully, but those that were close to the time limit may not get through at the moment.\n\nUpdate (September 3rd):\n- The good news: we have implemented a temporary measure now to mitigate the slowdown while we continue to investigate the root cause. Please let us know if you see any unexpected timeout errors with kernels submitted AFTER this post was made at roughly 17:30 PST.\n- The bad news: the mitigation involved migrating the submissions work to different machines, and we accidentally killed many running submissions in the process. I would recommend restarting any kernels that unexpectedly failed. Sorry for the inconvenience, but hopefully this leaves us on stable ground going into the last days of the competition. [We are extending the deadline by two days](https://www.kaggle.com/c/aptos2019-blindness-detection/discussion/107369#latest-617285) so impacted users have enough time to resubmit.\n\nOriginal post: As several of you have noticed, some kernels are running more slowly than usual. Our engineers are looking into the issue now. I'll post updates once we have more information.", "619790": "Kernel Timeout error again after 3 hours runing. Anyone else having the same problem?", "619109": "New submissions should run at normal speed again, please see the [update to the main post for details](https://www.kaggle.com/c/aptos2019-blindness-detection/discussion/107323).", "617276": "This situation is a mess, but to be honest, I don't want to have this competition extended because I'm getting tired and want to sleep.......", "617004": "Great to know that you are working on it. Please solve it asap.\nHave you thought about the possibility of giving us some extra time (some few more days) to compensate all the time we have wasted due to the different problems with commits and submittions (and also considering the 1 GPU limit)? ", "617245": "My kernel recently submitted was timed out without waiting 9 hours. I wonder why.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2208353%2F3727c4aca6848c33da8d3b3ed0506159%2Ftimeout_%202019-09-04%20085008.jpg?generation=1567554923612468&alt=media)", "618805": "It seems to work now but it is so slow: notebook running less than 1 hour is being submitted all 9 hours! ", "617003": "Is it possible to rerun that submissions without losing new attempts? ", "617199": "With this like the 5th time the kernels are not working in this kernels-only competition, I think Kaggle should consider the extending the deadline by a week as I discuss [here](https://www.kaggle.com/c/aptos2019-blindness-detection/discussion/105916#latest-609765)", "620724": "Same here\nKernel Timeout...", "619076": "my kernel eventually timed out.. i lost submission for the day and 1 more tomorrow..\nnot sure if it was thought initially that live run of kernel on huge private data could make things slow and would overload existing amount of resources..\nSuch kind of outages dont go well with expectation of smooth experience . \nParticipants have spent huge time in waiting for kernel to complete ,on the top of it on outages to get resolved.\n\n", "618819": "All subs running through... will report if something fails for me. Apparently, it is a problem for edge cases running long.", "618578": "submission kernel still running ...", "618812": "The issue still exists.. My kernels which usually takes around 2 hours to complete is still running since past 4 hours.", "618625": "Almost the same as the previous.\nI submitted two kernels with almost the same content roughly simultaneously. (15:19GMT+9 and 15:32GMT+9)\nOne has completed processing (successful) in 2 hours, but the other has been processing for more than 4 hours now.\n\nI would appreciate if you could tell us when the system is heavy.\nThank you.", "617795": "@sohier the problem hasn't gone. My kernel with commiting time ~20 min was being submitting during ~4 hours. Kernels with commiting time ~70 min just timed out (meanwhile almost the same ones were submitted earlier)", "617780": "I am still running kernels that take much more to complete than before - is the issue really fixed? I think they will time out...", "617234": "I lost a lot of submission in a few days.\nI will not be able to make the best submission.\nI would like to extend the due date a few days or increase the maximum number of submissions per day.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2930293%2Fae45ce748ad4a914f89e4eb1acb78e00%2F().png?generation=1567551903664904&alt=media)", "618603": "My kernel takes around 3 hours to run when i commit but when submitting it is still not completing with in 9 hours.", "617266": "Update:\n- The good news: we have implemented a temporary measure now to mitigate the slowdown while we continue to investigate the root cause. Please let us know if you see any unexpected timeout errors with kernels submitted AFTER this post was made at roughly 17:30 PST.\n- The bad news: the mitigation involved migrating the submissions work to different machines, and we accidentally killed many running submissions in the process. I would recommend restarting any kernels that unexpectedly failed. Sorry for the inconvenience, but hopefully this leaves us on stable ground going into the last 24 hours of the competition.\n", "617204": "All my subs finished fine today btw.", "620630": "I can't even open a kernel, \"this kernel is powered off\" banner keep showing ...", "619101": "@sohier, please let us know once the issue is fixed. Waiting for the resolution to press submit.", "618463": "I am unable to even open the kernel, let alone submit any thing. It's been like this since 24 hours at least.", "618454": "I'm facing this problem too, you should post this message in other competitions discussion sections to let people know that you're working on solving this issue.", "618062": "I keep getting \"ResourceExhaustedError\" just one hour after submitting. ", "617953": "i m unable to open the notebooks in view mode.. all thait is open is open but i cant open it the eixsting versions", "617766": "@sohier I report a suspicious behavior of kernels again.\nI submitted two kernels with almost the same content roughly simultaneously. (17:20GMT+9 and 17:40GMT+9)\nOne has completed processing (successful) in less than 2 hours, and the other has been processing for more than 4 hours now.\nAre there any abnormalities such as system load still high?\nThank you.\n\nEDITED: The kernel that took a long time has finally completed(success). It took at least twice as long.", "617750": "Thanks @sohier. \n\nMy kernel completed running in only 37 minutes yet my submission timed out with an error after 9hrs yesterday. I have re-submitted the same kernel after your post and will report back whether it succeeds or times out again.", "617679": "kernel that worked yesterday fails with exit code 134 on commit, and fails without explanation on submission", "617402": "I am still getting submission error ", "617354": "Thanks for informing give opportunity to rerun...\n\nBest", "617352": "Yes my last six submissions I observed unexpected timeouts... no possibility to be... even they run in half of the limit time... my older top versions were running almost two times slower.\n\nBest ", "616994": "Thanks, please pay attention at submission issue, this is critical for the competition I believe!", "620038": "", "618031": "" } }