{ "id": 102547, "title": "Note on last night's submission outage", "url": "/competitions/aptos2019-blindness-detection/discussion/102547", "author_name": "Sohier Dane", "post_date": "2019-08-02T16:53:35.057000", "votes": -30, "comment_count": 86, "views": 0, "content": "

Between roughly 9 PM PST last night and 9 AM PST this morning we had an outage that may have incorrectly marked valid submissions as errors. We apologize for the inconvenience and advise resubmitting any errored submissions from this timeframe that you believe may have been successful.

", "messages": [ { "id": 591362, "postDate": "2019-08-03T14:50:49.033Z", "content": "

Can someone from Kaggle at least confirm you are aware of the issue and working on it? This really puts me off kernel competitions ..

", "rawMarkdown": "Can someone from Kaggle at least confirm you are aware of the issue and working on it? This really puts me off kernel competitions ..", "votes": 7 }, { "id": 591594, "postDate": "2019-08-04T01:30:12.203Z", "content": "

Wow.. This is very frustrating. If there is an issue with the kernels or submission system there should be some kind of warning all over the competition page. And once it over we should know that the issue is resolved....

", "rawMarkdown": "Wow.. This is very frustrating. If there is an issue with the kernels or submission system there should be some kind of warning all over the competition page. And once it over we should know that the issue is resolved.... ", "votes": 5 }, { "id": 591169, "postDate": "2019-08-03T08:57:53.143Z", "content": "

Outage(without users knowing it's not their fault!) + very limited info in error messages + deducted submissions - wasn't best way to start my first Kernel competition. This should change if Kaggle wants people to participate in Kernel only competitions.

", "rawMarkdown": "Outage(without users knowing it's not their fault!) + very limited info in error messages + deducted submissions - wasn't best way to start my first Kernel competition. This should change if Kaggle wants people to participate in Kernel only competitions.", "votes": 6, "replies": [ { "id": 591188, "postDate": "2019-08-03T09:33:14.457Z", "content": "

Reason why this bothers me so much is there is no way of verifying model performance but to submit for public score, since local validation score doesn't seem to correlate. How are we supposed to know if we can't submit?

", "rawMarkdown": "Reason why this bothers me so much is there is no way of verifying model performance but to submit for public score, since local validation score doesn't seem to correlate. How are we supposed to know if we can't submit?" } ] }, { "id": 591524, "postDate": "2019-08-03T21:19:38.427Z", "content": "

Looks like similar issues in the other kernel competition:\nhttps://www.kaggle.com/c/severstal-steel-defect-detection/discussion/102595#latest-591382\nhttps://www.kaggle.com/c/generative-dog-images/discussion/102586#latest-591445\n:(

", "rawMarkdown": "Looks like similar issues in the other kernel competition:\nhttps://www.kaggle.com/c/severstal-steel-defect-detection/discussion/102595#latest-591382\nhttps://www.kaggle.com/c/generative-dog-images/discussion/102586#latest-591445\n:(", "votes": 3 }, { "id": 591395, "postDate": "2019-08-03T16:08:38.400Z", "content": "

don`t use the GPU in the kernel! I think it may be the problem of GPU driver.

", "rawMarkdown": "don`t use the GPU in the kernel! I think it may be the problem of GPU driver.", "votes": 3, "replies": [ { "id": 591431, "postDate": "2019-08-03T17:16:14.730Z", "content": "

It indeed seems to fail as soon as it reaches GPU code

", "rawMarkdown": "It indeed seems to fail as soon as it reaches GPU code" }, { "id": 591915, "postDate": "2019-08-04T13:13:02.970Z", "content": "

its too late for me πŸ˜‘ ... I have waste all of my times yesterday and today....

", "rawMarkdown": "its too late for me πŸ˜‘ ... I have waste all of my times yesterday and today...." } ] }, { "id": 591310, "postDate": "2019-08-03T13:29:58.167Z", "content": "
    \n
  1. All submissions are failing since last night 8/2 9 PM EST. Please fix asap. \n2.Please reset submission counter. All submissions errored out and Submission is no longer allowed for the day. Cannot even test whether issue has been resolved.
    2. \n
", "rawMarkdown": "1. All submissions are failing since last night 8/2 9 PM EST. Please fix asap. \n2.Please reset submission counter. All submissions errored out and Submission is no longer allowed for the day. Cannot even test whether issue has been resolved.", "votes": 3 }, { "id": 591227, "postDate": "2019-08-03T10:54:06.980Z", "content": "

Part of competition......test of patience πŸ˜ƒ

", "rawMarkdown": "Part of competition......test of patience πŸ˜ƒ ", "votes": 3, "replies": [ { "id": 591328, "postDate": "2019-08-03T14:00:45.567Z", "content": "

kaggle patience grandmaster

", "rawMarkdown": "kaggle patience grandmaster", "votes": 2 }, { "id": 591409, "postDate": "2019-08-03T16:44:16.777Z", "content": "

πŸ˜ƒ

", "rawMarkdown": "πŸ˜ƒ " }, { "id": 591832, "postDate": "2019-08-04T10:15:30.337Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 591004, "postDate": "2019-08-03T02:30:29.520Z", "content": "

Kernel Threw Exception?

", "rawMarkdown": "Kernel Threw Exception?", "votes": 3 }, { "id": 590996, "postDate": "2019-08-03T02:14:59.393Z", "content": "

This is still not fixed. I submitted a previously successful kernel and it failed.

", "rawMarkdown": "This is still not fixed. I submitted a previously successful kernel and it failed.", "votes": 3 }, { "id": 590982, "postDate": "2019-08-03T01:09:37.897Z", "content": "

No, I think it is still exist , after 2 minutes , it still report me a 'kernel exception'

", "rawMarkdown": "No, I think it is still exist , after 2 minutes , it still report me a 'kernel exception'", "votes": 3 }, { "id": 591810, "postDate": "2019-08-04T09:32:11.610Z", "content": "

Guide on submitting here:

\n\n

https://www.kaggle.com/c/aptos2019-blindness-detection/discussion/102718#latest-591805

", "rawMarkdown": "Guide on submitting here:\n\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/102718#latest-591805", "votes": 4, "replies": [ { "id": 592193, "postDate": "2019-08-05T01:46:25.830Z", "content": "

I was also doing this but still faced submission error, thanks to your script I found out that it has to be int64 instead of just int!

", "rawMarkdown": "I was also doing this but still faced submission error, thanks to your script I found out that it has to be int64 instead of just int!" } ] }, { "id": 591809, "postDate": "2019-08-04T09:30:20.370Z", "content": "

\"\"

\n\n

Looks like the Organiser's have given up on this too.

", "rawMarkdown": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F908229%2F9da3bff0a6a6020e0b6d07760f23a434%2FScreenshot%20from%202019-08-04%2014-52-56.jpg?generation=1564910967175542&alt=media)\n\nLooks like the Organiser's have given up on this too. ", "votes": 4 }, { "id": 591153, "postDate": "2019-08-03T08:12:59.067Z", "content": "

Can anyone successfully submit, I am still seeing the \"Kernel Threw Exception\" error? Wasted last few hours due to this.

", "rawMarkdown": "Can anyone successfully submit, I am still seeing the \"Kernel Threw Exception\" error? Wasted last few hours due to this. ", "votes": 4, "replies": [ { "id": 591285, "postDate": "2019-08-03T12:53:46.193Z", "content": "

still \"Kernel Threw Exception\" error

", "rawMarkdown": "still \"Kernel Threw Exception\" error" } ] }, { "id": 591081, "postDate": "2019-08-03T05:33:23.760Z", "content": "

I am still facing this issue and I can't even submit since my daily quota is over.

\n\n

\"\"

", "rawMarkdown": "I am still facing this issue and I can't even submit since my daily quota is over. \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F654629%2F2088658d873f72796eec24724d385e20%2F2019-08-03%2008_31_54-APTOS%202019%20Blindness%20Detection%20_%20Kaggle.png?generation=1564810357424792&alt=media)\n", "votes": 4 }, { "id": 591069, "postDate": "2019-08-03T04:59:42.140Z", "content": "

\"\"

\n\n

Now it changed to this. πŸ‘€

", "rawMarkdown": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F489993%2F75a1eac426f63a7e244c97bb932ea128%2FScreenshot%20from%202019-08-03%2012-58-37.png?generation=1564808341419040&alt=media)\n\nNow it changed to this. πŸ‘€ ", "votes": 4 }, { "id": 591671, "postDate": "2019-08-04T05:24:35.710Z", "content": "

:((

", "rawMarkdown": ":((", "votes": 1 }, { "id": 591657, "postDate": "2019-08-04T04:48:13.417Z", "content": "

special prize ..... one who debugs the exception.... πŸ˜ƒ

", "rawMarkdown": "special prize ..... one who debugs the exception.... πŸ˜ƒ ", "votes": 1, "replies": [ { "id": 591660, "postDate": "2019-08-04T04:54:43.170Z", "content": "

Turn off GPU before committing?

", "rawMarkdown": "Turn off GPU before committing?", "votes": 2 }, { "id": 591850, "postDate": "2019-08-04T11:16:28.993Z", "content": "

πŸ‘

", "rawMarkdown": "πŸ‘ " } ] }, { "id": 591596, "postDate": "2019-08-04T01:36:08.333Z", "content": "

:(

", "rawMarkdown": ":(", "votes": 1 }, { "id": 591593, "postDate": "2019-08-04T01:22:13.953Z", "content": "

Has this been fixed ?

", "rawMarkdown": "Has this been fixed ?", "votes": 1 }, { "id": 591548, "postDate": "2019-08-03T22:35:32.600Z", "content": "

we can commit, but can’t submit. is this means that the each environment is different without test data difference?

", "rawMarkdown": "we can commit, but can’t submit. is this means that the each environment is different without test data difference?", "votes": 1 }, { "id": 591493, "postDate": "2019-08-03T19:44:42.203Z", "content": "

Hey im facing this issue even now, has this been fixed?

", "rawMarkdown": "Hey im facing this issue even now, has this been fixed?", "votes": 1 }, { "id": 591463, "postDate": "2019-08-03T18:04:04.043Z", "content": "

Still facing it. Hope team fixes it quickly.

", "rawMarkdown": "Still facing it. Hope team fixes it quickly.", "votes": 1 }, { "id": 591455, "postDate": "2019-08-03T17:47:57.863Z", "content": "

@sohier can you please reset our submission count once this issue is fixed? thank you

", "rawMarkdown": "@sohier can you please reset our submission count once this issue is fixed? thank you", "votes": 1 }, { "id": 591299, "postDate": "2019-08-03T13:06:52.757Z", "content": "

Thats the reason why my position in the leaderboard hasnt dropped ,cause no one is able to submit.πŸ˜„

", "rawMarkdown": "Thats the reason why my position in the leaderboard hasnt dropped ,cause no one is able to submit.πŸ˜„ ", "votes": 1 }, { "id": 591217, "postDate": "2019-08-03T10:40:05.037Z", "content": "

Same......was checking my code and cant really find anything suspicious.....maybe some update ?

", "rawMarkdown": "Same......was checking my code and cant really find anything suspicious.....maybe some update ?", "votes": 1 }, { "id": 591163, "postDate": "2019-08-03T08:35:20.447Z", "content": "

I have already wasted 4 chances of submission today.

", "rawMarkdown": "I have already wasted 4 chances of submission today. ", "votes": 1, "replies": [ { "id": 591166, "postDate": "2019-08-03T08:48:44.347Z", "content": "

Why don't you play a game and ignore all about it?

", "rawMarkdown": "Why don't you play a game and ignore all about it?", "votes": 1 }, { "id": 591605, "postDate": "2019-08-04T02:14:40.023Z", "content": "

Good idea. So I give myself one day off to LOL :)

", "rawMarkdown": "Good idea. So I give myself one day off to LOL :)" } ] }, { "id": 591157, "postDate": "2019-08-03T08:23:37.843Z", "content": "

Still facing the issue.

", "rawMarkdown": "Still facing the issue.", "votes": 1 }, { "id": 590926, "postDate": "2019-08-02T20:50:48.653Z", "content": "

Are we having an outage again? I've been waiting for results for hours. They are usually delivered in a matter of minutes...

", "rawMarkdown": "Are we having an outage again? I've been waiting for results for hours. They are usually delivered in a matter of minutes...", "votes": 1, "replies": [ { "id": 590928, "postDate": "2019-08-02T20:52:22.760Z", "content": "

I'm experiencing the same problem. My submission is stuck for about 3 hours now

", "rawMarkdown": "I'm experiencing the same problem. My submission is stuck for about 3 hours now", "votes": 1 }, { "id": 590962, "postDate": "2019-08-02T23:11:14.180Z", "content": "

Facing the same issue - 4 hours and counting. Will be keeping an eye on this thread for an update.

", "rawMarkdown": "Facing the same issue - 4 hours and counting. Will be keeping an eye on this thread for an update. ", "votes": 1 }, { "id": 590963, "postDate": "2019-08-02T23:12:03.393Z", "content": "

I'm getting kernel errors on all my submissions. Pretty sure the issue is ongoing.

", "rawMarkdown": "I'm getting kernel errors on all my submissions. Pretty sure the issue is ongoing.", "votes": 1 }, { "id": 591423, "postDate": "2019-08-03T17:04:29.947Z", "content": "

Interestingly the same submission is still loading while my subsequent submission threw a kernel error. Hoping this will be fixed soon!

", "rawMarkdown": "Interestingly the same submission is still loading while my subsequent submission threw a kernel error. Hoping this will be fixed soon!" } ] }, { "id": 590827, "postDate": "2019-08-02T17:03:54.750Z", "content": "

Is it possible to recover submissions lost due to that outage?😊

", "rawMarkdown": "Is it possible to recover submissions lost due to that outage?😊 ", "votes": 1 }, { "id": 591951, "postDate": "2019-08-04T14:53:06.823Z", "content": "

Looks like test data set have lots of images , much more than other image based competitions , making gpu kernels or infact docker to shoot exception on not able to handle load ......πŸ€” ....howzzz that as reason for this failure...

", "rawMarkdown": "Looks like test data set have lots of images , much more than other image based competitions , making gpu kernels or infact docker to shoot exception on not able to handle load ......πŸ€” ....howzzz that as reason for this failure...", "votes": 2 }, { "id": 591754, "postDate": "2019-08-04T07:54:01.227Z", "content": "

Same here, Kernel Throw Exception.

\n\n

Perhaps is a good time to clear your mind for a while to re-study the problem from another perspective.

", "rawMarkdown": "Same here, Kernel Throw Exception.\n\nPerhaps is a good time to clear your mind for a while to re-study the problem from another perspective.", "votes": 2 }, { "id": 591610, "postDate": "2019-08-04T02:43:11.713Z", "content": "

I guess Kaggle team is taking holidays like us in weekend.\nSo fixing this problem would be the beginning of next week.

", "rawMarkdown": "I guess Kaggle team is taking holidays like us in weekend.\nSo fixing this problem would be the beginning of next week.", "votes": 2, "replies": [ { "id": 591617, "postDate": "2019-08-04T03:13:44.013Z", "content": "

Yes,I have the same thoughts with you at beginning , but in the past two days it was Thursday and Friday in America

", "rawMarkdown": "Yes,I have the same thoughts with you at beginning , but in the past two days it was Thursday and Friday in America ", "votes": 3 }, { "id": 591739, "postDate": "2019-08-04T07:20:25.883Z", "content": "

It's not like the error wasn't already present the whole Friday.

", "rawMarkdown": "It's not like the error wasn't already present the whole Friday.", "votes": 1 }, { "id": 591909, "postDate": "2019-08-04T13:08:08.947Z", "content": "

It doesn't make sense. There should be someone in charge of solving problems that arise on weekends. After all, on weekends is when many (most?) of us have time to work on Kaggle challenges.

", "rawMarkdown": "It doesn't make sense. There should be someone in charge of solving problems that arise on weekends. After all, on weekends is when many (most?) of us have time to work on Kaggle challenges. ", "votes": 2 } ] }, { "id": 591520, "postDate": "2019-08-03T21:12:48.093Z", "content": "

Sad :(

", "rawMarkdown": "Sad :(", "votes": 2 }, { "id": 591349, "postDate": "2019-08-03T14:40:11.663Z", "content": "

This problem seems to occur in all kernel competitions. It throws an error after the kernel has been running for a while. If this happens frequently it is a very serious problem.

", "rawMarkdown": "This problem seems to occur in all kernel competitions. It throws an error after the kernel has been running for a while. If this happens frequently it is a very serious problem.", "votes": 2 }, { "id": 591346, "postDate": "2019-08-03T14:37:08.510Z", "content": "

Not fixed - my valid kernels just failed (Kernel Threw Exception) in submission phase...

", "rawMarkdown": "Not fixed - my valid kernels just failed (Kernel Threw Exception) in submission phase...", "votes": 2 }, { "id": 591341, "postDate": "2019-08-03T14:24:02.953Z", "content": "

This error is still happening @sohier

", "rawMarkdown": "This error is still happening @sohier ", "votes": 2, "replies": [ { "id": 591358, "postDate": "2019-08-03T14:47:39.180Z", "content": "

Looks like we got an upgrade from Kernel Error to the more informative: Kernel Threw Exception, or is it just me?

", "rawMarkdown": "Looks like we got an upgrade from Kernel Error to the more informative: Kernel Threw Exception, or is it just me?", "votes": 6 }, { "id": 591366, "postDate": "2019-08-03T14:59:10.007Z", "content": "

It happens to everyone, I think so

", "rawMarkdown": "It happens to everyone, I think so", "votes": 2 } ] }, { "id": 591064, "postDate": "2019-08-03T04:46:46.123Z", "content": "

It seems this is still not fixed.

", "rawMarkdown": "It seems this is still not fixed.", "votes": 2 }, { "id": 591059, "postDate": "2019-08-03T04:28:49.033Z", "content": "

Kernel Error - previous error\nKernel Threw Exception ..... new error... new outage

", "rawMarkdown": "Kernel Error - previous error\nKernel Threw Exception ..... new error... new outage ", "votes": 2 }, { "id": 590988, "postDate": "2019-08-03T01:40:13.960Z", "content": "

Kernel Threw Exception! Same code committed successfully, I even double checked to spot any type of error occurred. Please fix this Kaggle team. Thank you!

", "rawMarkdown": "Kernel Threw Exception! Same code committed successfully, I even double checked to spot any type of error occurred. Please fix this Kaggle team. Thank you!", "votes": 2, "replies": [ { "id": 590990, "postDate": "2019-08-03T01:44:07.433Z", "content": "

The same to you

", "rawMarkdown": "The same to you ", "votes": 3 }, { "id": 590999, "postDate": "2019-08-03T02:23:17.027Z", "content": "

I know it's frustrating. It even counts on my daily submissions.

", "rawMarkdown": "I know it's frustrating. It even counts on my daily submissions.", "votes": 3 } ] }, { "id": 591518, "postDate": "2019-08-03T21:01:24.613Z", "content": "

I'm also seeing \"Kernel Threw Exception\".

", "rawMarkdown": "I'm also seeing \"Kernel Threw Exception\"." }, { "id": 591696, "postDate": "2019-08-04T06:09:21.977Z", "content": "

Not using GPU is the key. It seems error was thrown when it tried to use GPU.

", "rawMarkdown": "Not using GPU is the key. It seems error was thrown when it tried to use GPU.", "votes": 1, "replies": [ { "id": 591697, "postDate": "2019-08-04T06:13:04.360Z", "content": "

If you perform training on kernel, it will take a long time and potentially reach time limit.

", "rawMarkdown": "If you perform training on kernel, it will take a long time and potentially reach time limit." }, { "id": 591747, "postDate": "2019-08-04T07:44:10.960Z", "content": "

If you turn off GPU , how can you train your model?

", "rawMarkdown": "If you turn off GPU , how can you train your model?" }, { "id": 591748, "postDate": "2019-08-04T07:45:07.183Z", "content": "

I just load the weights and predict...

", "rawMarkdown": "I just load the weights and predict...", "votes": 1 }, { "id": 591749, "postDate": "2019-08-04T07:45:55.870Z", "content": "

Used to train on kernel but I have to adapt this method because of this error

", "rawMarkdown": "Used to train on kernel but I have to adapt this method because of this error" } ] }, { "id": 590822, "postDate": "2019-08-02T16:53:35.057Z", "content": "

Between roughly 9 PM PST last night and 9 AM PST this morning we had an outage that may have incorrectly marked valid submissions as errors. We apologize for the inconvenience and advise resubmitting any errored submissions from this timeframe that you believe may have been successful.

", "rawMarkdown": "Between roughly 9 PM PST last night and 9 AM PST this morning we had an outage that may have incorrectly marked valid submissions as errors. We apologize for the inconvenience and advise resubmitting any errored submissions from this timeframe that you believe may have been successful.", "votes": -29 }, { "id": 592237, "postDate": "2019-08-05T03:50:23.450Z", "content": "

Hello Folks !! Issue is finally fixed...... Cheers !!🀘

", "rawMarkdown": "Hello Folks !! Issue is finally fixed...... Cheers !!🀘 " }, { "id": 592194, "postDate": "2019-08-05T01:47:36.930Z", "content": "

Latest update here: https://www.kaggle.com/c/aptos2019-blindness-detection/discussion/102773#latest-592184

", "rawMarkdown": "Latest update here: https://www.kaggle.com/c/aptos2019-blindness-detection/discussion/102773#latest-592184" }, { "id": 592052, "postDate": "2019-08-04T18:09:05.897Z", "content": "

Submission at this shld go fine ?

", "rawMarkdown": "Submission at this shld go fine ?" }, { "id": 591994, "postDate": "2019-08-04T15:40:24.580Z", "content": "

It would also be useful if you could extend the competition deadline to compensate for the lost submissions. They did that at the Generative Dogs competition

", "rawMarkdown": "It would also be useful if you could extend the competition deadline to compensate for the lost submissions. They did that at the Generative Dogs competition " }, { "id": 591950, "postDate": "2019-08-04T14:47:58.313Z", "content": "

I still got the problem. Was anyone able to submit normally, like before?

", "rawMarkdown": "I still got the problem. Was anyone able to submit normally, like before?" }, { "id": 591798, "postDate": "2019-08-04T09:15:16.867Z", "content": "

Still facing issue after two days

", "rawMarkdown": "Still facing issue after two days " }, { "id": 591753, "postDate": "2019-08-04T07:54:01.217Z", "content": "

kernel throwing exception for same kernel that was working fine till last 2 days.... is any one facing the same ?

", "rawMarkdown": "kernel throwing exception for same kernel that was working fine till last 2 days.... is any one facing the same ?" }, { "id": 591722, "postDate": "2019-08-04T06:39:20.180Z", "content": "

Now it's still fail to submmit,kernel throw exception.

", "rawMarkdown": "Now it's still fail to submmit,kernel throw exception." }, { "id": 591337, "postDate": "2019-08-03T14:15:25.547Z", "content": "

Still having a problem? When will it be fixed?

", "rawMarkdown": "Still having a problem? When will it be fixed?" }, { "id": 591276, "postDate": "2019-08-03T12:45:26.393Z", "content": "

Still face the issue! only me still meet the issue?

", "rawMarkdown": "Still face the issue! only me still meet the issue?\n", "replies": [ { "id": 591279, "postDate": "2019-08-03T12:48:30.327Z", "content": "

You are not alone, the world is on your side

", "rawMarkdown": "You are not alone, the world is on your side" }, { "id": 591281, "postDate": "2019-08-03T12:51:11.700Z", "content": "

So , the kaggle have not fix the problem? that wasted one change :D

", "rawMarkdown": "So , the kaggle have not fix the problem? that wasted one change :D" } ] }, { "id": 591210, "postDate": "2019-08-03T10:29:29.003Z", "content": "

Still getting an error: Kernel threw exception... seems like it is not fixed still

", "rawMarkdown": "Still getting an error: Kernel threw exception... seems like it is not fixed still", "replies": [ { "id": 591278, "postDate": "2019-08-03T12:46:37.923Z", "content": "

me too..

", "rawMarkdown": "me too..\n" } ] }, { "id": 590845, "postDate": "2019-08-02T17:40:04.670Z", "content": "

Hello there, now I get an error \"Submission CSV Not Found\", is there still some official bug here? Or I make some mistakes?

", "rawMarkdown": "Hello there, now I get an error \"Submission CSV Not Found\", is there still some official bug here? Or I make some mistakes? ", "replies": [ { "id": 590849, "postDate": "2019-08-02T17:49:16.600Z", "content": "

That sounds like an error on your end. The kernel ran far enough to check for a submission, and I see recent valid submissions on the leaderboard.

", "rawMarkdown": "That sounds like an error on your end. The kernel ran far enough to check for a submission, and I see recent valid submissions on the leaderboard.", "votes": -3 }, { "id": 590880, "postDate": "2019-08-02T18:39:31.267Z", "content": "

Thank you a lot, I'm going to recheck my code :)

", "rawMarkdown": "Thank you a lot, I'm going to recheck my code :)" } ] }, { "id": 591225, "postDate": "2019-08-03T10:53:11.133Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 591362, "author_name": "Tom Aindow", "author_url": "", "post_date": "2019-08-03T14:50:49.033000", "content": "

Can someone from Kaggle at least confirm you are aware of the issue and working on it? This really puts me off kernel competitions ..

", "votes": 7, "replies": [] }, { "id": 591594, "author_name": "DrHB", "author_url": "", "post_date": "2019-08-04T01:30:12.203000", "content": "

Wow.. This is very frustrating. If there is an issue with the kernels or submission system there should be some kind of warning all over the competition page. And once it over we should know that the issue is resolved....

", "votes": 5, "replies": [] }, { "id": 591169, "author_name": "Moiz Saifee", "author_url": "", "post_date": "2019-08-03T08:57:53.143000", "content": "

Outage(without users knowing it's not their fault!) + very limited info in error messages + deducted submissions - wasn't best way to start my first Kernel competition. This should change if Kaggle wants people to participate in Kernel only competitions.

", "votes": 6, "replies": [ { "id": 591188, "author_name": "Quan", "author_url": "", "post_date": "2019-08-03T09:33:14.457000", "content": "

Reason why this bothers me so much is there is no way of verifying model performance but to submit for public score, since local validation score doesn't seem to correlate. How are we supposed to know if we can't submit?

", "votes": 0, "replies": [] } ] }, { "id": 591524, "author_name": "Tom Aindow", "author_url": "", "post_date": "2019-08-03T21:19:38.427000", "content": "

Looks like similar issues in the other kernel competition:\nhttps://www.kaggle.com/c/severstal-steel-defect-detection/discussion/102595#latest-591382\nhttps://www.kaggle.com/c/generative-dog-images/discussion/102586#latest-591445\n:(

", "votes": 3, "replies": [] }, { "id": 591395, "author_name": "XiaodongHe", "author_url": "", "post_date": "2019-08-03T16:08:38.400000", "content": "

don`t use the GPU in the kernel! I think it may be the problem of GPU driver.

", "votes": 3, "replies": [ { "id": 591431, "author_name": "Psi", "author_url": "", "post_date": "2019-08-03T17:16:14.730000", "content": "

It indeed seems to fail as soon as it reaches GPU code

", "votes": 0, "replies": [] }, { "id": 591915, "author_name": "YILING X", "author_url": "", "post_date": "2019-08-04T13:13:02.970000", "content": "

its too late for me πŸ˜‘ ... I have waste all of my times yesterday and today....

", "votes": 0, "replies": [] } ] }, { "id": 591310, "author_name": "RajA", "author_url": "", "post_date": "2019-08-03T13:29:58.167000", "content": "
    \n
  1. All submissions are failing since last night 8/2 9 PM EST. Please fix asap. \n2.Please reset submission counter. All submissions errored out and Submission is no longer allowed for the day. Cannot even test whether issue has been resolved.
    2. \n
", "votes": 3, "replies": [] }, { "id": 591227, "author_name": "ABHIJIT", "author_url": "", "post_date": "2019-08-03T10:54:06.980000", "content": "

Part of competition......test of patience πŸ˜ƒ

", "votes": 3, "replies": [ { "id": 591328, "author_name": "Endi Niu", "author_url": "", "post_date": "2019-08-03T14:00:45.567000", "content": "

kaggle patience grandmaster

", "votes": 2, "replies": [] }, { "id": 591409, "author_name": "ABHIJIT", "author_url": "", "post_date": "2019-08-03T16:44:16.777000", "content": "

πŸ˜ƒ

", "votes": 0, "replies": [] }, { "id": 591832, "author_name": "", "author_url": "", "post_date": "2019-08-04T10:15:30.337000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 591004, "author_name": "", "author_url": "", "post_date": "2019-08-03T02:30:29.520000", "content": "

Kernel Threw Exception?

", "votes": 3, "replies": [] }, { "id": 590996, "author_name": "ManishYathnalli", "author_url": "", "post_date": "2019-08-03T02:14:59.393000", "content": "

This is still not fixed. I submitted a previously successful kernel and it failed.

", "votes": 3, "replies": [] }, { "id": 590982, "author_name": "ε“ˆε°”ηš„η§»εŠ¨εŸŽε ‘", "author_url": "", "post_date": "2019-08-03T01:09:37.897000", "content": "

No, I think it is still exist , after 2 minutes , it still report me a 'kernel exception'

", "votes": 3, "replies": [] }, { "id": 591810, "author_name": "Tom Aindow", "author_url": "", "post_date": "2019-08-04T09:32:11.610000", "content": "

Guide on submitting here:

\n\n

https://www.kaggle.com/c/aptos2019-blindness-detection/discussion/102718#latest-591805

", "votes": 4, "replies": [ { "id": 592193, "author_name": "ChinHuiC", "author_url": "", "post_date": "2019-08-05T01:46:25.830000", "content": "

I was also doing this but still faced submission error, thanks to your script I found out that it has to be int64 instead of just int!

", "votes": 0, "replies": [] } ] }, { "id": 591809, "author_name": "DecentMakeover", "author_url": "", "post_date": "2019-08-04T09:30:20.370000", "content": "

\"\"

\n\n

Looks like the Organiser's have given up on this too.

", "votes": 4, "replies": [] }, { "id": 591153, "author_name": "Moiz Saifee", "author_url": "", "post_date": "2019-08-03T08:12:59.067000", "content": "

Can anyone successfully submit, I am still seeing the \"Kernel Threw Exception\" error? Wasted last few hours due to this.

", "votes": 4, "replies": [ { "id": 591285, "author_name": "Endi Niu", "author_url": "", "post_date": "2019-08-03T12:53:46.193000", "content": "

still \"Kernel Threw Exception\" error

", "votes": 0, "replies": [] } ] }, { "id": 591081, "author_name": "QuantScientist", "author_url": "", "post_date": "2019-08-03T05:33:23.760000", "content": "

I am still facing this issue and I can't even submit since my daily quota is over.

\n\n

\"\"

", "votes": 4, "replies": [] }, { "id": 591069, "author_name": "sosuneko", "author_url": "", "post_date": "2019-08-03T04:59:42.140000", "content": "

\"\"

\n\n

Now it changed to this. πŸ‘€

", "votes": 4, "replies": [] }, { "id": 591671, "author_name": "Brian Lee", "author_url": "", "post_date": "2019-08-04T05:24:35.710000", "content": "

:((

", "votes": 1, "replies": [] }, { "id": 591657, "author_name": "ABHIJIT", "author_url": "", "post_date": "2019-08-04T04:48:13.417000", "content": "

special prize ..... one who debugs the exception.... πŸ˜ƒ

", "votes": 1, "replies": [ { "id": 591660, "author_name": "Quan", "author_url": "", "post_date": "2019-08-04T04:54:43.170000", "content": "

Turn off GPU before committing?

", "votes": 2, "replies": [] }, { "id": 591850, "author_name": "ABHIJIT", "author_url": "", "post_date": "2019-08-04T11:16:28.993000", "content": "

πŸ‘

", "votes": 0, "replies": [] } ] }, { "id": 591596, "author_name": "Wonho Song", "author_url": "", "post_date": "2019-08-04T01:36:08.333000", "content": "

:(

", "votes": 1, "replies": [] }, { "id": 591593, "author_name": "Wang Xinliang", "author_url": "", "post_date": "2019-08-04T01:22:13.953000", "content": "

Has this been fixed ?

", "votes": 1, "replies": [] }, { "id": 591548, "author_name": "hirune924", "author_url": "", "post_date": "2019-08-03T22:35:32.600000", "content": "

we can commit, but can’t submit. is this means that the each environment is different without test data difference?

", "votes": 1, "replies": [] }, { "id": 591493, "author_name": "DecentMakeover", "author_url": "", "post_date": "2019-08-03T19:44:42.203000", "content": "

Hey im facing this issue even now, has this been fixed?

", "votes": 1, "replies": [] }, { "id": 591463, "author_name": "shbhshs", "author_url": "", "post_date": "2019-08-03T18:04:04.043000", "content": "

Still facing it. Hope team fixes it quickly.

", "votes": 1, "replies": [] }, { "id": 591455, "author_name": "Verne", "author_url": "", "post_date": "2019-08-03T17:47:57.863000", "content": "

@sohier can you please reset our submission count once this issue is fixed? thank you

", "votes": 1, "replies": [] }, { "id": 591299, "author_name": "AmardeepGanguly", "author_url": "", "post_date": "2019-08-03T13:06:52.757000", "content": "

Thats the reason why my position in the leaderboard hasnt dropped ,cause no one is able to submit.πŸ˜„

", "votes": 1, "replies": [] }, { "id": 591217, "author_name": "Hao Wu", "author_url": "", "post_date": "2019-08-03T10:40:05.037000", "content": "

Same......was checking my code and cant really find anything suspicious.....maybe some update ?

", "votes": 1, "replies": [] }, { "id": 591163, "author_name": "", "author_url": "", "post_date": "2019-08-03T08:35:20.447000", "content": "

I have already wasted 4 chances of submission today.

", "votes": 1, "replies": [ { "id": 591166, "author_name": "ε“ˆε°”ηš„η§»εŠ¨εŸŽε ‘", "author_url": "", "post_date": "2019-08-03T08:48:44.347000", "content": "

Why don't you play a game and ignore all about it?

", "votes": 1, "replies": [] }, { "id": 591605, "author_name": "", "author_url": "", "post_date": "2019-08-04T02:14:40.023000", "content": "

Good idea. So I give myself one day off to LOL :)

", "votes": 0, "replies": [] } ] }, { "id": 591157, "author_name": "iamkhader", "author_url": "", "post_date": "2019-08-03T08:23:37.843000", "content": "

Still facing the issue.

", "votes": 1, "replies": [] }, { "id": 590926, "author_name": "Peter Nemeth", "author_url": "", "post_date": "2019-08-02T20:50:48.653000", "content": "

Are we having an outage again? I've been waiting for results for hours. They are usually delivered in a matter of minutes...

", "votes": 1, "replies": [ { "id": 590928, "author_name": "Rhuax", "author_url": "", "post_date": "2019-08-02T20:52:22.760000", "content": "

I'm experiencing the same problem. My submission is stuck for about 3 hours now

", "votes": 1, "replies": [] }, { "id": 590962, "author_name": "ChinHuiC", "author_url": "", "post_date": "2019-08-02T23:11:14.180000", "content": "

Facing the same issue - 4 hours and counting. Will be keeping an eye on this thread for an update.

", "votes": 1, "replies": [] }, { "id": 590963, "author_name": "Lex Toumbourou", "author_url": "", "post_date": "2019-08-02T23:12:03.393000", "content": "

I'm getting kernel errors on all my submissions. Pretty sure the issue is ongoing.

", "votes": 1, "replies": [] }, { "id": 591423, "author_name": "ChinHuiC", "author_url": "", "post_date": "2019-08-03T17:04:29.947000", "content": "

Interestingly the same submission is still loading while my subsequent submission threw a kernel error. Hoping this will be fixed soon!

", "votes": 0, "replies": [] } ] }, { "id": 590827, "author_name": "Sam Rodriguez", "author_url": "", "post_date": "2019-08-02T17:03:54.750000", "content": "

Is it possible to recover submissions lost due to that outage?😊

", "votes": 1, "replies": [] }, { "id": 591951, "author_name": "ABHIJIT", "author_url": "", "post_date": "2019-08-04T14:53:06.823000", "content": "

Looks like test data set have lots of images , much more than other image based competitions , making gpu kernels or infact docker to shoot exception on not able to handle load ......πŸ€” ....howzzz that as reason for this failure...

", "votes": 2, "replies": [] }, { "id": 591754, "author_name": "", "author_url": "", "post_date": "2019-08-04T07:54:01.227000", "content": "", "votes": 2, "replies": [] }, { "id": 591610, "author_name": "", "author_url": "", "post_date": "2019-08-04T02:43:11.713000", "content": "", "votes": 2, "replies": [ { "id": 591617, "author_name": "", "author_url": "", "post_date": "2019-08-04T03:13:44.013000", "content": "", "votes": 3, "replies": [] }, { "id": 591739, "author_name": "", "author_url": "", "post_date": "2019-08-04T07:20:25.883000", "content": "", "votes": 1, "replies": [] }, { "id": 591909, "author_name": "", "author_url": "", "post_date": "2019-08-04T13:08:08.947000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 591520, "author_name": "", "author_url": "", "post_date": "2019-08-03T21:12:48.093000", "content": "", "votes": 2, "replies": [] }, { "id": 591349, "author_name": "", "author_url": "", "post_date": "2019-08-03T14:40:11.663000", "content": "", "votes": 2, "replies": [] }, { "id": 591346, "author_name": "", "author_url": "", "post_date": "2019-08-03T14:37:08.510000", "content": "", "votes": 2, "replies": [] }, { "id": 591341, "author_name": "", "author_url": "", "post_date": "2019-08-03T14:24:02.953000", "content": "", "votes": 2, "replies": [ { "id": 591358, "author_name": "", "author_url": "", "post_date": "2019-08-03T14:47:39.180000", "content": "", "votes": 6, "replies": [] }, { "id": 591366, "author_name": "", "author_url": "", "post_date": "2019-08-03T14:59:10.007000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 591064, "author_name": "", "author_url": "", "post_date": "2019-08-03T04:46:46.123000", "content": "", "votes": 2, "replies": [] }, { "id": 591059, "author_name": "", "author_url": "", "post_date": "2019-08-03T04:28:49.033000", "content": "", "votes": 2, "replies": [] }, { "id": 590988, "author_name": "", "author_url": "", "post_date": "2019-08-03T01:40:13.960000", "content": "", "votes": 2, "replies": [ { "id": 590990, "author_name": "", "author_url": "", "post_date": "2019-08-03T01:44:07.433000", "content": "", "votes": 3, "replies": [] }, { "id": 590999, "author_name": "", "author_url": "", "post_date": "2019-08-03T02:23:17.027000", "content": "", "votes": 3, "replies": [] } ] }, { "id": 591518, "author_name": "", "author_url": "", "post_date": "2019-08-03T21:01:24.613000", "content": "", "votes": 0, "replies": [] }, { "id": 591696, "author_name": "", "author_url": "", "post_date": "2019-08-04T06:09:21.977000", "content": "", "votes": 1, "replies": [ { "id": 591697, "author_name": "", "author_url": "", "post_date": "2019-08-04T06:13:04.360000", "content": "", "votes": 0, "replies": [] }, { "id": 591747, "author_name": "", "author_url": "", "post_date": "2019-08-04T07:44:10.960000", "content": "", "votes": 0, "replies": [] }, { "id": 591748, "author_name": "", "author_url": "", "post_date": "2019-08-04T07:45:07.183000", "content": "", "votes": 1, "replies": [] }, { "id": 591749, "author_name": "", "author_url": "", "post_date": "2019-08-04T07:45:55.870000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 592237, "author_name": "", "author_url": "", "post_date": "2019-08-05T03:50:23.450000", "content": "", "votes": 0, "replies": [] }, { "id": 592194, "author_name": "", "author_url": "", "post_date": "2019-08-05T01:47:36.930000", "content": "", "votes": 0, "replies": [] }, { "id": 592052, "author_name": "", "author_url": "", "post_date": "2019-08-04T18:09:05.897000", "content": "", "votes": 0, "replies": [] }, { "id": 591994, "author_name": "", "author_url": "", "post_date": "2019-08-04T15:40:24.580000", "content": "", "votes": 0, "replies": [] }, { "id": 591950, "author_name": "", "author_url": "", "post_date": "2019-08-04T14:47:58.313000", "content": "", "votes": 0, "replies": [] }, { "id": 591798, "author_name": "", "author_url": "", "post_date": "2019-08-04T09:15:16.867000", "content": "", "votes": 0, "replies": [] }, { "id": 591753, "author_name": "", "author_url": "", "post_date": "2019-08-04T07:54:01.217000", "content": "", "votes": 0, "replies": [] }, { "id": 591722, "author_name": "", "author_url": "", "post_date": "2019-08-04T06:39:20.180000", "content": "", "votes": 0, "replies": [] }, { "id": 591337, "author_name": "", "author_url": "", "post_date": "2019-08-03T14:15:25.547000", "content": "", "votes": 0, "replies": [] }, { "id": 591276, "author_name": "", "author_url": "", "post_date": "2019-08-03T12:45:26.393000", "content": "", "votes": 0, "replies": [ { "id": 591279, "author_name": "", "author_url": "", "post_date": "2019-08-03T12:48:30.327000", "content": "", "votes": 0, "replies": [] }, { "id": 591281, "author_name": "", "author_url": "", "post_date": "2019-08-03T12:51:11.700000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 591210, "author_name": "", "author_url": "", "post_date": "2019-08-03T10:29:29.003000", "content": "", "votes": 0, "replies": [ { "id": 591278, "author_name": "", "author_url": "", "post_date": "2019-08-03T12:46:37.923000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 590845, "author_name": "", "author_url": "", "post_date": "2019-08-02T17:40:04.670000", "content": "", "votes": 0, "replies": [ { "id": 590849, "author_name": "", "author_url": "", "post_date": "2019-08-02T17:49:16.600000", "content": "", "votes": -3, "replies": [] }, { "id": 590880, "author_name": "", "author_url": "", "post_date": "2019-08-02T18:39:31.267000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 591225, "author_name": "", "author_url": "", "post_date": "2019-08-03T10:53:11.133000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "591362": "Can someone from Kaggle at least confirm you are aware of the issue and working on it? This really puts me off kernel competitions ..", "591594": "Wow.. This is very frustrating. If there is an issue with the kernels or submission system there should be some kind of warning all over the competition page. And once it over we should know that the issue is resolved.... ", "591169": "Outage(without users knowing it's not their fault!) + very limited info in error messages + deducted submissions - wasn't best way to start my first Kernel competition. This should change if Kaggle wants people to participate in Kernel only competitions.", "591524": "Looks like similar issues in the other kernel competition:\nhttps://www.kaggle.com/c/severstal-steel-defect-detection/discussion/102595#latest-591382\nhttps://www.kaggle.com/c/generative-dog-images/discussion/102586#latest-591445\n:(", "591395": "don`t use the GPU in the kernel! I think it may be the problem of GPU driver.", "591310": "1. All submissions are failing since last night 8/2 9 PM EST. Please fix asap. \n2.Please reset submission counter. All submissions errored out and Submission is no longer allowed for the day. Cannot even test whether issue has been resolved.", "591227": "Part of competition......test of patience πŸ˜ƒ ", "591004": "Kernel Threw Exception?", "590996": "This is still not fixed. I submitted a previously successful kernel and it failed.", "590982": "No, I think it is still exist , after 2 minutes , it still report me a 'kernel exception'", "591810": "Guide on submitting here:\n\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/102718#latest-591805", "591809": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F908229%2F9da3bff0a6a6020e0b6d07760f23a434%2FScreenshot%20from%202019-08-04%2014-52-56.jpg?generation=1564910967175542&alt=media)\n\nLooks like the Organiser's have given up on this too. ", "591153": "Can anyone successfully submit, I am still seeing the \"Kernel Threw Exception\" error? Wasted last few hours due to this. ", "591081": "I am still facing this issue and I can't even submit since my daily quota is over. \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F654629%2F2088658d873f72796eec24724d385e20%2F2019-08-03%2008_31_54-APTOS%202019%20Blindness%20Detection%20_%20Kaggle.png?generation=1564810357424792&alt=media)\n", "591069": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F489993%2F75a1eac426f63a7e244c97bb932ea128%2FScreenshot%20from%202019-08-03%2012-58-37.png?generation=1564808341419040&alt=media)\n\nNow it changed to this. πŸ‘€ ", "591671": ":((", "591657": "special prize ..... one who debugs the exception.... πŸ˜ƒ ", "591596": ":(", "591593": "Has this been fixed ?", "591548": "we can commit, but can’t submit. is this means that the each environment is different without test data difference?", "591493": "Hey im facing this issue even now, has this been fixed?", "591463": "Still facing it. Hope team fixes it quickly.", "591455": "@sohier can you please reset our submission count once this issue is fixed? thank you", "591299": "Thats the reason why my position in the leaderboard hasnt dropped ,cause no one is able to submit.πŸ˜„ ", "591217": "Same......was checking my code and cant really find anything suspicious.....maybe some update ?", "591163": "I have already wasted 4 chances of submission today. ", "591157": "Still facing the issue.", "590926": "Are we having an outage again? I've been waiting for results for hours. They are usually delivered in a matter of minutes...", "590827": "Is it possible to recover submissions lost due to that outage?😊 ", "591951": "Looks like test data set have lots of images , much more than other image based competitions , making gpu kernels or infact docker to shoot exception on not able to handle load ......πŸ€” ....howzzz that as reason for this failure...", "591754": "Same here, Kernel Throw Exception.\n\nPerhaps is a good time to clear your mind for a while to re-study the problem from another perspective.", "591610": "I guess Kaggle team is taking holidays like us in weekend.\nSo fixing this problem would be the beginning of next week.", "591520": "Sad :(", "591349": "This problem seems to occur in all kernel competitions. It throws an error after the kernel has been running for a while. If this happens frequently it is a very serious problem.", "591346": "Not fixed - my valid kernels just failed (Kernel Threw Exception) in submission phase...", "591341": "This error is still happening @sohier ", "591064": "It seems this is still not fixed.", "591059": "Kernel Error - previous error\nKernel Threw Exception ..... new error... new outage ", "590988": "Kernel Threw Exception! Same code committed successfully, I even double checked to spot any type of error occurred. Please fix this Kaggle team. Thank you!", "591518": "I'm also seeing \"Kernel Threw Exception\".", "591696": "Not using GPU is the key. It seems error was thrown when it tried to use GPU.", "590822": "Between roughly 9 PM PST last night and 9 AM PST this morning we had an outage that may have incorrectly marked valid submissions as errors. We apologize for the inconvenience and advise resubmitting any errored submissions from this timeframe that you believe may have been successful.", "592237": "Hello Folks !! Issue is finally fixed...... Cheers !!🀘 ", "592194": "Latest update here: https://www.kaggle.com/c/aptos2019-blindness-detection/discussion/102773#latest-592184", "592052": "Submission at this shld go fine ?", "591994": "It would also be useful if you could extend the competition deadline to compensate for the lost submissions. They did that at the Generative Dogs competition ", "591950": "I still got the problem. Was anyone able to submit normally, like before?", "591798": "Still facing issue after two days ", "591753": "kernel throwing exception for same kernel that was working fine till last 2 days.... is any one facing the same ?", "591722": "Now it's still fail to submmit,kernel throw exception.", "591337": "Still having a problem? When will it be fixed?", "591276": "Still face the issue! only me still meet the issue?\n", "591210": "Still getting an error: Kernel threw exception... seems like it is not fixed still", "590845": "Hello there, now I get an error \"Submission CSV Not Found\", is there still some official bug here? Or I make some mistakes? ", "591225": "" } }