![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F684740%2F8ec4755077e8b2cc20ea4cf061bf89e8%2FScreen%20Shot%202020-03-31%20at%2010.04.40%20PM.png?generation=1585663699012625&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F684740%2Ff5624b1e89509fd0b0091aed3744f25a%2FScreen%20Shot%202020-03-31%20at%2010.04.53%20PM.png?generation=1585663701848121&alt=media\")
\n
It actually takes 600+ seconds to run the model, and the logs also show about 600 seconds. But I have to wait 2900+ seconds to see the results. What happened?
", "messages": [ { "id": "792737", "postDate": "03/31/2020 14:15:06", "content": "\n
It actually takes 600+ seconds to run the model, and the logs also show about 600 seconds. But I have to wait 2900+ seconds to see the results. What happened?
", "rawMarkdown": "\n\n\nIt actually takes 600+ seconds to run the model, and the logs also show about 600 seconds. But I have to wait 2900+ seconds to see the results. What happened?", "votes": null }, { "id": "793184", "postDate": "03/31/2020 21:37:04", "content": "Thanks for reporting, there was an outage this morning with the service which tracks the timing of your notebook so it ended up taking ~40m before it picked it back up and finalized the timing for your notebook.
\n\nThat's since been fixed so future runs should show the correct time, I'm also going to fix up the timing code a bit so that even if the service goes down it gets the correct timing afterwards.
", "rawMarkdown": "Thanks for reporting, there was an outage this morning with the service which tracks the timing of your notebook so it ended up taking ~40m before it picked it back up and finalized the timing for your notebook.\n\nThat's since been fixed so future runs should show the correct time, I'm also going to fix up the timing code a bit so that even if the service goes down it gets the correct timing afterwards.", "votes": null }, { "id": "793381", "postDate": "04/01/2020 01:48:21", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": null }, { "id": "815848", "postDate": "04/21/2020 22:06:02", "content": "Hi @herbison , I have a similar problem and leads that the notebook failed due to time exceeding (The logs indicates that it takes 543 seconds while notebook time exceeded with 10800 )\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F619930%2F3ef3a6f678028304b738e71b60510861%2FRuntime_1.png?generation=1587506601745921&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F619930%2F37fb24144410308005b519bb507f094d%2FRuntime_2.png?generation=1587506643158928&alt=media\")
It looks like the system was actively monitoring it for 3hrs (10800s), my best guess is that the code got stuck in that 'waiting to connect' code at 543s and hung there for a few hours without printing more logs until it was killed by timeout. I'm adding some more logging code on our side to have more forensics if this happens again so I can confirm this was the case.
\n\nThe final 'Failed. Exited with code 137.' is actually a fake log line, it wasn't actually reported at 543s and is just tagged on with the exit code in the viewer, we should change that to properly show it.
", "rawMarkdown": "It looks like the system was actively monitoring it for 3hrs (10800s), my best guess is that the code got stuck in that 'waiting to connect' code at 543s and hung there for a few hours without printing more logs until it was killed by timeout. I'm adding some more logging code on our side to have more forensics if this happens again so I can confirm this was the case.\n\nThe final 'Failed. Exited with code 137.' is actually a fake log line, it wasn't actually reported at 543s and is just tagged on with the exit code in the viewer, we should change that to properly show it.", "votes": null } ], "comments": [ { "id": 793184, "author_name": "herbison", "author_url": "", "post_date": "03/31/2020 21:37:04", "content": "Thanks for reporting, there was an outage this morning with the service which tracks the timing of your notebook so it ended up taking ~40m before it picked it back up and finalized the timing for your notebook.
\n\nThat's since been fixed so future runs should show the correct time, I'm also going to fix up the timing code a bit so that even if the service goes down it gets the correct timing afterwards.
", "votes": null, "replies": [ { "id": 793381, "author_name": "newpiegyc", "author_url": "", "post_date": "04/01/2020 01:48:21", "content": "Thank you!
", "votes": null, "replies": [] } ] }, { "id": 815848, "author_name": "osamir", "author_url": "", "post_date": "04/21/2020 22:06:02", "content": "Hi @herbison , I have a similar problem and leads that the notebook failed due to time exceeding (The logs indicates that it takes 543 seconds while notebook time exceeded with 10800 )\n\n
It looks like the system was actively monitoring it for 3hrs (10800s), my best guess is that the code got stuck in that 'waiting to connect' code at 543s and hung there for a few hours without printing more logs until it was killed by timeout. I'm adding some more logging code on our side to have more forensics if this happens again so I can confirm this was the case.
\n\nThe final 'Failed. Exited with code 137.' is actually a fake log line, it wasn't actually reported at 543s and is just tagged on with the exit code in the viewer, we should change that to properly show it.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "792737": "\n\n\nIt actually takes 600+ seconds to run the model, and the logs also show about 600 seconds. But I have to wait 2900+ seconds to see the results. What happened?", "793184": "Thanks for reporting, there was an outage this morning with the service which tracks the timing of your notebook so it ended up taking ~40m before it picked it back up and finalized the timing for your notebook.\n\nThat's since been fixed so future runs should show the correct time, I'm also going to fix up the timing code a bit so that even if the service goes down it gets the correct timing afterwards.", "793381": "Thank you!", "815848": "Hi @herbison , I have a similar problem and leads that the notebook failed due to time exceeding (The logs indicates that it takes 543 seconds while notebook time exceeded with 10800 )\n\n", "815914": "It looks like the system was actively monitoring it for 3hrs (10800s), my best guess is that the code got stuck in that 'waiting to connect' code at 543s and hung there for a few hours without printing more logs until it was killed by timeout. I'm adding some more logging code on our side to have more forensics if this happens again so I can confirm this was the case.\n\nThe final 'Failed. Exited with code 137.' is actually a fake log line, it wasn't actually reported at 543s and is just tagged on with the exit code in the viewer, we should change that to properly show it." }, "source": "meta" }