Looks like TPUs availability has become challenge since yesterday. My kernel dint start because of TPU unavailability yesterday, my valuable TPU hours got lapsed . Today also its same, they are not available for run because of Traffic.
\nCan kaggle Team look into this. Generally availability resolves in 20-30 min but this time its taking too long.
\nProbable availability quota should get increased seeing more demand for TPU which is generally not the case in competitions.
Looks like TPUs availability has become challenge since yesterday. My kernel dint start because of TPU unavailability yesterday, my valuable TPU hours got lapsed . Today also its same, they are not available for run because of Traffic.
\nCan kaggle Team look into this. Generally availability resolves in 20-30 min but this time its taking too long.
\nProbable availability quota should get increased seeing more demand for TPU which is generally not the case in competitions.
seems there is a lot of traffic and all Kagglers are on heavy job:)
\nI'm on TPU queue for ~4h now… what is the longer waiting you have experienced so far?
waiting for 6 hours
", "rawMarkdown": "waiting for 6 hours", "votes": null }, { "id": "1525620", "postDate": "09/27/2021 14:29:15", "content": "thx for letting know, same here 6h+
", "rawMarkdown": "thx for letting know, same here 6h+", "votes": null }, { "id": "1525623", "postDate": "09/27/2021 14:32:08", "content": "I've faced multiple TPU start errors and heavy queuing yesterday.
\nNow waiting for 3h in the queue.
8+ hours …
", "rawMarkdown": "8+ hours ...", "votes": null }, { "id": "1525943", "postDate": "09/27/2021 18:30:40", "content": "looks like every one crazily brazing for century mark for no of models in ensemble :)
", "rawMarkdown": "looks like every one crazily brazing for century mark for no of models in ensemble :)", "votes": null }, { "id": "1526075", "postDate": "09/27/2021 20:40:54", "content": "UPD: it finally started, after nearly 9 hours in the queue.
", "rawMarkdown": "UPD: it finally started, after nearly 9 hours in the queue.", "votes": null }, { "id": "1526290", "postDate": "09/28/2021 02:26:46", "content": "Nothing started even after 12hrs of wait
", "rawMarkdown": "Nothing started even after 12hrs of wait", "votes": null }, { "id": "1526308", "postDate": "09/28/2021 02:57:18", "content": "Kaggle should prioritize TPU usage for nearly ending challenges! I can't even use TPU in the interactive mode.
", "rawMarkdown": "Kaggle should prioritize TPU usage for nearly ending challenges! I can't even use TPU in the interactive mode.", "votes": null }, { "id": "1526369", "postDate": "09/28/2021 04:19:14", "content": "After 17 Hrs of wait, notebook failed because of:
\n---> GCS_PATH = KaggleDatasets().get_gcs_path('xxxxxxxxxxxxxxxxxxxxxxxxxxx')
BackendError: Unexpected response from the service. Response: {'errors': ['Unauthenticated'], 'error': {'code': 16, 'details': []}, 'wasSuccessful': False}.
\nSame line of code that executed successfully at least 10 times ….
", "rawMarkdown": "After 17 Hrs of wait, notebook failed because of:\n---> GCS_PATH = KaggleDatasets().get_gcs_path('xxxxxxxxxxxxxxxxxxxxxxxxxxx')\n\nBackendError: Unexpected response from the service. Response: {'errors': ['Unauthenticated'], 'error': {'code': 16, 'details': []}, 'wasSuccessful': False}.\n\nSame line of code that executed successfully at least 10 times ....", "votes": null }, { "id": "1526604", "postDate": "09/28/2021 08:24:29", "content": "Even though I've waited over 13 hours, it was still queued.
\nSo I got angry and stopped committing.
… I am regret a little bit now.
", "rawMarkdown": "Even though I've waited over 13 hours, it was still queued.\nSo I got angry and stopped committing.\n\n... I am regret a little bit now.", "votes": null }, { "id": "1526677", "postDate": "09/28/2021 09:31:50", "content": "10h+ waiting and then an 'Unauthenticated' error which doesn't happen in the interactive mode..
", "rawMarkdown": "10h+ waiting and then an 'Unauthenticated' error which doesn't happen in the interactive mode..", "votes": null }, { "id": "1526687", "postDate": "09/28/2021 09:35:42", "content": "Copy of my post in the other thread. This is just economics. There is a resource with limited supply, it is in demand. If everyone tries to use their TPU quota at once, there will be queues. No different from trying to drive to work in the morning peak. Given that Santa Claus isn't going to plug in a bunch of extra TPUs for free, the only options would be for Kaggle to further ration the resource (say cut everyone's TPU quota to 20h for the last week) or else to put up with the queues.
", "rawMarkdown": "Copy of my post in [the other thread](https://www.kaggle.com/c/g2net-gravitational-wave-detection/discussion/274930). This is just economics. There is a resource with limited supply, it is in demand. If everyone tries to use their TPU quota at once, there will be queues. No different from trying to drive to work in the morning peak. Given that Santa Claus isn't going to plug in a bunch of extra TPUs for free, the only options would be for Kaggle to further ration the resource (say cut everyone's TPU quota to 20h for the last week) or else to put up with the queues.", "votes": null }, { "id": "1527014", "postDate": "09/28/2021 13:24:26", "content": "16 Hrs … still waiting. could not train a single model in whole day!!!
", "rawMarkdown": "16 Hrs ... still waiting. could not train a single model in whole day!!!", "votes": null }, { "id": "1528469", "postDate": "09/29/2021 16:51:27", "content": "After waiting 1 day. Then my notebook has the same error
", "rawMarkdown": "After waiting 1 day. Then my notebook has the same error", "votes": null }, { "id": "1559744", "postDate": "10/27/2021 07:11:12", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
Nothing started even after 12hrs of wait
", "votes": null, "replies": [] }, { "id": 1527014, "author_name": "raghaw", "author_url": "", "post_date": "09/28/2021 13:24:26", "content": "16 Hrs … still waiting. could not train a single model in whole day!!!
", "votes": null, "replies": [] } ] }, { "id": 1525432, "author_name": "imeintanis", "author_url": "", "post_date": "09/27/2021 12:30:43", "content": "seems there is a lot of traffic and all Kagglers are on heavy job:)
\nI'm on TPU queue for ~4h now… what is the longer waiting you have experienced so far?
waiting for 6 hours
", "votes": null, "replies": [ { "id": 1525620, "author_name": "imeintanis", "author_url": "", "post_date": "09/27/2021 14:29:15", "content": "thx for letting know, same here 6h+
", "votes": null, "replies": [] }, { "id": 1525682, "author_name": "raghaw", "author_url": "", "post_date": "09/27/2021 15:12:09", "content": "8+ hours …
", "votes": null, "replies": [] } ] }, { "id": 1525623, "author_name": "rolandw0w", "author_url": "", "post_date": "09/27/2021 14:32:08", "content": "I've faced multiple TPU start errors and heavy queuing yesterday.
\nNow waiting for 3h in the queue.
UPD: it finally started, after nearly 9 hours in the queue.
", "votes": null, "replies": [] } ] }, { "id": 1525943, "author_name": "jaideepvalani", "author_url": "", "post_date": "09/27/2021 18:30:40", "content": "looks like every one crazily brazing for century mark for no of models in ensemble :)
", "votes": null, "replies": [] }, { "id": 1526308, "author_name": "solosquad1999", "author_url": "", "post_date": "09/28/2021 02:57:18", "content": "Kaggle should prioritize TPU usage for nearly ending challenges! I can't even use TPU in the interactive mode.
", "votes": null, "replies": [] }, { "id": 1526369, "author_name": "raghaw", "author_url": "", "post_date": "09/28/2021 04:19:14", "content": "After 17 Hrs of wait, notebook failed because of:
\n---> GCS_PATH = KaggleDatasets().get_gcs_path('xxxxxxxxxxxxxxxxxxxxxxxxxxx')
BackendError: Unexpected response from the service. Response: {'errors': ['Unauthenticated'], 'error': {'code': 16, 'details': []}, 'wasSuccessful': False}.
\nSame line of code that executed successfully at least 10 times ….
", "votes": null, "replies": [] }, { "id": 1526604, "author_name": "yamanity", "author_url": "", "post_date": "09/28/2021 08:24:29", "content": "Even though I've waited over 13 hours, it was still queued.
\nSo I got angry and stopped committing.
… I am regret a little bit now.
", "votes": null, "replies": [] }, { "id": 1526677, "author_name": "hannes82", "author_url": "", "post_date": "09/28/2021 09:31:50", "content": "10h+ waiting and then an 'Unauthenticated' error which doesn't happen in the interactive mode..
", "votes": null, "replies": [] }, { "id": 1526687, "author_name": "jbomitchell", "author_url": "", "post_date": "09/28/2021 09:35:42", "content": "Copy of my post in the other thread. This is just economics. There is a resource with limited supply, it is in demand. If everyone tries to use their TPU quota at once, there will be queues. No different from trying to drive to work in the morning peak. Given that Santa Claus isn't going to plug in a bunch of extra TPUs for free, the only options would be for Kaggle to further ration the resource (say cut everyone's TPU quota to 20h for the last week) or else to put up with the queues.
", "votes": null, "replies": [] }, { "id": 1528469, "author_name": "khoaphng", "author_url": "", "post_date": "09/29/2021 16:51:27", "content": "After waiting 1 day. Then my notebook has the same error
", "votes": null, "replies": [] }, { "id": 1559744, "author_name": "zerafachris", "author_url": "", "post_date": "10/27/2021 07:11:12", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris