With two Google Landmark competitions ending on the same day, I guess the load on TPU resources is very high. My TPU training kernel is in queue for training for last 5 hours. Anyone else is facing this issue? Few days back we had issues where kernel editor was down for more than 12+ hours in a day…please plan your TPU commits in advance to leave enough time for training and experimentation.
\nGood luck everyone!!! Four more days to go ✌️
", "messages": [ { "id": "1525605", "postDate": "09/27/2021 14:21:29", "content": "With two Google Landmark competitions ending on the same day, I guess the load on TPU resources is very high. My TPU training kernel is in queue for training for last 5 hours. Anyone else is facing this issue? Few days back we had issues where kernel editor was down for more than 12+ hours in a day…please plan your TPU commits in advance to leave enough time for training and experimentation.
\nGood luck everyone!!! Four more days to go ✌️
", "rawMarkdown": "With two Google Landmark competitions ending on the same day, I guess the load on TPU resources is very high. My TPU training kernel is in queue for training for last 5 hours. Anyone else is facing this issue? Few days back we had issues where kernel editor was down for more than 12+ hours in a day...please plan your TPU commits in advance to leave enough time for training and experimentation.\n\nGood luck everyone!!! Four more days to go ✌️", "votes": null }, { "id": "1525939", "postDate": "09/27/2021 18:19:23", "content": "Update: the TPU training commit is still in queue for 8+ hours 😭😭
", "rawMarkdown": "Update: the TPU training commit is still in queue for 8+ hours 😭😭", "votes": null }, { "id": "1525946", "postDate": "09/27/2021 18:33:37", "content": "I have the same issue, but after ~10 hours it is running. Soon people will run out of TPU hours and the problem will solve itself.
", "rawMarkdown": "I have the same issue, but after ~10 hours it is running. Soon people will run out of TPU hours and the problem will solve itself.", "votes": null }, { "id": "1526312", "postDate": "09/28/2021 03:00:08", "content": "my first commit ran after 9 hours and has finished running…the one made after that is in queue for 11 hours as of now…😅
", "rawMarkdown": "my first commit ran after 9 hours and has finished running...the one made after that is in queue for 11 hours as of now...😅", "votes": null }, { "id": "1527991", "postDate": "09/29/2021 09:02:00", "content": "Do you have any progress so far? I currently have a notebook stuck at \"starting\" for 16 hours now…
", "rawMarkdown": "Do you have any progress so far? I currently have a notebook stuck at \"starting\" for 16 hours now...", "votes": null }, { "id": "1528025", "postDate": "09/29/2021 09:49:45", "content": "mine ran after 13 hours of wait time…only to see a memory related error 3 hours into the run…😂
", "rawMarkdown": "mine ran after 13 hours of wait time...only to see a memory related error 3 hours into the run...😂", "votes": null }, { "id": "1528164", "postDate": "09/29/2021 12:25:29", "content": "Try to open it in edit mode and run the training, no queue(anyway much shorter) in that mode, just remember to save and download the training in time as it only is temp saved while running the session, and also remember to keep the session alive :)
", "rawMarkdown": "Try to open it in edit mode and run the training, no queue(anyway much shorter) in that mode, just remember to save and download the training in time as it only is temp saved while running the session, and also remember to keep the session alive :)", "votes": null }, { "id": "1528258", "postDate": "09/29/2021 14:02:21", "content": "keeping the session alive is the problem for me as I hardly get any time between work and generally leave the training job submitted overnight…😅 that's how I have managed for the last month or so…well..i guess one can't have everything 😁
", "rawMarkdown": "keeping the session alive is the problem for me as I hardly get any time between work and generally leave the training job submitted overnight...😅 that's how I have managed for the last month or so...well..i guess one can't have everything 😁", "votes": null }, { "id": "1528263", "postDate": "09/29/2021 14:12:34", "content": "What, do you sleep now when its closing day for the Kaggle competitions, what a risk! ;) To be serious I had session running for ~6 hours without touching it, maybe luck so cant promising it will work again, and 6-9h TPUv3 is worth gold these days so maybe better be safe than sorry I guess and wait for the queue line to run through :)
", "rawMarkdown": "What, do you sleep now when its closing day for the Kaggle competitions, what a risk! ;) To be serious I had session running for ~6 hours without touching it, maybe luck so cant promising it will work again, and 6-9h TPUv3 is worth gold these days so maybe better be safe than sorry I guess and wait for the queue line to run through :)", "votes": null }, { "id": "1528268", "postDate": "09/29/2021 14:17:58", "content": "My country timezone doesn't help me much…the competitions generally end at 530AM in the morning for me…so yes I have to go to sleep with a lot of suspense… especially on last day of competition…many times had heartbreak in the morning due to fall in positions…but sometimes it has been good as well…all in all that's what makes Kaggle fun…i am almost addicted to it 🤪
", "rawMarkdown": "My country timezone doesn't help me much...the competitions generally end at 530AM in the morning for me...so yes I have to go to sleep with a lot of suspense... especially on last day of competition...many times had heartbreak in the morning due to fall in positions...but sometimes it has been good as well...all in all that's what makes Kaggle fun...i am almost addicted to it 🤪", "votes": null } ], "comments": [ { "id": 1525939, "author_name": "sandy1112", "author_url": "", "post_date": "09/27/2021 18:19:23", "content": "Update: the TPU training commit is still in queue for 8+ hours 😭😭
", "votes": null, "replies": [] }, { "id": 1525946, "author_name": "markwijkhuizen", "author_url": "", "post_date": "09/27/2021 18:33:37", "content": "I have the same issue, but after ~10 hours it is running. Soon people will run out of TPU hours and the problem will solve itself.
", "votes": null, "replies": [ { "id": 1526312, "author_name": "sandy1112", "author_url": "", "post_date": "09/28/2021 03:00:08", "content": "my first commit ran after 9 hours and has finished running…the one made after that is in queue for 11 hours as of now…😅
", "votes": null, "replies": [] }, { "id": 1527991, "author_name": "markwijkhuizen", "author_url": "", "post_date": "09/29/2021 09:02:00", "content": "Do you have any progress so far? I currently have a notebook stuck at \"starting\" for 16 hours now…
", "votes": null, "replies": [] }, { "id": 1528025, "author_name": "sandy1112", "author_url": "", "post_date": "09/29/2021 09:49:45", "content": "mine ran after 13 hours of wait time…only to see a memory related error 3 hours into the run…😂
", "votes": null, "replies": [] } ] }, { "id": 1528164, "author_name": "kirderf", "author_url": "", "post_date": "09/29/2021 12:25:29", "content": "Try to open it in edit mode and run the training, no queue(anyway much shorter) in that mode, just remember to save and download the training in time as it only is temp saved while running the session, and also remember to keep the session alive :)
", "votes": null, "replies": [ { "id": 1528258, "author_name": "sandy1112", "author_url": "", "post_date": "09/29/2021 14:02:21", "content": "keeping the session alive is the problem for me as I hardly get any time between work and generally leave the training job submitted overnight…😅 that's how I have managed for the last month or so…well..i guess one can't have everything 😁
", "votes": null, "replies": [] }, { "id": 1528263, "author_name": "kirderf", "author_url": "", "post_date": "09/29/2021 14:12:34", "content": "What, do you sleep now when its closing day for the Kaggle competitions, what a risk! ;) To be serious I had session running for ~6 hours without touching it, maybe luck so cant promising it will work again, and 6-9h TPUv3 is worth gold these days so maybe better be safe than sorry I guess and wait for the queue line to run through :)
", "votes": null, "replies": [] }, { "id": 1528268, "author_name": "sandy1112", "author_url": "", "post_date": "09/29/2021 14:17:58", "content": "My country timezone doesn't help me much…the competitions generally end at 530AM in the morning for me…so yes I have to go to sleep with a lot of suspense… especially on last day of competition…many times had heartbreak in the morning due to fall in positions…but sometimes it has been good as well…all in all that's what makes Kaggle fun…i am almost addicted to it 🤪
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1525605": "With two Google Landmark competitions ending on the same day, I guess the load on TPU resources is very high. My TPU training kernel is in queue for training for last 5 hours. Anyone else is facing this issue? Few days back we had issues where kernel editor was down for more than 12+ hours in a day...please plan your TPU commits in advance to leave enough time for training and experimentation.\n\nGood luck everyone!!! Four more days to go ✌️", "1525939": "Update: the TPU training commit is still in queue for 8+ hours 😭😭", "1525946": "I have the same issue, but after ~10 hours it is running. Soon people will run out of TPU hours and the problem will solve itself.", "1526312": "my first commit ran after 9 hours and has finished running...the one made after that is in queue for 11 hours as of now...😅", "1527991": "Do you have any progress so far? I currently have a notebook stuck at \"starting\" for 16 hours now...", "1528025": "mine ran after 13 hours of wait time...only to see a memory related error 3 hours into the run...😂", "1528164": "Try to open it in edit mode and run the training, no queue(anyway much shorter) in that mode, just remember to save and download the training in time as it only is temp saved while running the session, and also remember to keep the session alive :)", "1528258": "keeping the session alive is the problem for me as I hardly get any time between work and generally leave the training job submitted overnight...😅 that's how I have managed for the last month or so...well..i guess one can't have everything 😁", "1528263": "What, do you sleep now when its closing day for the Kaggle competitions, what a risk! ;) To be serious I had session running for ~6 hours without touching it, maybe luck so cant promising it will work again, and 6-9h TPUv3 is worth gold these days so maybe better be safe than sorry I guess and wait for the queue line to run through :)", "1528268": "My country timezone doesn't help me much...the competitions generally end at 530AM in the morning for me...so yes I have to go to sleep with a lot of suspense... especially on last day of competition...many times had heartbreak in the morning due to fall in positions...but sometimes it has been good as well...all in all that's what makes Kaggle fun...i am almost addicted to it 🤪" }, "source": "meta" }