Hey there, I have been trying to train my model on TPU and it suddenly stops after a few epochs sometimes. Did somebody encounter the same problem?
\nNote : I ran the notebooks and it did run a couple of times and resulted in successful submissions as well but sometimes it abruptly stops after a few epochs (no error or any pop-up notification). I believe it's happening after an inactivity of a couple of seconds on the screen. But, that should not be the case. I'd be glad if somebody could guide me if I am doing something wrong. Already wasted a lot of TPU quota for the week 😢
Hey there, I have been trying to train my model on TPU and it suddenly stops after a few epochs sometimes. Did somebody encounter the same problem?
\nNote : I ran the notebooks and it did run a couple of times and resulted in successful submissions as well but sometimes it abruptly stops after a few epochs (no error or any pop-up notification). I believe it's happening after an inactivity of a couple of seconds on the screen. But, that should not be the case. I'd be glad if somebody could guide me if I am doing something wrong. Already wasted a lot of TPU quota for the week 😢
It really helps if you go to the version that you ran to identify the specific error. Without knowing what error arose, it is very difficult to diagnose this
", "rawMarkdown": "It really helps if you go to the version that you ran to identify the specific error. Without knowing what error arose, it is very difficult to diagnose this", "votes": null }, { "id": "2796726", "postDate": "05/06/2024 11:35:08", "content": "Sometimes you have to explicitly open the version that threw an error btw. Look at your version history on the notebook.
", "rawMarkdown": "Sometimes you have to explicitly open the version that threw an error btw. Look at your version history on the notebook.", "votes": null }, { "id": "2805561", "postDate": "05/10/2024 15:54:32", "content": "I believe you're dealing with interactive mode issues, correct? No issues if you 'save and run all'?
\nMy most likely guess is there's client-side issues I hit semi-often, and haven't figured out what makes it more likely vs less likely or root cause, but sometimes you get a '*' instead of a spinning circle, and if so it's iffy whether you'll get the output from the running operation. I usually refresh the screen to get past it, but you lose whatever was running, so that's not great solution for long training.
\nConsider using save and run all for all but the quickest training experiments?
", "rawMarkdown": "I believe you're dealing with interactive mode issues, correct? No issues if you 'save and run all'?\n\nMy most likely guess is there's client-side issues I hit semi-often, and haven't figured out what makes it more likely vs less likely or root cause, but sometimes you get a '*' instead of a spinning circle, and if so it's iffy whether you'll get the output from the running operation. I usually refresh the screen to get past it, but you lose whatever was running, so that's not great solution for long training.\n\nConsider using save and run all for all but the quickest training experiments?", "votes": null }, { "id": "2806280", "postDate": "05/11/2024 02:22:21", "content": "I am not sure if we are talking about the same issue, but when my interactive session freezes and I see the '*' it typically helps to run an empty print() statement in a fresh line to unfreeze the session.
Hey @roberthatch , thanks for answering. You're right this is what happens sometimes.
\nFor me, I discovered playing with learning rate might have caused the issue. Because, changing it back to the original one, my same model worked, it's strange but still I can't say certainly.
Hey @frenio Thanks for the trick. I tried it and it certainly did work, lol xP
", "rawMarkdown": "Hey @frenio Thanks for the trick. I tried it and it certainly did work, lol xP", "votes": null }, { "id": "2810851", "postDate": "05/13/2024 13:30:53", "content": "Great! I'm glad it worked for you.
", "rawMarkdown": "Great! I'm glad it worked for you.", "votes": null } ], "comments": [ { "id": 2796718, "author_name": "chemdatafarmer", "author_url": "", "post_date": "05/06/2024 11:33:27", "content": "It really helps if you go to the version that you ran to identify the specific error. Without knowing what error arose, it is very difficult to diagnose this
", "votes": null, "replies": [ { "id": 2796726, "author_name": "chemdatafarmer", "author_url": "", "post_date": "05/06/2024 11:35:08", "content": "Sometimes you have to explicitly open the version that threw an error btw. Look at your version history on the notebook.
", "votes": null, "replies": [] } ] }, { "id": 2805561, "author_name": "roberthatch", "author_url": "", "post_date": "05/10/2024 15:54:32", "content": "I believe you're dealing with interactive mode issues, correct? No issues if you 'save and run all'?
\nMy most likely guess is there's client-side issues I hit semi-often, and haven't figured out what makes it more likely vs less likely or root cause, but sometimes you get a '*' instead of a spinning circle, and if so it's iffy whether you'll get the output from the running operation. I usually refresh the screen to get past it, but you lose whatever was running, so that's not great solution for long training.
\nConsider using save and run all for all but the quickest training experiments?
", "votes": null, "replies": [ { "id": 2806280, "author_name": "frenio", "author_url": "", "post_date": "05/11/2024 02:22:21", "content": "I am not sure if we are talking about the same issue, but when my interactive session freezes and I see the '*' it typically helps to run an empty print() statement in a fresh line to unfreeze the session.
Hey @frenio Thanks for the trick. I tried it and it certainly did work, lol xP
", "votes": null, "replies": [ { "id": 2810851, "author_name": "frenio", "author_url": "", "post_date": "05/13/2024 13:30:53", "content": "Great! I'm glad it worked for you.
", "votes": null, "replies": [] } ] } ] }, { "id": 2810788, "author_name": "ahsuna123", "author_url": "", "post_date": "05/13/2024 12:57:19", "content": "Hey @roberthatch , thanks for answering. You're right this is what happens sometimes.
\nFor me, I discovered playing with learning rate might have caused the issue. Because, changing it back to the original one, my same model worked, it's strange but still I can't say certainly.