Over the past one week or so, I have been facing the issue of models stopping training and then each time, I have to end the session and restart? How to fix this issue; here is my notebook for reference pytorch-lightning. I have also tried with pytorch
", "messages": [ { "id": "2869445", "postDate": "06/13/2024 05:32:11", "content": "Over the past one week or so, I have been facing the issue of models stopping training and then each time, I have to end the session and restart? How to fix this issue; here is my notebook for reference pytorch-lightning. I have also tried with pytorch
", "rawMarkdown": "Over the past one week or so, I have been facing the issue of models stopping training and then each time, I have to end the session and restart? How to fix this issue; here is my notebook for reference [pytorch-lightning](https://www.kaggle.com/code/samu2505/rsna-pytorchlightning-train). I have also tried with pytorch", "votes": null }, { "id": "2869452", "postDate": "06/13/2024 05:35:09", "content": "Is your Internet off? Check that once.
", "rawMarkdown": "Is your Internet off? Check that once.", "votes": null }, { "id": "2869484", "postDate": "06/13/2024 05:59:10", "content": "Yeah, I have been having some internet issues.
", "rawMarkdown": "Yeah, I have been having some internet issues.", "votes": null }, { "id": "2870292", "postDate": "06/13/2024 13:53:35", "content": "If you have intermittent internet outages, you can submit a version of the notebook with version type \"Save & Run All (Commit)\". The result (such as best model) can be saved to output folder for later retrieval.
", "rawMarkdown": "If you have intermittent internet outages, you can submit a version of the notebook with version type \"Save & Run All (Commit)\". The result (such as best model) can be saved to output folder for later retrieval.", "votes": null }, { "id": "2870396", "postDate": "06/13/2024 15:02:55", "content": "Bro and bro, I meant there's a section on the right side, which has Internet settings, that should be off for this competition. @samu2505 @coderrkj
", "rawMarkdown": "Bro and bro, I meant there's a section on the right side, which has Internet settings, that should be off for this competition. @samu2505 @coderrkj", "votes": null }, { "id": "2871060", "postDate": "06/14/2024 03:43:00", "content": "Oh i understand :). But i have also been having real internet connection issues over this past week
", "rawMarkdown": "Oh i understand :). But i have also been having real internet connection issues over this past week", "votes": null }, { "id": "2871061", "postDate": "06/14/2024 03:43:04", "content": "I haven't tried this, i will try. According to some people, the issues can be solved by lowering the learning rate and changing the random seed. Currently experimenting with them.
", "rawMarkdown": "I haven't tried this, i will try. According to some people, the issues can be solved by lowering the learning rate and changing the random seed. Currently experimenting with them.", "votes": null }, { "id": "2873608", "postDate": "06/15/2024 16:06:37", "content": "Do the GPU/CPU still run? In that case the model would be running but progress won't be shown l. Faced it several times, you my setup wandb to track training even in such instances.
\nEven if you don't setup wandb, the checkpoint will still be saved. You just need to turn on persistence tab and reload the kernel to access the checkpoint.
", "rawMarkdown": "Do the GPU/CPU still run? In that case the model would be running but progress won't be shown l. Faced it several times, you my setup wandb to track training even in such instances. \n\nEven if you don't setup wandb, the checkpoint will still be saved. You just need to turn on persistence tab and reload the kernel to access the checkpoint.", "votes": null }, { "id": "2874168", "postDate": "06/16/2024 05:06:32", "content": "That notebook is not meant to be submitted. The internet connection in that notebook is to upload the training losses, validation losses, and artefacts to wandb, to choose the best model.
", "rawMarkdown": "That notebook is not meant to be submitted. The internet connection in that notebook is to upload the training losses, validation losses, and artefacts to wandb, to choose the best model.", "votes": null }, { "id": "2874170", "postDate": "06/16/2024 05:14:19", "content": "Over the past one week or so, I have been facing the issue of models stopping training and then each time
\nI had used your notebook for a long time, but I have not faced any issues related to model stopping training.
\nCan you clarify, what you mean by \" models stopping training and then each time, I have to end the session and restart\"?
\nOptions:
\n1) Do you mean that the model is being auto-stopped even if the model seems to be improving on validation_epoch_loss ?
\n2) You are stopping the training at will and it is being stopped successfully but when you are trying to restart it with a different configuration, it freezes or fails and you have to restart the notebook.
Yes, but running it on the background using save and run works
", "rawMarkdown": "Yes, but running it on the background using save and run works", "votes": null } ], "comments": [ { "id": 2869452, "author_name": "shubhamcodez", "author_url": "", "post_date": "06/13/2024 05:35:09", "content": "Is your Internet off? Check that once.
", "votes": null, "replies": [ { "id": 2869484, "author_name": "samu2505", "author_url": "", "post_date": "06/13/2024 05:59:10", "content": "Yeah, I have been having some internet issues.
", "votes": null, "replies": [ { "id": 2870292, "author_name": "coderrkj", "author_url": "", "post_date": "06/13/2024 13:53:35", "content": "If you have intermittent internet outages, you can submit a version of the notebook with version type \"Save & Run All (Commit)\". The result (such as best model) can be saved to output folder for later retrieval.
", "votes": null, "replies": [ { "id": 2870396, "author_name": "shubhamcodez", "author_url": "", "post_date": "06/13/2024 15:02:55", "content": "Bro and bro, I meant there's a section on the right side, which has Internet settings, that should be off for this competition. @samu2505 @coderrkj
", "votes": null, "replies": [ { "id": 2871060, "author_name": "samu2505", "author_url": "", "post_date": "06/14/2024 03:43:00", "content": "Oh i understand :). But i have also been having real internet connection issues over this past week
", "votes": null, "replies": [] }, { "id": 2874168, "author_name": "devsya", "author_url": "", "post_date": "06/16/2024 05:06:32", "content": "That notebook is not meant to be submitted. The internet connection in that notebook is to upload the training losses, validation losses, and artefacts to wandb, to choose the best model.
", "votes": null, "replies": [] } ] }, { "id": 2871061, "author_name": "samu2505", "author_url": "", "post_date": "06/14/2024 03:43:04", "content": "I haven't tried this, i will try. According to some people, the issues can be solved by lowering the learning rate and changing the random seed. Currently experimenting with them.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2873608, "author_name": "sandeepmnair", "author_url": "", "post_date": "06/15/2024 16:06:37", "content": "Do the GPU/CPU still run? In that case the model would be running but progress won't be shown l. Faced it several times, you my setup wandb to track training even in such instances.
\nEven if you don't setup wandb, the checkpoint will still be saved. You just need to turn on persistence tab and reload the kernel to access the checkpoint.
", "votes": null, "replies": [] }, { "id": 2874170, "author_name": "devsya", "author_url": "", "post_date": "06/16/2024 05:14:19", "content": "Over the past one week or so, I have been facing the issue of models stopping training and then each time
\nI had used your notebook for a long time, but I have not faced any issues related to model stopping training.
\nCan you clarify, what you mean by \" models stopping training and then each time, I have to end the session and restart\"?
\nOptions:
\n1) Do you mean that the model is being auto-stopped even if the model seems to be improving on validation_epoch_loss ?
\n2) You are stopping the training at will and it is being stopped successfully but when you are trying to restart it with a different configuration, it freezes or fails and you have to restart the notebook.
Yes, but running it on the background using save and run works
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2869445": "Over the past one week or so, I have been facing the issue of models stopping training and then each time, I have to end the session and restart? How to fix this issue; here is my notebook for reference [pytorch-lightning](https://www.kaggle.com/code/samu2505/rsna-pytorchlightning-train). I have also tried with pytorch", "2869452": "Is your Internet off? Check that once.", "2869484": "Yeah, I have been having some internet issues.", "2870292": "If you have intermittent internet outages, you can submit a version of the notebook with version type \"Save & Run All (Commit)\". The result (such as best model) can be saved to output folder for later retrieval.", "2870396": "Bro and bro, I meant there's a section on the right side, which has Internet settings, that should be off for this competition. @samu2505 @coderrkj", "2871060": "Oh i understand :). But i have also been having real internet connection issues over this past week", "2871061": "I haven't tried this, i will try. According to some people, the issues can be solved by lowering the learning rate and changing the random seed. Currently experimenting with them.", "2873608": "Do the GPU/CPU still run? In that case the model would be running but progress won't be shown l. Faced it several times, you my setup wandb to track training even in such instances. \n\nEven if you don't setup wandb, the checkpoint will still be saved. You just need to turn on persistence tab and reload the kernel to access the checkpoint.", "2874168": "That notebook is not meant to be submitted. The internet connection in that notebook is to upload the training losses, validation losses, and artefacts to wandb, to choose the best model.", "2874170": "Over the past one week or so, I have been facing the issue of models stopping training and then each time\n\nI had used your notebook for a long time, but I have not faced any issues related to model stopping training. \n\nCan you clarify, what you mean by \" models stopping training and then each time, I have to end the session and restart\"?\nOptions:\n1) Do you mean that the model is being auto-stopped even if the model seems to be improving on validation_epoch_loss ?\n2) You are stopping the training at will and it is being stopped successfully but when you are trying to restart it with a different configuration, it freezes or fails and you have to restart the notebook.", "2875392": "Yes, but running it on the background using save and run works" }, "source": "meta" }