Now, I am unsure if this has been answered, I remember my buddy telling me that anecdotally, one can \"reset\" the LR a little when one resumes training from a model's checkpoint.
\nIt is a bit counter-intuitive to me, but say I trained a model for 16 epochs, with a custom scheduler, say OneCycleLr + Adam or something, then when you resume training, should we reset the initial learning rate? I would think resetting is counter-intuitive as the purpose of the learning rate is to tune it such that your model can slowly converge to the minima (global one if the function is convex).
So the question is:
\nShould I reset the learning rate when resume training, if yes, reset to what?
If we should not reset, what is a good way to \"extract\" the last learning rate, as some scheduler depends on factors like epochs…
You have to save your optimizer parameters as you save your model weights when you checkpoint.
\nThen when you resume you load them.
\nFor instance here are functions I use to save and restore checkpoints with pytorch. Of course you need to adjust it to your code. It is important that you create your model, optimizer, etc in the load checkpoint function exactly as you create them in your training loop. The scaler is only useful if you use mixed precision (torch.cuda.amp).
\ndef save_checkpoint(model, optimizer, scheduler, scaler, epoch, fold, seed, fname=fname):\n checkpoint = {\n 'model': model.state_dict(),\n 'optimizer': optimizer.state_dict(),\n 'scheduler': scheduler.state_dict(),\n 'scaler': scaler.state_dict(),\n 'epoch': epoch,\n 'fold':fold,\n 'seed':seed,\n }\n torch.save(checkpoint, '../checkpoints/%s/%s_%d_%d.pt' % (fname, fname, fold, seed))\n\ndef load_checkpoint(fold, seed, fname):\n model = create_model().to(device)\n optimizer = optimizer = torch.optim.Adam(model.parameters(), lr=MAX_LR)\n scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer=optimizer, \n pct_start=PCT_START, \n div_factor=DIV_FACTOR \n max_lr=MAX_LR, \n epochs=EPOCHS, \n steps_per_epoch=int(np.ceil(len(train_data_loader)/GRADIENT_ACCUMULATION)))\n scaler = GradScaler()\n checkpoint = torch.load('../checkpoints/%s/%s_%d_%d.pt' % (fname, fname, fold, seed))\n model.load_state_dict(checkpoint['model'])\n optimizer.load_state_dict(checkpoint['optimizer'])\n scheduler.load_state_dict(checkpoint['scheduler'])\n scaler.load_state_dict(checkpoint['scaler'])\n return model, optimizer, scheduler, scaler, epoch\n@cpmpml Thanks, this clears the air, so we still resume in an almost verbatim manner when we resume training. I got it now :)
", "rawMarkdown": "@cpmpml Thanks, this clears the air, so we still resume in an almost verbatim manner when we resume training. I got it now :)", "votes": 1 }, { "id": 1358809, "postDate": "2021-06-20T18:50:57.543Z", "content": "@cpmpml Thanks, quite useful. One question on the seed, we would need to checkpoint the last state of random generator right? so when we resume training fresh random samples are created and not what were already created in the previous epochs (using same seed again).
", "rawMarkdown": "@cpmpml Thanks, quite useful. One question on the seed, we would need to checkpoint the last state of random generator right? so when we resume training fresh random samples are created and not what were already created in the previous epochs (using same seed again).", "votes": 3 }, { "id": 1358863, "postDate": "2021-06-20T20:36:54.110Z", "content": "@ankitsajwan I guess you're right. Initializing the pytorch random generator would ensure path would be even closer to what it would have been without the restart.
", "rawMarkdown": "@ankitsajwan I guess you're right. Initializing the pytorch random generator would ensure path would be even closer to what it would have been without the restart.", "votes": 1 } ] }, { "id": 1358123, "postDate": "2021-06-20T07:59:33.440Z", "content": "Now, I am unsure if this has been answered, I remember my buddy telling me that anecdotally, one can \"reset\" the LR a little when one resumes training from a model's checkpoint.
\nIt is a bit counter-intuitive to me, but say I trained a model for 16 epochs, with a custom scheduler, say OneCycleLr + Adam or something, then when you resume training, should we reset the initial learning rate? I would think resetting is counter-intuitive as the purpose of the learning rate is to tune it such that your model can slowly converge to the minima (global one if the function is convex).
So the question is:
\nShould I reset the learning rate when resume training, if yes, reset to what?
If we should not reset, what is a good way to \"extract\" the last learning rate, as some scheduler depends on factors like epochs…
i hope this discussion post of @raghaw is useful : https://www.kaggle.com/c/bengaliai-cv19/discussion/129323
", "rawMarkdown": "i hope this discussion post of @raghaw is useful : https://www.kaggle.com/c/bengaliai-cv19/discussion/129323", "votes": 3, "replies": [ { "id": 1358342, "postDate": "2021-06-20T11:33:06.703Z", "content": "Thanks @mobassir , I also encountered this before, I think @cpmpml code should solve this issue as well. It would be a bummer when \"resume\" training is not really a verbatim resumption.
", "rawMarkdown": "Thanks @mobassir , I also encountered this before, I think @cpmpml code should solve this issue as well. It would be a bummer when \"resume\" training is not really a verbatim resumption.", "votes": 2 }, { "id": 1395800, "postDate": "2021-07-21T14:44:05.997Z", "content": "@mobassir @reighns, The steps given by CPMP will solve your problem and it will work. I also solved my problem that I encountered in bengali AI competition, by a minor modification.
\nProblematic sequences that I used in bengali AI competition:
\nProblem in above sequence was at step 3. Once scheduler initlized, it resets the current learning rate in optimizer and initializes it to the learning rate of epoch 0. So I swapped step 2 and 3 that solved my problem. So the correct step that is working for all optimizers and schedulers that I have used till now is:
\nThanks Gao. I learned a lot from your work.
\nI usually training CNN model 5 epoch / 1day.
\nWhen I reload the model's state, the training rate was unstable sometimes.
I didn't think I need to save optimizer state. Thank you.
", "rawMarkdown": "Thanks Gao. I learned a lot from your work.\nI usually training CNN model 5 epoch / 1day.\nWhen I reload the model's state, the training rate was unstable sometimes.\n \nI didn't think I need to save optimizer state. Thank you.", "votes": 1 } ], "comments": [ { "id": 1358199, "author_name": "CPMP", "author_url": "", "post_date": "2021-06-20T09:05:36.080000", "content": "You have to save your optimizer parameters as you save your model weights when you checkpoint.
\nThen when you resume you load them.
\nFor instance here are functions I use to save and restore checkpoints with pytorch. Of course you need to adjust it to your code. It is important that you create your model, optimizer, etc in the load checkpoint function exactly as you create them in your training loop. The scaler is only useful if you use mixed precision (torch.cuda.amp).
\ndef save_checkpoint(model, optimizer, scheduler, scaler, epoch, fold, seed, fname=fname):\n checkpoint = {\n 'model': model.state_dict(),\n 'optimizer': optimizer.state_dict(),\n 'scheduler': scheduler.state_dict(),\n 'scaler': scaler.state_dict(),\n 'epoch': epoch,\n 'fold':fold,\n 'seed':seed,\n }\n torch.save(checkpoint, '../checkpoints/%s/%s_%d_%d.pt' % (fname, fname, fold, seed))\n\ndef load_checkpoint(fold, seed, fname):\n model = create_model().to(device)\n optimizer = optimizer = torch.optim.Adam(model.parameters(), lr=MAX_LR)\n scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer=optimizer, \n pct_start=PCT_START, \n div_factor=DIV_FACTOR \n max_lr=MAX_LR, \n epochs=EPOCHS, \n steps_per_epoch=int(np.ceil(len(train_data_loader)/GRADIENT_ACCUMULATION)))\n scaler = GradScaler()\n checkpoint = torch.load('../checkpoints/%s/%s_%d_%d.pt' % (fname, fname, fold, seed))\n model.load_state_dict(checkpoint['model'])\n optimizer.load_state_dict(checkpoint['optimizer'])\n scheduler.load_state_dict(checkpoint['scheduler'])\n scaler.load_state_dict(checkpoint['scaler'])\n return model, optimizer, scheduler, scaler, epoch\n@cpmpml Thanks, this clears the air, so we still resume in an almost verbatim manner when we resume training. I got it now :)
", "votes": 1, "replies": [] }, { "id": 1358809, "author_name": "sajwankit", "author_url": "", "post_date": "2021-06-20T18:50:57.543000", "content": "@cpmpml Thanks, quite useful. One question on the seed, we would need to checkpoint the last state of random generator right? so when we resume training fresh random samples are created and not what were already created in the previous epochs (using same seed again).
", "votes": 3, "replies": [] }, { "id": 1358863, "author_name": "CPMP", "author_url": "", "post_date": "2021-06-20T20:36:54.110000", "content": "@ankitsajwan I guess you're right. Initializing the pytorch random generator would ensure path would be even closer to what it would have been without the restart.
", "votes": 1, "replies": [] } ] }, { "id": 1358255, "author_name": "Mobassir", "author_url": "", "post_date": "2021-06-20T10:01:22.527000", "content": "i hope this discussion post of @raghaw is useful : https://www.kaggle.com/c/bengaliai-cv19/discussion/129323
", "votes": 3, "replies": [ { "id": 1358342, "author_name": "gao-hongnan", "author_url": "", "post_date": "2021-06-20T11:33:06.703000", "content": "Thanks @mobassir , I also encountered this before, I think @cpmpml code should solve this issue as well. It would be a bummer when \"resume\" training is not really a verbatim resumption.
", "votes": 2, "replies": [] }, { "id": 1395800, "author_name": "Raghawendra Singh", "author_url": "", "post_date": "2021-07-21T14:44:05.997000", "content": "@mobassir @reighns, The steps given by CPMP will solve your problem and it will work. I also solved my problem that I encountered in bengali AI competition, by a minor modification.
\nProblematic sequences that I used in bengali AI competition:
\nProblem in above sequence was at step 3. Once scheduler initlized, it resets the current learning rate in optimizer and initializes it to the learning rate of epoch 0. So I swapped step 2 and 3 that solved my problem. So the correct step that is working for all optimizers and schedulers that I have used till now is:
\nThanks Gao. I learned a lot from your work.
\nI usually training CNN model 5 epoch / 1day.
\nWhen I reload the model's state, the training rate was unstable sometimes.
I didn't think I need to save optimizer state. Thank you.
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1358199": "You have to save your optimizer parameters as you save your model weights when you checkpoint.\n\nThen when you resume you load them.\n\nFor instance here are functions I use to save and restore checkpoints with pytorch. Of course you need to adjust it to your code. It is important that you create your model, optimizer, etc in the load checkpoint function exactly as you create them in your training loop. The scaler is only useful if you use mixed precision (torch.cuda.amp).\n\n```\n\ndef save_checkpoint(model, optimizer, scheduler, scaler, epoch, fold, seed, fname=fname):\n checkpoint = {\n 'model': model.state_dict(),\n 'optimizer': optimizer.state_dict(),\n 'scheduler': scheduler.state_dict(),\n 'scaler': scaler.state_dict(),\n 'epoch': epoch,\n 'fold':fold,\n 'seed':seed,\n }\n torch.save(checkpoint, '../checkpoints/%s/%s_%d_%d.pt' % (fname, fname, fold, seed))\n\ndef load_checkpoint(fold, seed, fname):\n model = create_model().to(device)\n optimizer = optimizer = torch.optim.Adam(model.parameters(), lr=MAX_LR)\n scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer=optimizer, \n pct_start=PCT_START, \n div_factor=DIV_FACTOR \n max_lr=MAX_LR, \n epochs=EPOCHS, \n steps_per_epoch=int(np.ceil(len(train_data_loader)/GRADIENT_ACCUMULATION)))\n scaler = GradScaler()\n checkpoint = torch.load('../checkpoints/%s/%s_%d_%d.pt' % (fname, fname, fold, seed))\n model.load_state_dict(checkpoint['model'])\n optimizer.load_state_dict(checkpoint['optimizer'])\n scheduler.load_state_dict(checkpoint['scheduler'])\n scaler.load_state_dict(checkpoint['scaler'])\n return model, optimizer, scheduler, scaler, epoch\n\n```", "1358123": "Now, I am unsure if this has been answered, I remember my buddy telling me that anecdotally, one can \"reset\" the LR a little when one resumes training from a model's checkpoint.\n\nIt is a bit counter-intuitive to me, but say I trained a model for 16 epochs, with a custom scheduler, say `OneCycleLr` + `Adam` or something, then when you resume training, should we reset the initial learning rate? I would think resetting is counter-intuitive as the purpose of the learning rate is to tune it such that your model can slowly converge to the minima (global one if the function is convex). \n\nSo the question is: \n\n1. Should I reset the learning rate when resume training, if yes, reset to what?\n\n2. If we should not reset, what is a good way to \"extract\" the last learning rate, as some scheduler depends on factors like epochs...\n", "1358255": "i hope this discussion post of @raghaw is useful : https://www.kaggle.com/c/bengaliai-cv19/discussion/129323", "1359403": "Thanks Gao. I learned a lot from your work.\nI usually training CNN model 5 epoch / 1day.\nWhen I reload the model's state, the training rate was unstable sometimes.\n \nI didn't think I need to save optimizer state. Thank you." } }