I often read winning solutions from great people like @haqishen and @underwearfitting, I observed they like to make use of the Gradual warmup Scheduler as in this paper. Although I am unsure if gradual warmup has any significant effects on my measly GPU who can at most fit batch size of 32, but it does not prevent me from learning from them.
\nI see that the implementation here may potentially have some bugs, and seeing Qishen's solution in Melanoma, he made one or two changes. I am still confused, though, in this script here he changed only one line (line 14) from get_last_lr() to get_lr(). And in his pipeline, he also used this
scheduler_warmup.step() \n if epoch==2: scheduler_warmup.step() # bug workaround \nI tried to run both versions, with and without the workaround, and find very little intuition on what the \"bug\" actually was.
\nHere is my script, wish someone could enlighten me as although the \"intuition/idea\" of this gradual warmup approach seems simple, but I cannot find out where is the major bug:
\nimport torch\nfrom torch.optim.lr_scheduler import StepLR, ExponentialLR\nfrom torch.optim.sgd import SGD\nfrom gradual_warmup import *\nfrom typing import *\n\n# from warmup_scheduler import GradualWarmupScheduler\n\n# 1. n = minibatch size\n# 2. k = multiplier of minibatch size\n# 3. For eg, if n = 32, and k = 2, then the new batch size is 64.\n# 4. Intuition: In the first few steps, the LR is set to lower than the base LR, and increased gradually (linearly) to approach it\n# in a number of epochs.\n# 5. Example:\n# - base_lr = 1\n# - total_epoch (Warmup Epochs) = 10\n# - [base_lr * (float(self.last_epoch) / self.total_epoch) for base_lr in self.base_lrs]\n# - [1 * (0/10) for [all base_lrs in which case here is just 1]] -> [0],[0.1], [0.2] etc\n#\n\n\ndef get_lr(optimizer: torch.optim):\n for param_group in optimizer.param_groups:\n return param_group['lr']\n\n\ndef normal_scheduler(optimizer: torch.optim, scheduler: torch.optim.lr_scheduler):\n\n for epoch in range(1, 100):\n optimizer.zero_grad()\n curr_epoch_lr = optimizer.param_groups[0]['lr']\n print(epoch, curr_epoch_lr)\n\n optimizer.step()\n scheduler.step()\n\n\ndef warmup_scheduler(optimizer: torch.optim, scheduler: torch.optim.lr_scheduler):\n\n warmup: List = []\n\n for epoch in range(1, 101):\n optimizer.zero_grad()\n curr_epoch_lr = optimizer.param_groups[0]['lr']\n\n print(epoch, curr_epoch_lr)\n warmup.append(curr_epoch_lr)\n\n optimizer.step()\n scheduler.step()\n\n # if epoch == 2:\n # scheduler.step()\n\n textfile = open(\"warmup.txt\", \"w\")\n for element in warmup:\n textfile.write(str(element) + \"\\n\")\n textfile.close()\n\n\nif __name__ == '__main__':\n # create a naive model\n model = [torch.nn.Parameter(torch.randn(2, 2, requires_grad=True))]\n\n initial_lr = 10\n optimizer = SGD(model, initial_lr)\n\n # scheduler_warmup is chained with schduler_steplr\n scheduler_steplr = StepLR(optimizer, step_size=20, gamma=0.5)\n scheduler_warmup = GradualWarmupScheduler(\n optimizer, multiplier=1, total_epoch=10, after_scheduler=scheduler_steplr)\n\n scheduler_warmup_v2 = GradualWarmupSchedulerV2(\n optimizer, multiplier=1, total_epoch=10, after_scheduler=scheduler_steplr)\n\n warmup_scheduler(optimizer=optimizer, scheduler=scheduler_warmup_v2)\nIt is worth putting here both the GradualWarmUp function here for your reference:
from torch.optim.lr_scheduler import _LRScheduler\nfrom torch.optim.lr_scheduler import ReduceLROnPlateau\n\n\nclass GradualWarmupScheduler(_LRScheduler):\n \"\"\" Gradually warm-up(increasing) learning rate in optimizer.\n Proposed in 'Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour'.\n Args:\n optimizer (Optimizer): Wrapped optimizer.\n multiplier: target learning rate = base lr * multiplier if multiplier > 1.0. if multiplier = 1.0, lr starts from 0 and ends up with the base_lr.\n total_epoch: target learning rate is reached at total_epoch, gradually\n after_scheduler: after target_epoch, use this scheduler(eg. ReduceLROnPlateau)\n \"\"\"\n\n def __init__(self, optimizer, multiplier, total_epoch, after_scheduler=None):\n self.multiplier = multiplier\n if self.multiplier < 1.:\n raise ValueError(\n 'multiplier should be greater thant or equal to 1.')\n self.total_epoch = total_epoch\n self.after_scheduler = after_scheduler\n self.finished = False\n super(GradualWarmupScheduler, self).__init__(optimizer)\n\n def get_lr(self):\n\n if self.last_epoch > self.total_epoch:\n if self.after_scheduler:\n if not self.finished:\n print(\"AAAAAAAAAAA\", self.base_lrs)\n self.after_scheduler.base_lrs = [\n base_lr * self.multiplier for base_lr in self.base_lrs]\n self.finished = True\n return self.after_scheduler.get_last_lr()\n return [base_lr * self.multiplier for base_lr in self.base_lrs]\n\n if self.multiplier == 1.0:\n return [base_lr * (float(self.last_epoch) / self.total_epoch) for base_lr in self.base_lrs]\n else:\n return [base_lr * ((self.multiplier - 1.) * self.last_epoch / self.total_epoch + 1.) for base_lr in self.base_lrs]\n\n def step_ReduceLROnPlateau(self, metrics, epoch=None):\n if epoch is None:\n epoch = self.last_epoch + 1\n # ReduceLROnPlateau is called at the end of epoch, whereas others are called at beginning\n self.last_epoch = epoch if epoch != 0 else 1\n if self.last_epoch <= self.total_epoch:\n warmup_lr = [base_lr * ((self.multiplier - 1.) * self.last_epoch /\n self.total_epoch + 1.) for base_lr in self.base_lrs]\n for param_group, lr in zip(self.optimizer.param_groups, warmup_lr):\n param_group['lr'] = lr\n else:\n if epoch is None:\n self.after_scheduler.step(metrics, None)\n else:\n self.after_scheduler.step(metrics, epoch - self.total_epoch)\n\n def step(self, epoch=None, metrics=None):\n if type(self.after_scheduler) != ReduceLROnPlateau:\n if self.finished and self.after_scheduler:\n if epoch is None:\n self.after_scheduler.step(None)\n else:\n self.after_scheduler.step(epoch - self.total_epoch)\n # get last lr\n self._last_lr = self.after_scheduler.get_last_lr()\n print(self._last_lr)\n else:\n return super(GradualWarmupScheduler, self).step(epoch)\n else:\n self.step_ReduceLROnPlateau(metrics, epoch)\n\n\nclass GradualWarmupSchedulerV2(GradualWarmupScheduler):\n def __init__(self, optimizer, multiplier, total_epoch, after_scheduler=None):\n super(GradualWarmupSchedulerV2, self).__init__(\n optimizer, multiplier, total_epoch, after_scheduler\n )\n\n def get_lr(self):\n if self.last_epoch > self.total_epoch:\n if self.after_scheduler:\n if not self.finished:\n print(\"AAAAAAAAAAA\", self.base_lrs)\n self.after_scheduler.base_lrs = [\n base_lr * self.multiplier for base_lr in self.base_lrs\n ]\n self.finished = True\n return self.after_scheduler.get_lr()\n return [base_lr * self.multiplier for base_lr in self.base_lrs]\n if self.multiplier == 1.0:\n return [\n base_lr * (float(self.last_epoch) / self.total_epoch)\n for base_lr in self.base_lrs\n ]\n else:\n return [\n base_lr\n * ((self.multiplier - 1.0) * self.last_epoch / self.total_epoch + 1.0)\n for base_lr in self.base_lrs\n ]\nI often read winning solutions from great people like @haqishen and @underwearfitting, I observed they like to make use of the Gradual warmup Scheduler as in this paper. Although I am unsure if gradual warmup has any significant effects on my measly GPU who can at most fit batch size of 32, but it does not prevent me from learning from them.
\nI see that the implementation here may potentially have some bugs, and seeing Qishen's solution in Melanoma, he made one or two changes. I am still confused, though, in this script here he changed only one line (line 14) from get_last_lr() to get_lr(). And in his pipeline, he also used this
scheduler_warmup.step() \n if epoch==2: scheduler_warmup.step() # bug workaround \nI tried to run both versions, with and without the workaround, and find very little intuition on what the \"bug\" actually was.
\nHere is my script, wish someone could enlighten me as although the \"intuition/idea\" of this gradual warmup approach seems simple, but I cannot find out where is the major bug:
\nimport torch\nfrom torch.optim.lr_scheduler import StepLR, ExponentialLR\nfrom torch.optim.sgd import SGD\nfrom gradual_warmup import *\nfrom typing import *\n\n# from warmup_scheduler import GradualWarmupScheduler\n\n# 1. n = minibatch size\n# 2. k = multiplier of minibatch size\n# 3. For eg, if n = 32, and k = 2, then the new batch size is 64.\n# 4. Intuition: In the first few steps, the LR is set to lower than the base LR, and increased gradually (linearly) to approach it\n# in a number of epochs.\n# 5. Example:\n# - base_lr = 1\n# - total_epoch (Warmup Epochs) = 10\n# - [base_lr * (float(self.last_epoch) / self.total_epoch) for base_lr in self.base_lrs]\n# - [1 * (0/10) for [all base_lrs in which case here is just 1]] -> [0],[0.1], [0.2] etc\n#\n\n\ndef get_lr(optimizer: torch.optim):\n for param_group in optimizer.param_groups:\n return param_group['lr']\n\n\ndef normal_scheduler(optimizer: torch.optim, scheduler: torch.optim.lr_scheduler):\n\n for epoch in range(1, 100):\n optimizer.zero_grad()\n curr_epoch_lr = optimizer.param_groups[0]['lr']\n print(epoch, curr_epoch_lr)\n\n optimizer.step()\n scheduler.step()\n\n\ndef warmup_scheduler(optimizer: torch.optim, scheduler: torch.optim.lr_scheduler):\n\n warmup: List = []\n\n for epoch in range(1, 101):\n optimizer.zero_grad()\n curr_epoch_lr = optimizer.param_groups[0]['lr']\n\n print(epoch, curr_epoch_lr)\n warmup.append(curr_epoch_lr)\n\n optimizer.step()\n scheduler.step()\n\n # if epoch == 2:\n # scheduler.step()\n\n textfile = open(\"warmup.txt\", \"w\")\n for element in warmup:\n textfile.write(str(element) + \"\\n\")\n textfile.close()\n\n\nif __name__ == '__main__':\n # create a naive model\n model = [torch.nn.Parameter(torch.randn(2, 2, requires_grad=True))]\n\n initial_lr = 10\n optimizer = SGD(model, initial_lr)\n\n # scheduler_warmup is chained with schduler_steplr\n scheduler_steplr = StepLR(optimizer, step_size=20, gamma=0.5)\n scheduler_warmup = GradualWarmupScheduler(\n optimizer, multiplier=1, total_epoch=10, after_scheduler=scheduler_steplr)\n\n scheduler_warmup_v2 = GradualWarmupSchedulerV2(\n optimizer, multiplier=1, total_epoch=10, after_scheduler=scheduler_steplr)\n\n warmup_scheduler(optimizer=optimizer, scheduler=scheduler_warmup_v2)\nIt is worth putting here both the GradualWarmUp function here for your reference:
from torch.optim.lr_scheduler import _LRScheduler\nfrom torch.optim.lr_scheduler import ReduceLROnPlateau\n\n\nclass GradualWarmupScheduler(_LRScheduler):\n \"\"\" Gradually warm-up(increasing) learning rate in optimizer.\n Proposed in 'Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour'.\n Args:\n optimizer (Optimizer): Wrapped optimizer.\n multiplier: target learning rate = base lr * multiplier if multiplier > 1.0. if multiplier = 1.0, lr starts from 0 and ends up with the base_lr.\n total_epoch: target learning rate is reached at total_epoch, gradually\n after_scheduler: after target_epoch, use this scheduler(eg. ReduceLROnPlateau)\n \"\"\"\n\n def __init__(self, optimizer, multiplier, total_epoch, after_scheduler=None):\n self.multiplier = multiplier\n if self.multiplier < 1.:\n raise ValueError(\n 'multiplier should be greater thant or equal to 1.')\n self.total_epoch = total_epoch\n self.after_scheduler = after_scheduler\n self.finished = False\n super(GradualWarmupScheduler, self).__init__(optimizer)\n\n def get_lr(self):\n\n if self.last_epoch > self.total_epoch:\n if self.after_scheduler:\n if not self.finished:\n print(\"AAAAAAAAAAA\", self.base_lrs)\n self.after_scheduler.base_lrs = [\n base_lr * self.multiplier for base_lr in self.base_lrs]\n self.finished = True\n return self.after_scheduler.get_last_lr()\n return [base_lr * self.multiplier for base_lr in self.base_lrs]\n\n if self.multiplier == 1.0:\n return [base_lr * (float(self.last_epoch) / self.total_epoch) for base_lr in self.base_lrs]\n else:\n return [base_lr * ((self.multiplier - 1.) * self.last_epoch / self.total_epoch + 1.) for base_lr in self.base_lrs]\n\n def step_ReduceLROnPlateau(self, metrics, epoch=None):\n if epoch is None:\n epoch = self.last_epoch + 1\n # ReduceLROnPlateau is called at the end of epoch, whereas others are called at beginning\n self.last_epoch = epoch if epoch != 0 else 1\n if self.last_epoch <= self.total_epoch:\n warmup_lr = [base_lr * ((self.multiplier - 1.) * self.last_epoch /\n self.total_epoch + 1.) for base_lr in self.base_lrs]\n for param_group, lr in zip(self.optimizer.param_groups, warmup_lr):\n param_group['lr'] = lr\n else:\n if epoch is None:\n self.after_scheduler.step(metrics, None)\n else:\n self.after_scheduler.step(metrics, epoch - self.total_epoch)\n\n def step(self, epoch=None, metrics=None):\n if type(self.after_scheduler) != ReduceLROnPlateau:\n if self.finished and self.after_scheduler:\n if epoch is None:\n self.after_scheduler.step(None)\n else:\n self.after_scheduler.step(epoch - self.total_epoch)\n # get last lr\n self._last_lr = self.after_scheduler.get_last_lr()\n print(self._last_lr)\n else:\n return super(GradualWarmupScheduler, self).step(epoch)\n else:\n self.step_ReduceLROnPlateau(metrics, epoch)\n\n\nclass GradualWarmupSchedulerV2(GradualWarmupScheduler):\n def __init__(self, optimizer, multiplier, total_epoch, after_scheduler=None):\n super(GradualWarmupSchedulerV2, self).__init__(\n optimizer, multiplier, total_epoch, after_scheduler\n )\n\n def get_lr(self):\n if self.last_epoch > self.total_epoch:\n if self.after_scheduler:\n if not self.finished:\n print(\"AAAAAAAAAAA\", self.base_lrs)\n self.after_scheduler.base_lrs = [\n base_lr * self.multiplier for base_lr in self.base_lrs\n ]\n self.finished = True\n return self.after_scheduler.get_lr()\n return [base_lr * self.multiplier for base_lr in self.base_lrs]\n if self.multiplier == 1.0:\n return [\n base_lr * (float(self.last_epoch) / self.total_epoch)\n for base_lr in self.base_lrs\n ]\n else:\n return [\n base_lr\n * ((self.multiplier - 1.0) * self.last_epoch / self.total_epoch + 1.0)\n for base_lr in self.base_lrs\n ]\nThe lr shcheduler is not a particularly important point for improving model performance.
\nAbout the if epoch == 2 part, rather than a bug, it just didn't work as I expected it to, so I made some slight modifications to it. It's more like a personal habit.
Even if lr 0.1 is repeated 2 times, the model doesn't get suck because of it right?
", "rawMarkdown": "The lr shcheduler is not a particularly important point for improving model performance.\n\nAbout the `if epoch == 2` part, rather than a bug, it just didn't work as I expected it to, so I made some slight modifications to it. It's more like a personal habit.\n\nEven if lr 0.1 is repeated 2 times, the model doesn't get suck because of it right?\n\n", "votes": 2, "replies": [ { "id": 1463442, "postDate": "2021-08-10T07:33:59.790Z", "content": "Dear @haqishen , thanks for the reply, this clears the air.
", "rawMarkdown": "Dear @haqishen , thanks for the reply, this clears the air. " } ] }, { "id": 1457248, "postDate": "2021-08-07T10:13:59.993Z", "content": "Answering myself: I think I got why qishen used the if epoch == 2 then step. My hypothesis is because he only used 1 epoch to warmup. And thus the library he used will always not step properly when the first epoch end (or whichever number you set the “total epoch to warmup” to).
\nAs an example: if our base learning rate is 0.1. And he set warmup epoch to just 1 epoch, with no multiplier, at the second epoch it should already be at 0.1, and third epoch it should be using the learning rate from the original scheduler. But I think the bug is at the 3rd epoch it repeats 0.1 again.
", "rawMarkdown": "Answering myself: I think I got why qishen used the if epoch == 2 then step. My hypothesis is because he only used 1 epoch to warmup. And thus the library he used will always not step properly when the first epoch end (or whichever number you set the “total epoch to warmup” to). \n\nAs an example: if our base learning rate is 0.1. And he set warmup epoch to just 1 epoch, with no multiplier, at the second epoch it should already be at 0.1, and third epoch it should be using the learning rate from the original scheduler. But I think the bug is at the 3rd epoch it repeats 0.1 again. " } ], "comments": [ { "id": 1462047, "author_name": "Qishen Ha", "author_url": "", "post_date": "2021-08-09T17:07:17.103000", "content": "The lr shcheduler is not a particularly important point for improving model performance.
\nAbout the if epoch == 2 part, rather than a bug, it just didn't work as I expected it to, so I made some slight modifications to it. It's more like a personal habit.
Even if lr 0.1 is repeated 2 times, the model doesn't get suck because of it right?
", "votes": 2, "replies": [ { "id": 1463442, "author_name": "gao-hongnan", "author_url": "", "post_date": "2021-08-10T07:33:59.790000", "content": "Dear @haqishen , thanks for the reply, this clears the air.
", "votes": 0, "replies": [] } ] }, { "id": 1457248, "author_name": "gao-hongnan", "author_url": "", "post_date": "2021-08-07T10:13:59.993000", "content": "Answering myself: I think I got why qishen used the if epoch == 2 then step. My hypothesis is because he only used 1 epoch to warmup. And thus the library he used will always not step properly when the first epoch end (or whichever number you set the “total epoch to warmup” to).
\nAs an example: if our base learning rate is 0.1. And he set warmup epoch to just 1 epoch, with no multiplier, at the second epoch it should already be at 0.1, and third epoch it should be using the learning rate from the original scheduler. But I think the bug is at the 3rd epoch it repeats 0.1 again.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1454939": "I often read winning solutions from great people like @haqishen and @underwearfitting, I observed they like to make use of the [Gradual warmup Scheduler](https://research.fb.com/wp-content/uploads/2017/06/imagenet1kin1h5.pdf) as in this paper. Although I am unsure if gradual warmup has any significant effects on my measly GPU who can at most fit batch size of 32, but it does not prevent me from learning from them.\n\nI see that the implementation [here](https://github.com/ildoonet/pytorch-gradual-warmup-lr/blob/master/warmup_scheduler/scheduler.py) may potentially have some bugs, and seeing Qishen's [solution in Melanoma](https://github.com/haqishen/SIIM-ISIC-Melanoma-Classification-1st-Place-Solution/blob/2c4a5428c6d410e97d2a74aacd5f86b3750d32cf/util.py#L5), he made one or two changes. I am still confused, though, in this script here he changed only one line (line 14) from `get_last_lr()` to `get_lr()`. And in his pipeline, he also used this\n\n```\n scheduler_warmup.step() \n if epoch==2: scheduler_warmup.step() # bug workaround \n```\n\nI tried to run both versions, with and without the workaround, and find very little intuition on what the \"bug\" actually was. \n\nHere is my script, wish someone could enlighten me as although the \"intuition/idea\" of this gradual warmup approach seems simple, but I cannot find out where is the major bug:\n\n```\nimport torch\nfrom torch.optim.lr_scheduler import StepLR, ExponentialLR\nfrom torch.optim.sgd import SGD\nfrom gradual_warmup import *\nfrom typing import *\n\n# from warmup_scheduler import GradualWarmupScheduler\n\n# 1. n = minibatch size\n# 2. k = multiplier of minibatch size\n# 3. For eg, if n = 32, and k = 2, then the new batch size is 64.\n# 4. Intuition: In the first few steps, the LR is set to lower than the base LR, and increased gradually (linearly) to approach it\n# in a number of epochs.\n# 5. Example:\n# - base_lr = 1\n# - total_epoch (Warmup Epochs) = 10\n# - [base_lr * (float(self.last_epoch) / self.total_epoch) for base_lr in self.base_lrs]\n# - [1 * (0/10) for [all base_lrs in which case here is just 1]] -> [0],[0.1], [0.2] etc\n#\n\n\ndef get_lr(optimizer: torch.optim):\n for param_group in optimizer.param_groups:\n return param_group['lr']\n\n\ndef normal_scheduler(optimizer: torch.optim, scheduler: torch.optim.lr_scheduler):\n\n for epoch in range(1, 100):\n optimizer.zero_grad()\n curr_epoch_lr = optimizer.param_groups[0]['lr']\n print(epoch, curr_epoch_lr)\n\n optimizer.step()\n scheduler.step()\n\n\ndef warmup_scheduler(optimizer: torch.optim, scheduler: torch.optim.lr_scheduler):\n\n warmup: List = []\n\n for epoch in range(1, 101):\n optimizer.zero_grad()\n curr_epoch_lr = optimizer.param_groups[0]['lr']\n\n print(epoch, curr_epoch_lr)\n warmup.append(curr_epoch_lr)\n\n optimizer.step()\n scheduler.step()\n\n # if epoch == 2:\n # scheduler.step()\n\n textfile = open(\"warmup.txt\", \"w\")\n for element in warmup:\n textfile.write(str(element) + \"\\n\")\n textfile.close()\n\n\nif __name__ == '__main__':\n # create a naive model\n model = [torch.nn.Parameter(torch.randn(2, 2, requires_grad=True))]\n\n initial_lr = 10\n optimizer = SGD(model, initial_lr)\n\n # scheduler_warmup is chained with schduler_steplr\n scheduler_steplr = StepLR(optimizer, step_size=20, gamma=0.5)\n scheduler_warmup = GradualWarmupScheduler(\n optimizer, multiplier=1, total_epoch=10, after_scheduler=scheduler_steplr)\n\n scheduler_warmup_v2 = GradualWarmupSchedulerV2(\n optimizer, multiplier=1, total_epoch=10, after_scheduler=scheduler_steplr)\n\n warmup_scheduler(optimizer=optimizer, scheduler=scheduler_warmup_v2)\n```\n\nIt is worth putting here both the `GradualWarmUp` function here for your reference:\n\n```\nfrom torch.optim.lr_scheduler import _LRScheduler\nfrom torch.optim.lr_scheduler import ReduceLROnPlateau\n\n\nclass GradualWarmupScheduler(_LRScheduler):\n \"\"\" Gradually warm-up(increasing) learning rate in optimizer.\n Proposed in 'Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour'.\n Args:\n optimizer (Optimizer): Wrapped optimizer.\n multiplier: target learning rate = base lr * multiplier if multiplier > 1.0. if multiplier = 1.0, lr starts from 0 and ends up with the base_lr.\n total_epoch: target learning rate is reached at total_epoch, gradually\n after_scheduler: after target_epoch, use this scheduler(eg. ReduceLROnPlateau)\n \"\"\"\n\n def __init__(self, optimizer, multiplier, total_epoch, after_scheduler=None):\n self.multiplier = multiplier\n if self.multiplier < 1.:\n raise ValueError(\n 'multiplier should be greater thant or equal to 1.')\n self.total_epoch = total_epoch\n self.after_scheduler = after_scheduler\n self.finished = False\n super(GradualWarmupScheduler, self).__init__(optimizer)\n\n def get_lr(self):\n\n if self.last_epoch > self.total_epoch:\n if self.after_scheduler:\n if not self.finished:\n print(\"AAAAAAAAAAA\", self.base_lrs)\n self.after_scheduler.base_lrs = [\n base_lr * self.multiplier for base_lr in self.base_lrs]\n self.finished = True\n return self.after_scheduler.get_last_lr()\n return [base_lr * self.multiplier for base_lr in self.base_lrs]\n\n if self.multiplier == 1.0:\n return [base_lr * (float(self.last_epoch) / self.total_epoch) for base_lr in self.base_lrs]\n else:\n return [base_lr * ((self.multiplier - 1.) * self.last_epoch / self.total_epoch + 1.) for base_lr in self.base_lrs]\n\n def step_ReduceLROnPlateau(self, metrics, epoch=None):\n if epoch is None:\n epoch = self.last_epoch + 1\n # ReduceLROnPlateau is called at the end of epoch, whereas others are called at beginning\n self.last_epoch = epoch if epoch != 0 else 1\n if self.last_epoch <= self.total_epoch:\n warmup_lr = [base_lr * ((self.multiplier - 1.) * self.last_epoch /\n self.total_epoch + 1.) for base_lr in self.base_lrs]\n for param_group, lr in zip(self.optimizer.param_groups, warmup_lr):\n param_group['lr'] = lr\n else:\n if epoch is None:\n self.after_scheduler.step(metrics, None)\n else:\n self.after_scheduler.step(metrics, epoch - self.total_epoch)\n\n def step(self, epoch=None, metrics=None):\n if type(self.after_scheduler) != ReduceLROnPlateau:\n if self.finished and self.after_scheduler:\n if epoch is None:\n self.after_scheduler.step(None)\n else:\n self.after_scheduler.step(epoch - self.total_epoch)\n # get last lr\n self._last_lr = self.after_scheduler.get_last_lr()\n print(self._last_lr)\n else:\n return super(GradualWarmupScheduler, self).step(epoch)\n else:\n self.step_ReduceLROnPlateau(metrics, epoch)\n\n\nclass GradualWarmupSchedulerV2(GradualWarmupScheduler):\n def __init__(self, optimizer, multiplier, total_epoch, after_scheduler=None):\n super(GradualWarmupSchedulerV2, self).__init__(\n optimizer, multiplier, total_epoch, after_scheduler\n )\n\n def get_lr(self):\n if self.last_epoch > self.total_epoch:\n if self.after_scheduler:\n if not self.finished:\n print(\"AAAAAAAAAAA\", self.base_lrs)\n self.after_scheduler.base_lrs = [\n base_lr * self.multiplier for base_lr in self.base_lrs\n ]\n self.finished = True\n return self.after_scheduler.get_lr()\n return [base_lr * self.multiplier for base_lr in self.base_lrs]\n if self.multiplier == 1.0:\n return [\n base_lr * (float(self.last_epoch) / self.total_epoch)\n for base_lr in self.base_lrs\n ]\n else:\n return [\n base_lr\n * ((self.multiplier - 1.0) * self.last_epoch / self.total_epoch + 1.0)\n for base_lr in self.base_lrs\n ]\n```", "1462047": "The lr shcheduler is not a particularly important point for improving model performance.\n\nAbout the `if epoch == 2` part, rather than a bug, it just didn't work as I expected it to, so I made some slight modifications to it. It's more like a personal habit.\n\nEven if lr 0.1 is repeated 2 times, the model doesn't get suck because of it right?\n\n", "1457248": "Answering myself: I think I got why qishen used the if epoch == 2 then step. My hypothesis is because he only used 1 epoch to warmup. And thus the library he used will always not step properly when the first epoch end (or whichever number you set the “total epoch to warmup” to). \n\nAs an example: if our base learning rate is 0.1. And he set warmup epoch to just 1 epoch, with no multiplier, at the second epoch it should already be at 0.1, and third epoch it should be using the learning rate from the original scheduler. But I think the bug is at the 3rd epoch it repeats 0.1 again. " } }