{ "id": 316853, "title": "Idea for the Learning Rate Callback ", "url": "/competitions/happy-whale-and-dolphin/discussion/316853", "author_name": "", "post_date": "2022-04-04T08:42:51.942408900Z", "votes": 3, "comment_count": 7, "views": 0, "content": "

Many notebooks (in this competition) are used this function for **Learning Rate **

\n
def get_lr_callback(plot=False):\n    lr_start   = 0.000001\n    lr_max     = 0.000005 * config.BATCH_SIZE  \n    lr_min     = 0.000001\n    lr_ramp_ep = 4\n    lr_sus_ep  = 0\n    lr_decay   = 0.9\n\n    def lrfn(epoch):\n        if config.RESUME:\n            epoch = epoch + config.RESUME_EPOCH\n        if epoch < lr_ramp_ep:\n            lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n\n        elif epoch < lr_ramp_ep + lr_sus_ep:\n            lr = lr_max\n\n        else:\n            lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min\n\n        return lr\n\n    if plot:\n        epochs = list(range(config.EPOCHS))\n        learning_rates = [lrfn(x) for x in epochs]\n        plt.scatter(epochs,learning_rates)\n        plt.show()\n\n    lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=False)\n    return lr_callback\n\nget_lr_callback(plot=True)\n
\n

I do not know when is the idea for this implementation?
\nAnd what is useful to EfficientNet Architecture or in larger?

\n

P/S : I try to find out the answer in comments but maybe it is not exist, please show me if you know.

", "messages": [ { "id": "1744727", "postDate": "04/04/2022 08:42:51", "content": "

Many notebooks (in this competition) are used this function for **Learning Rate **

\n
def get_lr_callback(plot=False):\n    lr_start   = 0.000001\n    lr_max     = 0.000005 * config.BATCH_SIZE  \n    lr_min     = 0.000001\n    lr_ramp_ep = 4\n    lr_sus_ep  = 0\n    lr_decay   = 0.9\n\n    def lrfn(epoch):\n        if config.RESUME:\n            epoch = epoch + config.RESUME_EPOCH\n        if epoch < lr_ramp_ep:\n            lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n\n        elif epoch < lr_ramp_ep + lr_sus_ep:\n            lr = lr_max\n\n        else:\n            lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min\n\n        return lr\n\n    if plot:\n        epochs = list(range(config.EPOCHS))\n        learning_rates = [lrfn(x) for x in epochs]\n        plt.scatter(epochs,learning_rates)\n        plt.show()\n\n    lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=False)\n    return lr_callback\n\nget_lr_callback(plot=True)\n
\n

I do not know when is the idea for this implementation?
\nAnd what is useful to EfficientNet Architecture or in larger?

\n

P/S : I try to find out the answer in comments but maybe it is not exist, please show me if you know.

", "rawMarkdown": "Many notebooks (in this competition) are used this function for **Learning Rate **\n```\ndef get_lr_callback(plot=False):\n lr_start = 0.000001\n lr_max = 0.000005 * config.BATCH_SIZE \n lr_min = 0.000001\n lr_ramp_ep = 4\n lr_sus_ep = 0\n lr_decay = 0.9\n \n def lrfn(epoch):\n if config.RESUME:\n epoch = epoch + config.RESUME_EPOCH\n if epoch < lr_ramp_ep:\n lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n \n elif epoch < lr_ramp_ep + lr_sus_ep:\n lr = lr_max\n \n else:\n lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min\n \n return lr\n \n if plot:\n epochs = list(range(config.EPOCHS))\n learning_rates = [lrfn(x) for x in epochs]\n plt.scatter(epochs,learning_rates)\n plt.show()\n\n lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=False)\n return lr_callback\n\nget_lr_callback(plot=True)\n```\n\nI do not know when is the idea for this implementation?\nAnd what is useful to EfficientNet Architecture or in larger? \n\nP/S : I try to find out the answer in comments but maybe it is not exist, please show me if you know.", "votes": null }, { "id": "1744735", "postDate": "04/04/2022 08:59:44", "content": "

This is simple ExponentialLR with warmup, it looks like this:
\n\"lr\"
\nIt is not the best and only scheduler for this task, just one of.

", "rawMarkdown": "This is simple [ExponentialLR ](https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.ExponentialLR.html) with [warmup](https://stackoverflow.com/questions/55933867/what-does-learning-rate-warm-up-mean), it looks like this:\n![lr](https://i.ibb.co/vzkWWwW/image.png)\nIt is not the best and only scheduler for this task, just one of.", "votes": null }, { "id": "1745346", "postDate": "04/04/2022 21:10:33", "content": "

I would recommend checking out the lr scheduler in fastai, it can give a slightly better performance

", "rawMarkdown": "I would recommend checking out the lr scheduler in fastai, it can give a slightly better performance", "votes": null }, { "id": "1746746", "postDate": "04/06/2022 04:22:24", "content": "

A lot of the code shared in the public kernels comes from the Google Collab Flowers TPU example, including the LR scheduler.

\n

https://colab.research.google.com/notebooks/tpu.ipynb

", "rawMarkdown": "A lot of the code shared in the public kernels comes from the Google Collab Flowers TPU example, including the LR scheduler.\n\nhttps://colab.research.google.com/notebooks/tpu.ipynb", "votes": null }, { "id": "1746772", "postDate": "04/06/2022 04:56:29", "content": "

@kwentar what is your recommendation?

", "rawMarkdown": "kwentar what is your recommendation?", "votes": null }, { "id": "1746873", "postDate": "04/06/2022 07:26:47", "content": "

for me works good CosineAnnealingLR and LR scheduler from FastAI

", "rawMarkdown": "for me works good [CosineAnnealingLR ](https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.CosineAnnealingLR.html) and LR [scheduler](https://fastai1.fast.ai/callbacks.one_cycle.html) from FastAI", "votes": null }, { "id": "1746885", "postDate": "04/06/2022 07:33:30", "content": "

and that from experience or paper, or any algorithm? @kwentar

", "rawMarkdown": "and that from experience or paper, or any algorithm? @kwentar", "votes": null }, { "id": "1746894", "postDate": "04/06/2022 07:40:50", "content": "

I mean in this competition from my experience, but fastai says it works everywhere, also paper about it

", "rawMarkdown": "I mean in this competition from my experience, but fastai says it works everywhere, also [paper ](https://arxiv.org/pdf/1803.09820.pdf) about it", "votes": null } ], "comments": [ { "id": 1744735, "author_name": "kwentar", "author_url": "", "post_date": "04/04/2022 08:59:44", "content": "

This is simple ExponentialLR with warmup, it looks like this:
\n\"lr\"
\nIt is not the best and only scheduler for this task, just one of.

", "votes": null, "replies": [ { "id": 1746772, "author_name": "remekkinas", "author_url": "", "post_date": "04/06/2022 04:56:29", "content": "

@kwentar what is your recommendation?

", "votes": null, "replies": [] }, { "id": 1746873, "author_name": "kwentar", "author_url": "", "post_date": "04/06/2022 07:26:47", "content": "

for me works good CosineAnnealingLR and LR scheduler from FastAI

", "votes": null, "replies": [] }, { "id": 1746885, "author_name": "phanttan", "author_url": "", "post_date": "04/06/2022 07:33:30", "content": "

and that from experience or paper, or any algorithm? @kwentar

", "votes": null, "replies": [] }, { "id": 1746894, "author_name": "kwentar", "author_url": "", "post_date": "04/06/2022 07:40:50", "content": "

I mean in this competition from my experience, but fastai says it works everywhere, also paper about it

", "votes": null, "replies": [] } ] }, { "id": 1745346, "author_name": "init27", "author_url": "", "post_date": "04/04/2022 21:10:33", "content": "

I would recommend checking out the lr scheduler in fastai, it can give a slightly better performance

", "votes": null, "replies": [] }, { "id": 1746746, "author_name": "lextoumbourou", "author_url": "", "post_date": "04/06/2022 04:22:24", "content": "

A lot of the code shared in the public kernels comes from the Google Collab Flowers TPU example, including the LR scheduler.

\n

https://colab.research.google.com/notebooks/tpu.ipynb

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1744727": "Many notebooks (in this competition) are used this function for **Learning Rate **\n```\ndef get_lr_callback(plot=False):\n lr_start = 0.000001\n lr_max = 0.000005 * config.BATCH_SIZE \n lr_min = 0.000001\n lr_ramp_ep = 4\n lr_sus_ep = 0\n lr_decay = 0.9\n \n def lrfn(epoch):\n if config.RESUME:\n epoch = epoch + config.RESUME_EPOCH\n if epoch < lr_ramp_ep:\n lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n \n elif epoch < lr_ramp_ep + lr_sus_ep:\n lr = lr_max\n \n else:\n lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min\n \n return lr\n \n if plot:\n epochs = list(range(config.EPOCHS))\n learning_rates = [lrfn(x) for x in epochs]\n plt.scatter(epochs,learning_rates)\n plt.show()\n\n lr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=False)\n return lr_callback\n\nget_lr_callback(plot=True)\n```\n\nI do not know when is the idea for this implementation?\nAnd what is useful to EfficientNet Architecture or in larger? \n\nP/S : I try to find out the answer in comments but maybe it is not exist, please show me if you know.", "1744735": "This is simple [ExponentialLR ](https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.ExponentialLR.html) with [warmup](https://stackoverflow.com/questions/55933867/what-does-learning-rate-warm-up-mean), it looks like this:\n![lr](https://i.ibb.co/vzkWWwW/image.png)\nIt is not the best and only scheduler for this task, just one of.", "1745346": "I would recommend checking out the lr scheduler in fastai, it can give a slightly better performance", "1746746": "A lot of the code shared in the public kernels comes from the Google Collab Flowers TPU example, including the LR scheduler.\n\nhttps://colab.research.google.com/notebooks/tpu.ipynb", "1746772": "kwentar what is your recommendation?", "1746873": "for me works good [CosineAnnealingLR ](https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.CosineAnnealingLR.html) and LR [scheduler](https://fastai1.fast.ai/callbacks.one_cycle.html) from FastAI", "1746885": "and that from experience or paper, or any algorithm? @kwentar", "1746894": "I mean in this competition from my experience, but fastai says it works everywhere, also [paper ](https://arxiv.org/pdf/1803.09820.pdf) about it" }, "source": "meta" }