{ "id": 77065, "title": "different LR for different layers in Keras", "url": "/competitions/human-protein-atlas-image-classification/discussion/77065", "author_name": "", "post_date": "2019-01-09T06:48:36.370578800Z", "votes": 4, "comment_count": 12, "views": 0, "content": "

I use transfer learning for this competition and want to realize different learning rates for different layers. For example, learning rate of pretrained layer need to slower than next higher layer each epoch. But I can't find anything about this in Keras（tensorflow backend）. How to realize different learning rate for different layers in Keras.

", "messages": [ { "id": "452788", "postDate": "01/09/2019 06:48:36", "content": "

I use transfer learning for this competition and want to realize different learning rates for different layers. For example, learning rate of pretrained layer need to slower than next higher layer each epoch. But I can't find anything about this in Keras（tensorflow backend）. How to realize different learning rate for different layers in Keras.

", "rawMarkdown": "I use transfer learning for this competition and want to realize different learning rates for different layers. For example, learning rate of pretrained layer need to slower than next higher layer each epoch. But I can't find anything about this in Keras（tensorflow backend）. How to realize different learning rate for different layers in Keras.", "votes": null }, { "id": "452815", "postDate": "01/09/2019 07:33:42", "content": "

Unfortunately, there is no out-of-box differential learning rate for Keras.

", "rawMarkdown": "Unfortunately, there is no out-of-box differential learning rate for Keras.", "votes": null }, { "id": "453131", "postDate": "01/09/2019 18:10:51", "content": "

yes. I am really waiting for that.

", "rawMarkdown": "yes. I am really waiting for that.", "votes": null }, { "id": "453133", "postDate": "01/09/2019 18:14:14", "content": "

For the next image competition, I will switch to pytorch :P

", "rawMarkdown": "For the next image competition, I will switch to pytorch :P", "votes": null }, { "id": "453146", "postDate": "01/09/2019 18:30:02", "content": "

I did it too, and differential learning rate was one of the reasons.\nAlso, PyTorch results are more stable in terms of reproducibility.

", "rawMarkdown": "I did it too, and differential learning rate was one of the reasons.\nAlso, PyTorch results are more stable in terms of reproducibility.", "votes": null }, { "id": "453190", "postDate": "01/09/2019 20:39:02", "content": "

I was a little discourage at first in attempting to learn Pytorch. I feel it's time to switch after seeing the results on kaggle and noticing a lot of researchers using it. I guess you have to start somewhere.

", "rawMarkdown": "I was a little discourage at first in attempting to learn Pytorch. I feel it's time to switch after seeing the results on kaggle and noticing a lot of researchers using it. I guess you have to start somewhere.", "votes": null }, { "id": "453331", "postDate": "01/10/2019 02:46:23", "content": "

thx, William. It seems that i have to learn pytorch and fastai.

", "rawMarkdown": "thx, William. It seems that i have to learn pytorch and fastai.", "votes": null }, { "id": "453333", "postDate": "01/10/2019 02:47:56", "content": "

thx for your reply!

", "rawMarkdown": "thx for your reply!", "votes": null }, { "id": "453670", "postDate": "01/10/2019 15:23:55", "content": "

Why do you think, that differential learning will be helpfull ?

", "rawMarkdown": "Why do you think, that differential learning will be helpfull ?", "votes": null }, { "id": "453682", "postDate": "01/10/2019 15:48:33", "content": "

Because it allows you to apply smooth transfer learning.

", "rawMarkdown": "Because it allows you to apply smooth transfer learning.", "votes": null }, { "id": "453837", "postDate": "01/10/2019 21:46:21", "content": "

You can Start With this https://www.kaggle.com/ashishpatel26/zero-to-hero-in-pytorch-ml-dl-rl\nI find it very useful.

", "rawMarkdown": "You can Start With this https://www.kaggle.com/ashishpatel26/zero-to-hero-in-pytorch-ml-dl-rl\nI find it very useful.", "votes": null }, { "id": "453851", "postDate": "01/10/2019 22:27:14", "content": "

aha, thank you for your sharing! : )

", "rawMarkdown": "aha, thank you for your sharing! : )", "votes": null }, { "id": "454030", "postDate": "01/11/2019 03:30:01", "content": "

https://github.com/qrfaction/Learning-Rate-multipliers-in-Keras\ndiff lr in keras

", "rawMarkdown": "https://github.com/qrfaction/Learning-Rate-multipliers-in-Keras\ndiff lr in keras", "votes": null } ], "comments": [ { "id": 452815, "author_name": "hokmund", "author_url": "", "post_date": "01/09/2019 07:33:42", "content": "

Unfortunately, there is no out-of-box differential learning rate for Keras.

", "votes": null, "replies": [ { "id": 453131, "author_name": "christofhenkel", "author_url": "", "post_date": "01/09/2019 18:10:51", "content": "

yes. I am really waiting for that.

", "votes": null, "replies": [] }, { "id": 453133, "author_name": "christofhenkel", "author_url": "", "post_date": "01/09/2019 18:14:14", "content": "

For the next image competition, I will switch to pytorch :P

", "votes": null, "replies": [] }, { "id": 453146, "author_name": "hokmund", "author_url": "", "post_date": "01/09/2019 18:30:02", "content": "

I did it too, and differential learning rate was one of the reasons.\nAlso, PyTorch results are more stable in terms of reproducibility.

", "votes": null, "replies": [] }, { "id": 453333, "author_name": "shangweichen", "author_url": "", "post_date": "01/10/2019 02:47:56", "content": "

thx for your reply!

", "votes": null, "replies": [] }, { "id": 453670, "author_name": "unofloristo", "author_url": "", "post_date": "01/10/2019 15:23:55", "content": "

Why do you think, that differential learning will be helpfull ?

", "votes": null, "replies": [] }, { "id": 453682, "author_name": "hokmund", "author_url": "", "post_date": "01/10/2019 15:48:33", "content": "

Because it allows you to apply smooth transfer learning.

", "votes": null, "replies": [] } ] }, { "id": 453190, "author_name": "dskswu", "author_url": "", "post_date": "01/09/2019 20:39:02", "content": "

I was a little discourage at first in attempting to learn Pytorch. I feel it's time to switch after seeing the results on kaggle and noticing a lot of researchers using it. I guess you have to start somewhere.

", "votes": null, "replies": [ { "id": 453331, "author_name": "shangweichen", "author_url": "", "post_date": "01/10/2019 02:46:23", "content": "

thx, William. It seems that i have to learn pytorch and fastai.

", "votes": null, "replies": [] } ] }, { "id": 453837, "author_name": "jmourad100", "author_url": "", "post_date": "01/10/2019 21:46:21", "content": "

You can Start With this https://www.kaggle.com/ashishpatel26/zero-to-hero-in-pytorch-ml-dl-rl\nI find it very useful.

", "votes": null, "replies": [ { "id": 453851, "author_name": "shangweichen", "author_url": "", "post_date": "01/10/2019 22:27:14", "content": "

aha, thank you for your sharing! : )

", "votes": null, "replies": [] } ] }, { "id": 454030, "author_name": "action", "author_url": "", "post_date": "01/11/2019 03:30:01", "content": "

https://github.com/qrfaction/Learning-Rate-multipliers-in-Keras\ndiff lr in keras

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "452788": "I use transfer learning for this competition and want to realize different learning rates for different layers. For example, learning rate of pretrained layer need to slower than next higher layer each epoch. But I can't find anything about this in Keras（tensorflow backend）. How to realize different learning rate for different layers in Keras.", "452815": "Unfortunately, there is no out-of-box differential learning rate for Keras.", "453131": "yes. I am really waiting for that.", "453133": "For the next image competition, I will switch to pytorch :P", "453146": "I did it too, and differential learning rate was one of the reasons.\nAlso, PyTorch results are more stable in terms of reproducibility.", "453190": "I was a little discourage at first in attempting to learn Pytorch. I feel it's time to switch after seeing the results on kaggle and noticing a lot of researchers using it. I guess you have to start somewhere.", "453331": "thx, William. It seems that i have to learn pytorch and fastai.", "453333": "thx for your reply!", "453670": "Why do you think, that differential learning will be helpfull ?", "453682": "Because it allows you to apply smooth transfer learning.", "453837": "You can Start With this https://www.kaggle.com/ashishpatel26/zero-to-hero-in-pytorch-ml-dl-rl\nI find it very useful.", "453851": "aha, thank you for your sharing! : )", "454030": "https://github.com/qrfaction/Learning-Rate-multipliers-in-Keras\ndiff lr in keras" }, "source": "meta" }