Has anyone tried optimizers other than Adam or SGD momentum? If yes, do share your experiments.
\ntorch-optimizer - This package contains implementation of a lot of optimizers
Read about Adabound some time back
\n\n\nAn optimizer that trains as fast as Adam and as good as SGD for developing state-of-the-art deep learning models on a wide variety of popular tasks in the field of CV, NLP, etc.
\n
Official Repo - https://github.com/Luolc/AdaBound
\nI haven't tried any except Adam yet.
", "messages": [ { "id": 1130463, "postDate": "2020-12-29T04:37:19.883Z", "content": "Has anyone tried optimizers other than Adam or SGD momentum? If yes, do share your experiments.
\ntorch-optimizer - This package contains implementation of a lot of optimizers
Read about Adabound some time back
\n\n\nAn optimizer that trains as fast as Adam and as good as SGD for developing state-of-the-art deep learning models on a wide variety of popular tasks in the field of CV, NLP, etc.
\n
Official Repo - https://github.com/Luolc/AdaBound
\nI haven't tried any except Adam yet.
", "rawMarkdown": "Has anyone tried optimizers other than Adam or SGD momentum? If yes, do share your experiments.\n[torch-optimizer](https://github.com/jettify/pytorch-optimizer) - This package contains implementation of a lot of optimizers\n\nRead about [Adabound](https://arxiv.org/abs/1902.09843) some time back \n> An optimizer that trains as fast as Adam and as good as SGD for developing state-of-the-art deep learning models on a wide variety of popular tasks in the field of CV, NLP, etc.\n\nOfficial Repo - [https://github.com/Luolc/AdaBound](https://github.com/Luolc/AdaBound)\n\nI haven't tried any except Adam yet.", "votes": 16 }, { "id": 1130684, "postDate": "2020-12-29T08:25:55.360Z", "content": "yeah ranger is doing pretty good actually, i am using it too
", "rawMarkdown": "yeah ranger is doing pretty good actually, i am using it too", "votes": 3 }, { "id": 1130528, "postDate": "2020-12-29T05:42:54.257Z", "content": "AdamW and Ranger are my go-to optimizers and here Ranger has done pretty decently.
", "rawMarkdown": "AdamW and Ranger are my go-to optimizers and here Ranger has done pretty decently.", "votes": 3, "replies": [ { "id": 1130607, "postDate": "2020-12-29T06:57:14.860Z", "content": "Never heard about Ranger! I Will surely give it a try
", "rawMarkdown": "Never heard about Ranger! I Will surely give it a try", "votes": 1 } ] }, { "id": 1147486, "postDate": "2021-01-10T14:38:44.817Z", "content": "Try AdamP and SGDP, I haven’t tried it here yet but worked well in other problems.
\nhttps://arxiv.org/abs/2006.08217
\nhttps://github.com/clovaai/AdamP
\nhttps://catalyst-team.github.io/catalyst/api/contrib.html#module-catalyst.contrib.nn.optimizers.adamp
Thank you for sharing
", "rawMarkdown": "Thank you for sharing" } ] }, { "id": 1136768, "postDate": "2021-01-03T12:06:13.570Z", "content": "Thank you for share @debarshichanda . For Keras implementation Check This.
", "rawMarkdown": "Thank you for share @debarshichanda . For Keras implementation Check [This](https://github.com/titu1994/keras-adabound).", "votes": 1, "replies": [ { "id": 1139438, "postDate": "2021-01-05T11:49:37.913Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ] }, { "id": 1130778, "postDate": "2020-12-29T09:52:32.867Z", "content": "Is there a faster method to experiment with different optimizers without training on whole data? That would speed up the process.
", "rawMarkdown": "Is there a faster method to experiment with different optimizers without training on whole data? That would speed up the process.", "votes": 1, "replies": [ { "id": 1130875, "postDate": "2020-12-29T11:14:06.747Z", "content": "There's the option of training in small networks (e.g. EfficientNet B0, B1 or B2). That's what I've done for experimentation and I am taking a bet that the same things will work for B5 to B7 that I might use at the end.
", "rawMarkdown": "There's the option of training in small networks (e.g. EfficientNet B0, B1 or B2). That's what I've done for experimentation and I am taking a bet that the same things will work for B5 to B7 that I might use at the end.", "votes": 2 }, { "id": 1130894, "postDate": "2020-12-29T11:50:04.647Z", "content": "that's a good point, I will try it
", "rawMarkdown": "that's a good point, I will try it" }, { "id": 1130897, "postDate": "2020-12-29T11:51:59.993Z", "content": "@bjoernholzhauer Looks like a good idea!
", "rawMarkdown": "@bjoernholzhauer Looks like a good idea!" }, { "id": 1131352, "postDate": "2020-12-29T17:12:04.453Z", "content": "If you scroll down on the page mentioned in this topic: https://github.com/jettify/pytorch-optimizer
\nThe creator does also compare the optimizers based on a set amount of iteration (501 iterations)
\nYou can do the same for your model with different optimizers (set an amount of iterations and check the loss) or on certain batch-sizes train your model for N certain batches with an optimizer and check which has the lowest loss.
One optimizer claimed to be better than AdaBound is the AdaBelief https://juntang-zhuang.github.io/adabelief/
\nIt just came out in NeurIPS 2020
Thank you!
\nThis looks promising
There is a Keras compatible update if anyone is looking for as per Author's response here: https://github.com/juntang-zhuang/Adabelief-Optimizer/issues/2#issuecomment-721498692
\nLink: https://github.com/juntang-zhuang/Adabelief-Optimizer
", "rawMarkdown": "There is a Keras compatible update if anyone is looking for as per Author's response here: https://github.com/juntang-zhuang/Adabelief-Optimizer/issues/2#issuecomment-721498692\n\nLink: https://github.com/juntang-zhuang/Adabelief-Optimizer", "votes": 1 } ] }, { "id": 1133761, "postDate": "2020-12-31T14:03:01.370Z", "content": "For me, Adam work's well. Haven't tried others.
\nIn case anyone wants Keras code for ranger optimizer:
radam = tfa.optimizers.RectifiedAdam()\nranger = tfa.optimizers.Lookahead(radam, sync_period=6, slow_step_size=0.5)\nYou can find it here
", "rawMarkdown": "For me, Adam work's well. Haven't tried others.\nIn case anyone wants Keras code for `ranger` optimizer:\n\n```\nradam = tfa.optimizers.RectifiedAdam()\nranger = tfa.optimizers.Lookahead(radam, sync_period=6, slow_step_size=0.5)\n```\nYou can find it [here](https://www.tensorflow.org/addons/api_docs/python/tfa/optimizers/RectifiedAdam)", "votes": 2 }, { "id": 1131435, "postDate": "2020-12-29T18:12:27.903Z", "content": "any implementation of Ranger on Pytorch?
\nthanks
You can find it under: https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer
", "rawMarkdown": "You can find it under: https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer", "votes": 2 }, { "id": 1131452, "postDate": "2020-12-29T18:22:55.587Z", "content": "This package also has its implementation
\nhttps://github.com/jettify/pytorch-optimizer
Thanks for sharing.
", "rawMarkdown": "Thanks for sharing.", "votes": 1 }, { "id": 1131506, "postDate": "2020-12-29T19:15:18.563Z", "content": "nice. Thanks for the share.
", "rawMarkdown": "nice. Thanks for the share.", "votes": 2 }, { "id": 1133553, "postDate": "2020-12-31T09:56:30.953Z", "content": "Thanks for the share.
", "rawMarkdown": "Thanks for the share." } ], "comments": [ { "id": 1130684, "author_name": "Eren Tekin", "author_url": "", "post_date": "2020-12-29T08:25:55.360000", "content": "yeah ranger is doing pretty good actually, i am using it too
", "votes": 3, "replies": [] }, { "id": 1130528, "author_name": "ilovescience", "author_url": "", "post_date": "2020-12-29T05:42:54.257000", "content": "AdamW and Ranger are my go-to optimizers and here Ranger has done pretty decently.
", "votes": 3, "replies": [ { "id": 1130607, "author_name": "Debarshi Chanda", "author_url": "", "post_date": "2020-12-29T06:57:14.860000", "content": "Never heard about Ranger! I Will surely give it a try
", "votes": 1, "replies": [] } ] }, { "id": 1147486, "author_name": "Kirderf", "author_url": "", "post_date": "2021-01-10T14:38:44.817000", "content": "Try AdamP and SGDP, I haven’t tried it here yet but worked well in other problems.
\nhttps://arxiv.org/abs/2006.08217
\nhttps://github.com/clovaai/AdamP
\nhttps://catalyst-team.github.io/catalyst/api/contrib.html#module-catalyst.contrib.nn.optimizers.adamp
Thank you for sharing
", "votes": 0, "replies": [] } ] }, { "id": 1136768, "author_name": "AIFahim", "author_url": "", "post_date": "2021-01-03T12:06:13.570000", "content": "Thank you for share @debarshichanda . For Keras implementation Check This.
", "votes": 1, "replies": [ { "id": 1139438, "author_name": "", "author_url": "", "post_date": "2021-01-05T11:49:37.913000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 1130778, "author_name": "Fares Lassoued", "author_url": "", "post_date": "2020-12-29T09:52:32.867000", "content": "Is there a faster method to experiment with different optimizers without training on whole data? That would speed up the process.
", "votes": 1, "replies": [ { "id": 1130875, "author_name": "Björn", "author_url": "", "post_date": "2020-12-29T11:14:06.747000", "content": "There's the option of training in small networks (e.g. EfficientNet B0, B1 or B2). That's what I've done for experimentation and I am taking a bet that the same things will work for B5 to B7 that I might use at the end.
", "votes": 2, "replies": [] }, { "id": 1130894, "author_name": "Fares Lassoued", "author_url": "", "post_date": "2020-12-29T11:50:04.647000", "content": "that's a good point, I will try it
", "votes": 0, "replies": [] }, { "id": 1130897, "author_name": "Debarshi Chanda", "author_url": "", "post_date": "2020-12-29T11:51:59.993000", "content": "@bjoernholzhauer Looks like a good idea!
", "votes": 0, "replies": [] }, { "id": 1131352, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-12-29T17:12:04.453000", "content": "If you scroll down on the page mentioned in this topic: https://github.com/jettify/pytorch-optimizer
\nThe creator does also compare the optimizers based on a set amount of iteration (501 iterations)
\nYou can do the same for your model with different optimizers (set an amount of iterations and check the loss) or on certain batch-sizes train your model for N certain batches with an optimizer and check which has the lowest loss.
One optimizer claimed to be better than AdaBound is the AdaBelief https://juntang-zhuang.github.io/adabelief/
\nIt just came out in NeurIPS 2020
Thank you!
\nThis looks promising
There is a Keras compatible update if anyone is looking for as per Author's response here: https://github.com/juntang-zhuang/Adabelief-Optimizer/issues/2#issuecomment-721498692
\nLink: https://github.com/juntang-zhuang/Adabelief-Optimizer
", "votes": 1, "replies": [] } ] }, { "id": 1133761, "author_name": "Prateek Mishra", "author_url": "", "post_date": "2020-12-31T14:03:01.370000", "content": "For me, Adam work's well. Haven't tried others.
\nIn case anyone wants Keras code for ranger optimizer:
radam = tfa.optimizers.RectifiedAdam()\nranger = tfa.optimizers.Lookahead(radam, sync_period=6, slow_step_size=0.5)\nYou can find it here
", "votes": 2, "replies": [] }, { "id": 1131435, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2020-12-29T18:12:27.903000", "content": "any implementation of Ranger on Pytorch?
\nthanks
You can find it under: https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer
", "votes": 2, "replies": [] }, { "id": 1131452, "author_name": "Debarshi Chanda", "author_url": "", "post_date": "2020-12-29T18:22:55.587000", "content": "This package also has its implementation
\nhttps://github.com/jettify/pytorch-optimizer
Thanks for sharing.
", "votes": 1, "replies": [] }, { "id": 1131506, "author_name": "Ashik M", "author_url": "", "post_date": "2020-12-29T19:15:18.563000", "content": "nice. Thanks for the share.
", "votes": 2, "replies": [] }, { "id": 1133553, "author_name": "Bhaskar Sarma", "author_url": "", "post_date": "2020-12-31T09:56:30.953000", "content": "Thanks for the share.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1130463": "Has anyone tried optimizers other than Adam or SGD momentum? If yes, do share your experiments.\n[torch-optimizer](https://github.com/jettify/pytorch-optimizer) - This package contains implementation of a lot of optimizers\n\nRead about [Adabound](https://arxiv.org/abs/1902.09843) some time back \n> An optimizer that trains as fast as Adam and as good as SGD for developing state-of-the-art deep learning models on a wide variety of popular tasks in the field of CV, NLP, etc.\n\nOfficial Repo - [https://github.com/Luolc/AdaBound](https://github.com/Luolc/AdaBound)\n\nI haven't tried any except Adam yet.", "1130684": "yeah ranger is doing pretty good actually, i am using it too", "1130528": "AdamW and Ranger are my go-to optimizers and here Ranger has done pretty decently.", "1147486": "Try AdamP and SGDP, I haven’t tried it here yet but worked well in other problems.\n\nhttps://arxiv.org/abs/2006.08217\nhttps://github.com/clovaai/AdamP\nhttps://catalyst-team.github.io/catalyst/api/contrib.html#module-catalyst.contrib.nn.optimizers.adamp\n", "1136768": "Thank you for share @debarshichanda . For Keras implementation Check [This](https://github.com/titu1994/keras-adabound).", "1130778": "Is there a faster method to experiment with different optimizers without training on whole data? That would speed up the process.", "1137427": "One optimizer claimed to be better than AdaBound is the AdaBelief https://juntang-zhuang.github.io/adabelief/\nIt just came out in NeurIPS 2020", "1133761": "For me, Adam work's well. Haven't tried others.\nIn case anyone wants Keras code for `ranger` optimizer:\n\n```\nradam = tfa.optimizers.RectifiedAdam()\nranger = tfa.optimizers.Lookahead(radam, sync_period=6, slow_step_size=0.5)\n```\nYou can find it [here](https://www.tensorflow.org/addons/api_docs/python/tfa/optimizers/RectifiedAdam)", "1131435": "any implementation of Ranger on Pytorch?\nthanks", "1130571": "Thanks for sharing.", "1131506": "nice. Thanks for the share.", "1133553": "Thanks for the share." } }