Which optimization algo works best for training your UNet?
", "messages": [ { "id": 216040, "postDate": "2017-08-24T04:45:41.113Z", "content": "Which optimization algo works best for training your UNet?
", "rawMarkdown": "Which optimization algo works best for training your UNet?", "votes": 11 }, { "id": 216084, "postDate": "2017-08-24T08:31:48.267Z", "content": "Adam seems to work better compared to SGD for me. I've yet to try RMSProp.
", "rawMarkdown": "Adam seems to work better compared to SGD for me. I've yet to try RMSProp.", "votes": 4 }, { "id": 216057, "postDate": "2017-08-24T06:36:29.733Z", "content": "I got better results with RMSProp and SGD than with Adam.
", "rawMarkdown": "I got better results with RMSProp and SGD than with Adam. ", "votes": 2 }, { "id": 218363, "postDate": "2017-09-04T00:59:49.173Z", "content": "Adam was the best for me so far. SGD came in second, with RMSprop in 3rd. I might return to RMSprop later since I didn't give it that much time
", "rawMarkdown": "Adam was the best for me so far. SGD came in second, with RMSprop in 3rd. I might return to RMSprop later since I didn't give it that much time", "votes": 1, "replies": [ { "id": 218382, "postDate": "2017-09-04T02:26:16.587Z", "content": "Could you please show your batch size and input size?
", "rawMarkdown": "Could you please show your batch size and input size?" }, { "id": 218388, "postDate": "2017-09-04T03:04:18.923Z", "content": "Sure thing. My batch size is 2 for training and 8 for inference, using bn. My input size is 1024x1024, using hq images. My hypothesis is that smaller batch sizes actually perform well here since pixel positional information is important in this challenge
", "rawMarkdown": "Sure thing. My batch size is 2 for training and 8 for inference, using bn. My input size is 1024x1024, using hq images. My hypothesis is that smaller batch sizes actually perform well here since pixel positional information is important in this challenge" }, { "id": 221131, "postDate": "2017-09-14T08:04:25.857Z", "content": "Have you ever tried BR layer. In my net , BR layer got 0.0002 higher LB score than BN layer.
", "rawMarkdown": "Have you ever tried BR layer. In my net , BR layer got 0.0002 higher LB score than BN layer.", "votes": 2 }, { "id": 221410, "postDate": "2017-09-15T06:54:00.470Z", "content": "No I haven't actually https://arxiv.org/pdf/1702.03275.pdf
\n\nI think this is a pretty good idea for several image recognition competitions, not just this one, since it appears the higher resolution models that sacrifice batch size do better from my experience
", "rawMarkdown": "No I haven't actually https://arxiv.org/pdf/1702.03275.pdf\n\nI think this is a pretty good idea for several image recognition competitions, not just this one, since it appears the higher resolution models that sacrifice batch size do better from my experience" } ] }, { "id": 216154, "postDate": "2017-08-24T14:49:50.467Z", "content": "Depending on your hardware and batch_size. You try and test it on your system and you will find the answer lay within your notebook.
", "rawMarkdown": "Depending on your hardware and batch_size. You try and test it on your system and you will find the answer lay within your notebook." }, { "id": 217872, "postDate": "2017-09-01T09:52:58.027Z", "content": "http://ruder.io/optimizing-gradient-descent/index.html#gradientdescentoptimizationalgorithms
\n\nI hope this helps to clear your understanding on optimizers.
", "rawMarkdown": "http://ruder.io/optimizing-gradient-descent/index.html#gradientdescentoptimizationalgorithms\n\nI hope this helps to clear your understanding on optimizers.", "replies": [ { "id": 218364, "postDate": "2017-09-04T01:05:24.550Z", "content": "Dont think the OP was asking for clarification, but i do think this is a good post here. thanks
", "rawMarkdown": "Dont think the OP was asking for clarification, but i do think this is a good post here. thanks " } ] }, { "id": 1531533, "postDate": "2021-10-02T05:22:36.100Z", "content": "see this : An overview of gradient descent optimization algorithms
", "rawMarkdown": "see this : [An overview of gradient descent optimization algorithms](https://ruder.io/optimizing-gradient-descent/index.html)" }, { "id": 887486, "postDate": "2020-06-15T17:38:15.223Z", "content": "For me, SGD work pretty good, if you face any issue with SGD try with bit low learning rate and use the value of momentum around 0.85 to 0.95
", "rawMarkdown": "For me, SGD work pretty good, if you face any issue with SGD try with bit low learning rate and use the value of momentum around 0.85 to 0.95" }, { "id": 218601, "postDate": "2017-09-05T05:08:27.343Z", "content": "rmsprop uses only sign and not magnitude of the computed gradients for descend. A intermediate solution is gradient capping:
\n\n loss.backward()\n # accumulate gradients\n if (it+1)%num_grad_acc==0:\n torch.nn.utils.clip_grad_norm(net.parameters(), 1)\n\n optimizer.step()\n optimizer.zero_grad() # assume no effects on bn for accumulating grad\nI find this effective in preventing oscillation in loss.
", "rawMarkdown": "rmsprop uses only sign and not magnitude of the computed gradients for descend. A intermediate solution is gradient capping:\n\n loss.backward()\n # accumulate gradients\n if (it+1)%num_grad_acc==0:\n torch.nn.utils.clip_grad_norm(net.parameters(), 1)\n\n optimizer.step()\n optimizer.zero_grad() # assume no effects on bn for accumulating grad\n\nI find this effective in preventing oscillation in loss.", "replies": [ { "id": 218794, "postDate": "2017-09-05T22:42:09.947Z", "content": "This is for pytorch right? How do you do that with keras?
", "rawMarkdown": "This is for pytorch right? How do you do that with keras?" }, { "id": 218811, "postDate": "2017-09-06T00:09:55.147Z", "content": "\n\n\nrmsprop uses only sign and not magnitude of the computed gradients for descend.
\n
I don't understand this well, so could you elaborate this mean?
\n\n@YaGana Sheriff-Hussaini\nProbably we can get by this code
\n\n\noptimizer = RMSprop(lr=0.0001, clipnorm=1.)\n
I'm worried about that this gradient capping causes worse effect to learning.
\n\nTo get more stability of RMSprop, enlarging epsilon (1e-8 is keras default. change 1e-6.) is also a way.
", "rawMarkdown": " \n\n> rmsprop uses only sign and not magnitude of the computed gradients for descend.\n\nI don't understand this well, so could you elaborate this mean?\n\n@YaGana Sheriff-Hussaini\nProbably we can get by this code \n\n```\noptimizer = RMSprop(lr=0.0001, clipnorm=1.)\n```\n\nI'm worried about that this gradient capping causes worse effect to learning. \n\nTo get more stability of RMSprop, enlarging epsilon (1e-8 is keras default. change 1e-6.) is also a way.", "votes": 3 }, { "id": 218873, "postDate": "2017-09-06T06:27:36.520Z", "content": "@Heng, I have not used pytorch so I do not know how RMSprop is implemented there. But I took a look at the implementation of RMSprop in keras, and I do not think the oscillations you mentioned would be a problem. By definition, RMSprop uses oscillation reduction and momentum in tandem to achieve the desired effect.
\n\nThanks @Iyakaap. I have not tried it, but for those using Keras, I am also of the belief that it may very well result in an unwanted effect.
\n\nIn Keras, RMSprop is implemented according to http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf
", "rawMarkdown": "@Heng, I have not used pytorch so I do not know how RMSprop is implemented there. But I took a look at the implementation of RMSprop in keras, and I do not think the oscillations you mentioned would be a problem. By definition, RMSprop uses oscillation reduction and momentum in tandem to achieve the desired effect.\n\nThanks @Iyakaap. I have not tried it, but for those using Keras, I am also of the belief that it may very well result in an unwanted effect.\n\nIn Keras, RMSprop is implemented according to http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf" }, { "id": 219707, "postDate": "2017-09-09T08:23:08.863Z", "content": "Why enlarging epsilon will make RMSprop more stable? Thanks for more explaining~
\nAnd Keras is not recommending to change the eplison value. Any benefits can get according to your experiment?
RMSprop
\n\nkeras.optimizers.RMSprop(lr=0.001, rho=0.9, epsilon=1e-08, decay=0.0)\nRMSProp optimizer.
\n\nIt is recommended to leave the parameters of this optimizer at their default values (except the learning rate, which can be freely tuned).\nupdate:\nsame that epsilon here is added for non-zero dividing and so... it makes sense
", "rawMarkdown": "Why enlarging epsilon will make RMSprop more stable? Thanks for more explaining~ \nAnd Keras is not recommending to change the eplison value. Any benefits can get according to your experiment?\n\nRMSprop\n\nkeras.optimizers.RMSprop(lr=0.001, rho=0.9, epsilon=1e-08, decay=0.0)\nRMSProp optimizer.\n\nIt is recommended to leave the parameters of this optimizer at their default values (except the learning rate, which can be freely tuned).\nupdate:\nsame that epsilon here is added for non-zero dividing and so... it makes sense\n" }, { "id": 221129, "postDate": "2017-09-14T07:51:18.800Z", "content": "I also have the same question. Could you please show more details about it?
", "rawMarkdown": " I also have the same question. Could you please show more details about it?" }, { "id": 221515, "postDate": "2017-09-15T14:25:14.777Z", "content": "I reviewed the definition and found that epsilon does not matter so much. The most straight way is shrinking lr moderately. And if we want to do training model more conservatively, enlarging rho might help.
", "rawMarkdown": "I reviewed the definition and found that epsilon does not matter so much. The most straight way is shrinking lr moderately. And if we want to do training model more conservatively, enlarging rho might help." } ] }, { "id": 217882, "postDate": "2017-09-01T11:03:48.353Z", "content": "For me only SGD works right away... mhhm
", "rawMarkdown": "For me only SGD works right away... mhhm" }, { "id": 217820, "postDate": "2017-09-01T04:43:39.703Z", "content": "Many people reported RMSProp works better than Adam for this contest so I tried RMSProp after using Adam exclusively. For me, my RMSProp result was slightly worse than Adam. I am using batch_size = 1 if that matters. I will stick with Adam for now.
", "rawMarkdown": "Many people reported RMSProp works better than Adam for this contest so I tried RMSProp after using Adam exclusively. For me, my RMSProp result was slightly worse than Adam. I am using batch_size = 1 if that matters. I will stick with Adam for now." }, { "id": 1292351, "postDate": "2021-05-03T21:08:53.590Z", "content": "I prefer adam, but it is important to pay attention to Beta_2 h-params as it affects your momentum vector and has a huge influence.
\nRegarding RMSProp, one could use adam instead. As both of them are in the same category and consider the Exponential moving average for previous gradients.
\nSo in my opinion is a matter of ADAGRAD, SGD vanilla, or RMSPROP family!
Adam seems to work better compared to SGD for me. I've yet to try RMSProp.
", "votes": 4, "replies": [] }, { "id": 216057, "author_name": "James Requa", "author_url": "", "post_date": "2017-08-24T06:36:29.733000", "content": "I got better results with RMSProp and SGD than with Adam.
", "votes": 2, "replies": [] }, { "id": 218363, "author_name": "SpruceMoose", "author_url": "", "post_date": "2017-09-04T00:59:49.173000", "content": "Adam was the best for me so far. SGD came in second, with RMSprop in 3rd. I might return to RMSprop later since I didn't give it that much time
", "votes": 1, "replies": [ { "id": 218382, "author_name": "Will", "author_url": "", "post_date": "2017-09-04T02:26:16.587000", "content": "Could you please show your batch size and input size?
", "votes": 0, "replies": [] }, { "id": 218388, "author_name": "SpruceMoose", "author_url": "", "post_date": "2017-09-04T03:04:18.923000", "content": "Sure thing. My batch size is 2 for training and 8 for inference, using bn. My input size is 1024x1024, using hq images. My hypothesis is that smaller batch sizes actually perform well here since pixel positional information is important in this challenge
", "votes": 0, "replies": [] }, { "id": 221131, "author_name": "Will", "author_url": "", "post_date": "2017-09-14T08:04:25.857000", "content": "Have you ever tried BR layer. In my net , BR layer got 0.0002 higher LB score than BN layer.
", "votes": 2, "replies": [] }, { "id": 221410, "author_name": "SpruceMoose", "author_url": "", "post_date": "2017-09-15T06:54:00.470000", "content": "No I haven't actually https://arxiv.org/pdf/1702.03275.pdf
\n\nI think this is a pretty good idea for several image recognition competitions, not just this one, since it appears the higher resolution models that sacrifice batch size do better from my experience
", "votes": 0, "replies": [] } ] }, { "id": 216154, "author_name": "Kaggoo", "author_url": "", "post_date": "2017-08-24T14:49:50.467000", "content": "Depending on your hardware and batch_size. You try and test it on your system and you will find the answer lay within your notebook.
", "votes": 0, "replies": [] }, { "id": 217872, "author_name": "remidi", "author_url": "", "post_date": "2017-09-01T09:52:58.027000", "content": "http://ruder.io/optimizing-gradient-descent/index.html#gradientdescentoptimizationalgorithms
\n\nI hope this helps to clear your understanding on optimizers.
", "votes": 0, "replies": [ { "id": 218364, "author_name": "SpruceMoose", "author_url": "", "post_date": "2017-09-04T01:05:24.550000", "content": "Dont think the OP was asking for clarification, but i do think this is a good post here. thanks
", "votes": 0, "replies": [] } ] }, { "id": 1531533, "author_name": "M.Akbari42", "author_url": "", "post_date": "2021-10-02T05:22:36.100000", "content": "see this : An overview of gradient descent optimization algorithms
", "votes": 0, "replies": [] }, { "id": 887486, "author_name": "Daksh Bhatia", "author_url": "", "post_date": "2020-06-15T17:38:15.223000", "content": "For me, SGD work pretty good, if you face any issue with SGD try with bit low learning rate and use the value of momentum around 0.85 to 0.95
", "votes": 0, "replies": [] }, { "id": 218601, "author_name": "hengck23", "author_url": "", "post_date": "2017-09-05T05:08:27.343000", "content": "rmsprop uses only sign and not magnitude of the computed gradients for descend. A intermediate solution is gradient capping:
\n\n loss.backward()\n # accumulate gradients\n if (it+1)%num_grad_acc==0:\n torch.nn.utils.clip_grad_norm(net.parameters(), 1)\n\n optimizer.step()\n optimizer.zero_grad() # assume no effects on bn for accumulating grad\nI find this effective in preventing oscillation in loss.
", "votes": 0, "replies": [ { "id": 218794, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2017-09-05T22:42:09.947000", "content": "This is for pytorch right? How do you do that with keras?
", "votes": 0, "replies": [] }, { "id": 218811, "author_name": "lyakaap", "author_url": "", "post_date": "2017-09-06T00:09:55.147000", "content": "\n\n\nrmsprop uses only sign and not magnitude of the computed gradients for descend.
\n
I don't understand this well, so could you elaborate this mean?
\n\n@YaGana Sheriff-Hussaini\nProbably we can get by this code
\n\n\noptimizer = RMSprop(lr=0.0001, clipnorm=1.)\n
I'm worried about that this gradient capping causes worse effect to learning.
\n\nTo get more stability of RMSprop, enlarging epsilon (1e-8 is keras default. change 1e-6.) is also a way.
", "votes": 3, "replies": [] }, { "id": 218873, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2017-09-06T06:27:36.520000", "content": "@Heng, I have not used pytorch so I do not know how RMSprop is implemented there. But I took a look at the implementation of RMSprop in keras, and I do not think the oscillations you mentioned would be a problem. By definition, RMSprop uses oscillation reduction and momentum in tandem to achieve the desired effect.
\n\nThanks @Iyakaap. I have not tried it, but for those using Keras, I am also of the belief that it may very well result in an unwanted effect.
\n\nIn Keras, RMSprop is implemented according to http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf
", "votes": 0, "replies": [] }, { "id": 219707, "author_name": "Canoe", "author_url": "", "post_date": "2017-09-09T08:23:08.863000", "content": "Why enlarging epsilon will make RMSprop more stable? Thanks for more explaining~
\nAnd Keras is not recommending to change the eplison value. Any benefits can get according to your experiment?
RMSprop
\n\nkeras.optimizers.RMSprop(lr=0.001, rho=0.9, epsilon=1e-08, decay=0.0)\nRMSProp optimizer.
\n\nIt is recommended to leave the parameters of this optimizer at their default values (except the learning rate, which can be freely tuned).\nupdate:\nsame that epsilon here is added for non-zero dividing and so... it makes sense
", "votes": 0, "replies": [] }, { "id": 221129, "author_name": "Will", "author_url": "", "post_date": "2017-09-14T07:51:18.800000", "content": "I also have the same question. Could you please show more details about it?
", "votes": 0, "replies": [] }, { "id": 221515, "author_name": "lyakaap", "author_url": "", "post_date": "2017-09-15T14:25:14.777000", "content": "I reviewed the definition and found that epsilon does not matter so much. The most straight way is shrinking lr moderately. And if we want to do training model more conservatively, enlarging rho might help.
", "votes": 0, "replies": [] } ] }, { "id": 217882, "author_name": "Tim Joseph", "author_url": "", "post_date": "2017-09-01T11:03:48.353000", "content": "For me only SGD works right away... mhhm
", "votes": 0, "replies": [] }, { "id": 217820, "author_name": "jackkwok", "author_url": "", "post_date": "2017-09-01T04:43:39.703000", "content": "Many people reported RMSProp works better than Adam for this contest so I tried RMSProp after using Adam exclusively. For me, my RMSProp result was slightly worse than Adam. I am using batch_size = 1 if that matters. I will stick with Adam for now.
", "votes": 0, "replies": [] }, { "id": 1292351, "author_name": "Saman", "author_url": "", "post_date": "2021-05-03T21:08:53.590000", "content": "I prefer adam, but it is important to pay attention to Beta_2 h-params as it affects your momentum vector and has a huge influence.
\nRegarding RMSProp, one could use adam instead. As both of them are in the same category and consider the Exponential moving average for previous gradients.
\nSo in my opinion is a matter of ADAGRAD, SGD vanilla, or RMSPROP family!