@brendanartley and @harshitsheoran shared it, and it was the first thing I modified in public notebooks as well: it is important to have a decreasing learning rate when optimizing MAE.
\nHere is a very (too?) simple example to show why.
\nLet's say we optimize the simplest possible model: a constant prediction, using MAE.
\nMore formally, our model is a constant function f(x) = a for all input x. Let's assume the label is b = 0.75.
Then we want to minimize MAE(f(x), b).
Let's assume we use gradient descent, with a learning rate of 1.
The gradient of fis:
-1if a <ba = b1 if a > bLets assume that fis initialized with a = 0.
The gradient of f at 0 is -1. We make a step of minus gradient times the learning rate, i.e. a step of 1. This yields a = 1.
The gradient of f at 1 is 1 hence we make a step of -1. This yields a = 0.
The value of a will oscillate between 0 and 1 without ever converging to b.
It is easy to check that there will be an oscillation with whatever constant learning rate is used, except if by chance a becomes equal to b at some point. And when oscillation starts it keeps on forever.
My example is simplistic, but this oscillation happens with more complex models as well.
\nReducing the learning rate fixes that. Say learning rate is decreased by 0.1 at each iteration. Then we would get these steps:
| a | \ngradient | \nlr | \nstep | \n
|---|---|---|---|
| 0 | \n-1 | \n1 | \n1 | \n
| 1 | \n1 | \n0.9 | \n-0.9 | \n
| 0.1 | \n-1 | \n0.8 | \n0.8 | \n
| 0.9 | \n1 | \n0.7 | \n-0.7 | \n
| 0.2 | \n-1 | \n0.6 | \n0.6 | \n
| 0.8 | \n1 | \n0.5 | \n-0.5 | \n
| 0.3 | \n-1 | \n0.4 | \n0.4 | \n
| 0.7 | \n-1 | \n0.3 | \n0.3 | \n
| 1 | \n1 | \n0.2 | \n-0.2 | \n
| 0.8 | \n1 | \n0.1 | \n-0.1 | \n
| 0.7 | \n1 | \n0 | \n0 | \n
We end up much closer to the ground truth.
\nConclusion: use a learning rate scheduler to decrease your learning rate over time.
", "messages": [ { "id": "3216895", "postDate": "06/04/2025 08:51:12", "content": "@brendanartley and @harshitsheoran shared it, and it was the first thing I modified in public notebooks as well: it is important to have a decreasing learning rate when optimizing MAE.
\nHere is a very (too?) simple example to show why.
\nLet's say we optimize the simplest possible model: a constant prediction, using MAE.
\nMore formally, our model is a constant function f(x) = a for all input x. Let's assume the label is b = 0.75.
Then we want to minimize MAE(f(x), b).
Let's assume we use gradient descent, with a learning rate of 1.
The gradient of fis:
-1if a <ba = b1 if a > bLets assume that fis initialized with a = 0.
The gradient of f at 0 is -1. We make a step of minus gradient times the learning rate, i.e. a step of 1. This yields a = 1.
The gradient of f at 1 is 1 hence we make a step of -1. This yields a = 0.
The value of a will oscillate between 0 and 1 without ever converging to b.
It is easy to check that there will be an oscillation with whatever constant learning rate is used, except if by chance a becomes equal to b at some point. And when oscillation starts it keeps on forever.
My example is simplistic, but this oscillation happens with more complex models as well.
\nReducing the learning rate fixes that. Say learning rate is decreased by 0.1 at each iteration. Then we would get these steps:
| a | \ngradient | \nlr | \nstep | \n
|---|---|---|---|
| 0 | \n-1 | \n1 | \n1 | \n
| 1 | \n1 | \n0.9 | \n-0.9 | \n
| 0.1 | \n-1 | \n0.8 | \n0.8 | \n
| 0.9 | \n1 | \n0.7 | \n-0.7 | \n
| 0.2 | \n-1 | \n0.6 | \n0.6 | \n
| 0.8 | \n1 | \n0.5 | \n-0.5 | \n
| 0.3 | \n-1 | \n0.4 | \n0.4 | \n
| 0.7 | \n-1 | \n0.3 | \n0.3 | \n
| 1 | \n1 | \n0.2 | \n-0.2 | \n
| 0.8 | \n1 | \n0.1 | \n-0.1 | \n
| 0.7 | \n1 | \n0 | \n0 | \n
We end up much closer to the ground truth.
\nConclusion: use a learning rate scheduler to decrease your learning rate over time.
", "rawMarkdown": "brendanartley and @harshitsheoran shared it, and it was the first thing I modified in public notebooks as well: it is important to have a decreasing learning rate when optimizing MAE.\n\nHere is a very (too?) simple example to show why.\n\nLet's say we optimize the simplest possible model: a constant prediction, using MAE.\n\nMore formally, our model is a constant function `f(x) = a` for all input `x`. Let's assume the label is `b = 0.75`.\n\nThen we want to minimize `MAE(f(x), b).`\n\nLet's assume we use gradient descent, with a learning rate of `1`.\n\nThe gradient of `f `is:\n- `-1 `if `aThank you @cpmpml - for the explanation … I didnt know earlier why the decreasing learning rate is working in this problem .
\nAlso thank you for the nice ensemble implementation , it helped almost + 0.5 CV for me ..
", "rawMarkdown": "Thank you @cpmpml - for the explanation ... I didnt know earlier why the decreasing learning rate is working in this problem . \n\nAlso thank you for the nice ensemble implementation , it helped almost + 0.5 CV for me ..", "votes": null }, { "id": "3217041", "postDate": "06/04/2025 13:05:21", "content": "Glad what I share helps! Thanks for the feedback.
", "rawMarkdown": "Glad what I share helps! Thanks for the feedback.", "votes": null }, { "id": "3218954", "postDate": "06/07/2025 01:12:47", "content": "Theoretically, warm-up can also be helpful, as it prevents erratic behavior when the optimizer first navigates sharp transitions or abrupt changes in the loss landscape, where gradients are non-smooth or even discontinuous. Additionally, gradient clipping can complement warm-up by helping to prevent instability during the early stages of training.
", "rawMarkdown": "Theoretically, warm-up can also be helpful, as it prevents erratic behavior when the optimizer first navigates sharp transitions or abrupt changes in the loss landscape, where gradients are non-smooth or even discontinuous. Additionally, gradient clipping can complement warm-up by helping to prevent instability during the early stages of training.", "votes": null }, { "id": "3218968", "postDate": "06/07/2025 01:49:05", "content": "Sure, but warming up helping is not specific to MAE.
", "rawMarkdown": "Sure, but warming up helping is not specific to MAE.", "votes": null }, { "id": "3228317", "postDate": "06/20/2025 03:29:31", "content": "thanks for sharing @cpmpml it is someway hard to implement in spark due to limited naive support. any notebook suggestion would be perfect
", "rawMarkdown": "thanks for sharing @cpmpml it is someway hard to implement in spark due to limited naive support. any notebook suggestion would be perfect", "votes": null } ], "comments": [ { "id": 3217031, "author_name": "phoenix9032", "author_url": "", "post_date": "06/04/2025 12:45:37", "content": "Thank you @cpmpml - for the explanation … I didnt know earlier why the decreasing learning rate is working in this problem .
\nAlso thank you for the nice ensemble implementation , it helped almost + 0.5 CV for me ..
", "votes": null, "replies": [ { "id": 3217041, "author_name": "cpmpml", "author_url": "", "post_date": "06/04/2025 13:05:21", "content": "Glad what I share helps! Thanks for the feedback.
", "votes": null, "replies": [] } ] }, { "id": 3218954, "author_name": "yekenot", "author_url": "", "post_date": "06/07/2025 01:12:47", "content": "Theoretically, warm-up can also be helpful, as it prevents erratic behavior when the optimizer first navigates sharp transitions or abrupt changes in the loss landscape, where gradients are non-smooth or even discontinuous. Additionally, gradient clipping can complement warm-up by helping to prevent instability during the early stages of training.
", "votes": null, "replies": [ { "id": 3218968, "author_name": "cpmpml", "author_url": "", "post_date": "06/07/2025 01:49:05", "content": "Sure, but warming up helping is not specific to MAE.
", "votes": null, "replies": [] } ] }, { "id": 3228317, "author_name": "ibozkurt79", "author_url": "", "post_date": "06/20/2025 03:29:31", "content": "thanks for sharing @cpmpml it is someway hard to implement in spark due to limited naive support. any notebook suggestion would be perfect
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3216895": "brendanartley and @harshitsheoran shared it, and it was the first thing I modified in public notebooks as well: it is important to have a decreasing learning rate when optimizing MAE.\n\nHere is a very (too?) simple example to show why.\n\nLet's say we optimize the simplest possible model: a constant prediction, using MAE.\n\nMore formally, our model is a constant function `f(x) = a` for all input `x`. Let's assume the label is `b = 0.75`.\n\nThen we want to minimize `MAE(f(x), b).`\n\nLet's assume we use gradient descent, with a learning rate of `1`.\n\nThe gradient of `f `is:\n- `-1 `if `a